2024-09-16T06:26:26.959 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T06:26:26.976 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T06:26:27.057 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553 branch: reef description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/many workloads/rados_mon_workunits} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906553' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) mon_bind_msgr2: false sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T06:26:27.058 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T06:26:27.058 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T06:26:27.058 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T06:26:27.060 INFO:teuthology.task.internal:Checking packages... 2024-09-16T06:26:27.081 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T06:26:27.081 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T06:26:27.081 INFO:teuthology.packaging:ref: None 2024-09-16T06:26:27.081 INFO:teuthology.packaging:tag: None 2024-09-16T06:26:27.081 INFO:teuthology.packaging:branch: reef 2024-09-16T06:26:27.082 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T06:26:27.082 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-16T06:26:27.341 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1focal 2024-09-16T06:26:27.343 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T06:26:27.349 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T06:26:27.349 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T06:26:27.365 INFO:teuthology.task.internal:Saving configuration 2024-09-16T06:26:27.379 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T06:26:27.386 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T06:26:27.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 06:19:49.685358', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T06:26:27.433 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 06:19:49.687257', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T06:26:27.433 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T06:26:27.439 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T06:26:27.439 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T06:26:27.439 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T06:26:27.505 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa7915d6ef0>, signals=[15]) 2024-09-16T06:26:27.505 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T06:26:27.511 INFO:teuthology.task.internal:Opening connections... 2024-09-16T06:26:27.511 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T06:26:27.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:26:27.615 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T06:26:27.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:26:27.715 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T06:26:27.721 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-09-16T06:26:27.728 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-09-16T06:26:27.729 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-09-16T06:26:27.776 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="Ubuntu" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:ID=ubuntu 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE=debian 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="20.04" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_CODENAME=focal 2024-09-16T06:26:27.777 INFO:teuthology.orchestra.run.smithi076.stdout:UBUNTU_CODENAME=focal 2024-09-16T06:26:27.778 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-09-16T06:26:27.801 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-09-16T06:26:27.807 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-09-16T06:26:27.807 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="20.04" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T06:26:27.855 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T06:26:27.856 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=focal 2024-09-16T06:26:27.856 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=focal 2024-09-16T06:26:27.856 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-09-16T06:26:27.876 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T06:26:27.883 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T06:26:27.888 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T06:26:27.888 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-09-16T06:26:27.890 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-09-16T06:26:27.896 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T06:26:27.901 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T06:26:27.902 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-09-16T06:26:27.940 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-09-16T06:26:27.954 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T06:26:28.003 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T06:26:28.020 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T06:26:28.020 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T06:26:28.020 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T06:26:28.020 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T06:26:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:26:28.026 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T06:26:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:26:28.027 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-16T06:26:28.033 INFO:teuthology.orchestra.run.smithi189.stdout:5.4.0-124-generic 2024-09-16T06:26:28.033 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.4.0-124-generic 2024-09-16T06:26:28.033 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2024-09-16T06:26:28.072 INFO:teuthology.orchestra.run.smithi076.stdout:5.4.0-124-generic 2024-09-16T06:26:28.072 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.4.0-124-generic 2024-09-16T06:26:28.073 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get clean 2024-09-16T06:26:28.144 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T06:26:28.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2024-09-16T06:26:28.445 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:26:28.445 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T06:26:28.449 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:26:28.449 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T06:26:28.509 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T06:26:28.510 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T06:26:28.863 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T06:26:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T06:26:28.927 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T06:26:28.931 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T06:26:29.065 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T06:26:29.074 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T06:26:29.407 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T06:26:29.414 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T06:26:29.417 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T06:26:29.417 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T06:26:29.428 INFO:teuthology.orchestra.run.smithi076.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T06:26:29.430 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T06:26:29.436 INFO:teuthology.orchestra.run.smithi076.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T06:26:29.436 INFO:teuthology.orchestra.run.smithi076.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T06:26:29.439 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T06:26:29.439 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T06:26:29.440 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T06:26:29.526 INFO:teuthology.orchestra.run.smithi076.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T06:26:29.526 INFO:teuthology.orchestra.run.smithi076.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T06:26:29.528 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T06:26:29.533 INFO:teuthology.orchestra.run.smithi076.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T06:26:29.533 INFO:teuthology.orchestra.run.smithi076.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T06:26:29.536 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T06:26:29.536 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T06:26:29.545 INFO:teuthology.orchestra.run.smithi076.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T06:26:29.586 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T06:26:29.594 INFO:teuthology.orchestra.run.smithi076.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T06:26:29.598 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T06:26:29.598 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T06:26:29.599 INFO:teuthology.orchestra.run.smithi076.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T06:26:29.600 INFO:teuthology.orchestra.run.smithi076.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T06:26:29.601 INFO:teuthology.orchestra.run.smithi076.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T06:26:29.602 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T06:26:29.602 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T06:26:29.603 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T06:26:29.603 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T06:26:29.612 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T06:26:29.612 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T06:26:29.612 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T06:26:29.615 INFO:teuthology.orchestra.run.smithi076.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T06:26:29.629 INFO:teuthology.orchestra.run.smithi076.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T06:26:29.629 INFO:teuthology.orchestra.run.smithi076.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T06:26:29.673 INFO:teuthology.orchestra.run.smithi076.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T06:26:29.673 INFO:teuthology.orchestra.run.smithi076.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T06:26:29.675 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T06:26:29.704 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T06:26:29.715 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T06:26:29.715 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T06:26:29.733 INFO:teuthology.orchestra.run.smithi076.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T06:26:29.734 INFO:teuthology.orchestra.run.smithi076.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T06:26:29.746 INFO:teuthology.orchestra.run.smithi076.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T06:26:29.747 INFO:teuthology.orchestra.run.smithi076.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T06:26:29.748 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T06:26:29.778 INFO:teuthology.orchestra.run.smithi076.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T06:26:29.779 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T06:26:29.785 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T06:26:29.785 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T06:26:29.786 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T06:26:29.789 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T06:26:29.793 INFO:teuthology.orchestra.run.smithi076.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T06:26:29.800 INFO:teuthology.orchestra.run.smithi076.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T06:26:29.803 INFO:teuthology.orchestra.run.smithi076.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T06:26:29.803 INFO:teuthology.orchestra.run.smithi076.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T06:26:29.846 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T06:26:29.846 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T06:26:29.847 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T06:26:29.848 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T06:26:29.860 INFO:teuthology.orchestra.run.smithi076.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T06:26:29.861 INFO:teuthology.orchestra.run.smithi076.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T06:26:29.861 INFO:teuthology.orchestra.run.smithi076.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T06:26:29.862 INFO:teuthology.orchestra.run.smithi076.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T06:26:29.927 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T06:26:29.928 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T06:26:29.928 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T06:26:29.942 INFO:teuthology.orchestra.run.smithi076.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T06:26:29.942 INFO:teuthology.orchestra.run.smithi076.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T06:26:29.942 INFO:teuthology.orchestra.run.smithi076.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T06:26:29.943 INFO:teuthology.orchestra.run.smithi076.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T06:26:33.715 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 21.8 MB in 4s (6,219 kB/s) 2024-09-16T06:26:33.896 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 21.8 MB in 4s (6,158 kB/s) 2024-09-16T06:26:35.309 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:26:35.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T06:26:35.411 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:26:35.500 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:26:35.529 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T06:26:35.600 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:26:35.629 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2024-09-16T06:26:35.629 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2024-09-16T06:26:35.818 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T06:26:35.819 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout:The following additional packages will be installed: 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout:Suggested packages: 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout:Recommended packages: 2024-09-16T06:26:35.833 INFO:teuthology.orchestra.run.smithi076.stdout: thermald 2024-09-16T06:26:35.889 INFO:teuthology.orchestra.run.smithi076.stdout:The following NEW packages will be installed: 2024-09-16T06:26:35.889 INFO:teuthology.orchestra.run.smithi076.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T06:26:35.889 INFO:teuthology.orchestra.run.smithi076.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T06:26:35.890 INFO:teuthology.orchestra.run.smithi076.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T06:26:35.891 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages will be upgraded: 2024-09-16T06:26:35.892 INFO:teuthology.orchestra.run.smithi076.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T06:26:36.030 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2024-09-16T06:26:36.030 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T06:26:36.030 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T06:26:36.030 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T06:26:36.032 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2024-09-16T06:26:36.032 INFO:teuthology.orchestra.run.smithi189.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T06:26:36.032 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2024-09-16T06:26:36.032 INFO:teuthology.orchestra.run.smithi189.stdout: thermald 2024-09-16T06:26:36.072 INFO:teuthology.orchestra.run.smithi076.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T06:26:36.072 INFO:teuthology.orchestra.run.smithi076.stdout:Need to get 77.2 MB of archives. 2024-09-16T06:26:36.072 INFO:teuthology.orchestra.run.smithi076.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T06:26:36.072 INFO:teuthology.orchestra.run.smithi076.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T06:26:36.093 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2024-09-16T06:26:36.093 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T06:26:36.093 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T06:26:36.094 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T06:26:36.095 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2024-09-16T06:26:36.095 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T06:26:36.277 INFO:teuthology.orchestra.run.smithi189.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T06:26:36.278 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 77.2 MB of archives. 2024-09-16T06:26:36.278 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T06:26:36.278 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T06:26:37.239 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T06:26:37.399 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T06:26:37.576 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T06:26:37.621 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T06:26:38.177 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T06:26:38.195 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T06:26:38.213 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T06:26:38.329 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T06:26:38.362 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T06:26:38.903 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T06:26:38.903 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T06:26:38.903 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T06:26:39.125 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 77.2 MB in 2s (34.1 MB/s) 2024-09-16T06:26:39.183 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T06:26:39.225 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T06:26:39.321 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T06:26:39.904 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 77.2 MB in 3s (23.2 MB/s) 2024-09-16T06:26:40.102 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T06:26:41.139 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T06:26:41.144 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:41.427 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:41.824 INFO:teuthology.orchestra.run.smithi076.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T06:26:41.829 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:41.907 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:43.370 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T06:26:43.382 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:43.478 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:43.801 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T06:26:43.815 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:43.920 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:44.581 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T06:26:44.594 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:44.664 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:45.029 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T06:26:45.045 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:45.087 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:49.484 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:26:49.611 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:26:49.893 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:26:49.981 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:26:50.019 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:26:50.107 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:26:50.371 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:26:50.460 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T06:26:50.475 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T06:26:50.498 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:26:50.518 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T06:26:50.825 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T06:26:50.842 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T06:26:50.883 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T06:26:56.075 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T06:26:56.096 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:56.133 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:56.281 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T06:26:56.303 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T06:26:56.347 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:26:59.102 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:26:59.264 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:26:59.621 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:00.012 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T06:27:00.140 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T06:27:00.500 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:01.711 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:01.920 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:02.870 INFO:teuthology.orchestra.run.smithi076.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:02.870 INFO:teuthology.orchestra.run.smithi076.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:02.988 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:03.136 INFO:teuthology.orchestra.run.smithi189.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:03.136 INFO:teuthology.orchestra.run.smithi189.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:03.306 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:04.516 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T06:27:04.642 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T06:27:04.768 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:04.902 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T06:27:04.940 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T06:27:05.020 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T06:27:05.066 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T06:27:05.163 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:05.191 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:05.228 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T06:27:05.228 INFO:teuthology.orchestra.run.smithi189.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:05.317 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T06:27:05.443 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T06:27:05.588 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T06:27:05.666 INFO:teuthology.orchestra.run.smithi076.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T06:27:05.667 INFO:teuthology.orchestra.run.smithi076.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:27.206 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T06:27:27.207 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub' 2024-09-16T06:27:27.223 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T06:27:27.243 INFO:teuthology.orchestra.run.smithi189.stdout:Generating grub configuration file ... 2024-09-16T06:27:27.594 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:27.625 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:27.849 INFO:teuthology.orchestra.run.smithi076.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T06:27:27.849 INFO:teuthology.orchestra.run.smithi076.stdout:Sourcing file `/etc/default/grub' 2024-09-16T06:27:27.861 INFO:teuthology.orchestra.run.smithi076.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T06:27:27.887 INFO:teuthology.orchestra.run.smithi076.stdout:Generating grub configuration file ... 2024-09-16T06:27:28.037 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T06:27:28.049 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T06:27:28.194 INFO:teuthology.orchestra.run.smithi076.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:28.222 INFO:teuthology.orchestra.run.smithi076.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:28.644 INFO:teuthology.orchestra.run.smithi076.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T06:27:28.656 INFO:teuthology.orchestra.run.smithi076.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T06:27:29.860 INFO:teuthology.orchestra.run.smithi189.stdout:done 2024-09-16T06:27:30.286 DEBUG:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2024-09-16T06:27:30.339 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2024-09-16T06:27:30.339 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2024-09-16T06:27:30.339 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2024-09-16T06:27:30.339 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2024-09-16T06:27:30.339 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 20 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 5.4.0.195.193 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2024-09-16T06:27:30.340 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2024-09-16T06:27:30.340 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T06:27:30.341 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T06:27:30.341 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T06:27:30.341 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T06:27:30.341 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T06:27:30.341 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2024-09-16T06:27:30.425 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T06:27:30.523 INFO:teuthology.orchestra.run.smithi076.stdout:done 2024-09-16T06:27:30.687 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T06:27:30.693 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:27:30.775 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T06:27:30.857 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T06:27:30.933 DEBUG:teuthology.orchestra.run.smithi076:> dpkg -s linux-image-generic 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Package: linux-image-generic 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Status: install ok installed 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Priority: optional 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Section: kernel 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Installed-Size: 20 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: amd64 2024-09-16T06:27:30.986 INFO:teuthology.orchestra.run.smithi076.stdout:Source: linux-meta 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:Version: 5.4.0.195.193 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:Recommends: thermald 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:Description: Generic Linux kernel image 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout: This package will always depend on the latest generic kernel image 2024-09-16T06:27:30.987 INFO:teuthology.orchestra.run.smithi076.stdout: available. 2024-09-16T06:27:30.987 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T06:27:30.987 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T06:27:30.987 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T06:27:30.988 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T06:27:30.988 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T06:27:30.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get clean 2024-09-16T06:27:31.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2024-09-16T06:27:31.226 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T06:27:31.342 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:27:31.424 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T06:27:31.513 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T06:27:32.556 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:27:32.582 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T06:27:32.647 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:27:32.842 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T06:27:32.842 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T06:27:33.063 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:27:33.081 INFO:teuthology.orchestra.run.smithi189.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T06:27:33.081 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T06:27:33.083 DEBUG:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2024-09-16T06:27:33.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T06:27:33.104 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 20 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 5.4.0.195.193 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2024-09-16T06:27:33.105 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2024-09-16T06:27:33.106 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2024-09-16T06:27:33.106 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-16T06:27:33.112 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.sRzVJbNGqw 2024-09-16T06:27:33.112 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub/grub.cfg /tmp/tmp.sRzVJbNGqw 2024-09-16T06:27:33.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.sRzVJbNGqw 2024-09-16T06:27:33.159 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:27:33.209 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.sRzVJbNGqw is 9KB 2024-09-16T06:27:33.224 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.sRzVJbNGqw 2024-09-16T06:27:33.230 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.288 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:27:33.288 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.288 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.323 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T06:27:33.323 DEBUG:teuthology.orchestra.run.smithi189:> sudo update-grub 2024-09-16T06:27:33.352 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2024-09-16T06:27:33.353 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2024-09-16T06:27:33.580 INFO:teuthology.orchestra.run.smithi076.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T06:27:33.580 INFO:teuthology.orchestra.run.smithi076.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T06:27:33.582 DEBUG:teuthology.orchestra.run.smithi076:> dpkg -s linux-image-generic 2024-09-16T06:27:33.602 INFO:teuthology.orchestra.run.smithi076.stdout:Package: linux-image-generic 2024-09-16T06:27:33.602 INFO:teuthology.orchestra.run.smithi076.stdout:Status: install ok installed 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Priority: optional 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Section: kernel 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Installed-Size: 20 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: amd64 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Source: linux-meta 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Version: 5.4.0.195.193 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Recommends: thermald 2024-09-16T06:27:33.603 INFO:teuthology.orchestra.run.smithi076.stdout:Description: Generic Linux kernel image 2024-09-16T06:27:33.604 INFO:teuthology.orchestra.run.smithi076.stdout: This package will always depend on the latest generic kernel image 2024-09-16T06:27:33.604 INFO:teuthology.orchestra.run.smithi076.stdout: available. 2024-09-16T06:27:33.604 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T06:27:33.609 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.a81jNRGGpQ 2024-09-16T06:27:33.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub/grub.cfg /tmp/tmp.a81jNRGGpQ 2024-09-16T06:27:33.622 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.a81jNRGGpQ 2024-09-16T06:27:33.726 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.a81jNRGGpQ is 9KB 2024-09-16T06:27:33.739 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.a81jNRGGpQ 2024-09-16T06:27:33.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.802 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:27:33.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T06:27:33.846 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T06:27:33.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo update-grub 2024-09-16T06:27:34.726 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub' 2024-09-16T06:27:34.730 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T06:27:34.743 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-09-16T06:27:35.039 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:35.051 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:35.158 INFO:teuthology.orchestra.run.smithi076.stderr:Sourcing file `/etc/default/grub' 2024-09-16T06:27:35.165 INFO:teuthology.orchestra.run.smithi076.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T06:27:35.177 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-09-16T06:27:35.428 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T06:27:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T06:27:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T06:27:35.455 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T06:27:35.796 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T06:27:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T06:27:36.233 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-09-16T06:27:36.235 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-09-16T06:27:36.565 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-09-16T06:27:36.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-09-16T06:28:06.265 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T06:28:06.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:28:06.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:06.570 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T06:28:06.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:28:06.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:24.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T06:28:25.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T06:28:33.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:28:33.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:34.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:28:34.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:36.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-16T06:28:37.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T06:28:48.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:28:48.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:49.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:28:49.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:28:52.157 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T06:29:07.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:29:07.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:29:25.697 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-09-16T06:29:43.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:29:43.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:29:43.966 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T06:29:44.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:29:44.513 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-16T06:29:44.513 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-09-16T06:29:44.520 INFO:teuthology.orchestra.run.smithi076.stdout:5.4.0-195-generic 2024-09-16T06:29:44.520 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T06:29:44.521 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T06:29:44.521 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T06:29:45.522 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T06:29:45.523 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T06:29:45.598 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2024-09-16T06:29:45.637 DEBUG:teuthology.parallel:result is None 2024-09-16T06:29:48.844 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T06:30:03.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:30:03.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:30:04.095 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-16T06:30:04.660 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:30:04.661 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-16T06:30:04.661 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-16T06:30:04.667 INFO:teuthology.orchestra.run.smithi189.stdout:5.4.0-195-generic 2024-09-16T06:30:04.667 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T06:30:04.667 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T06:30:04.668 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T06:30:05.668 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T06:30:05.668 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T06:30:05.757 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-09-16T06:30:05.778 DEBUG:teuthology.parallel:result is None 2024-09-16T06:30:05.779 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T06:30:05.786 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T06:30:05.786 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T06:30:05.789 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T06:30:05.795 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T06:30:05.836 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T06:30:05.845 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T06:30:05.845 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T06:30:05.848 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T06:30:05.869 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T06:30:05.875 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T06:30:05.876 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T06:30:05.878 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T06:30:05.896 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:30:05.906 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:30:05.925 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:30:05.936 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:30:05.937 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T06:30:05.943 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T06:30:05.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T06:30:05.946 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T06:30:05.961 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T06:30:05.970 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T06:30:05.970 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T06:30:06.004 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T06:30:06.010 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T06:30:06.020 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T06:30:06.026 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:30:06.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T06:30:06.099 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T06:30:06.106 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T06:30:06.155 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:30:06.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T06:30:06.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-09-16T06:30:06.176 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-09-16T06:30:06.226 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T06:30:06.233 INFO:teuthology.task.internal:Starting timer... 2024-09-16T06:30:06.233 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T06:30:06.246 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T06:30:06.254 DEBUG:teuthology.task.selinux:Excluding smithi076: OS 'ubuntu' does not support SELinux 2024-09-16T06:30:06.254 DEBUG:teuthology.task.selinux:Excluding smithi189: OS 'ubuntu' does not support SELinux 2024-09-16T06:30:06.254 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T06:30:06.254 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T06:30:06.254 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T06:30:06.254 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T06:30:06.265 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T06:30:06.268 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T06:30:06.382 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T06:30:06.394 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T06:30:06.395 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T06:35:40.846 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-09-16T06:35:40.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:35:40.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:35:40.954 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-09-16T06:35:41.079 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-09-16T06:35:41.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:35:41.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T06:35:41.175 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-16T06:35:41.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-16T06:35:41.304 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T06:35:41.315 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T06:35:41.315 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T06:35:41.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T06:35:41.318 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T06:35:41.318 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T06:35:41.349 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T06:35:41.349 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Command line: ntpd -gq 2024-09-16T06:35:41.350 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: proto: precision = 0.196 usec (-22) 2024-09-16T06:35:41.350 INFO:teuthology.orchestra.run.smithi076.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T06:35:41.351 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T06:35:41.351 INFO:teuthology.orchestra.run.smithi076.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen normally on 3 enp3s0f1 172.21.15.76:123 2024-09-16T06:35:41.352 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen normally on 4 lo [::1]:123 2024-09-16T06:35:41.353 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47e1%5]:123 2024-09-16T06:35:41.353 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:41 ntpd[13785]: Listening on routing socket on fd #22 for interface updates 2024-09-16T06:35:41.353 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T06:35:41.353 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Command line: ntpd -gq 2024-09-16T06:35:41.353 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: proto: precision = 0.082 usec (-23) 2024-09-16T06:35:41.354 INFO:teuthology.orchestra.run.smithi189.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T06:35:41.354 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T06:35:41.354 INFO:teuthology.orchestra.run.smithi189.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T06:35:41.355 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T06:35:41.355 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T06:35:41.355 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T06:35:41.356 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T06:35:41.356 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen normally on 3 enp3s0f1 172.21.15.189:123 2024-09-16T06:35:41.356 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen normally on 4 lo [::1]:123 2024-09-16T06:35:41.356 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df1%5]:123 2024-09-16T06:35:41.356 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:41 ntpd[13749]: Listening on routing socket on fd #22 for interface updates 2024-09-16T06:35:42.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:42.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:42.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:42.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:42.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:42.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:42 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:43.352 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:43 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:43.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:43 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:44.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:44 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:44.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:44 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:45 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:45 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:45.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:45 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:46 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:46.355 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.355 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.355 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:46.356 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:46.357 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:46.357 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:46.357 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:46 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:47 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:48.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.350 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stdout:16 Sep 06:35:48 ntpd[13785]: ntpd: time slew -0.003603 s 2024-09-16T06:35:48.351 INFO:teuthology.orchestra.run.smithi076.stdout:ntpd: time slew -0.003603s 2024-09-16T06:35:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T06:35:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:16 Sep 06:35:48 ntpd[13785]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T06:35:48.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:48.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:48 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:48.417 INFO:teuthology.orchestra.run.smithi076.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T06:35:48.417 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================== 2024-09-16T06:35:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:48.418 INFO:teuthology.orchestra.run.smithi076.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:49.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:49 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:50.353 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T06:35:50.354 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T06:35:50.355 INFO:teuthology.orchestra.run.smithi189.stdout:16 Sep 06:35:50 ntpd[13749]: ntpd: time slew +0.004161 s 2024-09-16T06:35:50.355 INFO:teuthology.orchestra.run.smithi189.stdout:ntpd: time slew +0.004161s 2024-09-16T06:35:50.355 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T06:35:50.355 INFO:teuthology.orchestra.run.smithi189.stderr:16 Sep 06:35:50 ntpd[13749]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:50.429 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T06:35:50.430 INFO:teuthology.run_tasks:Running task install... 2024-09-16T06:35:50.441 DEBUG:teuthology.task.install:project ceph 2024-09-16T06:35:50.441 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T06:35:50.441 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T06:35:50.441 INFO:teuthology.task.install:Using flavor: default 2024-09-16T06:35:50.450 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T06:35:50.450 INFO:teuthology.task.install:extra packages: [] 2024-09-16T06:35:50.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-key list | grep Ceph 2024-09-16T06:35:50.452 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-key list | grep Ceph 2024-09-16T06:35:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T06:35:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T06:35:50.612 INFO:teuthology.orchestra.run.smithi076.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T06:35:50.612 INFO:teuthology.orchestra.run.smithi076.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T06:35:50.613 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T06:35:50.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T06:35:50.634 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T06:35:50.634 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T06:35:50.635 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T06:35:50.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T06:35:50.761 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T06:35:50.761 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T06:35:50.783 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T06:35:50.783 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T06:35:50.877 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:35:50.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T06:35:50.894 DEBUG:teuthology.orchestra.run.smithi076:> sudo apt-get update 2024-09-16T06:35:50.898 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:35:50.898 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T06:35:50.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo apt-get update 2024-09-16T06:35:51.092 INFO:teuthology.orchestra.run.smithi076.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T06:35:51.113 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T06:35:51.114 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T06:35:51.135 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T06:35:51.137 INFO:teuthology.orchestra.run.smithi076.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T06:35:51.159 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T06:35:51.167 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T06:35:51.167 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:35:51.168 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T06:35:51.186 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T06:35:51.187 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T06:35:51.187 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T06:35:51.201 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T06:35:51.219 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T06:35:51.238 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T06:35:51.256 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T06:35:51.319 INFO:teuthology.orchestra.run.smithi076.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T06:35:51.338 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T06:35:51.828 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 28.7 kB in 1s (37.5 kB/s) 2024-09-16T06:35:51.848 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 28.7 kB in 1s (37.3 kB/s) 2024-09-16T06:35:52.981 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:35:53.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T06:35:53.077 INFO:teuthology.orchestra.run.smithi076.stdout:Reading package lists... 2024-09-16T06:35:53.106 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:35:53.130 DEBUG:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T06:35:53.204 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2024-09-16T06:35:53.270 INFO:teuthology.orchestra.run.smithi076.stdout:Building dependency tree... 2024-09-16T06:35:53.271 INFO:teuthology.orchestra.run.smithi076.stdout:Reading state information... 2024-09-16T06:35:53.422 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2024-09-16T06:35:53.423 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2024-09-16T06:35:53.464 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout:The following additional packages will be installed: 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T06:35:53.466 INFO:teuthology.orchestra.run.smithi076.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T06:35:53.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T06:35:53.468 INFO:teuthology.orchestra.run.smithi076.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout:Suggested packages: 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T06:35:53.469 INFO:teuthology.orchestra.run.smithi076.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T06:35:53.470 INFO:teuthology.orchestra.run.smithi076.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T06:35:53.470 INFO:teuthology.orchestra.run.smithi076.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T06:35:53.470 INFO:teuthology.orchestra.run.smithi076.stdout:Recommended packages: 2024-09-16T06:35:53.470 INFO:teuthology.orchestra.run.smithi076.stdout: btrfs-tools 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout:The following NEW packages will be installed: 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T06:35:53.587 INFO:teuthology.orchestra.run.smithi076.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T06:35:53.588 INFO:teuthology.orchestra.run.smithi076.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T06:35:53.588 INFO:teuthology.orchestra.run.smithi076.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T06:35:53.588 INFO:teuthology.orchestra.run.smithi076.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T06:35:53.589 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T06:35:53.590 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T06:35:53.590 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet 2024-09-16T06:35:53.590 INFO:teuthology.orchestra.run.smithi076.stdout:The following packages will be upgraded: 2024-09-16T06:35:53.590 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 librbd1 2024-09-16T06:35:53.630 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T06:35:53.631 INFO:teuthology.orchestra.run.smithi189.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T06:35:53.632 INFO:teuthology.orchestra.run.smithi189.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T06:35:53.632 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2024-09-16T06:35:53.632 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T06:35:53.632 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T06:35:53.632 INFO:teuthology.orchestra.run.smithi189.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T06:35:53.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T06:35:53.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T06:35:53.636 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2024-09-16T06:35:53.637 INFO:teuthology.orchestra.run.smithi189.stdout: btrfs-tools 2024-09-16T06:35:53.656 INFO:teuthology.orchestra.run.smithi076.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T06:35:53.656 INFO:teuthology.orchestra.run.smithi076.stdout:Need to get 199 MB of archives. 2024-09-16T06:35:53.656 INFO:teuthology.orchestra.run.smithi076.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T06:35:53.656 INFO:teuthology.orchestra.run.smithi076.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T06:35:53.740 INFO:teuthology.orchestra.run.smithi076.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T06:35:53.741 INFO:teuthology.orchestra.run.smithi076.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T06:35:53.743 INFO:teuthology.orchestra.run.smithi076.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T06:35:53.753 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2024-09-16T06:35:53.753 INFO:teuthology.orchestra.run.smithi189.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T06:35:53.753 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T06:35:53.753 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T06:35:53.753 INFO:teuthology.orchestra.run.smithi189.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T06:35:53.754 INFO:teuthology.orchestra.run.smithi189.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T06:35:53.754 INFO:teuthology.orchestra.run.smithi189.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T06:35:53.754 INFO:teuthology.orchestra.run.smithi189.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T06:35:53.756 INFO:teuthology.orchestra.run.smithi189.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T06:35:53.757 INFO:teuthology.orchestra.run.smithi189.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T06:35:53.757 INFO:teuthology.orchestra.run.smithi189.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T06:35:53.757 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T06:35:53.757 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T06:35:53.757 INFO:teuthology.orchestra.run.smithi189.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet 2024-09-16T06:35:53.758 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2024-09-16T06:35:53.759 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 librbd1 2024-09-16T06:35:53.761 INFO:teuthology.orchestra.run.smithi076.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T06:35:53.814 INFO:teuthology.orchestra.run.smithi076.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T06:35:53.816 INFO:teuthology.orchestra.run.smithi076.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T06:35:53.823 INFO:teuthology.orchestra.run.smithi189.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T06:35:53.823 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 199 MB of archives. 2024-09-16T06:35:53.823 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T06:35:53.823 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T06:35:53.823 INFO:teuthology.orchestra.run.smithi076.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T06:35:53.825 INFO:teuthology.orchestra.run.smithi076.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T06:35:53.825 INFO:teuthology.orchestra.run.smithi076.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T06:35:53.826 INFO:teuthology.orchestra.run.smithi076.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T06:35:53.826 INFO:teuthology.orchestra.run.smithi076.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T06:35:53.829 INFO:teuthology.orchestra.run.smithi076.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T06:35:53.830 INFO:teuthology.orchestra.run.smithi076.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T06:35:53.832 INFO:teuthology.orchestra.run.smithi076.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T06:35:53.849 INFO:teuthology.orchestra.run.smithi076.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T06:35:53.849 INFO:teuthology.orchestra.run.smithi076.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T06:35:53.850 INFO:teuthology.orchestra.run.smithi076.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T06:35:53.851 INFO:teuthology.orchestra.run.smithi076.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T06:35:53.852 INFO:teuthology.orchestra.run.smithi076.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T06:35:53.852 INFO:teuthology.orchestra.run.smithi076.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T06:35:53.854 INFO:teuthology.orchestra.run.smithi076.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T06:35:53.855 INFO:teuthology.orchestra.run.smithi076.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T06:35:53.857 INFO:teuthology.orchestra.run.smithi076.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T06:35:53.866 INFO:teuthology.orchestra.run.smithi076.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T06:35:53.877 INFO:teuthology.orchestra.run.smithi076.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T06:35:53.882 INFO:teuthology.orchestra.run.smithi076.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T06:35:53.882 INFO:teuthology.orchestra.run.smithi076.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T06:35:53.883 INFO:teuthology.orchestra.run.smithi076.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T06:35:53.883 INFO:teuthology.orchestra.run.smithi076.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T06:35:53.884 INFO:teuthology.orchestra.run.smithi076.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T06:35:53.887 INFO:teuthology.orchestra.run.smithi076.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T06:35:53.887 INFO:teuthology.orchestra.run.smithi076.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T06:35:53.889 INFO:teuthology.orchestra.run.smithi076.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T06:35:53.901 INFO:teuthology.orchestra.run.smithi076.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T06:35:53.902 INFO:teuthology.orchestra.run.smithi076.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T06:35:53.904 INFO:teuthology.orchestra.run.smithi076.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T06:35:53.904 INFO:teuthology.orchestra.run.smithi076.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T06:35:53.905 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T06:35:53.906 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T06:35:53.909 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T06:35:53.909 INFO:teuthology.orchestra.run.smithi076.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T06:35:53.918 INFO:teuthology.orchestra.run.smithi076.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T06:35:53.918 INFO:teuthology.orchestra.run.smithi076.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T06:35:53.920 INFO:teuthology.orchestra.run.smithi076.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T06:35:53.920 INFO:teuthology.orchestra.run.smithi076.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T06:35:53.921 INFO:teuthology.orchestra.run.smithi076.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T06:35:53.921 INFO:teuthology.orchestra.run.smithi076.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T06:35:53.926 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T06:35:53.936 INFO:teuthology.orchestra.run.smithi076.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T06:35:53.936 INFO:teuthology.orchestra.run.smithi076.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T06:35:53.938 INFO:teuthology.orchestra.run.smithi076.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T06:35:53.940 INFO:teuthology.orchestra.run.smithi076.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T06:35:53.953 INFO:teuthology.orchestra.run.smithi076.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T06:35:53.953 INFO:teuthology.orchestra.run.smithi076.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T06:35:53.954 INFO:teuthology.orchestra.run.smithi076.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T06:35:53.969 INFO:teuthology.orchestra.run.smithi076.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T06:35:53.979 INFO:teuthology.orchestra.run.smithi076.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T06:35:53.981 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T06:35:53.983 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T06:35:53.987 INFO:teuthology.orchestra.run.smithi076.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T06:35:53.988 INFO:teuthology.orchestra.run.smithi076.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T06:35:53.989 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T06:35:53.991 INFO:teuthology.orchestra.run.smithi076.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T06:35:53.991 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T06:35:53.992 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T06:35:53.992 INFO:teuthology.orchestra.run.smithi076.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T06:35:53.993 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T06:35:53.993 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T06:35:53.996 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T06:35:53.996 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T06:35:53.999 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T06:35:53.999 INFO:teuthology.orchestra.run.smithi076.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T06:35:54.016 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T06:35:54.016 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T06:35:54.016 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T06:35:54.017 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T06:35:54.018 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T06:35:54.018 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T06:35:54.020 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T06:35:54.020 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T06:35:54.022 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T06:35:54.033 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T06:35:54.036 INFO:teuthology.orchestra.run.smithi076.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T06:35:54.041 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T06:35:54.047 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T06:35:54.047 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T06:35:54.048 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T06:35:54.048 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T06:35:54.050 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T06:35:54.053 INFO:teuthology.orchestra.run.smithi076.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T06:35:54.053 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T06:35:54.053 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T06:35:54.055 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T06:35:54.067 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T06:35:54.069 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T06:35:54.070 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T06:35:54.071 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T06:35:54.071 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T06:35:54.084 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T06:35:54.085 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T06:35:54.086 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T06:35:54.086 INFO:teuthology.orchestra.run.smithi189.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T06:35:54.087 INFO:teuthology.orchestra.run.smithi189.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T06:35:54.101 INFO:teuthology.orchestra.run.smithi189.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T06:35:54.102 INFO:teuthology.orchestra.run.smithi189.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T06:35:54.104 INFO:teuthology.orchestra.run.smithi189.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T06:35:54.106 INFO:teuthology.orchestra.run.smithi189.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T06:35:54.108 INFO:teuthology.orchestra.run.smithi189.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T06:35:54.119 INFO:teuthology.orchestra.run.smithi189.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T06:35:54.119 INFO:teuthology.orchestra.run.smithi189.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T06:35:54.124 INFO:teuthology.orchestra.run.smithi076.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T06:35:54.124 INFO:teuthology.orchestra.run.smithi076.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T06:35:54.126 INFO:teuthology.orchestra.run.smithi076.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T06:35:54.131 INFO:teuthology.orchestra.run.smithi076.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T06:35:54.141 INFO:teuthology.orchestra.run.smithi076.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T06:35:54.141 INFO:teuthology.orchestra.run.smithi076.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T06:35:54.142 INFO:teuthology.orchestra.run.smithi076.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T06:35:54.143 INFO:teuthology.orchestra.run.smithi076.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T06:35:54.143 INFO:teuthology.orchestra.run.smithi076.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T06:35:54.143 INFO:teuthology.orchestra.run.smithi076.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T06:35:54.143 INFO:teuthology.orchestra.run.smithi076.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T06:35:54.192 INFO:teuthology.orchestra.run.smithi076.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T06:35:54.192 INFO:teuthology.orchestra.run.smithi076.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T06:35:54.193 INFO:teuthology.orchestra.run.smithi076.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T06:35:54.198 INFO:teuthology.orchestra.run.smithi076.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T06:35:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T06:35:54.200 INFO:teuthology.orchestra.run.smithi076.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T06:35:54.202 INFO:teuthology.orchestra.run.smithi076.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T06:35:54.203 INFO:teuthology.orchestra.run.smithi076.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T06:35:54.205 INFO:teuthology.orchestra.run.smithi076.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T06:35:54.208 INFO:teuthology.orchestra.run.smithi076.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T06:35:54.217 INFO:teuthology.orchestra.run.smithi189.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T06:35:54.218 INFO:teuthology.orchestra.run.smithi189.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T06:35:54.222 INFO:teuthology.orchestra.run.smithi076.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T06:35:54.225 INFO:teuthology.orchestra.run.smithi076.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T06:35:54.227 INFO:teuthology.orchestra.run.smithi076.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T06:35:54.229 INFO:teuthology.orchestra.run.smithi189.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T06:35:54.230 INFO:teuthology.orchestra.run.smithi189.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T06:35:54.269 INFO:teuthology.orchestra.run.smithi076.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T06:35:54.270 INFO:teuthology.orchestra.run.smithi076.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T06:35:54.270 INFO:teuthology.orchestra.run.smithi076.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T06:35:54.271 INFO:teuthology.orchestra.run.smithi076.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T06:35:54.282 INFO:teuthology.orchestra.run.smithi076.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T06:35:54.282 INFO:teuthology.orchestra.run.smithi076.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T06:35:54.283 INFO:teuthology.orchestra.run.smithi076.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T06:35:54.283 INFO:teuthology.orchestra.run.smithi076.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T06:35:54.284 INFO:teuthology.orchestra.run.smithi076.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T06:35:54.289 INFO:teuthology.orchestra.run.smithi189.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T06:35:54.290 INFO:teuthology.orchestra.run.smithi189.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T06:35:54.290 INFO:teuthology.orchestra.run.smithi189.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T06:35:54.291 INFO:teuthology.orchestra.run.smithi189.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T06:35:54.291 INFO:teuthology.orchestra.run.smithi189.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T06:35:54.293 INFO:teuthology.orchestra.run.smithi189.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T06:35:54.293 INFO:teuthology.orchestra.run.smithi189.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T06:35:54.306 INFO:teuthology.orchestra.run.smithi189.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T06:35:54.311 INFO:teuthology.orchestra.run.smithi189.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T06:35:54.313 INFO:teuthology.orchestra.run.smithi189.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T06:35:54.323 INFO:teuthology.orchestra.run.smithi189.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T06:35:54.324 INFO:teuthology.orchestra.run.smithi189.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T06:35:54.328 INFO:teuthology.orchestra.run.smithi189.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T06:35:54.331 INFO:teuthology.orchestra.run.smithi189.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T06:35:54.347 INFO:teuthology.orchestra.run.smithi189.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T06:35:54.349 INFO:teuthology.orchestra.run.smithi189.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T06:35:54.351 INFO:teuthology.orchestra.run.smithi189.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T06:35:54.395 INFO:teuthology.orchestra.run.smithi189.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T06:35:54.396 INFO:teuthology.orchestra.run.smithi189.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T06:35:54.396 INFO:teuthology.orchestra.run.smithi189.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T06:35:54.465 INFO:teuthology.orchestra.run.smithi189.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T06:35:54.486 INFO:teuthology.orchestra.run.smithi189.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T06:35:54.486 INFO:teuthology.orchestra.run.smithi189.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T06:35:54.488 INFO:teuthology.orchestra.run.smithi189.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T06:35:54.488 INFO:teuthology.orchestra.run.smithi189.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T06:35:54.488 INFO:teuthology.orchestra.run.smithi189.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T06:35:54.706 INFO:teuthology.orchestra.run.smithi076.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T06:35:54.824 INFO:teuthology.orchestra.run.smithi076.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T06:35:54.825 INFO:teuthology.orchestra.run.smithi076.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T06:35:54.827 INFO:teuthology.orchestra.run.smithi076.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T06:35:54.853 INFO:teuthology.orchestra.run.smithi076.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T06:35:54.875 INFO:teuthology.orchestra.run.smithi189.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T06:35:54.972 INFO:teuthology.orchestra.run.smithi076.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T06:35:55.483 INFO:teuthology.orchestra.run.smithi189.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T06:35:55.504 INFO:teuthology.orchestra.run.smithi076.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T06:35:55.505 INFO:teuthology.orchestra.run.smithi076.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T06:35:55.516 INFO:teuthology.orchestra.run.smithi076.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T06:35:55.519 INFO:teuthology.orchestra.run.smithi189.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T06:35:55.532 INFO:teuthology.orchestra.run.smithi076.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T06:35:55.534 INFO:teuthology.orchestra.run.smithi076.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T06:35:55.535 INFO:teuthology.orchestra.run.smithi076.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T06:35:55.537 INFO:teuthology.orchestra.run.smithi189.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T06:35:55.538 INFO:teuthology.orchestra.run.smithi189.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T06:35:55.545 INFO:teuthology.orchestra.run.smithi189.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T06:35:55.546 INFO:teuthology.orchestra.run.smithi189.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T06:35:55.579 INFO:teuthology.orchestra.run.smithi189.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T06:35:55.650 INFO:teuthology.orchestra.run.smithi076.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T06:35:55.804 INFO:teuthology.orchestra.run.smithi076.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T06:35:55.804 INFO:teuthology.orchestra.run.smithi076.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T06:35:56.191 INFO:teuthology.orchestra.run.smithi189.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T06:35:56.194 INFO:teuthology.orchestra.run.smithi189.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T06:35:56.223 INFO:teuthology.orchestra.run.smithi189.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T06:35:57.068 INFO:teuthology.orchestra.run.smithi076.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T06:35:57.071 INFO:teuthology.orchestra.run.smithi076.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T06:35:57.072 INFO:teuthology.orchestra.run.smithi076.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T06:35:57.272 INFO:teuthology.orchestra.run.smithi189.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T06:35:57.408 INFO:teuthology.orchestra.run.smithi076.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T06:35:57.460 INFO:teuthology.orchestra.run.smithi189.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T06:35:57.463 INFO:teuthology.orchestra.run.smithi189.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T06:35:57.465 INFO:teuthology.orchestra.run.smithi189.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T06:35:57.491 INFO:teuthology.orchestra.run.smithi189.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T06:35:57.601 INFO:teuthology.orchestra.run.smithi189.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T06:35:58.050 INFO:teuthology.orchestra.run.smithi076.stdout:Fetched 199 MB in 4s (52.2 MB/s) 2024-09-16T06:35:58.104 INFO:teuthology.orchestra.run.smithi189.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T06:35:58.105 INFO:teuthology.orchestra.run.smithi189.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T06:35:58.119 INFO:teuthology.orchestra.run.smithi189.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T06:35:58.136 INFO:teuthology.orchestra.run.smithi189.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T06:35:58.137 INFO:teuthology.orchestra.run.smithi189.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T06:35:58.138 INFO:teuthology.orchestra.run.smithi189.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T06:35:58.219 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T06:35:58.275 INFO:teuthology.orchestra.run.smithi076.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T06:35:58.279 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T06:35:58.296 INFO:teuthology.orchestra.run.smithi189.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T06:35:58.368 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T06:35:58.487 INFO:teuthology.orchestra.run.smithi189.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T06:35:58.487 INFO:teuthology.orchestra.run.smithi189.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T06:35:58.746 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T06:35:58.762 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T06:35:58.804 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T06:35:59.158 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T06:35:59.174 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T06:35:59.216 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T06:35:59.644 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T06:35:59.660 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:35:59.705 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:35:59.958 INFO:teuthology.orchestra.run.smithi189.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T06:35:59.960 INFO:teuthology.orchestra.run.smithi189.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T06:35:59.961 INFO:teuthology.orchestra.run.smithi189.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T06:36:00.137 INFO:teuthology.orchestra.run.smithi189.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T06:36:00.223 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T06:36:00.241 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:36:00.281 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:36:00.651 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T06:36:00.669 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:36:00.709 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:36:00.783 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 199 MB in 6s (31.2 MB/s) 2024-09-16T06:36:00.978 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T06:36:01.033 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T06:36:01.038 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T06:36:01.112 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T06:36:01.118 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T06:36:01.128 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T06:36:01.162 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T06:36:01.480 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T06:36:01.497 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T06:36:01.538 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T06:36:01.556 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:01.682 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T06:36:01.858 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T06:36:01.878 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T06:36:01.917 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T06:36:02.378 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T06:36:02.394 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:36:02.412 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:02.472 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:36:02.570 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T06:36:02.981 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T06:36:02.999 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:36:03.040 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:36:03.307 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T06:36:03.328 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:03.357 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:03.376 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T06:36:03.395 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T06:36:03.434 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:36:03.710 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rados. 2024-09-16T06:36:03.726 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:03.768 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:03.821 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T06:36:03.837 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T06:36:03.879 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T06:36:04.146 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T06:36:04.162 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:04.213 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:04.273 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:04.382 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T06:36:04.490 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T06:36:04.508 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:04.548 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:04.901 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T06:36:04.917 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:04.959 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:05.105 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:05.230 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T06:36:05.279 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T06:36:05.297 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T06:36:05.339 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T06:36:05.642 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T06:36:05.705 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:05.705 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:05.942 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T06:36:05.963 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:05.992 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:06.061 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T06:36:06.078 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T06:36:06.119 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T06:36:06.311 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rados. 2024-09-16T06:36:06.328 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:06.370 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:06.448 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T06:36:06.463 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T06:36:06.506 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T06:36:06.755 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T06:36:06.772 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:06.813 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:06.851 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T06:36:06.869 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T06:36:06.909 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T06:36:07.049 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T06:36:07.065 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:07.107 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:07.278 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package librgw2. 2024-09-16T06:36:07.295 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:07.336 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:07.435 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T06:36:07.455 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:07.493 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:07.813 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T06:36:07.829 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T06:36:07.871 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T06:36:08.165 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T06:36:08.183 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:08.189 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T06:36:08.208 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:08.223 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:08.239 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:08.509 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T06:36:08.528 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T06:36:08.559 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T06:36:08.600 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liboath0. 2024-09-16T06:36:08.616 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T06:36:08.659 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T06:36:08.945 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T06:36:08.961 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T06:36:09.003 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T06:36:09.003 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T06:36:09.019 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:09.062 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:09.425 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T06:36:09.443 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T06:36:09.581 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T06:36:09.640 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-common. 2024-09-16T06:36:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:09.690 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:09.901 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librgw2. 2024-09-16T06:36:09.917 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:09.950 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:10.828 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T06:36:10.847 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:10.887 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:11.231 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liboath0. 2024-09-16T06:36:11.247 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T06:36:11.289 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T06:36:11.600 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T06:36:11.619 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:11.658 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-common. 2024-09-16T06:36:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:12.212 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-base. 2024-09-16T06:36:12.227 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:12.306 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:13.575 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T06:36:13.593 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T06:36:13.633 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T06:36:13.987 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T06:36:14.004 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T06:36:14.037 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T06:36:14.373 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T06:36:14.389 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T06:36:14.440 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T06:36:14.524 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-base. 2024-09-16T06:36:14.543 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:14.593 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:14.784 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T06:36:14.802 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T06:36:14.842 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T06:36:15.187 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-mako. 2024-09-16T06:36:15.203 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T06:36:15.245 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T06:36:15.632 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T06:36:15.648 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T06:36:15.723 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T06:36:15.804 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T06:36:15.819 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T06:36:15.862 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T06:36:16.026 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T06:36:16.042 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T06:36:16.109 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T06:36:16.198 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T06:36:16.214 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T06:36:16.256 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T06:36:16.412 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-webob. 2024-09-16T06:36:16.429 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T06:36:16.495 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T06:36:16.601 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T06:36:16.617 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T06:36:16.659 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T06:36:16.852 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T06:36:16.872 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T06:36:16.915 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T06:36:16.962 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T06:36:16.978 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T06:36:17.020 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T06:36:17.239 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T06:36:17.257 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T06:36:17.301 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T06:36:17.340 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-mako. 2024-09-16T06:36:17.356 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T06:36:17.398 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T06:36:17.671 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T06:36:17.689 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T06:36:17.726 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T06:36:17.732 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T06:36:17.744 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T06:36:17.776 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T06:36:18.074 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T06:36:18.093 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T06:36:18.112 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T06:36:18.131 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T06:36:18.132 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T06:36:18.170 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T06:36:18.460 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-paste. 2024-09-16T06:36:18.472 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webob. 2024-09-16T06:36:18.477 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T06:36:18.488 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T06:36:18.518 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T06:36:18.531 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T06:36:18.880 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T06:36:18.883 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T06:36:18.897 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T06:36:18.900 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T06:36:18.938 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T06:36:18.941 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T06:36:19.245 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T06:36:19.258 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T06:36:19.261 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T06:36:19.276 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T06:36:19.303 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T06:36:19.316 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T06:36:19.652 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T06:36:19.670 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T06:36:19.680 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T06:36:19.697 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T06:36:19.710 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T06:36:19.742 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T06:36:20.062 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T06:36:20.075 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T06:36:20.078 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T06:36:20.092 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T06:36:20.120 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T06:36:20.133 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T06:36:20.453 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-paste. 2024-09-16T06:36:20.471 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T06:36:20.500 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T06:36:20.511 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T06:36:20.517 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T06:36:20.558 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T06:36:20.805 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T06:36:20.823 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T06:36:20.864 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T06:36:20.878 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T06:36:20.895 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:20.936 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:21.183 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T06:36:21.200 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T06:36:21.241 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T06:36:21.333 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T06:36:21.352 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:21.391 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:21.519 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T06:36:21.539 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T06:36:21.577 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T06:36:21.752 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T06:36:21.769 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T06:36:21.810 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T06:36:21.863 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T06:36:21.883 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T06:36:21.944 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T06:36:22.139 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T06:36:22.157 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T06:36:22.197 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T06:36:22.293 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T06:36:22.309 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T06:36:22.351 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T06:36:22.576 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T06:36:22.595 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T06:36:22.658 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T06:36:22.661 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T06:36:22.676 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:22.738 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:23.024 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T06:36:23.044 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:23.082 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:23.121 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T06:36:23.140 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:23.179 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:23.507 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T06:36:23.527 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T06:36:23.565 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T06:36:23.603 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T06:36:23.619 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:23.661 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:23.843 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T06:36:23.863 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T06:36:23.901 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T06:36:24.272 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T06:36:24.289 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T06:36:24.355 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T06:36:24.683 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T06:36:24.700 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:24.713 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T06:36:24.730 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:24.741 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:24.772 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:26.111 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T06:36:26.129 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:26.169 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:27.101 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T06:36:27.117 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:27.159 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:27.309 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph. 2024-09-16T06:36:27.326 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:27.367 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:27.695 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T06:36:27.710 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:27.753 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:28.198 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T06:36:28.217 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:28.256 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:28.818 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package cephadm. 2024-09-16T06:36:28.836 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:28.876 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:29.229 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T06:36:29.248 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T06:36:29.287 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T06:36:29.312 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph. 2024-09-16T06:36:29.328 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:29.370 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:29.657 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T06:36:29.676 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T06:36:29.697 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T06:36:29.713 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:29.740 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T06:36:29.755 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:30.076 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T06:36:30.093 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:30.141 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T06:36:30.142 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:30.158 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:30.199 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:30.521 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T06:36:30.538 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T06:36:30.579 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T06:36:30.702 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package cephadm. 2024-09-16T06:36:30.720 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:30.760 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-attr. 2024-09-16T06:36:30.923 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T06:36:30.957 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T06:36:31.130 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T06:36:31.146 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T06:36:31.188 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T06:36:31.369 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T06:36:31.385 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T06:36:31.427 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T06:36:31.558 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T06:36:31.574 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T06:36:31.616 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T06:36:31.763 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-py. 2024-09-16T06:36:31.779 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T06:36:31.821 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T06:36:31.969 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T06:36:31.985 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:32.027 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:32.174 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T06:36:32.191 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T06:36:32.232 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T06:36:32.380 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T06:36:32.397 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T06:36:32.438 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T06:36:32.561 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T06:36:32.577 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T06:36:32.620 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T06:36:32.758 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-attr. 2024-09-16T06:36:32.776 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T06:36:32.816 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T06:36:32.946 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T06:36:32.962 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T06:36:33.004 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T06:36:33.127 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T06:36:33.144 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T06:36:33.185 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T06:36:33.342 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T06:36:33.358 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T06:36:33.400 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T06:36:33.505 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-py. 2024-09-16T06:36:33.521 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T06:36:33.563 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T06:36:33.770 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T06:36:33.786 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T06:36:33.828 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T06:36:33.891 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T06:36:33.911 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T06:36:33.949 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T06:36:34.106 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T06:36:34.124 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T06:36:34.164 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T06:36:34.277 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T06:36:34.293 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T06:36:34.335 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T06:36:34.466 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T06:36:34.482 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T06:36:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T06:36:34.604 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T06:36:34.621 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T06:36:34.662 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T06:36:34.819 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-routes. 2024-09-16T06:36:34.837 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T06:36:34.869 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T06:36:34.998 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T06:36:35.014 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T06:36:35.056 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T06:36:35.197 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T06:36:35.216 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:35.255 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:35.410 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T06:36:35.426 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T06:36:35.468 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T06:36:35.704 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T06:36:35.721 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T06:36:35.762 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T06:36:36.049 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T06:36:36.065 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T06:36:36.107 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T06:36:36.426 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-routes. 2024-09-16T06:36:36.444 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T06:36:36.484 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T06:36:36.744 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T06:36:36.762 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T06:36:36.802 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T06:36:36.854 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T06:36:36.874 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:36.912 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:37.080 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T06:36:37.097 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T06:36:37.138 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T06:36:37.482 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T06:36:37.498 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T06:36:37.540 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T06:36:38.334 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T06:36:38.352 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T06:36:38.392 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T06:36:38.694 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T06:36:38.714 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T06:36:38.753 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T06:36:38.833 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T06:36:38.850 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T06:36:38.891 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T06:36:39.047 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T06:36:39.068 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T06:36:39.097 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T06:36:39.461 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T06:36:39.478 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T06:36:39.520 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T06:36:39.906 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T06:36:39.922 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T06:36:39.964 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T06:36:40.380 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T06:36:40.400 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T06:36:40.447 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T06:36:40.716 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T06:36:40.736 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:40.774 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:41.141 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T06:36:41.159 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T06:36:41.408 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T06:36:41.762 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T06:36:41.780 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T06:36:41.828 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T06:36:42.135 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T06:36:42.153 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T06:36:42.185 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T06:36:42.497 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T06:36:42.502 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T06:36:42.516 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T06:36:42.518 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:42.555 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T06:36:42.560 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:42.900 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T06:36:42.920 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T06:36:42.958 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T06:36:43.385 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T06:36:43.402 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T06:36:43.443 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T06:36:43.666 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T06:36:43.684 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T06:36:43.716 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T06:36:43.755 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T06:36:43.775 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T06:36:43.813 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T06:36:44.035 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T06:36:44.055 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T06:36:44.093 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T06:36:44.100 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T06:36:44.118 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T06:36:44.158 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T06:36:44.438 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T06:36:44.454 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T06:36:44.496 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T06:36:44.511 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T06:36:44.527 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T06:36:44.578 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T06:36:44.919 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T06:36:44.932 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T06:36:44.935 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T06:36:44.949 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T06:36:44.976 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T06:36:44.990 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T06:36:45.289 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T06:36:45.305 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T06:36:45.335 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T06:36:45.347 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T06:36:45.351 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T06:36:45.393 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T06:36:45.687 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T06:36:45.705 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T06:36:45.745 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T06:36:45.749 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T06:36:45.768 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T06:36:45.807 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T06:36:46.057 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T06:36:46.073 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T06:36:46.115 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T06:36:46.370 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T06:36:46.389 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:46.437 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:46.444 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T06:36:46.463 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T06:36:46.502 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T06:36:46.797 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T06:36:46.814 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T06:36:46.838 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T06:36:46.855 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T06:36:46.856 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T06:36:46.896 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T06:36:47.232 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T06:36:47.235 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T06:36:47.249 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T06:36:47.253 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T06:36:47.290 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T06:36:47.293 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:36:47.587 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package jq. 2024-09-16T06:36:47.603 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T06:36:47.654 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:36:47.788 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T06:36:47.805 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:47.872 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:47.940 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package socat. 2024-09-16T06:36:47.957 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T06:36:47.998 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T06:36:48.202 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T06:36:48.218 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T06:36:48.260 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T06:36:48.334 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T06:36:48.351 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T06:36:48.384 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T06:36:48.647 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T06:36:48.664 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T06:36:48.704 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:36:48.730 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-test. 2024-09-16T06:36:48.746 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:48.788 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:48.958 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package jq. 2024-09-16T06:36:48.976 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T06:36:49.015 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:36:49.276 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package socat. 2024-09-16T06:36:49.293 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T06:36:49.334 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T06:36:49.656 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T06:36:49.672 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T06:36:49.714 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T06:36:50.061 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-test. 2024-09-16T06:36:50.077 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:50.118 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:52.917 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T06:36:52.936 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:52.975 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:53.287 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T06:36:53.305 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T06:36:53.345 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:36:53.857 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T06:36:53.876 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:53.915 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:54.335 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T06:36:54.351 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T06:36:54.363 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T06:36:54.381 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T06:36:54.413 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:54.445 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T06:36:54.709 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T06:36:54.725 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T06:36:54.750 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T06:36:54.767 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:36:54.767 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T06:36:54.850 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T06:36:55.227 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T06:36:55.245 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T06:36:55.285 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:36:55.295 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T06:36:55.311 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:36:55.353 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:36:55.748 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T06:36:55.768 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T06:36:55.815 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T06:36:56.152 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T06:36:56.170 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T06:36:56.211 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T06:36:56.589 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T06:36:56.608 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T06:36:56.649 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:36:56.855 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T06:36:56.874 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T06:36:56.912 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T06:36:57.274 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T06:36:57.290 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T06:36:57.332 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T06:36:57.668 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T06:36:57.684 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T06:36:57.726 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T06:36:58.087 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T06:36:58.106 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T06:36:58.145 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T06:36:58.262 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T06:36:58.282 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T06:36:58.320 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T06:36:58.649 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T06:36:58.665 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T06:36:58.706 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T06:36:58.707 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T06:36:58.725 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T06:36:58.764 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T06:36:59.051 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T06:36:59.071 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T06:36:59.075 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T06:36:59.094 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T06:36:59.109 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T06:36:59.133 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T06:36:59.465 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T06:36:59.483 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T06:36:59.503 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T06:36:59.521 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T06:36:59.523 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T06:36:59.561 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T06:36:59.851 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T06:36:59.869 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T06:36:59.909 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T06:37:00.039 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T06:37:00.056 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T06:37:00.097 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T06:37:00.203 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T06:37:00.222 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T06:37:00.261 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T06:37:00.426 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T06:37:00.443 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T06:37:00.484 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T06:37:00.837 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T06:37:00.856 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T06:37:00.895 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T06:37:01.014 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package radosgw. 2024-09-16T06:37:01.031 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:37:01.072 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:01.206 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T06:37:01.222 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T06:37:01.263 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T06:37:01.541 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T06:37:01.560 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T06:37:01.616 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T06:37:02.314 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package radosgw. 2024-09-16T06:37:02.332 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:37:02.372 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:02.526 INFO:teuthology.orchestra.run.smithi076.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T06:37:02.545 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:37:02.617 INFO:teuthology.orchestra.run.smithi076.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:02.929 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T06:37:03.265 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T06:37:03.425 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T06:37:03.550 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T06:37:03.760 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:03.767 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T06:37:03.785 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T06:37:03.833 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:04.165 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T06:37:04.328 INFO:teuthology.orchestra.run.smithi076.stdout:Adding system user cephadm....done 2024-09-16T06:37:04.454 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T06:37:04.517 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:37:04.580 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T06:37:04.626 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T06:37:04.706 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T06:37:04.925 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T06:37:05.312 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:37:05.438 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T06:37:05.469 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user cephadm....done 2024-09-16T06:37:05.589 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T06:37:05.655 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:37:05.784 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T06:37:05.949 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T06:37:06.111 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T06:37:06.368 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T06:37:06.395 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T06:37:06.554 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T06:37:06.669 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T06:37:06.680 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T06:37:06.954 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T06:37:07.008 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T06:37:07.230 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T06:37:07.401 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T06:37:07.531 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:07.694 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T06:37:07.865 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T06:37:07.979 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T06:37:08.016 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T06:37:08.280 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T06:37:08.292 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T06:37:08.581 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:08.601 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T06:37:08.761 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T06:37:08.890 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T06:37:09.016 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T06:37:09.153 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T06:37:09.284 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T06:37:09.455 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T06:37:09.576 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T06:37:09.744 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T06:37:09.764 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T06:37:10.120 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T06:37:10.132 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T06:37:10.421 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T06:37:10.424 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T06:37:10.722 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T06:37:10.759 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T06:37:11.056 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T06:37:11.060 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T06:37:11.332 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T06:37:11.369 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T06:37:11.520 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T06:37:11.629 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T06:37:11.675 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T06:37:11.914 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T06:37:11.960 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T06:37:12.532 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T06:37:12.611 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T06:37:12.736 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T06:37:12.880 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T06:37:12.917 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T06:37:13.172 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T06:37:13.311 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T06:37:13.469 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:13.557 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T06:37:13.796 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:13.893 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T06:37:13.930 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T06:37:14.250 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T06:37:14.253 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T06:37:14.376 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T06:37:14.379 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:14.521 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T06:37:14.731 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:14.804 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T06:37:14.865 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T06:37:15.147 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:15.183 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T06:37:15.342 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T06:37:15.348 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T06:37:15.469 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T06:37:15.482 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T06:37:15.753 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T06:37:15.810 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T06:37:16.085 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:16.102 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T06:37:16.214 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T06:37:16.340 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T06:37:16.387 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T06:37:16.546 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T06:37:16.666 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T06:37:16.826 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T06:37:16.950 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T06:37:17.116 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T06:37:17.252 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T06:37:17.378 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T06:37:17.467 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T06:37:17.683 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T06:37:17.840 INFO:teuthology.orchestra.run.smithi076.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T06:37:17.980 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T06:37:18.103 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T06:37:18.229 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T06:37:18.289 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T06:37:18.547 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T06:37:18.711 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T06:37:18.899 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T06:37:18.967 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T06:37:19.025 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T06:37:19.093 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T06:37:19.193 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T06:37:19.411 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T06:37:19.545 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T06:37:19.771 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T06:37:19.821 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T06:37:19.897 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T06:37:20.022 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T06:37:20.365 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T06:37:20.616 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T06:37:21.302 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:37:21.453 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T06:37:21.830 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T06:37:22.064 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:37:22.115 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T06:37:22.190 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T06:37:22.433 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:22.550 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T06:37:22.579 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T06:37:22.834 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T06:37:22.919 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T06:37:23.127 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T06:37:23.253 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T06:37:23.254 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T06:37:23.613 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T06:37:23.614 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T06:37:23.924 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T06:37:23.931 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T06:37:24.307 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T06:37:24.608 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T06:37:24.964 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T06:37:25.065 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T06:37:25.191 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:37:25.351 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T06:37:25.631 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T06:37:25.677 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T06:37:25.781 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T06:37:25.803 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T06:37:25.891 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T06:37:26.017 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T06:37:26.105 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T06:37:26.310 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T06:37:26.373 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T06:37:26.436 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T06:37:26.708 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T06:37:26.754 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T06:37:27.046 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T06:37:27.102 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:27.253 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T06:37:27.380 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:27.381 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T06:37:27.481 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:27.735 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:27.742 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:27.859 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T06:37:27.860 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T06:37:27.986 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.112 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.162 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.288 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T06:37:28.356 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.457 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T06:37:28.607 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.734 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.766 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:28.851 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T06:37:28.893 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.120 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T06:37:29.120 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T06:37:29.170 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.296 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.422 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.586 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.645 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T06:37:29.645 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T06:37:29.712 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:29.839 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T06:37:30.132 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:30.233 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:30.333 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T06:37:30.776 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:30.902 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.054 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.180 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.263 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.306 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.405 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.531 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.649 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.750 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:31.901 INFO:teuthology.orchestra.run.smithi076.stdout:Adding group ceph....done 2024-09-16T06:37:32.303 INFO:teuthology.orchestra.run.smithi189.stdout:Adding group ceph....done 2024-09-16T06:37:32.312 INFO:teuthology.orchestra.run.smithi076.stdout:Adding system user ceph....done 2024-09-16T06:37:32.480 INFO:teuthology.orchestra.run.smithi076.stdout:Setting system user ceph properties....done 2024-09-16T06:37:32.502 INFO:teuthology.orchestra.run.smithi076.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T06:37:32.651 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T06:37:32.720 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user ceph....done 2024-09-16T06:37:32.872 INFO:teuthology.orchestra.run.smithi189.stdout:Setting system user ceph properties....done 2024-09-16T06:37:32.895 INFO:teuthology.orchestra.run.smithi189.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T06:37:32.979 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T06:37:33.054 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T06:37:33.375 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T06:37:33.418 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:33.544 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:33.827 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:33.953 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:34.017 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T06:37:34.017 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T06:37:34.417 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T06:37:34.417 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T06:37:34.454 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:34.863 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:34.869 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T06:37:35.257 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T06:37:35.315 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:35.552 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T06:37:35.552 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T06:37:35.706 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:35.923 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T06:37:35.923 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T06:37:36.023 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:36.262 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T06:37:36.263 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T06:37:36.375 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:36.600 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T06:37:36.600 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T06:37:36.702 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:37.059 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:37.232 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T06:37:37.232 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T06:37:37.541 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T06:37:37.541 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T06:37:37.705 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:37.831 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:37.978 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:38.088 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:38.225 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:38.230 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:38.453 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T06:37:38.453 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T06:37:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T06:37:38.454 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T06:37:38.885 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:38.907 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.027 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.049 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.162 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.175 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.327 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.330 INFO:teuthology.orchestra.run.smithi076.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T06:37:39.692 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T06:37:39.730 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T06:37:39.843 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T06:37:39.881 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T06:37:40.137 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T06:37:40.199 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T06:37:43.104 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T06:37:43.123 INFO:teuthology.orchestra.run.smithi076.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T06:37:43.966 INFO:teuthology.orchestra.run.smithi189.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T06:37:43.972 DEBUG:teuthology.parallel:result is None 2024-09-16T06:37:44.143 INFO:teuthology.orchestra.run.smithi076.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T06:37:44.147 DEBUG:teuthology.parallel:result is None 2024-09-16T06:37:44.148 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T06:37:44.294 DEBUG:teuthology.orchestra.run.smithi076:> dpkg-query -W -f '${Version}' ceph 2024-09-16T06:37:44.318 INFO:teuthology.orchestra.run.smithi076.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T06:37:44.319 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T06:37:44.319 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T06:37:44.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T06:37:44.462 DEBUG:teuthology.orchestra.run.smithi189:> dpkg-query -W -f '${Version}' ceph 2024-09-16T06:37:44.486 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T06:37:44.486 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T06:37:44.486 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T06:37:44.488 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T06:37:44.489 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:44.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T06:37:44.505 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:44.505 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T06:37:44.544 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T06:37:44.544 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:44.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T06:37:44.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T06:37:44.617 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:44.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T06:37:44.632 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T06:37:44.643 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T06:37:44.644 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:44.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T06:37:44.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T06:37:44.675 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:44.675 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T06:37:44.701 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T06:37:44.711 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T06:37:44.711 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:44.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T06:37:44.734 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T06:37:44.744 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:44.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T06:37:44.760 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T06:37:44.817 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T06:37:44.900 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T06:37:44.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2024-09-16T06:37:44.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-09-16T06:37:44.918 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T06:37:44.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T06:37:44.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T06:37:44.976 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T06:37:44.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T06:37:44.979 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T06:37:44.998 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T06:37:44.998 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T06:37:44.998 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/many workloads/rados_mon_workunits}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906553', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'mon_bind_msgr2': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T06:37:44.999 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T06:37:45.036 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T06:37:45.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T06:37:45.045 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T06:37:45.059 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:45.059 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T06:37:45.104 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T06:37:45.104 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 345 Links: 1 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 06:34:15.595430711 +0000 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 06:34:15.591430803 +0000 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 06:34:15.591430803 +0000 2024-09-16T06:37:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2024-09-16T06:37:45.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T06:37:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T06:37:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T06:37:45.127 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000440073 s, 1.2 MB/s 2024-09-16T06:37:45.128 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T06:37:45.178 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 351 Links: 1 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 06:34:16.051420220 +0000 2024-09-16T06:37:45.185 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 06:34:16.047420312 +0000 2024-09-16T06:37:45.186 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 06:34:16.047420312 +0000 2024-09-16T06:37:45.186 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2024-09-16T06:37:45.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T06:37:45.197 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T06:37:45.197 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T06:37:45.198 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000400861 s, 1.3 MB/s 2024-09-16T06:37:45.198 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T06:37:45.250 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 357 Links: 1 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 06:34:16.683405679 +0000 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 06:34:16.491410097 +0000 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 06:34:16.491410097 +0000 2024-09-16T06:37:45.257 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2024-09-16T06:37:45.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T06:37:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T06:37:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T06:37:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000410494 s, 1.2 MB/s 2024-09-16T06:37:45.273 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T06:37:45.282 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-09-16T06:37:45.332 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-09-16 06:34:17.111395833 +0000 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-09-16 06:34:16.899400711 +0000 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-09-16 06:34:16.899400711 +0000 2024-09-16T06:37:45.333 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2024-09-16T06:37:45.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T06:37:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-09-16T06:37:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-09-16T06:37:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000362362 s, 1.4 MB/s 2024-09-16T06:37:45.346 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T06:37:45.353 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:45.353 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:45.353 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T06:37:45.360 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T06:37:45.360 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-09-16T06:37:45.368 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T06:37:45.368 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.368 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-09-16T06:37:45.368 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.368 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 06:34:14.875928394 +0000 2024-09-16T06:37:45.369 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 06:34:14.671933088 +0000 2024-09-16T06:37:45.369 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 06:34:14.671933088 +0000 2024-09-16T06:37:45.369 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T06:37:45.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T06:37:45.380 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T06:37:45.380 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T06:37:45.380 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000428894 s, 1.2 MB/s 2024-09-16T06:37:45.381 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T06:37:45.436 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 06:34:15.111922965 +0000 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 06:34:15.107923057 +0000 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 06:34:15.107923057 +0000 2024-09-16T06:37:45.444 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T06:37:45.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T06:37:45.457 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T06:37:45.457 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T06:37:45.457 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000317859 s, 1.6 MB/s 2024-09-16T06:37:45.458 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T06:37:45.466 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-09-16T06:37:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T06:37:45.516 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 372 Links: 1 2024-09-16T06:37:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.516 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 06:34:15.531913302 +0000 2024-09-16T06:37:45.517 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 06:34:15.527913394 +0000 2024-09-16T06:37:45.517 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 06:34:15.527913394 +0000 2024-09-16T06:37:45.517 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T06:37:45.517 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T06:37:45.531 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T06:37:45.532 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T06:37:45.532 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000398228 s, 1.3 MB/s 2024-09-16T06:37:45.533 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T06:37:45.539 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-16 06:34:16.155898945 +0000 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-16 06:34:15.943903823 +0000 2024-09-16T06:37:45.588 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-16 06:34:15.943903823 +0000 2024-09-16T06:37:45.589 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2024-09-16T06:37:45.589 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T06:37:45.603 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-16T06:37:45.603 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-16T06:37:45.603 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000373709 s, 1.4 MB/s 2024-09-16T06:37:45.604 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T06:37:45.612 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:45.612 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T06:37:45.612 INFO:tasks.ceph:Generating config... 2024-09-16T06:37:45.614 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T06:37:45.614 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T06:37:45.614 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T06:37:45.614 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T06:37:45.614 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T06:37:45.614 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T06:37:45.614 INFO:tasks.ceph:[global] ms type = async 2024-09-16T06:37:45.614 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T06:37:45.614 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T06:37:45.614 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T06:37:45.615 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T06:37:45.615 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T06:37:45.615 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-16T06:37:45.615 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T06:37:45.616 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T06:37:45.617 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T06:37:45.617 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T06:37:45.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T06:37:45.660 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T06:37:45.684 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T06:37:45.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T06:37:45.783 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.76:6789]'), ('mon.c', '[v1:172.21.15.76:6790]'), ('mon.b', '[v1:172.21.15.189:6789]')] 2024-09-16T06:37:45.784 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.76:6789],[v1:172.21.15.76:6790],[v1:172.21.15.189:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T06:37:45.785 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:45.785 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T06:37:45.790 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.76:6789]' --addv c '[v1:172.21.15.76:6790]' --addv b '[v1:172.21.15.189:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:45.877 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:45.877 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: generated fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:37:45.877 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = pacific 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 0 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:min_mon_release 16 (pacific) 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:election_strategy: 3 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:37:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T06:37:45.879 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T06:37:45.884 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 53fe3d11-e8d6-4d19-8f38-046f335b6765... 2024-09-16T06:37:45.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T06:37:45.887 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T06:37:45.919 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2024-09-16T06:37:45.920 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.921 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T06:37:45.922 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = "[v1:172.21.15.76:6789],[v1:172.21.15.76:6790],[v1:172.21.15.189:6789]" 2024-09-16T06:37:45.923 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: mon election default strategy = 3 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: ms bind msgr2 = False 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 5000 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: ms type = async 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.924 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 600 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2024-09-16T06:37:45.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2024-09-16T06:37:45.926 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore compression algorithm = zstd 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore compression mode = aggressive 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore zero block detection = True 2024-09-16T06:37:45.927 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 20 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 20 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 10 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: mon client ping interval = 4 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: mon client ping timeout = 12 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T06:37:45.928 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify cached snaps = True 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify missing on start = True 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock override recovery settings = True 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue = debug_random 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue cut off = debug_random 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2024-09-16T06:37:45.929 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2024-09-16T06:37:45.930 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T06:37:45.931 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.932 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2024-09-16T06:37:45.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon min osdmap epochs = 25 2024-09-16T06:37:45.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune interval = 2 2024-09-16T06:37:45.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune min = 15 2024-09-16T06:37:45.933 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune txsize = 2 2024-09-16T06:37:45.934 INFO:teuthology.orchestra.run.smithi076.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T06:37:45.934 INFO:teuthology.orchestra.run.smithi076.stdout: mon scrub inject missing keys = 0.05 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout: mon scrub interval = 300 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout: paxos service trim min = 5 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2024-09-16T06:37:45.935 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2024-09-16T06:37:45.936 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2024-09-16T06:37:45.936 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T06:37:45.936 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T06:37:45.936 INFO:teuthology.orchestra.run.smithi076.stdout: debug monc = 20 2024-09-16T06:37:45.936 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-09-16T06:37:45.937 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2024-09-16T06:37:45.937 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.c] 2024-09-16T06:37:45.937 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.b] 2024-09-16T06:37:45.940 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-16T06:37:45.940 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-16T06:37:45.940 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T06:37:45.940 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-16T06:37:45.941 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.941 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-16T06:37:45.941 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.941 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-16T06:37:45.941 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-16T06:37:45.942 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T06:37:45.943 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.944 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-16T06:37:45.944 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.944 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-16T06:37:45.944 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-16T06:37:45.944 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T06:37:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T06:37:45.946 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-16T06:37:45.946 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.946 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T06:37:45.946 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.946 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-16T06:37:45.947 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.947 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-16T06:37:45.947 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.947 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-16T06:37:45.947 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-16T06:37:45.948 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-16T06:37:45.948 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "[v1:172.21.15.76:6789],[v1:172.21.15.76:6790],[v1:172.21.15.189:6789]" 2024-09-16T06:37:45.948 INFO:teuthology.orchestra.run.smithi189.stdout: mon client directed command retry = 5 2024-09-16T06:37:45.948 INFO:teuthology.orchestra.run.smithi189.stdout: mon election default strategy = 3 2024-09-16T06:37:45.948 INFO:teuthology.orchestra.run.smithi189.stdout: ms bind msgr2 = False 2024-09-16T06:37:45.949 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject socket failures = 5000 2024-09-16T06:37:45.949 INFO:teuthology.orchestra.run.smithi189.stdout: ms type = async 2024-09-16T06:37:45.949 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:37:45.949 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.949 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-16T06:37:45.950 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-16T06:37:45.950 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.950 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-16T06:37:45.950 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-16T06:37:45.950 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-16T06:37:45.951 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-16T06:37:45.952 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-16T06:37:45.952 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.952 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-16T06:37:45.952 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-16T06:37:45.952 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.953 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-16T06:37:45.953 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.953 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-16T06:37:45.953 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.953 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-16T06:37:45.954 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-16T06:37:45.955 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-16T06:37:45.955 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression algorithm = zstd 2024-09-16T06:37:45.955 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression mode = aggressive 2024-09-16T06:37:45.955 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-16T06:37:45.955 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore zero block detection = True 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 20 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 20 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 10 2024-09-16T06:37:45.956 INFO:teuthology.orchestra.run.smithi189.stdout: mon client ping interval = 4 2024-09-16T06:37:45.957 INFO:teuthology.orchestra.run.smithi189.stdout: mon client ping timeout = 12 2024-09-16T06:37:45.957 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T06:37:45.957 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-16T06:37:45.957 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-16T06:37:45.957 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify cached snaps = True 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify missing on start = True 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock override recovery settings = True 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue = debug_random 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue cut off = debug_random 2024-09-16T06:37:45.958 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-16T06:37:45.959 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-16T06:37:45.960 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.961 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T06:37:45.962 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon min osdmap epochs = 25 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune interval = 2 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune min = 15 2024-09-16T06:37:45.963 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune txsize = 2 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub inject missing keys = 0.05 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub interval = 300 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: paxos service trim min = 5 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-16T06:37:45.964 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 20 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-16T06:37:45.965 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-16T06:37:45.965 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T06:37:45.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T06:37:46.036 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T06:37:46.037 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:46.037 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T06:37:46.047 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:46.047 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T06:37:46.053 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T06:37:46.053 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:46.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T06:37:46.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T06:37:46.118 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:37:46.119 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:46.125 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T06:37:46.126 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:46.126 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T06:37:46.126 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T06:37:46.153 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:37:46.153 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:46.204 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T06:37:46.204 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T06:37:46.204 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T06:37:46.254 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T06:37:46.264 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T06:37:46.264 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T06:37:46.264 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T06:37:46.304 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T06:37:46.317 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T06:37:46.318 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T06:37:46.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T06:37:46.334 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:46.334 INFO:tasks.ceph:role: osd.0 2024-09-16T06:37:46.335 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T06:37:46.335 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T06:37:46.883 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:46.884 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:46.886 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T06:37:46.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T06:37:46.987 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T06:37:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:47.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:47.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T06:37:47.058 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:47.058 INFO:tasks.ceph:role: osd.1 2024-09-16T06:37:47.058 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T06:37:47.058 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T06:37:47.565 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:47.565 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:47.566 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:47.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T06:37:47.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T06:37:47.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T06:37:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:47.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:47.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T06:37:47.670 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:47.670 INFO:tasks.ceph:role: osd.2 2024-09-16T06:37:47.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T06:37:47.670 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T06:37:48.134 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 2024-09-16T06:37:48.134 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:48.135 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:48.135 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:48.135 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:48.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-09-16T06:37:48.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T06:37:48.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T06:37:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:48.209 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:48.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:48.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T06:37:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T06:37:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T06:37:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T06:37:48.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T06:37:48.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:48.290+0000 7f75baee9880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T06:37:51.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T06:37:51.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T06:37:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T06:37:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T06:37:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T06:37:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T06:37:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:51.414+0000 7f1a8052b880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T06:37:54.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T06:37:54.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:54.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T06:37:54.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T06:37:54.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T06:37:54.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T06:37:54.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T06:37:54.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:37:54.474+0000 7faa9b359880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T06:37:57.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T06:37:57.466 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T06:37:57.484 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:57.484 INFO:tasks.ceph:role: osd.3 2024-09-16T06:37:57.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T06:37:57.484 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:58.066 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:58.068 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T06:37:58.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T06:37:58.168 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T06:37:58.181 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:58.182 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T06:37:58.193 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:58.193 INFO:tasks.ceph:role: osd.4 2024-09-16T06:37:58.193 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T06:37:58.194 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T06:37:58.734 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:58.734 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:58.734 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:58.734 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:58.735 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:58.736 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T06:37:58.736 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T06:37:58.772 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T06:37:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:58.781 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:58.782 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T06:37:58.839 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T06:37:58.839 INFO:tasks.ceph:role: osd.5 2024-09-16T06:37:58.839 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T06:37:58.840 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T06:37:59.338 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T06:37:59.338 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T06:37:59.338 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T06:37:59.339 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T06:37:59.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-16T06:37:59.340 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T06:37:59.403 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T06:37:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T06:37:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:37:59.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:37:59.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.490+0000 7f470f869880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T06:37:59.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.494+0000 7f470f869880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T06:37:59.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.494+0000 7f470f869880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T06:37:59.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.494+0000 7f470f869880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T06:37:59.495 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.494+0000 7f470f869880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T06:37:59.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:37:59.498+0000 7f470f869880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T06:38:02.489 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T06:38:02.504 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:38:02.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T06:38:02.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T06:38:02.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T06:38:02.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T06:38:02.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T06:38:02.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:02.594+0000 7f9948dea880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T06:38:05.567 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T06:38:05.583 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:38:05.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T06:38:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T06:38:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T06:38:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T06:38:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T06:38:05.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:05.626+0000 7fd5f1d15880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T06:38:08.624 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T06:38:08.641 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T06:38:08.641 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:08.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T06:38:08.659 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:08.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T06:38:08.675 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:08.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T06:38:08.689 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:08.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T06:38:08.705 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:08.705 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T06:38:08.765 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:08.765 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T06:38:08.775 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:08.775 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T06:38:08.786 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:08.786 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T06:38:08.793 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T06:38:08.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T06:38:08.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA80udmV/9oERAADp8g+S9PoQ1zzLd6givPug== 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA/0udmFXvlGBAA/NQmUWo5Mob4kgwFCJOGFQ== 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBC0udma7hyHBAAkAVCOMFkr7Z00rPGtkRh8A== 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA60udmZ5xQDxAA/njCPfpI5PnOkqqrBmkx8w== 2024-09-16T06:38:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBH0udmnYyOHRAA4ck4J46ARwp4j5AnHaUFMg== 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBK0udm1CCcIxAAZRc1R1RMY1eHMVVGhtwrDA== 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBN0udmkkiPJRAAOyZBXpuM3TF9LIRxXWNY+A== 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-09-16T06:38:08.807 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA60udm2vVIEhAAqKPkQamY9CooAiYGRGn57g== 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA80udmV/9oERAADp8g+S9PoQ1zzLd6givPug== 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA/0udmFXvlGBAA/NQmUWo5Mob4kgwFCJOGFQ== 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBC0udma7hyHBAAkAVCOMFkr7Z00rPGtkRh8A== 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA60udmZ5xQDxAA/njCPfpI5PnOkqqrBmkx8w== 2024-09-16T06:38:08.808 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.3] 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBH0udmnYyOHRAA4ck4J46ARwp4j5AnHaUFMg== 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.4] 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBK0udm1CCcIxAAZRc1R1RMY1eHMVVGhtwrDA== 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.5] 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBN0udmkkiPJRAAOyZBXpuM3TF9LIRxXWNY+A== 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-09-16T06:38:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA60udm2vVIEhAAqKPkQamY9CooAiYGRGn57g== 2024-09-16T06:38:08.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:08.812 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:08.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:08.906 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:08.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:08.956 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T06:38:09.005 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T06:38:09.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.091 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.147 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.197 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T06:38:09.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T06:38:09.292 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T06:38:09.343 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T06:38:09.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T06:38:09.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T06:38:09.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T06:38:09.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T06:38:09.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T06:38:09.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T06:38:09.615 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T06:38:09.633 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T06:38:09.767 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T06:38:09.784 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:38:09.787 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T06:38:09.792 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T06:38:09.792 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:38:09.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:38:09.795 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:38:09.795 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:38:09.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:38:09.797 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:38:09.797 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:38:09.797 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:38:09.835 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:38:09.835 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T06:38:09.836 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T06:38:09.836 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T06:38:09.838 INFO:tasks.ceph.mgr.x:Started 2024-09-16T06:38:09.838 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T06:38:09.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T06:38:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: get_monmap_and_config 2024-09-16T06:38:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: build_initial_monmap 2024-09-16T06:38:09.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: monmap: 2024-09-16T06:38:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:09.949255+0000 2024-09-16T06:38:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:09.949255+0000 2024-09-16T06:38:09.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:09.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 1 Processor -- start 2024-09-16T06:38:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 1 -- start start 2024-09-16T06:38:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: init 2024-09-16T06:38:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-c con 0x7fe2c80bb420 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-a con 0x7fe2c8029b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-b con 0x7fe2c80bb800 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c8024180 con 0x7fe2c8029b70 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c8024460 con 0x7fe2c80bb420 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c8027ac0 con 0x7fe2c80bb800 2024-09-16T06:38:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d6235700 10 monclient(hunting): authenticate will time out at 837.082590s 2024-09-16T06:38:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d5233700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2c80bb420 0x7fe2c8026fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:53258/0 (socket says 172.21.15.76:53258) 2024-09-16T06:38:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d5a34700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe2c80bb800 0x7fe2c80c3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60408/0 (socket says 172.21.15.76:60408) 2024-09-16T06:38:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:09.946+0000 7fe2d5233700 1 -- 172.21.15.76:0/1029739993 learned_addr learned my addr 172.21.15.76:0/1029739993 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:12.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): tick 2024-09-16T06:38:12.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:38:12.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6790/0 conn(0x7fe2c80bb800 legacy=0x7fe2c80c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.189:6789/0 conn(0x7fe2c80bb420 legacy=0x7fe2c8026fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6789/0 conn(0x7fe2c8029b70 legacy=0x7fe2c80c0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-b con 0x7fe2b4003760 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7fe2b4007420 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-c con 0x7fe2b400b150 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b400ee20 con 0x7fe2b4007420 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4010010 con 0x7fe2b400b150 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4011200 con 0x7fe2b4003760 2024-09-16T06:38:12.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:12.950+0000 7fe2c6ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): tick 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6790/0 conn(0x7fe2b4003760 legacy=0x7fe2b4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.189:6789/0 conn(0x7fe2b400b150 legacy=0x7fe2b400d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6789/0 conn(0x7fe2b4007420 legacy=0x7fe2b40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-b con 0x7fe2b40115e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7fe2b400efb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:15.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): picked mon.noname-c con 0x7fe2b400b150 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:15.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40123f0 con 0x7fe2b400efb0 2024-09-16T06:38:15.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b400fb90 con 0x7fe2b400b150 2024-09-16T06:38:15.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b400fd70 con 0x7fe2b40115e0 2024-09-16T06:38:15.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:15.950+0000 7fe2c6ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.158+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735513503 0 0) 0x7fe2b400fd70 con 0x7fe2b40115e0 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient(hunting): my global_id is 4100 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc0035d0 con 0x7fe2b40115e0 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473798981 0 0) 0x7fe2bc0035d0 con 0x7fe2b40115e0 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.189:6789/0 conn(0x7fe2b400b150 legacy=0x7fe2b400d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6789/0 conn(0x7fe2b400efb0 legacy=0x7fe2b400f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 monclient: found mon.noname-b 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40098c0 con 0x7fe2b40115e0 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 5 monclient: authenticate success, global_id 4100 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe2d0069400 con 0x7fe2b40115e0 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: dump: 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 1 -- 172.21.15.76:0/1029739993 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2d004ec20 con 0x7fe2b40115e0 2024-09-16T06:38:16.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:16.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 4 set_mon_vals no callback set 2024-09-16T06:38:16.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 10 monclient: shutdown 2024-09-16T06:38:16.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 -- 172.21.15.76:0/1029739993 >> v1:172.21.15.76:6790/0 conn(0x7fe2b40115e0 legacy=0x7fe2b4011a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 -- 172.21.15.76:0/1029739993 shutdown_connections 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 -- 172.21.15.76:0/1029739993 >> 172.21.15.76:0/1029739993 conn(0x7fe2c8006700 msgr2=0x7fe2c8028120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 -- 172.21.15.76:0/1029739993 shutdown_connections 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 -- 172.21.15.76:0/1029739993 wait complete. 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 10 monclient: monmap: 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:16.168 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.162+0000 7fe2d6235700 1 Processor -- start 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 1 -- start start 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient: init 2024-09-16T06:38:16.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-b con 0x7fe2b400b150 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-c con 0x7fe2b400efb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient(hunting): picked mon.noname-a con 0x7fe2b40115e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80a9470 con 0x7fe2b40115e0 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c8026850 con 0x7fe2b400b150 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80ab560 con 0x7fe2b400efb0 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d6235700 10 monclient(hunting): authenticate will time out at 843.300973s 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d4a32700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe2b400efb0 0x7fe2c8155760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33150/0 (socket says 172.21.15.76:33150) 2024-09-16T06:38:16.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2d4a32700 1 -- 172.21.15.76:0/3456163949 learned_addr learned my addr 172.21.15.76:0/3456163949 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325123355 0 0) 0x7fe2c80a9470 con 0x7fe2b40115e0 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): my global_id is 4104 2024-09-16T06:38:16.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b400fd70 con 0x7fe2b40115e0 2024-09-16T06:38:16.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 731471662 0 0) 0x7fe2b400fd70 con 0x7fe2b40115e0 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c80a9470 con 0x7fe2b40115e0 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe2cc001600 con 0x7fe2b40115e0 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:16.174 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.166+0000 7fe2c5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508886879 0 0) 0x7fe2c80ab560 con 0x7fe2b400efb0 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): my global_id is 4106 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b400fd70 con 0x7fe2b400efb0 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119477506 0 0) 0x7fe2c8026850 con 0x7fe2b400b150 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): my global_id is 4105 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c80ab560 con 0x7fe2b400b150 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470269404 0 0) 0x7fe2b400fd70 con 0x7fe2b400efb0 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c8026850 con 0x7fe2b400efb0 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe2c00016e0 con 0x7fe2b400efb0 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:16.176 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992935796 0 0) 0x7fe2c80ab560 con 0x7fe2b400b150 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b400fd70 con 0x7fe2b400b150 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe2d0069400 con 0x7fe2b400b150 2024-09-16T06:38:16.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 294235277 0 0) 0x7fe2c80a9470 con 0x7fe2b40115e0 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 >> v1:172.21.15.76:6790/0 conn(0x7fe2b400efb0 legacy=0x7fe2c8155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 >> v1:172.21.15.189:6789/0 conn(0x7fe2b400b150 legacy=0x7fe2c8150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 monclient: found mon.a 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c802cb50 con 0x7fe2b40115e0 2024-09-16T06:38:16.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 5 monclient: authenticate success, global_id 4104 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _renew_subs 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80a9980 con 0x7fe2b40115e0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _renew_subs 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80ab840 con 0x7fe2b40115e0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2cc0017e0 con 0x7fe2b40115e0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe2cc004890 con 0x7fe2b40115e0 2024-09-16T06:38:16.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c77fe700 4 set_mon_vals no callback set 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 10 monclient: dump: 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.181 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fe2cc004d70 con 0x7fe2b40115e0 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2141141204 0 0) 0x7fe2cc0060a0 con 0x7fe2b40115e0 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:16.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.170+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b0005100 con 0x7fe2b40115e0 2024-09-16T06:38:16.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.178+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fe2cc005250 con 0x7fe2b40115e0 2024-09-16T06:38:16.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.178+0000 7fe2c5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.178+0000 7fe2c5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:16.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.250+0000 7fe2d6235700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T06:38:16.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.250+0000 7fe2d6235700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T06:38:16.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.250+0000 7fe2d6235700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:16.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.250+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe2b00053b0 con 0x7fe2b40115e0 2024-09-16T06:38:16.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.254+0000 7fe2c5ffb700 1 -- 172.21.15.76:0/3456163949 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe2cc005470 con 0x7fe2b40115e0 2024-09-16T06:38:16.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.254+0000 7fe2c5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T06:38:16.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.254+0000 7fe2c5ffb700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T06:38:16.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 10 monclient: shutdown 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 >> v1:172.21.15.76:6789/0 conn(0x7fe2b40115e0 legacy=0x7fe2c8158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 shutdown_connections 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 >> 172.21.15.76:0/3456163949 conn(0x7fe2c8006700 msgr2=0x7fe2c8020a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 shutdown_connections 2024-09-16T06:38:16.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:16.258+0000 7fe2d6235700 1 -- 172.21.15.76:0/3456163949 wait complete. 2024-09-16T06:38:16.260 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-09-16T06:38:16.274 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T06:38:16.274 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T06:38:16.274 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:16.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T06:38:16.290 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:16.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T06:38:16.303 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-09-16T06:38:16.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T06:38:16.317 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:16.318 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T06:38:16.333 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:16.333 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T06:38:16.347 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:38:16.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T06:38:16.392 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:16.390+0000 7f623b64a2c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T06:38:16.405 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 85115802-a6ae-4a37-9bfa-b343e08d4648 0 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 10 monclient: monmap: 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:16.514730+0000 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:16.514730+0000 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.512 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.513 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:16.513 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:16.513 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 1 Processor -- start 2024-09-16T06:38:16.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 1 -- start start 2024-09-16T06:38:16.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.510+0000 7f51f9b4e700 10 monclient: init 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-b con 0x7f51f4132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-a con 0x7f51f406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-c con 0x7f51f406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f406c280 con 0x7f51f406d480 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f406bfa0 con 0x7f51f406cf50 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f409ace0 con 0x7f51f4132330 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient(hunting): authenticate will time out at 849.028233s 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f934d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51f406cf50 0x7f51f413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55142/0 (socket says 172.21.15.189:55142) 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f934d700 1 -- 172.21.15.189:0/2303690584 learned_addr learned my addr 172.21.15.189:0/2303690584 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667726584 0 0) 0x7f51f406bfa0 con 0x7f51f406cf50 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient(hunting): my global_id is 4111 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e40035d0 con 0x7f51f406cf50 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85223954 0 0) 0x7f51e40035d0 con 0x7f51f406cf50 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 >> v1:172.21.15.76:6790/0 conn(0x7f51f4132330 legacy=0x7f51f4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 >> v1:172.21.15.76:6789/0 conn(0x7f51f406d480 legacy=0x7f51f413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 monclient: found mon.noname-c 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f409ae70 con 0x7f51f406cf50 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 5 monclient: authenticate success, global_id 4111 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51ec001990 con 0x7f51f406cf50 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: dump: 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.519 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51ec001cb0 con 0x7f51f406cf50 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 4 set_mon_vals no callback set 2024-09-16T06:38:16.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 1 -- 172.21.15.189:0/2303690584 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51ec001ee0 con 0x7f51f406cf50 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f37fe700 10 monclient: dump: 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: shutdown 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- 172.21.15.189:0/2303690584 >> v1:172.21.15.189:6789/0 conn(0x7f51f406cf50 legacy=0x7f51f413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- 172.21.15.189:0/2303690584 shutdown_connections 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- 172.21.15.189:0/2303690584 >> 172.21.15.189:0/2303690584 conn(0x7f51f40a41c0 msgr2=0x7f51f40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- 172.21.15.189:0/2303690584 shutdown_connections 2024-09-16T06:38:16.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 1 -- 172.21.15.189:0/2303690584 wait complete. 2024-09-16T06:38:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.514+0000 7f51f9b4e700 10 monclient: monmap: 2024-09-16T06:38:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:16.526 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 Processor -- start 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- start start 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: init 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-a con 0x7f51f406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-b con 0x7f51f406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient(hunting): picked mon.noname-c con 0x7f51f4132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f4138f10 con 0x7f51f406cf50 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f41360e0 con 0x7f51f406d480 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f4137b70 con 0x7f51f4132330 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient(hunting): authenticate will time out at 849.033153s 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51f406d480 0x7f51f41359d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55150/0 (socket says 172.21.15.189:55150) 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f3fff700 1 -- 172.21.15.189:0/3493750649 learned_addr learned my addr 172.21.15.189:0/3493750649 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549777973 0 0) 0x7f51f41360e0 con 0x7f51f406d480 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.528 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): my global_id is 4114 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d80035d0 con 0x7f51f406d480 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531888239 0 0) 0x7f51f4137b70 con 0x7f51f4132330 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): my global_id is 4115 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f41360e0 con 0x7f51f4132330 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051714186 0 0) 0x7f51f4138f10 con 0x7f51f406cf50 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): my global_id is 4113 2024-09-16T06:38:16.529 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f4137b70 con 0x7f51f406cf50 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067568951 0 0) 0x7f51d80035d0 con 0x7f51f406d480 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51f4138f10 con 0x7f51f406d480 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51e00019c0 con 0x7f51f406d480 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): dump: 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.530 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3976653824 0 0) 0x7f51f4138f10 con 0x7f51f406d480 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 >> v1:172.21.15.76:6790/0 conn(0x7f51f4132330 legacy=0x7f51f41d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 >> v1:172.21.15.76:6789/0 conn(0x7f51f406cf50 legacy=0x7f51f41352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 monclient: found mon.b 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.531 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f4131160 con 0x7f51f406d480 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 5 monclient: authenticate success, global_id 4114 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _renew_subs 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f4137e30 con 0x7f51f406d480 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _renew_subs 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51e0001d30 con 0x7f51f406d480 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f4138c30 con 0x7f51f406d480 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51e0003b90 con 0x7f51f406d480 2024-09-16T06:38:16.532 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 10 monclient: dump: 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f2ffd700 4 set_mon_vals no callback set 2024-09-16T06:38:16.533 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f51e00040c0 con 0x7f51f406d480 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 507215682 0 0) 0x7f51e0006570 con 0x7f51f406d480 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.518+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f409dcd0 con 0x7f51f406d480 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.522+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f51e0005850 con 0x7f51f406d480 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.522+0000 7f51f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.534 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.522+0000 7f51f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:16.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.598+0000 7f51f9b4e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "85115802-a6ae-4a37-9bfa-b343e08d4648", "id": 0}] 2024-09-16T06:38:16.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.598+0000 7f51f9b4e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "85115802-a6ae-4a37-9bfa-b343e08d4648", "id": 0}] 2024-09-16T06:38:16.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.598+0000 7f51f9b4e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.598+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "85115802-a6ae-4a37-9bfa-b343e08d4648", "id": 0} v 0) v1 -- 0x7f51f41385c0 con 0x7f51f406d480 2024-09-16T06:38:16.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f17fa700 1 -- 172.21.15.189:0/3493750649 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "85115802-a6ae-4a37-9bfa-b343e08d4648", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 50128450 0 732707515) 0x7f51e00059f0 con 0x7f51f406d480 2024-09-16T06:38:16.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "85115802-a6ae-4a37-9bfa-b343e08d4648", "id": 0}] 2024-09-16T06:38:16.604 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:16.605 INFO:teuthology.orchestra.run.smithi189.stdout:0 2024-09-16T06:38:16.605 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f9b4e700 10 monclient: shutdown 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f9b4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 >> v1:172.21.15.189:6789/0 conn(0x7f51f406d480 legacy=0x7f51f41359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 shutdown_connections 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.602+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 >> 172.21.15.189:0/3493750649 conn(0x7f51f40a41c0 msgr2=0x7f51f40a1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.606+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 shutdown_connections 2024-09-16T06:38:16.606 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.606+0000 7f51f9b4e700 1 -- 172.21.15.189:0/3493750649 wait complete. 2024-09-16T06:38:16.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 99a99267-bac0-4365-af54-14423447fbc5 1 2024-09-16T06:38:16.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 10 monclient: get_monmap_and_config 2024-09-16T06:38:16.728 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 10 monclient: monmap: 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:16.731337+0000 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:16.731337+0000 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:16.729 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 1 Processor -- start 2024-09-16T06:38:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 1 -- start start 2024-09-16T06:38:16.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.726+0000 7f1fe6044700 10 monclient: init 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-a con 0x7f1fe0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-c con 0x7f1fe006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-b con 0x7f1fe006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006c280 con 0x7f1fe0132330 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006bfa0 con 0x7f1fe006d480 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe009ace0 con 0x7f1fe006cf50 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): authenticate will time out at 849.244765s 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe4841700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fe006d480 0x7f1fe013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55164/0 (socket says 172.21.15.189:55164) 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe4841700 1 -- 172.21.15.189:0/2797579255 learned_addr learned my addr 172.21.15.189:0/2797579255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361297180 0 0) 0x7f1fe006bfa0 con 0x7f1fe006d480 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient(hunting): my global_id is 4117 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc80035d0 con 0x7f1fe006d480 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244029216 0 0) 0x7f1fc80035d0 con 0x7f1fe006d480 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 >> v1:172.21.15.76:6790/0 conn(0x7f1fe006cf50 legacy=0x7f1fe013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 >> v1:172.21.15.76:6789/0 conn(0x7f1fe0132330 legacy=0x7f1fe0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 monclient: found mon.noname-c 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe009ae70 con 0x7f1fe006d480 2024-09-16T06:38:16.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1fd4001610 con 0x7f1fe006d480 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 5 monclient: authenticate success, global_id 4117 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: dump: 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.735 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.736 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.736 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.736 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.736 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd40018c0 con 0x7f1fe006d480 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 1 -- 172.21.15.189:0/2797579255 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1fd4001af0 con 0x7f1fe006d480 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fdffff700 10 monclient: dump: 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.737 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 4 set_mon_vals no callback set 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: shutdown 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- 172.21.15.189:0/2797579255 >> v1:172.21.15.189:6789/0 conn(0x7f1fe006d480 legacy=0x7f1fe013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- 172.21.15.189:0/2797579255 shutdown_connections 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- 172.21.15.189:0/2797579255 >> 172.21.15.189:0/2797579255 conn(0x7f1fe00a41c0 msgr2=0x7f1fe00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- 172.21.15.189:0/2797579255 shutdown_connections 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- 172.21.15.189:0/2797579255 wait complete. 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.738 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: monmap: 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:16.739 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:16.740 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 Processor -- start 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 1 -- start start 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: init 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.730+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-c con 0x7f1fe006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-a con 0x7f1fe006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient(hunting): picked mon.noname-b con 0x7f1fe0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe01303b0 con 0x7f1fe006d480 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe012e840 con 0x7f1fe0132330 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe012d450 con 0x7f1fe006cf50 2024-09-16T06:38:16.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient(hunting): authenticate will time out at 849.248097s 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe5843700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1fe0132330 0x7f1fe01d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55176/0 (socket says 172.21.15.189:55176) 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe5843700 1 -- 172.21.15.189:0/538440435 learned_addr learned my addr 172.21.15.189:0/538440435 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585118685 0 0) 0x7f1fe009ace0 con 0x7f1fe0132330 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): my global_id is 4120 2024-09-16T06:38:16.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc0035d0 con 0x7f1fe0132330 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602570883 0 0) 0x7f1fe012d450 con 0x7f1fe006cf50 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): my global_id is 4121 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe009ace0 con 0x7f1fe006cf50 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927091640 0 0) 0x7f1fe01303b0 con 0x7f1fe006d480 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): my global_id is 4119 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe012d450 con 0x7f1fe006d480 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069096801 0 0) 0x7f1fcc0035d0 con 0x7f1fe0132330 2024-09-16T06:38:16.747 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe01303b0 con 0x7f1fe0132330 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1fd0001680 con 0x7f1fe0132330 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): dump: 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.748 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325474785 0 0) 0x7f1fe01303b0 con 0x7f1fe0132330 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 >> v1:172.21.15.76:6790/0 conn(0x7f1fe006cf50 legacy=0x7f1fe012c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 >> v1:172.21.15.76:6789/0 conn(0x7f1fe006d480 legacy=0x7f1fe012cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 monclient: found mon.b 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe009e8b0 con 0x7f1fe0132330 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.749 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 5 monclient: authenticate success, global_id 4120 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _renew_subs 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe01313f0 con 0x7f1fe0132330 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _renew_subs 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe012fb10 con 0x7f1fe0132330 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd0001bf0 con 0x7f1fe0132330 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1fd0003860 con 0x7f1fe0132330 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fdf7fe700 4 set_mon_vals no callback set 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 10 monclient: dump: 2024-09-16T06:38:16.750 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f1fd0003cc0 con 0x7f1fe0132330 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1019711942 0 0) 0x7f1fd0006020 con 0x7f1fe0132330 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.734+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0005100 con 0x7f1fe0132330 2024-09-16T06:38:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.738+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f1fd0004150 con 0x7f1fe0132330 2024-09-16T06:38:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.738+0000 7f1fddffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.752 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.738+0000 7f1fddffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.814+0000 7f1fe6044700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "99a99267-bac0-4365-af54-14423447fbc5", "id": 1}] 2024-09-16T06:38:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.814+0000 7f1fe6044700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "99a99267-bac0-4365-af54-14423447fbc5", "id": 1}] 2024-09-16T06:38:16.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.814+0000 7f1fe6044700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.814+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "99a99267-bac0-4365-af54-14423447fbc5", "id": 1} v 0) v1 -- 0x7f1fc0005930 con 0x7f1fe0132330 2024-09-16T06:38:16.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.818+0000 7f1fddffb700 1 -- 172.21.15.189:0/538440435 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "99a99267-bac0-4365-af54-14423447fbc5", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2777715052 0 940483276) 0x7f1fd0004370 con 0x7f1fe0132330 2024-09-16T06:38:16.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.818+0000 7f1fddffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "99a99267-bac0-4365-af54-14423447fbc5", "id": 1}] 2024-09-16T06:38:16.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.818+0000 7f1fddffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:16.821 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 10 monclient: shutdown 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 >> v1:172.21.15.189:6789/0 conn(0x7f1fe0132330 legacy=0x7f1fe01d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 shutdown_connections 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 >> 172.21.15.189:0/538440435 conn(0x7f1fe00a41c0 msgr2=0x7f1fe0129e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 shutdown_connections 2024-09-16T06:38:16.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.822+0000 7f1fe6044700 1 -- 172.21.15.189:0/538440435 wait complete. 2024-09-16T06:38:16.835 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 6d2d3abd-a22c-443d-a7c7-8c53e04088ba 2 2024-09-16T06:38:16.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: get_monmap_and_config 2024-09-16T06:38:16.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.947 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: monmap: 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:16.949958+0000 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:16.949958+0000 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:16.948 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 1 Processor -- start 2024-09-16T06:38:16.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 1 -- start start 2024-09-16T06:38:16.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: init 2024-09-16T06:38:16.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.952 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-c con 0x7f6b580a47f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-b con 0x7f6b580a52e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-a con 0x7f6b5809e4e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5806cc40 con 0x7f6b5809e4e0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5806c530 con 0x7f6b580a47f0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5809e0c0 con 0x7f6b580a52e0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b56d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6b580a47f0 0x7f6b580a4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55186/0 (socket says 172.21.15.189:55186) 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b56d9d700 1 -- 172.21.15.189:0/2918191629 learned_addr learned my addr 172.21.15.189:0/2918191629 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b5d224700 10 monclient(hunting): authenticate will time out at 849.463572s 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000304100 0 0) 0x7f6b5806c530 con 0x7f6b580a47f0 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b55d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b55d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.953 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b55d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.946+0000 7f6b55d9b700 10 monclient(hunting): my global_id is 4123 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b440035d0 con 0x7f6b580a47f0 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136602294 0 0) 0x7f6b5806cc40 con 0x7f6b5809e4e0 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): my global_id is 4122 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5806c530 con 0x7f6b5809e4e0 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971784566 0 0) 0x7f6b5809e0c0 con 0x7f6b580a52e0 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.954 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient(hunting): my global_id is 4124 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5806cc40 con 0x7f6b580a52e0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 109596526 0 0) 0x7f6b440035d0 con 0x7f6b580a47f0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 >> v1:172.21.15.76:6790/0 conn(0x7f6b580a52e0 legacy=0x7f6b5813a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 >> v1:172.21.15.76:6789/0 conn(0x7f6b5809e4e0 legacy=0x7f6b5813dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 monclient: found mon.noname-c 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5809e910 con 0x7f6b580a47f0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 5 monclient: authenticate success, global_id 4123 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b48001610 con 0x7f6b580a47f0 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: dump: 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.956 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b480018f0 con 0x7f6b580a47f0 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 1 -- 172.21.15.189:0/2918191629 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b48001c80 con 0x7f6b580a47f0 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 4 set_mon_vals no callback set 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b55d9b700 10 monclient: dump: 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.957 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: shutdown 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- 172.21.15.189:0/2918191629 >> v1:172.21.15.189:6789/0 conn(0x7f6b580a47f0 legacy=0x7f6b580a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- 172.21.15.189:0/2918191629 shutdown_connections 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- 172.21.15.189:0/2918191629 >> 172.21.15.189:0/2918191629 conn(0x7f6b580a23e0 msgr2=0x7f6b580a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- 172.21.15.189:0/2918191629 shutdown_connections 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- 172.21.15.189:0/2918191629 wait complete. 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: build_initial_monmap 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: monmap: 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:16.959 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:16.960 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 Processor -- start 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 1 -- start start 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: init 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.950+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-a con 0x7f6b5809e4e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:16.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-b con 0x7f6b580a47f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient(hunting): picked mon.noname-c con 0x7f6b580a52e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5811d490 con 0x7f6b5809e4e0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580724e0 con 0x7f6b580a47f0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5813e570 con 0x7f6b580a52e0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient(hunting): authenticate will time out at 849.468173s 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6b580a47f0 0x7f6b581cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55190/0 (socket says 172.21.15.189:55190) 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5659c700 1 -- 172.21.15.189:0/1522867206 learned_addr learned my addr 172.21.15.189:0/1522867206 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581400443 0 0) 0x7f6b580724e0 con 0x7f6b580a47f0 2024-09-16T06:38:16.968 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): my global_id is 4126 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b380035d0 con 0x7f6b580a47f0 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791234487 0 0) 0x7f6b5811d490 con 0x7f6b5809e4e0 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): my global_id is 4125 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b580724e0 con 0x7f6b5809e4e0 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348945251 0 0) 0x7f6b5813e570 con 0x7f6b580a52e0 2024-09-16T06:38:16.969 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): my global_id is 4127 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5811d490 con 0x7f6b580a52e0 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925994693 0 0) 0x7f6b380035d0 con 0x7f6b580a47f0 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b5813e570 con 0x7f6b580a47f0 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b40001930 con 0x7f6b580a47f0 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.970 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975926965 0 0) 0x7f6b580724e0 con 0x7f6b5809e4e0 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b380035d0 con 0x7f6b5809e4e0 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b480015d0 con 0x7f6b5809e4e0 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.971 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375178100 0 0) 0x7f6b5811d490 con 0x7f6b580a52e0 2024-09-16T06:38:16.972 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b580724e0 con 0x7f6b580a52e0 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b50001bf0 con 0x7f6b580a52e0 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.973 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1732223495 0 0) 0x7f6b5813e570 con 0x7f6b580a47f0 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 >> v1:172.21.15.76:6790/0 conn(0x7f6b580a52e0 legacy=0x7f6b581d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 >> v1:172.21.15.76:6789/0 conn(0x7f6b5809e4e0 legacy=0x7f6b581ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 monclient: found mon.b 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580a1810 con 0x7f6b580a47f0 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:16.974 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 5 monclient: authenticate success, global_id 4126 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _renew_subs 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5811d770 con 0x7f6b580a47f0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _renew_subs 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5813ce30 con 0x7f6b580a47f0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b40001cc0 con 0x7f6b580a47f0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5559a700 4 set_mon_vals no callback set 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b40003b10 con 0x7f6b580a47f0 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:16.975 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 10 monclient: dump: 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f6b40003ff0 con 0x7f6b580a47f0 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2994956795 0 0) 0x7f6b40006540 con 0x7f6b580a47f0 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:16.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.954+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b580008d0 con 0x7f6b580a47f0 2024-09-16T06:38:16.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.958+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f6b400042a0 con 0x7f6b580a47f0 2024-09-16T06:38:16.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.958+0000 7f6b4f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:16.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:16.958+0000 7f6b4f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.034+0000 7f6b5d224700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d2d3abd-a22c-443d-a7c7-8c53e04088ba", "id": 2}] 2024-09-16T06:38:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.034+0000 7f6b5d224700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d2d3abd-a22c-443d-a7c7-8c53e04088ba", "id": 2}] 2024-09-16T06:38:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.034+0000 7f6b5d224700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.034+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6d2d3abd-a22c-443d-a7c7-8c53e04088ba", "id": 2} v 0) v1 -- 0x7f6b5813d0e0 con 0x7f6b580a47f0 2024-09-16T06:38:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.038+0000 7f6b4f7fe700 1 -- 172.21.15.189:0/1522867206 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d2d3abd-a22c-443d-a7c7-8c53e04088ba", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1848650676 0 216599125) 0x7f6b40006120 con 0x7f6b580a47f0 2024-09-16T06:38:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.038+0000 7f6b4f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d2d3abd-a22c-443d-a7c7-8c53e04088ba", "id": 2}] 2024-09-16T06:38:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.038+0000 7f6b4f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:17.042 INFO:teuthology.orchestra.run.smithi189.stdout:2 2024-09-16T06:38:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 10 monclient: shutdown 2024-09-16T06:38:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 >> v1:172.21.15.189:6789/0 conn(0x7f6b580a47f0 legacy=0x7f6b581cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 shutdown_connections 2024-09-16T06:38:17.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 >> 172.21.15.189:0/1522867206 conn(0x7f6b580a23e0 msgr2=0x7f6b5813ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 shutdown_connections 2024-09-16T06:38:17.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.042+0000 7f6b5d224700 1 -- 172.21.15.189:0/1522867206 wait complete. 2024-09-16T06:38:17.055 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new b39276aa-291e-4283-9006-7b82b23dd012 3 2024-09-16T06:38:17.139 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.138+0000 7f623b64a2c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T06:38:17.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: get_monmap_and_config 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: monmap: 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:17.171119+0000 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:17.171119+0000 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:17.169 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 1 Processor -- start 2024-09-16T06:38:17.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 1 -- start start 2024-09-16T06:38:17.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: init 2024-09-16T06:38:17.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.166+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-c con 0x7fb2b0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b006c280 con 0x7fb2b006d480 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b006bfa0 con 0x7fb2b0132330 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b009ace0 con 0x7fb2b006cf50 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient(hunting): authenticate will time out at 849.684128s 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2af7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb2b0132330 0x7fb2b0132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55192/0 (socket says 172.21.15.189:55192) 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2af7fe700 1 -- 172.21.15.189:0/2887473117 learned_addr learned my addr 172.21.15.189:0/2887473117 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252133047 0 0) 0x7fb2b006bfa0 con 0x7fb2b0132330 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): my global_id is 4129 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2980035d0 con 0x7fb2b0132330 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642550814 0 0) 0x7fb2b006c280 con 0x7fb2b006d480 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): my global_id is 4128 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b006bfa0 con 0x7fb2b006d480 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227698685 0 0) 0x7fb2b009ace0 con 0x7fb2b006cf50 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient(hunting): my global_id is 4130 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b006c280 con 0x7fb2b006cf50 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456044036 0 0) 0x7fb2980035d0 con 0x7fb2b0132330 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 >> v1:172.21.15.76:6790/0 conn(0x7fb2b006cf50 legacy=0x7fb2b013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 >> v1:172.21.15.76:6789/0 conn(0x7fb2b006d480 legacy=0x7fb2b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 monclient: found mon.noname-c 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b009ae70 con 0x7fb2b0132330 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 5 monclient: authenticate success, global_id 4129 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a0001610 con 0x7fb2b0132330 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: dump: 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a00018f0 con 0x7fb2b0132330 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 4 set_mon_vals no callback set 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 1 -- 172.21.15.189:0/2887473117 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a0001ae0 con 0x7fb2b0132330 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2ae7fc700 10 monclient: dump: 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient: shutdown 2024-09-16T06:38:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- 172.21.15.189:0/2887473117 >> v1:172.21.15.189:6789/0 conn(0x7fb2b0132330 legacy=0x7fb2b0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- 172.21.15.189:0/2887473117 shutdown_connections 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- 172.21.15.189:0/2887473117 >> 172.21.15.189:0/2887473117 conn(0x7fb2b00a41c0 msgr2=0x7fb2b00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- 172.21.15.189:0/2887473117 shutdown_connections 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- 172.21.15.189:0/2887473117 wait complete. 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient: monmap: 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 Processor -- start 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 1 -- start start 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.170+0000 7fb2b4abf700 10 monclient: init 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-c con 0x7fb2b006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b0131260 con 0x7fb2b0132330 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b012e350 con 0x7fb2b006cf50 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b012f510 con 0x7fb2b006d480 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient(hunting): authenticate will time out at 849.688202s 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2af7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb2b006cf50 0x7fb2b012d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55196/0 (socket says 172.21.15.189:55196) 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2af7fe700 1 -- 172.21.15.189:0/691076056 learned_addr learned my addr 172.21.15.189:0/691076056 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391093454 0 0) 0x7fb2b012e350 con 0x7fb2b006cf50 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): my global_id is 4132 2024-09-16T06:38:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c0035d0 con 0x7fb2b006cf50 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270691247 0 0) 0x7fb2b012f510 con 0x7fb2b006d480 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): my global_id is 4133 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b012e350 con 0x7fb2b006d480 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4266359825 0 0) 0x7fb29c0035d0 con 0x7fb2b006cf50 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b012f510 con 0x7fb2b006cf50 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a0004890 con 0x7fb2b006cf50 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:17.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): dump: 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569341333 0 0) 0x7fb2b0131260 con 0x7fb2b0132330 2024-09-16T06:38:17.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient(hunting): my global_id is 4131 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c0035d0 con 0x7fb2b0132330 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 784711743 0 0) 0x7fb2b012f510 con 0x7fb2b006cf50 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 >> v1:172.21.15.76:6790/0 conn(0x7fb2b006d480 legacy=0x7fb2b012dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 >> v1:172.21.15.76:6789/0 conn(0x7fb2b0132330 legacy=0x7fb2b01d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 monclient: found mon.b 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b0131770 con 0x7fb2b006cf50 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 5 monclient: authenticate success, global_id 4132 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _renew_subs 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b009e8b0 con 0x7fb2b006cf50 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _renew_subs 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b0133270 con 0x7fb2b006cf50 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2a0004ca0 con 0x7fb2b006cf50 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b0132710 con 0x7fb2b006cf50 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a0004fb0 con 0x7fb2b006cf50 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 10 monclient: dump: 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.193 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb2a0005410 con 0x7fb2b006cf50 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb2ad7fa700 4 set_mon_vals no callback set 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.174+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3733584740 0 0) 0x7fb2a0006840 con 0x7fb2b006cf50 2024-09-16T06:38:17.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.178+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fb2a0006b60 con 0x7fb2b006cf50 2024-09-16T06:38:17.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.178+0000 7fb297fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.178+0000 7fb297fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:17.200 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.198+0000 7f623b64a2c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T06:38:17.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.254+0000 7fb2b4abf700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b39276aa-291e-4283-9006-7b82b23dd012", "id": 3}] 2024-09-16T06:38:17.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.254+0000 7fb2b4abf700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b39276aa-291e-4283-9006-7b82b23dd012", "id": 3}] 2024-09-16T06:38:17.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.254+0000 7fb2b4abf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.256 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.254+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b39276aa-291e-4283-9006-7b82b23dd012", "id": 3} v 0) v1 -- 0x7fb2b00a3430 con 0x7fb2b006cf50 2024-09-16T06:38:17.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb297fff700 1 -- 172.21.15.189:0/691076056 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b39276aa-291e-4283-9006-7b82b23dd012", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2031319699 0 525046306) 0x7fb2b00a3430 con 0x7fb2b006cf50 2024-09-16T06:38:17.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb297fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b39276aa-291e-4283-9006-7b82b23dd012", "id": 3}] 2024-09-16T06:38:17.263 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb297fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:17.263 INFO:teuthology.orchestra.run.smithi189.stdout:3 2024-09-16T06:38:17.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 10 monclient: shutdown 2024-09-16T06:38:17.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.264 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 >> v1:172.21.15.189:6789/0 conn(0x7fb2b006cf50 legacy=0x7fb2b012d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 shutdown_connections 2024-09-16T06:38:17.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 >> 172.21.15.189:0/691076056 conn(0x7fb2b00a41c0 msgr2=0x7fb2b009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 shutdown_connections 2024-09-16T06:38:17.265 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.262+0000 7fb2b4abf700 1 -- 172.21.15.189:0/691076056 wait complete. 2024-09-16T06:38:17.276 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a 4 2024-09-16T06:38:17.378 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.378+0000 7f623b64a2c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T06:38:17.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: get_monmap_and_config 2024-09-16T06:38:17.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: monmap: 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:17.393332+0000 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:17.393332+0000 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:17.391 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:17.392 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 Processor -- start 2024-09-16T06:38:17.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- start start 2024-09-16T06:38:17.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: init 2024-09-16T06:38:17.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.398 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-a con 0x7f2dd8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-b con 0x7f2dd806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-c con 0x7f2dd806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd806c280 con 0x7f2dd8132330 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd806bfa0 con 0x7f2dd806cf50 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd809ace0 con 0x7f2dd806d480 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient(hunting): authenticate will time out at 849.905979s 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddc8c5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2dd806cf50 0x7f2dd813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55208/0 (socket says 172.21.15.189:55208) 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddc8c5700 1 -- 172.21.15.189:0/902650368 learned_addr learned my addr 172.21.15.189:0/902650368 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053587757 0 0) 0x7f2dd806bfa0 con 0x7f2dd806cf50 2024-09-16T06:38:17.399 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): my global_id is 4135 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc00035d0 con 0x7f2dd806cf50 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062530198 0 0) 0x7f2dd809ace0 con 0x7f2dd806d480 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): my global_id is 4136 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd806bfa0 con 0x7f2dd806d480 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128930313 0 0) 0x7f2dd806c280 con 0x7f2dd8132330 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient(hunting): my global_id is 4134 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd809ace0 con 0x7f2dd8132330 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255494134 0 0) 0x7f2dc00035d0 con 0x7f2dd806cf50 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 >> v1:172.21.15.76:6790/0 conn(0x7f2dd806d480 legacy=0x7f2dd813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 >> v1:172.21.15.76:6789/0 conn(0x7f2dd8132330 legacy=0x7f2dd8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 monclient: found mon.noname-c 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd809ae70 con 0x7f2dd806cf50 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.401 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 5 monclient: authenticate success, global_id 4135 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2dd0001990 con 0x7f2dd806cf50 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: dump: 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.402 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dd0001c70 con 0x7f2dd806cf50 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 1 -- 172.21.15.189:0/902650368 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2dd0001e60 con 0x7f2dd806cf50 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 4 set_mon_vals no callback set 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2dd6ffd700 10 monclient: dump: 2024-09-16T06:38:17.403 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 10 monclient: shutdown 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/902650368 >> v1:172.21.15.189:6789/0 conn(0x7f2dd806cf50 legacy=0x7f2dd813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/902650368 shutdown_connections 2024-09-16T06:38:17.404 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.390+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/902650368 >> 172.21.15.189:0/902650368 conn(0x7f2dd80a41c0 msgr2=0x7f2dd80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/902650368 shutdown_connections 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/902650368 wait complete. 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: monmap: 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:17.405 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 Processor -- start 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- start start 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: init 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-a con 0x7f2dd806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-b con 0x7f2dd806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient(hunting): picked mon.noname-c con 0x7f2dd8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd8131260 con 0x7f2dd806cf50 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd813ed30 con 0x7f2dd806d480 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd809e8b0 con 0x7f2dd8132330 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.406 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient(hunting): authenticate will time out at 849.909779s 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2dd806d480 0x7f2dd81ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55218/0 (socket says 172.21.15.189:55218) 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd77fe700 1 -- 172.21.15.189:0/3930854508 learned_addr learned my addr 172.21.15.189:0/3930854508 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523956377 0 0) 0x7f2dd813ed30 con 0x7f2dd806d480 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): my global_id is 4138 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc40035d0 con 0x7f2dd806d480 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196810087 0 0) 0x7f2dd8131260 con 0x7f2dd806cf50 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): my global_id is 4137 2024-09-16T06:38:17.407 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd813ed30 con 0x7f2dd806cf50 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341033989 0 0) 0x7f2dc40035d0 con 0x7f2dd806d480 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd8131260 con 0x7f2dd806d480 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2dc80018f0 con 0x7f2dd806d480 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.408 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808590344 0 0) 0x7f2dd809e8b0 con 0x7f2dd8132330 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient(hunting): my global_id is 4139 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc40035d0 con 0x7f2dd8132330 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1996302478 0 0) 0x7f2dd8131260 con 0x7f2dd806d480 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 >> v1:172.21.15.76:6790/0 conn(0x7f2dd8132330 legacy=0x7f2dd81d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.409 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 >> v1:172.21.15.76:6789/0 conn(0x7f2dd806cf50 legacy=0x7f2dd81c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 monclient: found mon.b 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd81317b0 con 0x7f2dd806d480 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 5 monclient: authenticate success, global_id 4138 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _renew_subs 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.410 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd8130b50 con 0x7f2dd806d480 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _renew_subs 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd812ed40 con 0x7f2dd806d480 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc80036a0 con 0x7f2dd806d480 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2dc80039b0 con 0x7f2dd806d480 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 10 monclient: dump: 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.411 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd67fc700 4 set_mon_vals no callback set 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f2dc8003e30 con 0x7f2dd806d480 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.394+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 842512886 0 0) 0x7f2dc8006500 con 0x7f2dd806d480 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2ddd0c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2ddd0c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2ddd0c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db8005100 con 0x7f2dd806d480 2024-09-16T06:38:17.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f2dc8004370 con 0x7f2dd806d480 2024-09-16T06:38:17.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2dd4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.413 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.398+0000 7f2dd4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:17.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.486+0000 7f2ddd0c6700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a", "id": 4}] 2024-09-16T06:38:17.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.486+0000 7f2ddd0c6700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a", "id": 4}] 2024-09-16T06:38:17.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.486+0000 7f2ddd0c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.486 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.486+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a", "id": 4} v 0) v1 -- 0x7f2db8005930 con 0x7f2dd806d480 2024-09-16T06:38:17.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2dd4ff9700 1 -- 172.21.15.189:0/3930854508 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3001663078 0 1697012583) 0x7f2dc8005870 con 0x7f2dd806d480 2024-09-16T06:38:17.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2dd4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a", "id": 4}] 2024-09-16T06:38:17.492 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2dd4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:17.492 INFO:teuthology.orchestra.run.smithi189.stdout:4 2024-09-16T06:38:17.493 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 10 monclient: shutdown 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 >> v1:172.21.15.189:6789/0 conn(0x7f2dd806d480 legacy=0x7f2dd81ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 shutdown_connections 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 >> 172.21.15.189:0/3930854508 conn(0x7f2dd80a41c0 msgr2=0x7f2dd809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 shutdown_connections 2024-09-16T06:38:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.490+0000 7f2ddd0c6700 1 -- 172.21.15.189:0/3930854508 wait complete. 2024-09-16T06:38:17.506 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 9b794539-b7ca-4f2b-8653-a98904253c60 5 2024-09-16T06:38:17.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.610+0000 7fc67ccc5700 10 monclient: get_monmap_and_config 2024-09-16T06:38:17.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.610+0000 7fc67ccc5700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.610+0000 7fc67ccc5700 10 monclient: monmap: 2024-09-16T06:38:17.613 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.613 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:38:17.616027+0000 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:38:17.616027+0000 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:17.614 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 Processor -- start 2024-09-16T06:38:17.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- start start 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient: init 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-c con 0x7fc678132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-b con 0x7fc67806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-a con 0x7fc67806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806c280 con 0x7fc67806cf50 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806bfa0 con 0x7fc678132330 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6777fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc678132330 0x7fc678132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55220/0 (socket says 172.21.15.189:55220) 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6777fe700 1 -- 172.21.15.189:0/2205149283 learned_addr learned my addr 172.21.15.189:0/2205149283 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67809ace0 con 0x7fc67806d480 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient(hunting): authenticate will time out at 850.129842s 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340885999 0 0) 0x7fc67806bfa0 con 0x7fc678132330 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): my global_id is 4141 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6600035d0 con 0x7fc678132330 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013002074 0 0) 0x7fc67809ace0 con 0x7fc67806d480 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): my global_id is 4142 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67806bfa0 con 0x7fc67806d480 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763171164 0 0) 0x7fc67806c280 con 0x7fc67806cf50 2024-09-16T06:38:17.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient(hunting): my global_id is 4140 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67809ace0 con 0x7fc67806cf50 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962919348 0 0) 0x7fc6600035d0 con 0x7fc678132330 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 >> v1:172.21.15.76:6790/0 conn(0x7fc67806d480 legacy=0x7fc67813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 >> v1:172.21.15.76:6789/0 conn(0x7fc67806cf50 legacy=0x7fc67813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 monclient: found mon.noname-c 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67809ae70 con 0x7fc678132330 2024-09-16T06:38:17.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 5 monclient: authenticate success, global_id 4141 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc668001610 con 0x7fc678132330 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: dump: 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.622 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6680018f0 con 0x7fc678132330 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 4 set_mon_vals no callback set 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 1 -- 172.21.15.189:0/2205149283 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc668001ae0 con 0x7fc678132330 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc6767fc700 10 monclient: dump: 2024-09-16T06:38:17.623 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 10 monclient: shutdown 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 >> v1:172.21.15.189:6789/0 conn(0x7fc678132330 legacy=0x7fc678132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 shutdown_connections 2024-09-16T06:38:17.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 >> 172.21.15.189:0/2205149283 conn(0x7fc6780a41c0 msgr2=0x7fc6780a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.614+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 shutdown_connections 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2205149283 wait complete. 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: monmap: 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:17.625 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 Processor -- start 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- start start 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: init 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-a con 0x7fc67806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-c con 0x7fc67806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient(hunting): picked mon.noname-b con 0x7fc678132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67812e6a0 con 0x7fc67806cf50 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67812f5a0 con 0x7fc678132330 2024-09-16T06:38:17.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc678130190 con 0x7fc67806d480 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient(hunting): authenticate will time out at 850.134145s 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc677fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc678132330 0x7fc6781d2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:55234/0 (socket says 172.21.15.189:55234) 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc677fff700 1 -- 172.21.15.189:0/2700811509 learned_addr learned my addr 172.21.15.189:0/2700811509 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216260718 0 0) 0x7fc67812f5a0 con 0x7fc678132330 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): my global_id is 4144 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6640035d0 con 0x7fc678132330 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392850867 0 0) 0x7fc678130190 con 0x7fc67806d480 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): my global_id is 4145 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67812f5a0 con 0x7fc67806d480 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038754153 0 0) 0x7fc67812e6a0 con 0x7fc67806cf50 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): my global_id is 4143 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc678130190 con 0x7fc67806cf50 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2026444174 0 0) 0x7fc6640035d0 con 0x7fc678132330 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc67812e6a0 con 0x7fc678132330 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc670001b80 con 0x7fc678132330 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): dump: 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754482126 0 0) 0x7fc67812e6a0 con 0x7fc678132330 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 >> v1:172.21.15.76:6790/0 conn(0x7fc67806d480 legacy=0x7fc6781cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 >> v1:172.21.15.76:6789/0 conn(0x7fc67806cf50 legacy=0x7fc67813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 monclient: found mon.b 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6781317e0 con 0x7fc678132330 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 5 monclient: authenticate success, global_id 4144 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _renew_subs 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781312d0 con 0x7fc678132330 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _renew_subs 2024-09-16T06:38:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67812fbb0 con 0x7fc678132330 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6700020f0 con 0x7fc678132330 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc670003bc0 con 0x7fc678132330 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc675ffb700 4 set_mon_vals no callback set 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 10 monclient: dump: 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 3 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fc6700040a0 con 0x7fc678132330 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.618+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 436655249 0 0) 0x7fc670005bf0 con 0x7fc678132330 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc67ccc5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc67ccc5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc67ccc5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658005100 con 0x7fc678132330 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fc670004350 con 0x7fc678132330 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc65ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.635 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.622+0000 7fc65ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:17.636 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.630+0000 7f623b64a2c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T06:38:17.684 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.682+0000 7f623b64a2c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T06:38:17.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.698+0000 7fc67ccc5700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9b794539-b7ca-4f2b-8653-a98904253c60", "id": 5}] 2024-09-16T06:38:17.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.698+0000 7fc67ccc5700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9b794539-b7ca-4f2b-8653-a98904253c60", "id": 5}] 2024-09-16T06:38:17.700 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.698+0000 7fc67ccc5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:17.701 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.698+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9b794539-b7ca-4f2b-8653-a98904253c60", "id": 5} v 0) v1 -- 0x7fc658005930 con 0x7fc678132330 2024-09-16T06:38:17.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc65ffff700 1 -- 172.21.15.189:0/2700811509 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b794539-b7ca-4f2b-8653-a98904253c60", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2951413880 0 1988412176) 0x7fc6700045c0 con 0x7fc678132330 2024-09-16T06:38:17.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc65ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9b794539-b7ca-4f2b-8653-a98904253c60", "id": 5}] 2024-09-16T06:38:17.710 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc65ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:17.711 INFO:teuthology.orchestra.run.smithi189.stdout:5 2024-09-16T06:38:17.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 10 monclient: shutdown 2024-09-16T06:38:17.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 >> v1:172.21.15.189:6789/0 conn(0x7fc678132330 legacy=0x7fc6781d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 shutdown_connections 2024-09-16T06:38:17.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 >> 172.21.15.189:0/2700811509 conn(0x7fc6780a41c0 msgr2=0x7fc67809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 shutdown_connections 2024-09-16T06:38:17.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-16T06:38:17.710+0000 7fc67ccc5700 1 -- 172.21.15.189:0/2700811509 wait complete. 2024-09-16T06:38:17.727 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T06:38:17.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T06:38:17.730 INFO:tasks.ceph.osd.0:Started 2024-09-16T06:38:17.730 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T06:38:17.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T06:38:17.732 INFO:tasks.ceph.osd.1:Started 2024-09-16T06:38:17.732 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T06:38:17.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T06:38:17.735 INFO:tasks.ceph.osd.2:Started 2024-09-16T06:38:17.735 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T06:38:17.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T06:38:17.737 INFO:tasks.ceph.osd.3:Started 2024-09-16T06:38:17.737 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T06:38:17.737 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T06:38:17.739 INFO:tasks.ceph.osd.4:Started 2024-09-16T06:38:17.739 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T06:38:17.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T06:38:17.741 INFO:tasks.ceph.osd.5:Started 2024-09-16T06:38:17.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.834+0000 7fe153219700 10 monclient: get_monmap_and_config 2024-09-16T06:38:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.834+0000 7fe153219700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.834+0000 7fe153219700 10 monclient: monmap: 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:17.839745+0000 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:17.839745+0000 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 Processor -- start 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- start start 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient: init 2024-09-16T06:38:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient(hunting): picked mon.noname-b con 0x7fe14c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient(hunting): picked mon.noname-c con 0x7fe14c09bc60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient(hunting): picked mon.noname-a con 0x7fe14c09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c09c7d0 con 0x7fe14c09fd20 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c071f30 con 0x7fe14c09bc60 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c09b4f0 con 0x7fe14c09f230 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient(hunting): authenticate will time out at 844.973244s 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe1517b6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe14c09fd20 0x7fe14c13a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45806/0 (socket says 172.21.15.76:45806) 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe150fb5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe14c09f230 0x7fe14c09f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33192/0 (socket says 172.21.15.76:33192) 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe1517b6700 1 -- 172.21.15.76:0/2025270877 learned_addr learned my addr 172.21.15.76:0/2025270877 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958917432 0 0) 0x7fe14c09b4f0 con 0x7fe14c09f230 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): my global_id is 4166 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1340035d0 con 0x7fe14c09f230 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177866532 0 0) 0x7fe14c09c7d0 con 0x7fe14c09fd20 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): my global_id is 4164 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c09b4f0 con 0x7fe14c09fd20 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308685371 0 0) 0x7fe14c071f30 con 0x7fe14c09bc60 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient(hunting): my global_id is 4165 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c09c7d0 con 0x7fe14c09bc60 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143137029 0 0) 0x7fe1340035d0 con 0x7fe14c09f230 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 >> v1:172.21.15.189:6789/0 conn(0x7fe14c09bc60 legacy=0x7fe14c137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 >> v1:172.21.15.76:6789/0 conn(0x7fe14c09fd20 legacy=0x7fe14c13a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c0a4570 con 0x7fe14c09f230 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 5 monclient: authenticate success, global_id 4166 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe144001990 con 0x7fe14c09f230 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: dump: 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe144001c70 con 0x7fe14c09f230 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 4 set_mon_vals no callback set 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 1 -- 172.21.15.76:0/2025270877 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe144002000 con 0x7fe14c09f230 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe14b7fe700 10 monclient: dump: 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 10 monclient: shutdown 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- 172.21.15.76:0/2025270877 >> v1:172.21.15.76:6790/0 conn(0x7fe14c09f230 legacy=0x7fe14c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- 172.21.15.76:0/2025270877 shutdown_connections 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- 172.21.15.76:0/2025270877 >> 172.21.15.76:0/2025270877 conn(0x7fe14c0a3e50 msgr2=0x7fe14c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- 172.21.15.76:0/2025270877 shutdown_connections 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.838+0000 7fe153219700 1 -- 172.21.15.76:0/2025270877 wait complete. 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: build_initial_monmap 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: monmap: 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:17.851 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 Processor -- start 2024-09-16T06:38:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- start start 2024-09-16T06:38:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: init 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient(hunting): picked mon.noname-b con 0x7fe14c09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient(hunting): picked mon.noname-c con 0x7fe14c09fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient(hunting): picked mon.noname-a con 0x7fe14c1c5ae0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c1a5150 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c072450 con 0x7fe14c09f230 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c0a2fd0 con 0x7fe14c09fd20 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient(hunting): authenticate will time out at 844.978080s 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe14bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe14c09fd20 0x7fe14c1cae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33198/0 (socket says 172.21.15.76:33198) 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe14bfff700 1 -- 172.21.15.76:0/2177914733 learned_addr learned my addr 172.21.15.76:0/2177914733 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199510602 0 0) 0x7fe14c1a5150 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): my global_id is 4167 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1380035d0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114364228 0 0) 0x7fe14c0a2fd0 con 0x7fe14c09fd20 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): my global_id is 4169 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c1a5150 con 0x7fe14c09fd20 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763092748 0 0) 0x7fe14c072450 con 0x7fe14c09f230 2024-09-16T06:38:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): my global_id is 4168 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c0a2fd0 con 0x7fe14c09f230 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121246072 0 0) 0x7fe1380035d0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe14c072450 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe13c0018b0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): dump: 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786582258 0 0) 0x7fe14c1a5150 con 0x7fe14c09fd20 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1380035d0 con 0x7fe14c09fd20 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1400018f0 con 0x7fe14c09fd20 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): dump: 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2377841432 0 0) 0x7fe14c0a2fd0 con 0x7fe14c09f230 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe14c1a5150 con 0x7fe14c09f230 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe144001950 con 0x7fe14c09f230 2024-09-16T06:38:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient(hunting): dump: 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560203036 0 0) 0x7fe14c072450 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 >> v1:172.21.15.76:6790/0 conn(0x7fe14c09fd20 legacy=0x7fe14c1cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 >> v1:172.21.15.189:6789/0 conn(0x7fe14c09f230 legacy=0x7fe14c1c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: _finish_hunting 0 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 monclient: found mon.a 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c1a6c40 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: _finish_auth 0 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: _check_auth_tickets 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 5 monclient: authenticate success, global_id 4167 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _renew_subs 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c1c22b0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _renew_subs 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe13c004890 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c1a5f40 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe13c004ba0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe1497fa700 10 monclient: dump: 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.842+0000 7fe14affd700 4 set_mon_vals no callback set 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fe13c005000 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2808375702 0 0) 0x7fe13c005990 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe153219700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe153219700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe153219700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:17.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c0678b0 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7fe13c005460 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe1497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:17.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.846+0000 7fe1497fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:17.889 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.886+0000 7f623b64a2c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T06:38:17.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe14c1a7390 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe1497fa700 1 -- 172.21.15.76:0/2177914733 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 2745663805) 0x7fe13c005730 con 0x7fe14c1c5ae0 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe1497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe1497fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:17.918 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":8,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:17.703691+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 10 monclient: shutdown 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 >> v1:172.21.15.76:6789/0 conn(0x7fe14c1c5ae0 legacy=0x7fe14c1ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 shutdown_connections 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.914+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 >> 172.21.15.76:0/2177914733 conn(0x7fe14c0a3e50 msgr2=0x7fe14c06ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.918+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 shutdown_connections 2024-09-16T06:38:17.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:17.918+0000 7fe153219700 1 -- 172.21.15.76:0/2177914733 wait complete. 2024-09-16T06:38:17.929 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T06:38:17.929 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T06:38:17.957 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:17.958+0000 7f623b64a2c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T06:38:18.027 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:18.026+0000 7f623b64a2c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T06:38:18.103 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T06:38:18.102+0000 7f95115b8880 -1 Falling back to public interface 2024-09-16T06:38:18.119 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T06:38:18.118+0000 7f7037b84880 -1 Falling back to public interface 2024-09-16T06:38:18.344 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T06:38:18.342+0000 7facd14ab880 -1 Falling back to public interface 2024-09-16T06:38:18.351 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T06:38:18.350+0000 7fe4e6210880 -1 Falling back to public interface 2024-09-16T06:38:18.367 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T06:38:18.366+0000 7fe2a35f2880 -1 Falling back to public interface 2024-09-16T06:38:18.379 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T06:38:18.378+0000 7f8b657ca880 -1 Falling back to public interface 2024-09-16T06:38:18.474 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:18.474+0000 7f623b64a2c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T06:38:18.533 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:18.604 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:18.602+0000 7f623b64a2c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: build_initial_monmap 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: monmap: 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:epoch 0 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.640 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:38:18.642259+0000 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:38:18.642259+0000 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:18.641 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.644 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 1 Processor -- start 2024-09-16T06:38:18.644 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 1 -- start start 2024-09-16T06:38:18.644 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: init 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f818006c2a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f81800a7050 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f818006cd90 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81800a6200 con 0x7f818006c2a0 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818009a300 con 0x7f818006cd90 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8180099a60 con 0x7f81800a7050 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.638+0000 7f817ffff700 10 monclient(hunting): authenticate will time out at 845.775844s 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f81800a7050 0x7f818013a400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33212/0 (socket says 172.21.15.76:33212) 2024-09-16T06:38:18.645 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f818006c2a0 0x7f818006c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45830/0 (socket says 172.21.15.76:45830) 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817e7fc700 1 -- 172.21.15.76:0/1304718370 learned_addr learned my addr 172.21.15.76:0/1304718370 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773302849 0 0) 0x7f81800a6200 con 0x7f818006c2a0 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): my global_id is 4170 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c0035d0 con 0x7f818006c2a0 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687870955 0 0) 0x7f8180099a60 con 0x7f81800a7050 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient(hunting): my global_id is 4172 2024-09-16T06:38:18.646 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81800a6200 con 0x7f81800a7050 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 726449291 0 0) 0x7f816c0035d0 con 0x7f818006c2a0 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 >> v1:172.21.15.76:6790/0 conn(0x7f81800a7050 legacy=0x7f818013a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 >> v1:172.21.15.189:6789/0 conn(0x7f818006cd90 legacy=0x7f818013db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 monclient: found mon.noname-a 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818009a720 con 0x7f818006c2a0 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:18.647 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8170001610 con 0x7f818006c2a0 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: dump: 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.648 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 5 monclient: authenticate success, global_id 4170 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81700018f0 con 0x7f818006c2a0 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 1 -- 172.21.15.76:0/1304718370 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8170001cc0 con 0x7f818006c2a0 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 4 set_mon_vals no callback set 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817dffb700 10 monclient: dump: 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.649 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: shutdown 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- 172.21.15.76:0/1304718370 >> v1:172.21.15.76:6789/0 conn(0x7f818006c2a0 legacy=0x7f818006c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- 172.21.15.76:0/1304718370 shutdown_connections 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- 172.21.15.76:0/1304718370 >> 172.21.15.76:0/1304718370 conn(0x7f81800a7a00 msgr2=0x7f81800a7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- 172.21.15.76:0/1304718370 shutdown_connections 2024-09-16T06:38:18.650 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- 172.21.15.76:0/1304718370 wait complete. 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: build_initial_monmap 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: monmap: 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:epoch 0 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:created 0.000000 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.651 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 Processor -- start 2024-09-16T06:38:18.652 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- start start 2024-09-16T06:38:18.652 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: init 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f818006c2a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f818006cd90 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f81800a7050 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818006d940 con 0x7f818006c2a0 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818009b040 con 0x7f81800a7050 2024-09-16T06:38:18.656 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818013d630 con 0x7f818006cd90 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817ffff700 10 monclient(hunting): authenticate will time out at 845.779470s 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f818006c2a0 0x7f8180132a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45840/0 (socket says 172.21.15.76:45840) 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.642+0000 7f817effd700 1 -- 172.21.15.76:0/55770221 learned_addr learned my addr 172.21.15.76:0/55770221 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718265665 0 0) 0x7f818006d940 con 0x7f818006c2a0 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): my global_id is 4173 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81600035d0 con 0x7f818006c2a0 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130195747 0 0) 0x7f818013d630 con 0x7f818006cd90 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:18.657 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): my global_id is 4175 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818006d940 con 0x7f818006cd90 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297624379 0 0) 0x7f818009b040 con 0x7f81800a7050 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): my global_id is 4174 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818013d630 con 0x7f81800a7050 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963835568 0 0) 0x7f81600035d0 con 0x7f818006c2a0 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818009b040 con 0x7f818006c2a0 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8170001c00 con 0x7f818006c2a0 2024-09-16T06:38:18.658 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): dump: 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2292478949 0 0) 0x7f818013d630 con 0x7f81800a7050 2024-09-16T06:38:18.659 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81600035d0 con 0x7f81800a7050 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8178001b70 con 0x7f81800a7050 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): dump: 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.660 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423138276 0 0) 0x7f818006d940 con 0x7f818006cd90 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818013d630 con 0x7f818006cd90 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81680018b0 con 0x7f818006cd90 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient(hunting): dump: 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.661 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1802830241 0 0) 0x7f818009b040 con 0x7f818006c2a0 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 >> v1:172.21.15.76:6790/0 conn(0x7f818006cd90 legacy=0x7f8180133190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 >> v1:172.21.15.189:6789/0 conn(0x7f81800a7050 legacy=0x7f81801d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 monclient: found mon.a 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818009aaf0 con 0x7f818006c2a0 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:18.662 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 5 monclient: authenticate success, global_id 4173 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _renew_subs 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81801366c0 con 0x7f818006c2a0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _renew_subs 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8180134d30 con 0x7f818006c2a0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81700011b0 con 0x7f818006c2a0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8170004e00 con 0x7f818006c2a0 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:18.663 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 10 monclient: dump: 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f81700052a0 con 0x7f818006c2a0 2024-09-16T06:38:18.664 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2808375702 0 0) 0x7f8170006830 con 0x7f818006c2a0 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817cff9700 4 set_mon_vals no callback set 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.646+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8164005100 con 0x7f818006c2a0 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.650+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (unknown 1092875540 0 2019800707) 0x7f8170006a60 con 0x7f818006c2a0 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.650+0000 7f81777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:18.665 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.650+0000 7f81777fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:18.717 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.714+0000 7f817ffff700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:18.718 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.714+0000 7f817ffff700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:18.718 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.714+0000 7f817ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:18.718 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.714+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81640053b0 con 0x7f818006c2a0 2024-09-16T06:38:18.718 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f81777fe700 1 -- 172.21.15.76:0/55770221 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (unknown 150273500 0 2745663805) 0x7f81700062e0 con 0x7f818006c2a0 2024-09-16T06:38:18.719 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f81777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:18.719 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f81777fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:18.719 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T06:38:18.719 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":8,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:17.703691+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 10 monclient: shutdown 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 >> v1:172.21.15.76:6789/0 conn(0x7f818006c2a0 legacy=0x7f8180132a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 shutdown_connections 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 >> 172.21.15.76:0/55770221 conn(0x7f81800a7a00 msgr2=0x7f818012f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 shutdown_connections 2024-09-16T06:38:18.720 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:18.718+0000 7f817ffff700 1 -- 172.21.15.76:0/55770221 wait complete. 2024-09-16T06:38:18.731 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T06:38:18.968 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:18.966+0000 7f623b64a2c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T06:38:19.154 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:19.154+0000 7f623b64a2c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T06:38:19.221 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:19.218+0000 7f623b64a2c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T06:38:19.383 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:19.382+0000 7f623b64a2c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T06:38:20.086 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.086+0000 7f623b64a2c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T06:38:20.165 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.162+0000 7f623b64a2c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T06:38:20.332 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.330+0000 7f623b64a2c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T06:38:20.397 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.394+0000 7f623b64a2c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T06:38:20.471 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.470+0000 7f623b64a2c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T06:38:20.798 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.798+0000 7f623b64a2c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T06:38:20.875 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:20.874+0000 7f623b64a2c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T06:38:20.946 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T06:38:20.946+0000 7fe2a35f2880 -1 osd.3 0 log_to_monitors true 2024-09-16T06:38:21.417 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T06:38:21.414+0000 7f7037b84880 -1 osd.1 0 log_to_monitors true 2024-09-16T06:38:21.605 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:21.602+0000 7f623b64a2c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T06:38:21.729 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T06:38:21.726+0000 7f95115b8880 -1 osd.5 0 log_to_monitors true 2024-09-16T06:38:21.870 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T06:38:21.870+0000 7facd14ab880 -1 osd.2 0 log_to_monitors true 2024-09-16T06:38:22.209 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T06:38:22.206+0000 7fe4e6210880 -1 osd.4 0 log_to_monitors true 2024-09-16T06:38:22.236 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T06:38:22.234+0000 7f8b657ca880 -1 osd.0 0 log_to_monitors true 2024-09-16T06:38:22.649 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:22.646+0000 7f62221c1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T06:38:22.650 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-16T06:38:22.650+0000 7f62221c1700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T06:38:22.913 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T06:38:22.910+0000 7fe29f431700 -1 osd.3 0 waiting for initial osdmap 2024-09-16T06:38:22.916 INFO:tasks.ceph.osd.3.smithi189.stderr:2024-09-16T06:38:22.914+0000 7fe296ee0700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:23.862 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T06:38:23.862+0000 7f950d3f7700 -1 osd.5 0 waiting for initial osdmap 2024-09-16T06:38:23.865 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-16T06:38:23.862+0000 7f9504ea6700 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:23.928 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T06:38:23.926+0000 7f703434a700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T06:38:23.932 INFO:tasks.ceph.osd.1.smithi076.stderr:2024-09-16T06:38:23.930+0000 7f702b472700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:23.940 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T06:38:23.938+0000 7faccd2ea700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T06:38:23.944 INFO:tasks.ceph.osd.2.smithi076.stderr:2024-09-16T06:38:23.942+0000 7facc4d99700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:24.054 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T06:38:24.054+0000 7f8b61f90700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T06:38:24.059 INFO:tasks.ceph.osd.0.smithi076.stderr:2024-09-16T06:38:24.058+0000 7f8b5d0c0700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:24.973 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T06:38:24.970+0000 7fe4e204f700 -1 osd.4 0 waiting for initial osdmap 2024-09-16T06:38:24.979 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-16T06:38:24.978+0000 7fe4d9afe700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T06:38:25.337 DEBUG:teuthology.orchestra.run.smithi076:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:25.440 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 10 monclient: get_monmap_and_config 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 10 monclient: build_initial_monmap 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 10 monclient: monmap: 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:epoch 0 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:38:25.442733+0000 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:38:25.442733+0000 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:25.441 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:25.442 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.444 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 1 Processor -- start 2024-09-16T06:38:25.444 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 1 -- start start 2024-09-16T06:38:25.444 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.438+0000 7f7679760700 10 monclient: init 2024-09-16T06:38:25.444 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:25.444 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient(hunting): picked mon.noname-a con 0x7f76740a0c00 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient(hunting): picked mon.noname-c con 0x7f767409c710 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient(hunting): picked mon.noname-b con 0x7f76741397c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740a0fe0 con 0x7f76740a0c00 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767409c360 con 0x7f767409c710 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767409fe40 con 0x7f76741397c0 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient(hunting): authenticate will time out at 852.576497s 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7673fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f76740a0c00 0x7f767409f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60322/0 (socket says 172.21.15.76:60322) 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7673fff700 1 -- 172.21.15.76:0/289057480 learned_addr learned my addr 172.21.15.76:0/289057480 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951712960 0 0) 0x7f76740a0fe0 con 0x7f76740a0c00 2024-09-16T06:38:25.445 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): my global_id is 4212 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c0035d0 con 0x7f76740a0c00 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331358449 0 0) 0x7f767409c360 con 0x7f767409c710 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient(hunting): my global_id is 4213 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76740a0fe0 con 0x7f767409c710 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573180301 0 0) 0x7f765c0035d0 con 0x7f76740a0c00 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 >> v1:172.21.15.76:6790/0 conn(0x7f76741397c0 legacy=0x7f767413bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 >> v1:172.21.15.189:6789/0 conn(0x7f767409c710 legacy=0x7f767409f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.446 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 monclient: found mon.noname-a 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767409ac60 con 0x7f76740a0c00 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7664001610 con 0x7f76740a0c00 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: dump: 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:25.447 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 5 monclient: authenticate success, global_id 4212 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7664001880 con 0x7f76740a0c00 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 1 -- 172.21.15.76:0/289057480 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7664001c10 con 0x7f76740a0c00 2024-09-16T06:38:25.448 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 4 set_mon_vals no callback set 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7672ffd700 10 monclient: dump: 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.449 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: shutdown 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- 172.21.15.76:0/289057480 >> v1:172.21.15.76:6789/0 conn(0x7f76740a0c00 legacy=0x7f767409f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- 172.21.15.76:0/289057480 shutdown_connections 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- 172.21.15.76:0/289057480 >> 172.21.15.76:0/289057480 conn(0x7f76740a0110 msgr2=0x7f76740a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- 172.21.15.76:0/289057480 shutdown_connections 2024-09-16T06:38:25.450 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 1 -- 172.21.15.76:0/289057480 wait complete. 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: build_initial_monmap 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.442+0000 7f7679760700 10 monclient: monmap: 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:epoch 0 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:created 0.000000 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:25.454 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 Processor -- start 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- start start 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: init 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient(hunting): picked mon.noname-a con 0x7f767409c710 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient(hunting): picked mon.noname-c con 0x7f76740a0c00 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient(hunting): picked mon.noname-b con 0x7f76741397c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7674071950 con 0x7f767409c710 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740a2fe0 con 0x7f76741397c0 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7674094b80 con 0x7f76740a0c00 2024-09-16T06:38:25.455 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient(hunting): authenticate will time out at 852.581382s 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7673fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f767409c710 0x7f76740960b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60324/0 (socket says 172.21.15.76:60324) 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7673fff700 1 -- 172.21.15.76:0/1186869239 learned_addr learned my addr 172.21.15.76:0/1186869239 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878743290 0 0) 0x7f7674094b80 con 0x7f76740a0c00 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): my global_id is 4214 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76600035d0 con 0x7f76740a0c00 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805358600 0 0) 0x7f7674071950 con 0x7f767409c710 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.456 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): my global_id is 4215 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7674094b80 con 0x7f767409c710 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914124054 0 0) 0x7f76740a2fe0 con 0x7f76741397c0 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): my global_id is 4216 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7674071950 con 0x7f76741397c0 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2639585961 0 0) 0x7f76600035d0 con 0x7f76740a0c00 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76740a2fe0 con 0x7f76740a0c00 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76680018b0 con 0x7f76740a0c00 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:25.457 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821472829 0 0) 0x7f7674071950 con 0x7f76741397c0 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76600035d0 con 0x7f76741397c0 2024-09-16T06:38:25.458 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483823243 0 0) 0x7f7674094b80 con 0x7f767409c710 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7674071950 con 0x7f767409c710 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1165330943 0 0) 0x7f76740a2fe0 con 0x7f76740a0c00 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 >> v1:172.21.15.189:6789/0 conn(0x7f76741397c0 legacy=0x7f76741d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 >> v1:172.21.15.76:6789/0 conn(0x7f767409c710 legacy=0x7f76740960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 monclient: found mon.c 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767409a800 con 0x7f76740a0c00 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 5 monclient: authenticate success, global_id 4214 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _renew_subs 2024-09-16T06:38:25.459 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76740948a0 con 0x7f76740a0c00 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _renew_subs 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76740967c0 con 0x7f76740a0c00 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76680036a0 con 0x7f76740a0c00 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76680039b0 con 0x7f76740a0c00 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f76727fc700 4 set_mon_vals no callback set 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.446+0000 7f7670ff9700 10 monclient: dump: 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:epoch 1 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.460 INFO:teuthology.misc.health.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr: 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7679760700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7679760700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7679760700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 3059230865 0 0) 0x7f766800e3f0 con 0x7f76740a0c00 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7654005100 con 0x7f76740a0c00 2024-09-16T06:38:25.461 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 2767+0+0 (unknown 353683001 0 0) 0x7f7668041be0 con 0x7f76740a0c00 2024-09-16T06:38:25.462 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.450+0000 7f7673fff700 10 monclient: get_auth_request con 0x7f7660036430 auth_method 0 2024-09-16T06:38:25.462 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.458+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f766800f9e0 con 0x7f76740a0c00 2024-09-16T06:38:25.462 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.458+0000 7f7670ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.462 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.458+0000 7f7670ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:25.633 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.630+0000 7f7679760700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:25.633 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.630+0000 7f7679760700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.630+0000 7f7679760700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.630+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76540053b0 con 0x7f76740a0c00 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.634+0000 7f7670ff9700 1 -- 172.21.15.76:0/1186869239 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7132 (unknown 62554612 0 2385011618) 0x7f7668011a60 con 0x7f76740a0c00 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.634+0000 7f7670ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.634+0000 7f7670ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:25.634 INFO:teuthology.misc.health.smithi076.stdout: 2024-09-16T06:38:25.635 INFO:teuthology.misc.health.smithi076.stdout:{"epoch":13,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:25.048660+0000","last_up_change":"2024-09-16T06:38:25.048660+0000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6808","nonce":3838562525}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6809","nonce":3838562525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6811","nonce":3838562525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6810","nonce":3838562525}]},"public_addr":"172.21.15.76:6808/3838562525","cluster_addr":"172.21.15.76:6809/3838562525","heartbeat_back_addr":"172.21.15.76:6811/3838562525","heartbeat_front_addr":"172.21.15.76:6810/3838562525","state":["exists","up"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6800","nonce":3502405955}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6801","nonce":3502405955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6803","nonce":3502405955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6802","nonce":3502405955}]},"public_addr":"172.21.15.76:6800/3502405955","cluster_addr":"172.21.15.76:6801/3502405955","heartbeat_back_addr":"172.21.15.76:6803/3502405955","heartbeat_front_addr":"172.21.15.76:6802/3502405955","state":["exists","up"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6804","nonce":779009499}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6805","nonce":779009499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6807","nonce":779009499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6806","nonce":779009499}]},"public_addr":"172.21.15.76:6804/779009499","cluster_addr":"172.21.15.76:6805/779009499","heartbeat_back_addr":"172.21.15.76:6807/779009499","heartbeat_front_addr":"172.21.15.76:6806/779009499","state":["exists","up"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3809009238}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3809009238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3809009238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3809009238}]},"public_addr":"172.21.15.189:6808/3809009238","cluster_addr":"172.21.15.189:6809/3809009238","heartbeat_back_addr":"172.21.15.189:6811/3809009238","heartbeat_front_addr":"172.21.15.189:6810/3809009238","state":["exists","up"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":407912501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":407912501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":407912501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":407912501}]},"public_addr":"172.21.15.189:6804/407912501","cluster_addr":"172.21.15.189:6805/407912501","heartbeat_back_addr":"172.21.15.189:6807/407912501","heartbeat_front_addr":"172.21.15.189:6806/407912501","state":["exists","up"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":2128366135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":2128366135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":2128366135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":2128366135}]},"public_addr":"172.21.15.189:6800/2128366135","cluster_addr":"172.21.15.189:6801/2128366135","heartbeat_back_addr":"172.21.15.189:6803/2128366135","heartbeat_front_addr":"172.21.15.189:6802/2128366135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.408945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.850780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:21.901722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.780816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7660036430 legacy=0x7f76600388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 10 monclient: shutdown 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 >> v1:172.21.15.76:6790/0 conn(0x7f76740a0c00 legacy=0x7f76741ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 shutdown_connections 2024-09-16T06:38:25.638 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 >> 172.21.15.76:0/1186869239 conn(0x7f76740a0110 msgr2=0x7f767406f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:25.639 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 shutdown_connections 2024-09-16T06:38:25.639 INFO:teuthology.misc.health.smithi076.stderr:2024-09-16T06:38:25.638+0000 7f7679760700 1 -- 172.21.15.76:0/1186869239 wait complete. 2024-09-16T06:38:25.653 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T06:38:25.653 INFO:tasks.ceph:Creating RBD pool 2024-09-16T06:38:25.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T06:38:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: get_monmap_and_config 2024-09-16T06:38:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: build_initial_monmap 2024-09-16T06:38:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: monmap: 2024-09-16T06:38:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:25.768301+0000 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:25.768301+0000 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:25.767 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 1 Processor -- start 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 1 -- start start 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: init 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-a con 0x7f8478132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:25.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-b con 0x7f847806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-c con 0x7f847806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847806c280 con 0x7f8478132330 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847806bfa0 con 0x7f847806cf50 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847809ace0 con 0x7f847806d480 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847ed31700 10 monclient(hunting): authenticate will time out at 852.902882s 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847d52e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f847806d480 0x7f847813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52738/0 (socket says 172.21.15.76:52738) 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847d52e700 1 -- 172.21.15.76:0/2940532607 learned_addr learned my addr 172.21.15.76:0/2940532607 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088588867 0 0) 0x7f847809ace0 con 0x7f847806d480 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): my global_id is 4217 2024-09-16T06:38:25.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84640035d0 con 0x7f847806d480 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442344781 0 0) 0x7f847806bfa0 con 0x7f847806cf50 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): my global_id is 4219 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847809ace0 con 0x7f847806cf50 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696591178 0 0) 0x7f847806c280 con 0x7f8478132330 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient(hunting): my global_id is 4218 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847806bfa0 con 0x7f8478132330 2024-09-16T06:38:25.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3378754532 0 0) 0x7f84640035d0 con 0x7f847806d480 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 >> v1:172.21.15.189:6789/0 conn(0x7f847806cf50 legacy=0x7f847813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 >> v1:172.21.15.76:6789/0 conn(0x7f8478132330 legacy=0x7f8478132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient: _finish_hunting 0 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 monclient: found mon.noname-b 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847809ae70 con 0x7f847806d480 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient: _finish_auth 0 2024-09-16T06:38:25.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient: _check_auth_tickets 2024-09-16T06:38:25.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:25.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8468001610 con 0x7f847806d480 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.766+0000 7f847cd2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: dump: 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 5 monclient: authenticate success, global_id 4217 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84680018f0 con 0x7f847806d480 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 1 -- 172.21.15.76:0/2940532607 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8468001ae0 con 0x7f847806d480 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847cd2d700 10 monclient: dump: 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.778 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 4 set_mon_vals no callback set 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 10 monclient: shutdown 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 -- 172.21.15.76:0/2940532607 >> v1:172.21.15.76:6790/0 conn(0x7f847806d480 legacy=0x7f847813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 -- 172.21.15.76:0/2940532607 shutdown_connections 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 -- 172.21.15.76:0/2940532607 >> 172.21.15.76:0/2940532607 conn(0x7f84780a41c0 msgr2=0x7f84780a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 -- 172.21.15.76:0/2940532607 shutdown_connections 2024-09-16T06:38:25.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 -- 172.21.15.76:0/2940532607 wait complete. 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 10 monclient: build_initial_monmap 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 10 monclient: monmap: 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.770+0000 7f847ed31700 1 Processor -- start 2024-09-16T06:38:25.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- start start 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: init 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-c con 0x7f847806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-b con 0x7f847806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient(hunting): picked mon.noname-a con 0x7f8478132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84781317b0 con 0x7f8478132330 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8478131260 con 0x7f847806d480 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847812f3d0 con 0x7f847806cf50 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient(hunting): authenticate will time out at 852.909288s 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847dd2f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f847806cf50 0x7f847812d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52742/0 (socket says 172.21.15.76:52742) 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847e530700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8478132330 0x7f847812e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60342/0 (socket says 172.21.15.76:60342) 2024-09-16T06:38:25.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847dd2f700 1 -- 172.21.15.76:0/3442751694 learned_addr learned my addr 172.21.15.76:0/3442751694 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867291600 0 0) 0x7f84781317b0 con 0x7f8478132330 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): my global_id is 4221 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c0035d0 con 0x7f8478132330 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656186021 0 0) 0x7f847812f3d0 con 0x7f847806cf50 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): my global_id is 4220 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84781317b0 con 0x7f847806cf50 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773939438 0 0) 0x7f8478131260 con 0x7f847806d480 2024-09-16T06:38:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): my global_id is 4222 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847812f3d0 con 0x7f847806d480 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685138332 0 0) 0x7f845c0035d0 con 0x7f8478132330 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8478131260 con 0x7f8478132330 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8470001c30 con 0x7f8478132330 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.788 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888038017 0 0) 0x7f84781317b0 con 0x7f847806cf50 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f845c0035d0 con 0x7f847806cf50 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84740018f0 con 0x7f847806cf50 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:25.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1343678131 0 0) 0x7f847812f3d0 con 0x7f847806d480 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84781317b0 con 0x7f847806d480 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8468001020 con 0x7f847806d480 2024-09-16T06:38:25.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3167881288 0 0) 0x7f8478131260 con 0x7f8478132330 2024-09-16T06:38:25.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 >> v1:172.21.15.76:6790/0 conn(0x7f847806cf50 legacy=0x7f847812d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 >> v1:172.21.15.189:6789/0 conn(0x7f847806d480 legacy=0x7f847812db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 monclient: found mon.a 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847809e8b0 con 0x7f8478132330 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 5 monclient: authenticate success, global_id 4221 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _renew_subs 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847813ddc0 con 0x7f8478132330 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _renew_subs 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f847812fea0 con 0x7f8478132330 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8470003a40 con 0x7f8478132330 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8470001360 con 0x7f8478132330 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846ffff700 4 set_mon_vals no callback set 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.774+0000 7f846e7fc700 10 monclient: dump: 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:25.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 3059230865 0 0) 0x7f8470003d50 con 0x7f8478132330 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f847ed31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f847ed31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f847ed31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8460005100 con 0x7f8478132330 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 2767+0+0 (unknown 353683001 0 0) 0x7f8470042ff0 con 0x7f8478132330 2024-09-16T06:38:25.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.778+0000 7f847dd2f700 10 monclient: get_auth_request con 0x7f845c036530 auth_method 0 2024-09-16T06:38:25.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.782+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8470014770 con 0x7f8478132330 2024-09-16T06:38:25.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.786+0000 7f846e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:25.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.786+0000 7f846e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:25.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.954+0000 7f847ed31700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T06:38:25.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.954+0000 7f847ed31700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T06:38:25.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.954+0000 7f847ed31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:25.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:25.954+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f8460005930 con 0x7f8478132330 2024-09-16T06:38:26.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.078+0000 7f846e7fc700 1 -- 172.21.15.76:0/3442751694 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (unknown 898139697 0 0) 0x7f8470012e10 con 0x7f8478132330 2024-09-16T06:38:26.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.078+0000 7f846e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T06:38:26.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.078+0000 7f846e7fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T06:38:26.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 >> v1:172.21.15.189:6812/3098810674 conn(0x7f845c036530 legacy=0x7f845c0389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 10 monclient: shutdown 2024-09-16T06:38:26.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:26.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 >> v1:172.21.15.76:6789/0 conn(0x7f8478132330 legacy=0x7f847812e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 shutdown_connections 2024-09-16T06:38:26.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 >> 172.21.15.76:0/3442751694 conn(0x7f84780a41c0 msgr2=0x7f847809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:26.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 shutdown_connections 2024-09-16T06:38:26.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.086+0000 7f847ed31700 1 -- 172.21.15.76:0/3442751694 wait complete. 2024-09-16T06:38:26.089 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'rbd' created 2024-09-16T06:38:26.101 DEBUG:teuthology.orchestra.run.smithi076:> rbd --cluster ceph pool init rbd 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: get_monmap_and_config 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: build_initial_monmap 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: monmap: 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:26.147987+0000 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:26.147987+0000 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:26.146 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.147 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:26.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:26.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:26.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 1 Processor -- start 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 1 -- start start 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: init 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: _reopen_session rank -1 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-a con 0x5592ef49b550 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-b con 0x5592ef49c040 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-c con 0x5592ef535a70 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef318ca0 con 0x5592ef49b550 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef2f1bb0 con 0x5592ef535a70 2024-09-16T06:38:26.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef2feb00 con 0x5592ef49c040 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient(hunting): _renew_subs 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient(hunting): authenticate will time out at 853.281702s 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x5592ef49b550 0x5592ef49b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60354/0 (socket says 172.21.15.76:60354) 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e3fff700 1 -- 172.21.15.76:0/713089437 learned_addr learned my addr 172.21.15.76:0/713089437 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11dbfff700 1 --1- 172.21.15.76:0/713089437 >> v1:172.21.15.76:6790/0 conn(0x5592ef49c040 0x5592ef534350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52750/0 (socket says 172.21.15.76:52750) 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737571801 0 0) 0x5592ef318ca0 con 0x5592ef49b550 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): my global_id is 4224 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d0001e10 con 0x5592ef49b550 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123711010 0 0) 0x5592ef2feb00 con 0x5592ef49c040 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): my global_id is 4223 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5592ef318ca0 con 0x5592ef49c040 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471664300 0 0) 0x5592ef2f1bb0 con 0x5592ef535a70 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient(hunting): my global_id is 4225 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5592ef2feb00 con 0x5592ef535a70 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1721577769 0 0) 0x7f11d0001e10 con 0x5592ef49b550 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 >> v1:172.21.15.76:6790/0 conn(0x5592ef49c040 legacy=0x5592ef534350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 >> v1:172.21.15.189:6789/0 conn(0x5592ef535a70 legacy=0x5592ef537e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 monclient: found mon.noname-a 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592ef2f1f40 con 0x5592ef49b550 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 5 monclient: authenticate success, global_id 4224 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11d4001540 con 0x5592ef49b550 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: dump: 2024-09-16T06:38:26.153 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d4001760 con 0x5592ef49b550 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 10 monclient: get_monmap_and_config success 2024-09-16T06:38:26.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e9ad8780 4 set_mon_vals no callback set 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 1 -- 172.21.15.76:0/713089437 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11d4001a70 con 0x5592ef49b550 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.146+0000 7f11e37fe700 10 monclient: dump: 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.155 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: shutdown 2024-09-16T06:38:26.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:26.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- 172.21.15.76:0/713089437 >> v1:172.21.15.76:6789/0 conn(0x5592ef49b550 legacy=0x5592ef49b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- 172.21.15.76:0/713089437 shutdown_connections 2024-09-16T06:38:26.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- 172.21.15.76:0/713089437 >> 172.21.15.76:0/713089437 conn(0x5592ef47dcf0 msgr2=0x5592ef47e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:26.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- 172.21.15.76:0/713089437 shutdown_connections 2024-09-16T06:38:26.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- 172.21.15.76:0/713089437 wait complete. 2024-09-16T06:38:26.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: build_initial_monmap 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: monmap: 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 Processor -- start 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- start start 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: init 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: _reopen_session rank -1 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-c con 0x5592ef49b550 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-b con 0x5592ef49c040 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient(hunting): picked mon.noname-a con 0x5592ef535a70 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef2f19a0 con 0x5592ef535a70 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef49af20 con 0x5592ef49c040 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5592ef496ea0 con 0x5592ef49b550 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient(hunting): _renew_subs 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9ad8780 10 monclient(hunting): authenticate will time out at 853.287767s 2024-09-16T06:38:26.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x5592ef49b550 0x5592ef53e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52762/0 (socket says 172.21.15.76:52762) 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e3fff700 1 -- 172.21.15.76:0/2142784857 learned_addr learned my addr 172.21.15.76:0/2142784857 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.150+0000 7f11e9871700 1 --1- 172.21.15.76:0/2142784857 >> v1:172.21.15.76:6789/0 conn(0x5592ef535a70 0x5592ef546840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60364/0 (socket says 172.21.15.76:60364) 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342235283 0 0) 0x5592ef2f19a0 con 0x5592ef535a70 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): my global_id is 4227 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4001e10 con 0x5592ef535a70 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362281680 0 0) 0x5592ef496ea0 con 0x5592ef49b550 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): my global_id is 4226 2024-09-16T06:38:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5592ef2f19a0 con 0x5592ef49b550 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806139826 0 0) 0x5592ef49af20 con 0x5592ef49c040 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): my global_id is 4228 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5592ef496ea0 con 0x5592ef49c040 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3573944987 0 0) 0x5592ef2f19a0 con 0x5592ef49b550 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5592ef49af20 con 0x5592ef49b550 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11d4001440 con 0x5592ef49b550 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777606395 0 0) 0x7f11c4001e10 con 0x5592ef535a70 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5592ef2f19a0 con 0x5592ef535a70 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3928759390 0 0) 0x5592ef496ea0 con 0x5592ef49c040 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c4001e10 con 0x5592ef49c040 2024-09-16T06:38:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11dc001c30 con 0x5592ef535a70 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11cc001830 con 0x5592ef49c040 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.167 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3789959726 0 0) 0x5592ef49af20 con 0x5592ef49b550 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.189:6789/0 conn(0x5592ef49c040 legacy=0x5592ef543110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.76:6789/0 conn(0x5592ef535a70 legacy=0x5592ef546840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 monclient: found mon.c 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592ef495670 con 0x5592ef49b550 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 5 monclient: authenticate success, global_id 4226 2024-09-16T06:38:26.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 10 monclient: _renew_subs 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x5592ef51f0d0 con 0x5592ef49b550 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 10 monclient: _renew_subs 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x5592ef51edf0 con 0x5592ef49b550 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d4003380 con 0x5592ef49b550 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11d4003690 con 0x5592ef49b550 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 10 monclient: dump: 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:26.169 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e2ffd700 4 set_mon_vals no callback set 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.154+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 4) v1 ==== 43216+0+0 (unknown 3059230865 0 0) 0x7f11d4003870 con 0x5592ef49b550 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 3484+0+0 (unknown 4246591831 0 0) 0x7f11d4041b20 con 0x5592ef49b550 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11e9ad8780 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11e9ad8780 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11e9ad8780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:26.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x5592ef51c200 con 0x5592ef49b550 2024-09-16T06:38:26.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:26.158+0000 7f11dbfff700 10 monclient: get_auth_request con 0x7f11c4034d50 auth_method 0 2024-09-16T06:38:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.090+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 141+0+0 (unknown 3216690297 0 0) 0x7f11d4013260 con 0x5592ef49b550 2024-09-16T06:38:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.090+0000 7f11e0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T06:38:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.090+0000 7f11e0ff9700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T06:38:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.090+0000 7f11e9ad8780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:27.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.090+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x5592ef317dd0 con 0x5592ef49b550 2024-09-16T06:38:27.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 2942983318 0 0) 0x7f11d4013260 con 0x5592ef49b550 2024-09-16T06:38:27.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e0ff9700 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-16T06:38:27.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e2ffd700 10 monclient: _renew_subs 2024-09-16T06:38:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e2ffd700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=15}) v3 -- 0x7f11c8002ae0 con 0x5592ef49b550 2024-09-16T06:38:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 11 ==== osd_map(15..15 src has 1..15) v4 ==== 1044+0+0 (unknown 3533265650 0 0) 0x7f11d4013b90 con 0x5592ef49b550 2024-09-16T06:38:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6800/3502405955 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x5592ef54c0f0 con 0x5592ef5485e0 2024-09-16T06:38:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.094+0000 7f11e9871700 10 monclient: get_auth_request con 0x5592ef5485e0 auth_method 0 2024-09-16T06:38:27.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.102+0000 7f11e9871700 1 -- 172.21.15.76:0/2142784857 <== osd.1 v1:172.21.15.76:6800/3502405955 1 ==== osd_op_reply(1 rbd_trash [create] v15'1 uv1 ondisk = 0) v8 ==== 153+0+0 (unknown 2722885549 0 0) 0x7f11dc004620 con 0x5592ef5485e0 2024-09-16T06:38:27.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.102+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6808/3838562525 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) v8 -- 0x5592ef5a0100 con 0x5592ef59c500 2024-09-16T06:38:27.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.102+0000 7f11e3fff700 10 monclient: get_auth_request con 0x5592ef59c500 auth_method 0 2024-09-16T06:38:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.106+0000 7f11e3fff700 1 -- 172.21.15.76:0/2142784857 <== osd.0 v1:172.21.15.76:6808/3838562525 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2354864950 0 0) 0x7f11d400fc50 con 0x5592ef59c500 2024-09-16T06:38:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.106+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6808/3838562525 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) v8 -- 0x5592ef59af90 con 0x5592ef59c500 2024-09-16T06:38:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.106+0000 7f11e3fff700 1 -- 172.21.15.76:0/2142784857 <== osd.0 v1:172.21.15.76:6808/3838562525 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 2366413114 0 0) 0x7f11d400fc50 con 0x5592ef59c500 2024-09-16T06:38:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.106+0000 7f11e27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:27.106+0000 7f11e27fc700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f11bc001f20 con 0x5592ef49b550 2024-09-16T06:38:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 12 ==== pool_op_reply(tid 4 (0) Success v16) v1 ==== 55+0+0 (unknown 3461865474 0 0) 0x7f11d40116d0 con 0x5592ef49b550 2024-09-16T06:38:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e0ff9700 10 monclient: _renew_subs 2024-09-16T06:38:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=16}) v3 -- 0x5592ef49af20 con 0x5592ef49b550 2024-09-16T06:38:28.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 13 ==== osd_map(16..16 src has 1..16) v4 ==== 996+0+0 (unknown 3394483122 0 0) 0x7f11d40136f0 con 0x5592ef49b550 2024-09-16T06:38:28.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.098+0000 7f11e2ffd700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6808/3838562525 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f11c8005600 con 0x5592ef59c500 2024-09-16T06:38:28.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.106+0000 7f11e3fff700 1 -- 172.21.15.76:0/2142784857 <== osd.0 v1:172.21.15.76:6808/3838562525 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v16'1 uv1 ondisk = 0) v8 ==== 194+0+0 (unknown 1230306805 0 0) 0x7f11d400fc50 con 0x5592ef59c500 2024-09-16T06:38:28.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.106+0000 7f11e27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:28.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.106+0000 7f11e27fc700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v16) v4 -- 0x7f11bc0033b0 con 0x5592ef49b550 2024-09-16T06:38:28.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:28.570+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 14 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f11d401b3c0 con 0x5592ef49b550 2024-09-16T06:38:29.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 15 ==== pool_op_reply(tid 6 (0) Success v17) v1 ==== 43+0+0 (unknown 2531423240 0 0) 0x7f11d4012e20 con 0x5592ef49b550 2024-09-16T06:38:29.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e0ff9700 10 monclient: _renew_subs 2024-09-16T06:38:29.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=17}) v3 -- 0x5592ef2f19a0 con 0x5592ef49b550 2024-09-16T06:38:29.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e0ff9700 1 -- 172.21.15.76:0/2142784857 <== mon.2 v1:172.21.15.76:6790/0 16 ==== osd_map(17..17 src has 1..17) v4 ==== 654+0+0 (unknown 2830632998 0 0) 0x7f11d4043350 con 0x5592ef49b550 2024-09-16T06:38:29.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.109+0000 7f11e2ffd700 1 -- 172.21.15.76:0/2142784857 --> v1:172.21.15.76:6808/3838562525 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x7f11c80059b0 con 0x5592ef59c500 2024-09-16T06:38:29.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e3fff700 1 -- 172.21.15.76:0/2142784857 <== osd.0 v1:172.21.15.76:6808/3838562525 4 ==== osd_op_reply(7 rbd_info [write 0~19] v17'2 uv2 ondisk = 0) v8 ==== 152+0+0 (unknown 3144429724 0 0) 0x7f11d400fc50 con 0x5592ef59c500 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.76:6808/3838562525 conn(0x5592ef59c500 legacy=0x5592ef59e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.76:6800/3502405955 conn(0x5592ef5485e0 legacy=0x5592ef54a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.189:6812/3098810674 conn(0x7f11c4034d50 legacy=0x7f11c4037210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 10 monclient: shutdown 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9871700 1 -- 172.21.15.76:0/2142784857 reap_dead start 2024-09-16T06:38:29.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 >> v1:172.21.15.76:6790/0 conn(0x5592ef49b550 legacy=0x5592ef53e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 shutdown_connections 2024-09-16T06:38:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 >> 172.21.15.76:0/2142784857 conn(0x5592ef47dcf0 msgr2=0x5592ef495bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 shutdown_connections 2024-09-16T06:38:29.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.113+0000 7f11e9ad8780 1 -- 172.21.15.76:0/2142784857 wait complete. 2024-09-16T06:38:29.121 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T06:38:29.121 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T06:38:29.121 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T06:38:29.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T06:38:29.122 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T06:38:29.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.225+0000 7fa771a37700 10 monclient: get_monmap_and_config 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.225+0000 7fa771a37700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.225+0000 7fa771a37700 10 monclient: monmap: 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:29.230785+0000 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:29.230785+0000 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:29.229 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:29.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 1 Processor -- start 2024-09-16T06:38:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 1 -- start start 2024-09-16T06:38:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient: init 2024-09-16T06:38:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-a con 0x7fa76c131910 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-b con 0x7fa76c1347f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-c con 0x7fa76c138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c132400 con 0x7fa76c131910 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c09fcd0 con 0x7fa76c138300 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c09b300 con 0x7fa76c1347f0 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient(hunting): authenticate will time out at 856.365720s 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa770a35700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa76c131910 0x7fa76c131cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60374/0 (socket says 172.21.15.76:60374) 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa770a35700 1 -- 172.21.15.76:0/3123166301 learned_addr learned my addr 172.21.15.76:0/3123166301 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424906819 0 0) 0x7fa76c132400 con 0x7fa76c131910 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): my global_id is 4239 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c0046e0 con 0x7fa76c131910 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504172062 0 0) 0x7fa76c09b300 con 0x7fa76c1347f0 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): my global_id is 4238 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c132400 con 0x7fa76c1347f0 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340646801 0 0) 0x7fa76c09fcd0 con 0x7fa76c138300 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient(hunting): my global_id is 4240 2024-09-16T06:38:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c09b300 con 0x7fa76c138300 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079164558 0 0) 0x7fa76c132400 con 0x7fa76c1347f0 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 >> v1:172.21.15.189:6789/0 conn(0x7fa76c138300 legacy=0x7fa76c13a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 >> v1:172.21.15.76:6789/0 conn(0x7fa76c131910 legacy=0x7fa76c131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c09b7e0 con 0x7fa76c1347f0 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 5 monclient: authenticate success, global_id 4238 2024-09-16T06:38:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa758001570 con 0x7fa76c1347f0 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: dump: 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.238 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa758001890 con 0x7fa76c1347f0 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 1 -- 172.21.15.76:0/3123166301 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa758001c60 con 0x7fa76c1347f0 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa771a37700 4 set_mon_vals no callback set 2024-09-16T06:38:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.229+0000 7fa76b7fe700 10 monclient: dump: 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: shutdown 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- 172.21.15.76:0/3123166301 >> v1:172.21.15.76:6790/0 conn(0x7fa76c1347f0 legacy=0x7fa76c136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- 172.21.15.76:0/3123166301 shutdown_connections 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- 172.21.15.76:0/3123166301 >> 172.21.15.76:0/3123166301 conn(0x7fa76c0a4430 msgr2=0x7fa76c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- 172.21.15.76:0/3123166301 shutdown_connections 2024-09-16T06:38:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- 172.21.15.76:0/3123166301 wait complete. 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: monmap: 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 Processor -- start 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- start start 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: init 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-c con 0x7fa76c131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-b con 0x7fa76c1347f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient(hunting): picked mon.noname-a con 0x7fa76c138300 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c11bb00 con 0x7fa76c138300 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c1380b0 con 0x7fa76c1347f0 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c11c310 con 0x7fa76c131910 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa771a37700 10 monclient(hunting): authenticate will time out at 856.371934s 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa770a35700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa76c131910 0x7fa76c12f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52784/0 (socket says 172.21.15.76:52784) 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.233+0000 7fa770a35700 1 -- 172.21.15.76:0/243425523 learned_addr learned my addr 172.21.15.76:0/243425523 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33008690 0 0) 0x7fa76c11c310 con 0x7fa76c131910 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): my global_id is 4241 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7500035d0 con 0x7fa76c131910 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532686041 0 0) 0x7fa76c11bb00 con 0x7fa76c138300 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): my global_id is 4242 2024-09-16T06:38:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c11c310 con 0x7fa76c138300 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333019194 0 0) 0x7fa76c1380b0 con 0x7fa76c1347f0 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): my global_id is 4243 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c11bb00 con 0x7fa76c1347f0 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907933570 0 0) 0x7fa76c11c310 con 0x7fa76c138300 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76c1380b0 con 0x7fa76c138300 2024-09-16T06:38:29.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa764001bb0 con 0x7fa76c138300 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): dump: 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3746500349 0 0) 0x7fa7500035d0 con 0x7fa76c131910 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76c11c310 con 0x7fa76c131910 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7600019f0 con 0x7fa76c131910 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): dump: 2024-09-16T06:38:29.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331826384 0 0) 0x7fa76c11bb00 con 0x7fa76c1347f0 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7500035d0 con 0x7fa76c1347f0 2024-09-16T06:38:29.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa758001530 con 0x7fa76c1347f0 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient(hunting): dump: 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 682926341 0 0) 0x7fa76c1380b0 con 0x7fa76c138300 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 >> v1:172.21.15.76:6790/0 conn(0x7fa76c131910 legacy=0x7fa76c12f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 >> v1:172.21.15.189:6789/0 conn(0x7fa76c1347f0 legacy=0x7fa76c1ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 monclient: found mon.a 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c071600 con 0x7fa76c138300 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 5 monclient: authenticate success, global_id 4242 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _renew_subs 2024-09-16T06:38:29.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c09d4c0 con 0x7fa76c138300 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _renew_subs 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c11c030 con 0x7fa76c138300 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa764003a40 con 0x7fa76c138300 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa76affd700 4 set_mon_vals no callback set 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa764003d50 con 0x7fa76c138300 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 10 monclient: dump: 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.257 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa76400e7c0 con 0x7fa76c138300 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c005100 con 0x7fa76c138300 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.237+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fa764042100 con 0x7fa76c138300 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.241+0000 7fa770a35700 10 monclient: get_auth_request con 0x7fa750036460 auth_method 0 2024-09-16T06:38:29.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.245+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa764013580 con 0x7fa76c138300 2024-09-16T06:38:29.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.245+0000 7fa7697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.245+0000 7fa7697fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:29.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.457+0000 7fa771a37700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T06:38:29.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.457+0000 7fa771a37700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T06:38:29.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.457+0000 7fa771a37700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.457+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa74c0053b0 con 0x7fa76c138300 2024-09-16T06:38:29.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa7697fa700 1 -- 172.21.15.76:0/243425523 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83059 (unknown 481731318 0 2229581682) 0x7fa764011c20 con 0x7fa76c138300 2024-09-16T06:38:29.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa7697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T06:38:29.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa7697fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:29.466 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:29.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa750036460 legacy=0x7fa750038920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 10 monclient: shutdown 2024-09-16T06:38:29.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 >> v1:172.21.15.76:6789/0 conn(0x7fa76c138300 legacy=0x7fa76c1d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 shutdown_connections 2024-09-16T06:38:29.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 >> 172.21.15.76:0/243425523 conn(0x7fa76c0a4430 msgr2=0x7fa76c09f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 shutdown_connections 2024-09-16T06:38:29.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.461+0000 7fa771a37700 1 -- 172.21.15.76:0/243425523 wait complete. 2024-09-16T06:38:29.478 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6812","nonce":3098810674}]},"active_addr":"172.21.15.189:6812/3098810674","active_change":"2024-09-16T06:38:21.631410+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":33431992}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2377186413}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2193630857}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":4071701924}]}]} 2024-09-16T06:38:29.480 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T06:38:29.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T06:38:29.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: get_monmap_and_config 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: monmap: 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:29.593927+0000 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:29.593927+0000 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.593 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:29.593 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:29.593 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 1 Processor -- start 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 1 -- start start 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: init 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-a con 0x7fad1409c8e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-c con 0x7fad1409f410 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-b con 0x7fad1409d3d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1409ed10 con 0x7fad1409c8e0 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1409dc40 con 0x7fad1409f410 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140a1c20 con 0x7fad1409d3d0 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad18d34700 10 monclient(hunting): authenticate will time out at 856.727550s 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad137fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fad1409c8e0 0x7fad1409ccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60396/0 (socket says 172.21.15.76:60396) 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad13fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fad1409d3d0 0x7fad1413bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52786/0 (socket says 172.21.15.76:52786) 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.589+0000 7fad137fe700 1 -- 172.21.15.76:0/1897846058 learned_addr learned my addr 172.21.15.76:0/1897846058 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681767574 0 0) 0x7fad140a1c20 con 0x7fad1409d3d0 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): my global_id is 4244 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc0035d0 con 0x7fad1409d3d0 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675110851 0 0) 0x7fad1409ed10 con 0x7fad1409c8e0 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): my global_id is 4245 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad140a1c20 con 0x7fad1409c8e0 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450499995 0 0) 0x7fad1409dc40 con 0x7fad1409f410 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient(hunting): my global_id is 4246 2024-09-16T06:38:29.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1409ed10 con 0x7fad1409f410 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427662725 0 0) 0x7facfc0035d0 con 0x7fad1409d3d0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 >> v1:172.21.15.189:6789/0 conn(0x7fad1409f410 legacy=0x7fad14138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 >> v1:172.21.15.76:6789/0 conn(0x7fad1409c8e0 legacy=0x7fad1409ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 monclient: found mon.noname-b 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1409c3e0 con 0x7fad1409d3d0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad0c001990 con 0x7fad1409d3d0 2024-09-16T06:38:29.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: dump: 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 5 monclient: authenticate success, global_id 4244 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad0c001c70 con 0x7fad1409d3d0 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 1 -- 172.21.15.76:0/1897846058 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad0c002040 con 0x7fad1409d3d0 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 4 set_mon_vals no callback set 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad127fc700 10 monclient: dump: 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.601 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 10 monclient: shutdown 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- 172.21.15.76:0/1897846058 >> v1:172.21.15.76:6790/0 conn(0x7fad1409d3d0 legacy=0x7fad1413bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- 172.21.15.76:0/1897846058 shutdown_connections 2024-09-16T06:38:29.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- 172.21.15.76:0/1897846058 >> 172.21.15.76:0/1897846058 conn(0x7fad140a2300 msgr2=0x7fad140a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- 172.21.15.76:0/1897846058 shutdown_connections 2024-09-16T06:38:29.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- 172.21.15.76:0/1897846058 wait complete. 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 10 monclient: monmap: 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:29.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 Processor -- start 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 1 -- start start 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.593+0000 7fad18d34700 10 monclient: init 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:29.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-c con 0x7fad1409c8e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-a con 0x7fad1409d3d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient(hunting): picked mon.noname-b con 0x7fad1409f410 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad141c87c0 con 0x7fad1409d3d0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140998d0 con 0x7fad1409f410 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad141b6020 con 0x7fad1409c8e0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient(hunting): authenticate will time out at 856.732663s 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad137fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fad1409c8e0 0x7fad141ba840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52790/0 (socket says 172.21.15.76:52790) 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad137fe700 1 -- 172.21.15.76:0/2485995916 learned_addr learned my addr 172.21.15.76:0/2485995916 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421064890 0 0) 0x7fad1409ed10 con 0x7fad1409d3d0 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): my global_id is 4248 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad000035d0 con 0x7fad1409d3d0 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050936527 0 0) 0x7fad141b6020 con 0x7fad1409c8e0 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): my global_id is 4247 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1409ed10 con 0x7fad1409c8e0 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075786975 0 0) 0x7fad1409c3e0 con 0x7fad1409f410 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): my global_id is 4249 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad141b6020 con 0x7fad1409f410 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222728869 0 0) 0x7fad1409ed10 con 0x7fad1409c8e0 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad1409c3e0 con 0x7fad1409c8e0 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58966884 0 0) 0x7fad000035d0 con 0x7fad1409d3d0 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad1409ed10 con 0x7fad1409d3d0 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad080019d0 con 0x7fad1409c8e0 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): dump: 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.610 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad04001710 con 0x7fad1409d3d0 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient(hunting): dump: 2024-09-16T06:38:29.611 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193133704 0 0) 0x7fad141b6020 con 0x7fad1409f410 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad000035d0 con 0x7fad1409f410 2024-09-16T06:38:29.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495695334 0 0) 0x7fad1409c3e0 con 0x7fad1409c8e0 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 >> v1:172.21.15.189:6789/0 conn(0x7fad1409f410 legacy=0x7fad141d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 >> v1:172.21.15.76:6789/0 conn(0x7fad1409d3d0 legacy=0x7fad141ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 monclient: found mon.c 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14099310 con 0x7fad1409c8e0 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 5 monclient: authenticate success, global_id 4247 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _renew_subs 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1409c5d0 con 0x7fad1409c8e0 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad08001cc0 con 0x7fad1409c8e0 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad08004ce0 con 0x7fad1409c8e0 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 10 monclient: dump: 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.614 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _renew_subs 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad141b4f90 con 0x7fad1409c8e0 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad117fa700 4 set_mon_vals no callback set 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fad0800f750 con 0x7fad1409c8e0 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fad08043290 con 0x7fad1409c8e0 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad140678b0 con 0x7fad1409c8e0 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.597+0000 7fad12ffd700 10 monclient: get_auth_request con 0x7fad00036490 auth_method 0 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.605+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fad08014610 con 0x7fad1409c8e0 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.605+0000 7facfbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.605+0000 7facfbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:29.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7fad18d34700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:29.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7fad18d34700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:29.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7fad18d34700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad1409e7e0 con 0x7fad1409c8e0 2024-09-16T06:38:29.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7facfbfff700 1 -- 172.21.15.76:0/2485995916 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+10832 (unknown 3225238715 0 4229994956) 0x7fad08012cb0 con 0x7fad1409c8e0 2024-09-16T06:38:29.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7facfbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:29.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.777+0000 7facfbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:29.780 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:29.780 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:29.102711+0000","last_up_change":"2024-09-16T06:38:25.048660+0000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T06:38:25.702868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T06:38:25.996213+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6808","nonce":3838562525}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6809","nonce":3838562525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6811","nonce":3838562525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6810","nonce":3838562525}]},"public_addr":"172.21.15.76:6808/3838562525","cluster_addr":"172.21.15.76:6809/3838562525","heartbeat_back_addr":"172.21.15.76:6811/3838562525","heartbeat_front_addr":"172.21.15.76:6810/3838562525","state":["exists","up"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6800","nonce":3502405955}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6801","nonce":3502405955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6803","nonce":3502405955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6802","nonce":3502405955}]},"public_addr":"172.21.15.76:6800/3502405955","cluster_addr":"172.21.15.76:6801/3502405955","heartbeat_back_addr":"172.21.15.76:6803/3502405955","heartbeat_front_addr":"172.21.15.76:6802/3502405955","state":["exists","up"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6804","nonce":779009499}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6805","nonce":779009499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6807","nonce":779009499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6806","nonce":779009499}]},"public_addr":"172.21.15.76:6804/779009499","cluster_addr":"172.21.15.76:6805/779009499","heartbeat_back_addr":"172.21.15.76:6807/779009499","heartbeat_front_addr":"172.21.15.76:6806/779009499","state":["exists","up"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3809009238}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3809009238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3809009238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3809009238}]},"public_addr":"172.21.15.189:6808/3809009238","cluster_addr":"172.21.15.189:6809/3809009238","heartbeat_back_addr":"172.21.15.189:6811/3809009238","heartbeat_front_addr":"172.21.15.189:6810/3809009238","state":["exists","up"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":407912501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":407912501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":407912501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":407912501}]},"public_addr":"172.21.15.189:6804/407912501","cluster_addr":"172.21.15.189:6805/407912501","heartbeat_back_addr":"172.21.15.189:6807/407912501","heartbeat_front_addr":"172.21.15.189:6806/407912501","state":["exists","up"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":2128366135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":2128366135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":2128366135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":2128366135}]},"public_addr":"172.21.15.189:6800/2128366135","cluster_addr":"172.21.15.189:6801/2128366135","heartbeat_back_addr":"172.21.15.189:6803/2128366135","heartbeat_front_addr":"172.21.15.189:6802/2128366135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.209148+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.408945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.850780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:21.901722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.166044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.780816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 >> v1:172.21.15.189:6812/3098810674 conn(0x7fad00036490 legacy=0x7fad00038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 10 monclient: shutdown 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 >> v1:172.21.15.76:6790/0 conn(0x7fad1409c8e0 legacy=0x7fad141ba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 shutdown_connections 2024-09-16T06:38:29.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 >> 172.21.15.76:0/2485995916 conn(0x7fad140a2300 msgr2=0x7fad14070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 shutdown_connections 2024-09-16T06:38:29.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.781+0000 7fad18d34700 1 -- 172.21.15.76:0/2485995916 wait complete. 2024-09-16T06:38:29.800 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T06:38:29.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: get_monmap_and_config 2024-09-16T06:38:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: monmap: 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:29.909529+0000 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:29.909529+0000 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:29.908 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 1 Processor -- start 2024-09-16T06:38:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 1 -- start start 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: init 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-a con 0x7f1e74070d10 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-b con 0x7f1e7409e920 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-c con 0x7f1e74071800 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e74073f50 con 0x7f1e74070d10 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7409efc0 con 0x7f1e74071800 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7409f2a0 con 0x7f1e7409e920 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e790b9700 10 monclient(hunting): authenticate will time out at 857.043283s 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e737fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1e7409e920 0x7f1e7413a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52796/0 (socket says 172.21.15.76:52796) 2024-09-16T06:38:29.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e737fe700 1 -- 172.21.15.76:0/3669961522 learned_addr learned my addr 172.21.15.76:0/3669961522 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036298196 0 0) 0x7f1e7409f2a0 con 0x7f1e7409e920 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): my global_id is 4250 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c0035d0 con 0x7f1e7409e920 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506810994 0 0) 0x7f1e74073f50 con 0x7f1e74070d10 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 10 monclient(hunting): my global_id is 4251 2024-09-16T06:38:29.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.905+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7409f2a0 con 0x7f1e74070d10 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003394022 0 0) 0x7f1e7409efc0 con 0x7f1e74071800 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient(hunting): my global_id is 4252 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e74073f50 con 0x7f1e74071800 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2562496334 0 0) 0x7f1e5c0035d0 con 0x7f1e7409e920 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 >> v1:172.21.15.189:6789/0 conn(0x7f1e74071800 legacy=0x7f1e7413db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 >> v1:172.21.15.76:6789/0 conn(0x7f1e74070d10 legacy=0x7f1e740710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 monclient: found mon.noname-b 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:29.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740a4ab0 con 0x7f1e7409e920 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 5 monclient: authenticate success, global_id 4250 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e64001650 con 0x7f1e7409e920 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: dump: 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.916 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e640018e0 con 0x7f1e7409e920 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 1 -- 172.21.15.76:0/3669961522 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e64001c70 con 0x7f1e7409e920 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 4 set_mon_vals no callback set 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e72ffd700 10 monclient: dump: 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: shutdown 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:29.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- 172.21.15.76:0/3669961522 >> v1:172.21.15.76:6790/0 conn(0x7f1e7409e920 legacy=0x7f1e7413a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- 172.21.15.76:0/3669961522 shutdown_connections 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- 172.21.15.76:0/3669961522 >> 172.21.15.76:0/3669961522 conn(0x7f1e740a4430 msgr2=0x7f1e740a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- 172.21.15.76:0/3669961522 shutdown_connections 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- 172.21.15.76:0/3669961522 wait complete. 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: build_initial_monmap 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: monmap: 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 Processor -- start 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- start start 2024-09-16T06:38:29.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: init 2024-09-16T06:38:29.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:29.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-b con 0x7f1e74070d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-a con 0x7f1e74071800 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient(hunting): picked mon.noname-c con 0x7f1e7413de20 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7412fc60 con 0x7f1e74071800 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7413a810 con 0x7f1e74070d10 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e74137680 con 0x7f1e7413de20 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.909+0000 7f1e790b9700 10 monclient(hunting): authenticate will time out at 857.047842s 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e788b8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1e7413de20 0x7f1e741d2370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52804/0 (socket says 172.21.15.76:52804) 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e788b8700 1 -- 172.21.15.76:0/15823865 learned_addr learned my addr 172.21.15.76:0/15823865 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e737fe700 1 --1- 172.21.15.76:0/15823865 >> v1:172.21.15.76:6789/0 conn(0x7f1e74071800 0x7f1e74136f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60422/0 (socket says 172.21.15.76:60422) 2024-09-16T06:38:29.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140496880 0 0) 0x7f1e74137680 con 0x7f1e7413de20 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): my global_id is 4253 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e600035d0 con 0x7f1e7413de20 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321339014 0 0) 0x7f1e7412fc60 con 0x7f1e74071800 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): my global_id is 4254 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e74137680 con 0x7f1e74071800 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523044875 0 0) 0x7f1e7413a810 con 0x7f1e74070d10 2024-09-16T06:38:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): my global_id is 4255 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7412fc60 con 0x7f1e74070d10 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777561972 0 0) 0x7f1e600035d0 con 0x7f1e7413de20 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e7413a810 con 0x7f1e7413de20 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e6c001c90 con 0x7f1e7413de20 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.926 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283787830 0 0) 0x7f1e74137680 con 0x7f1e74071800 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e600035d0 con 0x7f1e74071800 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e64001610 con 0x7f1e74071800 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378257224 0 0) 0x7f1e7412fc60 con 0x7f1e74070d10 2024-09-16T06:38:29.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e74137680 con 0x7f1e74070d10 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e680018b0 con 0x7f1e74070d10 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684925523 0 0) 0x7f1e600035d0 con 0x7f1e74071800 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 >> v1:172.21.15.76:6790/0 conn(0x7f1e7413de20 legacy=0x7f1e741d2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 >> v1:172.21.15.189:6789/0 conn(0x7f1e74070d10 legacy=0x7f1e74136860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 monclient: found mon.a 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74130190 con 0x7f1e74071800 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:29.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 5 monclient: authenticate success, global_id 4254 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _renew_subs 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7413a530 con 0x7f1e74071800 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _renew_subs 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7413d410 con 0x7f1e74071800 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e64004b40 con 0x7f1e74071800 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e64004e50 con 0x7f1e74071800 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e727fc700 4 set_mon_vals no callback set 2024-09-16T06:38:29.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 10 monclient: dump: 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:29.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54005100 con 0x7f1e74071800 2024-09-16T06:38:29.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1e6400f8c0 con 0x7f1e74071800 2024-09-16T06:38:29.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f1e64043280 con 0x7f1e74071800 2024-09-16T06:38:29.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.913+0000 7f1e73fff700 10 monclient: get_auth_request con 0x7f1e60036540 auth_method 0 2024-09-16T06:38:29.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.921+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1e640146c0 con 0x7f1e74071800 2024-09-16T06:38:29.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.921+0000 7f1e70ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:29.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:29.921+0000 7f1e70ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e790b9700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:30.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e790b9700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:30.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e790b9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e540053b0 con 0x7f1e74071800 2024-09-16T06:38:30.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e70ff9700 1 -- 172.21.15.76:0/15823865 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+10832 (unknown 3225238715 0 4229994956) 0x7f1e64012d60 con 0x7f1e74071800 2024-09-16T06:38:30.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e70ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:30.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.093+0000 7f1e70ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.097 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:30.097 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:29.102711+0000","last_up_change":"2024-09-16T06:38:25.048660+0000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T06:38:25.702868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T06:38:25.996213+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6808","nonce":3838562525}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6809","nonce":3838562525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6811","nonce":3838562525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6810","nonce":3838562525}]},"public_addr":"172.21.15.76:6808/3838562525","cluster_addr":"172.21.15.76:6809/3838562525","heartbeat_back_addr":"172.21.15.76:6811/3838562525","heartbeat_front_addr":"172.21.15.76:6810/3838562525","state":["exists","up"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6800","nonce":3502405955}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6801","nonce":3502405955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6803","nonce":3502405955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6802","nonce":3502405955}]},"public_addr":"172.21.15.76:6800/3502405955","cluster_addr":"172.21.15.76:6801/3502405955","heartbeat_back_addr":"172.21.15.76:6803/3502405955","heartbeat_front_addr":"172.21.15.76:6802/3502405955","state":["exists","up"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6804","nonce":779009499}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6805","nonce":779009499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6807","nonce":779009499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6806","nonce":779009499}]},"public_addr":"172.21.15.76:6804/779009499","cluster_addr":"172.21.15.76:6805/779009499","heartbeat_back_addr":"172.21.15.76:6807/779009499","heartbeat_front_addr":"172.21.15.76:6806/779009499","state":["exists","up"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3809009238}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3809009238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3809009238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3809009238}]},"public_addr":"172.21.15.189:6808/3809009238","cluster_addr":"172.21.15.189:6809/3809009238","heartbeat_back_addr":"172.21.15.189:6811/3809009238","heartbeat_front_addr":"172.21.15.189:6810/3809009238","state":["exists","up"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":407912501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":407912501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":407912501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":407912501}]},"public_addr":"172.21.15.189:6804/407912501","cluster_addr":"172.21.15.189:6805/407912501","heartbeat_back_addr":"172.21.15.189:6807/407912501","heartbeat_front_addr":"172.21.15.189:6806/407912501","state":["exists","up"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":2128366135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":2128366135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":2128366135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":2128366135}]},"public_addr":"172.21.15.189:6800/2128366135","cluster_addr":"172.21.15.189:6801/2128366135","heartbeat_back_addr":"172.21.15.189:6803/2128366135","heartbeat_front_addr":"172.21.15.189:6802/2128366135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.209148+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.408945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.850780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:21.901722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.166044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.780816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1e60036540 legacy=0x7f1e60038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 10 monclient: shutdown 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 >> v1:172.21.15.76:6789/0 conn(0x7f1e74071800 legacy=0x7f1e74136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 shutdown_connections 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 >> 172.21.15.76:0/15823865 conn(0x7f1e740a4430 msgr2=0x7f1e74131440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 shutdown_connections 2024-09-16T06:38:30.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.097+0000 7f1e790b9700 1 -- 172.21.15.76:0/15823865 wait complete. 2024-09-16T06:38:30.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T06:38:30.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T06:38:30.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T06:38:30.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T06:38:30.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T06:38:30.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.233+0000 7ff97ec71700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.233+0000 7ff97ec71700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.233+0000 7ff97ec71700 10 monclient: monmap: 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.239089+0000 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.239089+0000 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.237 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.238 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.238 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.238 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.238 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.233+0000 7ff97ec71700 1 Processor -- start 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- start start 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient: init 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-b con 0x7ff978132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-a con 0x7ff97806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-c con 0x7ff97806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97806c280 con 0x7ff97806d480 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97806bfa0 con 0x7ff97806cf50 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97809ace0 con 0x7ff978132330 2024-09-16T06:38:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient(hunting): authenticate will time out at 857.372987s 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97dc6f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff978132330 0x7ff978132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52818/0 (socket says 172.21.15.76:52818) 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97dc6f700 1 -- 172.21.15.76:0/2567787254 learned_addr learned my addr 172.21.15.76:0/2567787254 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016625125 0 0) 0x7ff97809ace0 con 0x7ff978132330 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): my global_id is 4256 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9640035d0 con 0x7ff978132330 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267848153 0 0) 0x7ff97806c280 con 0x7ff97806d480 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient(hunting): my global_id is 4257 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff97809ace0 con 0x7ff97806d480 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751397246 0 0) 0x7ff97809ace0 con 0x7ff97806d480 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 >> v1:172.21.15.76:6790/0 conn(0x7ff978132330 legacy=0x7ff978132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 >> v1:172.21.15.189:6789/0 conn(0x7ff97806cf50 legacy=0x7ff97813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 monclient: found mon.noname-a 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97809ae70 con 0x7ff97806d480 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff968001610 con 0x7ff97806d480 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: dump: 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 5 monclient: authenticate success, global_id 4257 2024-09-16T06:38:30.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff968001830 con 0x7ff97806d480 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 1 -- 172.21.15.76:0/2567787254 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff968001a20 con 0x7ff97806d480 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97cc6d700 10 monclient: dump: 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.244 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.245 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.245 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 4 set_mon_vals no callback set 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 10 monclient: shutdown 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- 172.21.15.76:0/2567787254 >> v1:172.21.15.76:6789/0 conn(0x7ff97806d480 legacy=0x7ff97813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- 172.21.15.76:0/2567787254 shutdown_connections 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- 172.21.15.76:0/2567787254 >> 172.21.15.76:0/2567787254 conn(0x7ff9780a41c0 msgr2=0x7ff9780a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- 172.21.15.76:0/2567787254 shutdown_connections 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.237+0000 7ff97ec71700 1 -- 172.21.15.76:0/2567787254 wait complete. 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: monmap: 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 Processor -- start 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- start start 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: init 2024-09-16T06:38:30.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: monmap: 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.244139+0000 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.244139+0000 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.248 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.249 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-c con 0x7ff97806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-a con 0x7ff97806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient(hunting): picked mon.noname-b con 0x7ff978132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9781306e0 con 0x7ff97806d480 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9781ca380 con 0x7ff978132330 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff978130950 con 0x7ff97806cf50 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient(hunting): authenticate will time out at 857.377102s 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97dc6f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff97806cf50 0x7ff9781c9560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52826/0 (socket says 172.21.15.76:52826) 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97dc6f700 1 -- 172.21.15.76:0/113579970 learned_addr learned my addr 172.21.15.76:0/113579970 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 1 Processor -- start 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826602984 0 0) 0x7ff978130950 con 0x7ff97806cf50 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): my global_id is 4259 2024-09-16T06:38:30.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff964005100 con 0x7ff97806cf50 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895553896 0 0) 0x7ff9781306e0 con 0x7ff97806d480 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): my global_id is 4260 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff978130950 con 0x7ff97806d480 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 1 -- start start 2024-09-16T06:38:30.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: init 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215821823 0 0) 0x7ff9781ca380 con 0x7ff978132330 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): my global_id is 4261 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9781306e0 con 0x7ff978132330 2024-09-16T06:38:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 409668465 0 0) 0x7ff978130950 con 0x7ff97806d480 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9781ca380 con 0x7ff97806d480 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff9680019a0 con 0x7ff97806d480 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435725583 0 0) 0x7ff9781ca380 con 0x7ff97806d480 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 >> v1:172.21.15.76:6790/0 conn(0x7ff97806cf50 legacy=0x7ff9781c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 >> v1:172.21.15.189:6789/0 conn(0x7ff978132330 legacy=0x7ff9781d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 monclient: found mon.a 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978130c30 con 0x7ff97806d480 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc98c1315f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff968003950 con 0x7ff97806d480 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 5 monclient: authenticate success, global_id 4260 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _renew_subs 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97812f340 con 0x7ff97806d480 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fc98c1320e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff968003ce0 con 0x7ff97806d480 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 10 monclient: dump: 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _renew_subs 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96ffff700 4 set_mon_vals no callback set 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97812ee20 con 0x7ff97806d480 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fc98c0983d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0996a0 con 0x7fc98c1320e0 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c098820 con 0x7fc98c0983d0 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c09fc30 con 0x7fc98c1315f0 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98bfff700 10 monclient(hunting): authenticate will time out at 857.378579s 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc98c0983d0 0x7fc98c139fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:52794/0 (socket says 172.21.15.76:52794) 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc98b7fe700 1 -- 172.21.15.76:0/2845716190 learned_addr learned my addr 172.21.15.76:0/2845716190 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372655164 0 0) 0x7fc98c09fc30 con 0x7fc98c1315f0 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): my global_id is 4262 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9740035d0 con 0x7fc98c1315f0 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004806031 0 0) 0x7fc98c098820 con 0x7fc98c0983d0 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff96800e7d0 con 0x7ff97806d480 2024-09-16T06:38:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 10 monclient(hunting): my global_id is 4264 2024-09-16T06:38:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.241+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98c09fc30 con 0x7fc98c0983d0 2024-09-16T06:38:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7ff9680420c0 con 0x7ff97806d480 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780279331 0 0) 0x7fc98c09fc30 con 0x7fc98c0983d0 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 >> v1:172.21.15.76:6790/0 conn(0x7fc98c1315f0 legacy=0x7fc98c1319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 >> v1:172.21.15.76:6789/0 conn(0x7fc98c1320e0 legacy=0x7fc98c1368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 monclient: found mon.noname-c 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0990e0 con 0x7fc98c0983d0 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 5 monclient: authenticate success, global_id 4264 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc984001990 con 0x7fc98c0983d0 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7ff97dc6f700 10 monclient: get_auth_request con 0x7ff9640365a0 auth_method 0 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: dump: 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc984001bb0 con 0x7fc98c0983d0 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 1 -- 172.21.15.76:0/2845716190 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc984001f40 con 0x7fc98c0983d0 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 4 set_mon_vals no callback set 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc989ffb700 10 monclient: dump: 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6804/779009499 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9780a0ae0 con 0x7ff978060db0 2024-09-16T06:38:30.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: shutdown 2024-09-16T06:38:30.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- 172.21.15.76:0/2845716190 >> v1:172.21.15.189:6789/0 conn(0x7fc98c0983d0 legacy=0x7fc98c139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- 172.21.15.76:0/2845716190 shutdown_connections 2024-09-16T06:38:30.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- 172.21.15.76:0/2845716190 >> 172.21.15.76:0/2845716190 conn(0x7fc98c0a0030 msgr2=0x7fc98c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7ff97e470700 10 monclient: get_auth_request con 0x7ff978060db0 auth_method 0 2024-09-16T06:38:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- 172.21.15.76:0/2845716190 shutdown_connections 2024-09-16T06:38:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- 172.21.15.76:0/2845716190 wait complete. 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: monmap: 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.272 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== osd.2 v1:172.21.15.76:6804/779009499 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7ff9780a0ae0 con 0x7ff978060db0 2024-09-16T06:38:30.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 Processor -- start 2024-09-16T06:38:30.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- start start 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: init 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc98c0983d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fc98c1315f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fc98c1320e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c130a90 con 0x7fc98c1320e0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c12f710 con 0x7fc98c0983d0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c09bab0 con 0x7fc98c1315f0 2024-09-16T06:38:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.245+0000 7fc98bfff700 10 monclient(hunting): authenticate will time out at 857.383908s 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc98c1320e0 0x7fc98c12e690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60458/0 (socket says 172.21.15.76:60458) 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98b7fe700 1 -- 172.21.15.76:0/759686145 learned_addr learned my addr 172.21.15.76:0/759686145 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517529482 0 0) 0x7fc98c130a90 con 0x7fc98c1320e0 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): my global_id is 4266 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9780035d0 con 0x7fc98c1320e0 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852874123 0 0) 0x7fc98c12f710 con 0x7fc98c0983d0 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): my global_id is 4267 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98c130a90 con 0x7fc98c0983d0 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 489584489 0 0) 0x7fc9780035d0 con 0x7fc98c1320e0 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc98c12f710 con 0x7fc98c1320e0 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc984001ec0 con 0x7fc98c1320e0 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): dump: 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1739426388 0 0) 0x7fc98c12f710 con 0x7fc98c1320e0 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 >> v1:172.21.15.76:6790/0 conn(0x7fc98c1315f0 legacy=0x7fc98c12df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 >> v1:172.21.15.189:6789/0 conn(0x7fc98c0983d0 legacy=0x7fc98c12d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 monclient: found mon.a 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c130d30 con 0x7fc98c1320e0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 5 monclient: authenticate success, global_id 4266 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 10 monclient: _renew_subs 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c131010 con 0x7fc98c1320e0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 10 monclient: _renew_subs 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c1d3070 con 0x7fc98c1320e0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc984004ec0 con 0x7fc98c1320e0 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc9840051d0 con 0x7fc98c1320e0 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 10 monclient: dump: 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9897fa700 4 set_mon_vals no callback set 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc98400fc40 con 0x7fc98c1320e0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fc984043600 con 0x7fc98c1320e0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.249+0000 7fc98affd700 10 monclient: get_auth_request con 0x7fc9780362d0 auth_method 0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.189:6800/2128366135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc96c005330 con 0x7fc96c0015c0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fc98a7fc700 10 monclient: get_auth_request con 0x7fc96c0015c0 auth_method 0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== osd.5 v1:172.21.15.189:6800/2128366135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fc96c005330 con 0x7fc96c0015c0 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: monmap: 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.260057+0000 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.260057+0000 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.287 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fd08359e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fd08359e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.253+0000 7fd08359e700 10 monclient: monmap: 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.259871+0000 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.259871+0000 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 1 Processor -- start 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 1 Processor -- start 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: monmap: 2024-09-16T06:38:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.260963+0000 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.260963+0000 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 1 -- start start 2024-09-16T06:38:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient: init 2024-09-16T06:38:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 1 -- start start 2024-09-16T06:38:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: init 2024-09-16T06:38:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd084070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd084070ea0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd08413b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084072610 con 0x7fd084070ac0 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08409f560 con 0x7fd08413b980 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084072330 con 0x7fd084070ea0 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 10 monclient(hunting): authenticate will time out at 857.394568s 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd084070ac0 0x7fd084097bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60462/0 (socket says 172.21.15.76:60462) 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd081d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd084070ea0 0x7fd0840712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52844/0 (socket says 172.21.15.76:52844) 2024-09-16T06:38:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08259c700 1 -- 172.21.15.76:0/3532711504 learned_addr learned my addr 172.21.15.76:0/3532711504 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 1 Processor -- start 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f4b9009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4b9009ed60 0x7f4b901315f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60466/0 (socket says 172.21.15.76:60466) 2024-09-16T06:38:30.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8e59c700 1 -- 172.21.15.76:0/2876890965 learned_addr learned my addr 172.21.15.76:0/2876890965 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108821704 0 0) 0x7fd084072330 con 0x7fd084070ea0 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): my global_id is 4265 2024-09-16T06:38:30.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd06c0035d0 con 0x7fd084070ea0 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f4b900984d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f4b901388c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b900996a0 con 0x7f4b9009ed60 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b90132410 con 0x7f4b901388c0 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9009fc30 con 0x7f4b900984d0 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8f59e700 10 monclient(hunting): authenticate will time out at 857.395306s 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308602799 0 0) 0x7fd084072610 con 0x7fd084070ac0 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient(hunting): my global_id is 4269 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd084072330 con 0x7fd084070ac0 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439228643 0 0) 0x7fd06c0035d0 con 0x7fd084070ea0 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 >> v1:172.21.15.189:6789/0 conn(0x7fd08413b980 legacy=0x7fd08413dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:38:30.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 >> v1:172.21.15.76:6789/0 conn(0x7fd084070ac0 legacy=0x7fd084097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840974f0 con 0x7fd084070ea0 2024-09-16T06:38:30.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd078001610 con 0x7fd084070ea0 2024-09-16T06:38:30.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: dump: 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 5 monclient: authenticate success, global_id 4265 2024-09-16T06:38:30.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 1 -- start start 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: init 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151435028 0 0) 0x7f4b900996a0 con 0x7f4b9009ed60 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 10 monclient(hunting): my global_id is 4272 2024-09-16T06:38:30.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b780035d0 con 0x7f4b9009ed60 2024-09-16T06:38:30.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd078001830 con 0x7fd084070ea0 2024-09-16T06:38:30.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7fd08159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5809ad50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-a con 0x7f5e5809d590 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-c con 0x7f5e5809b130 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5809df80 con 0x7f5e5809d590 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.257+0000 7f5e5f247700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5809c1a0 con 0x7f5e5809b130 2024-09-16T06:38:30.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5809a880 con 0x7f5e5809ad50 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 10 monclient(hunting): authenticate will time out at 857.396030s 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e57fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5e5809d590 0x7f5e5809a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60476/0 (socket says 172.21.15.76:60476) 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e57fff700 1 -- 172.21.15.76:0/4112669491 learned_addr learned my addr 172.21.15.76:0/4112669491 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2850923519 0 0) 0x7f4b780035d0 con 0x7f4b9009ed60 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 >> v1:172.21.15.76:6790/0 conn(0x7f4b900984d0 legacy=0x7f4b90131d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 >> v1:172.21.15.189:6789/0 conn(0x7f4b901388c0 legacy=0x7f4b9013acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 monclient: found mon.noname-a 2024-09-16T06:38:30.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b900990e0 con 0x7f4b9009ed60 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4b84001610 con 0x7f4b9009ed60 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: dump: 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.304 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 5 monclient: authenticate success, global_id 4272 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 4 set_mon_vals no callback set 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08159a700 1 -- 172.21.15.76:0/3532711504 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd078001bc0 con 0x7fd084070ea0 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08159a700 10 monclient: dump: 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.306 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b84001930 con 0x7f4b9009ed60 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 1 -- 172.21.15.76:0/2876890965 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4b84001cc0 con 0x7f4b9009ed60 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8d59a700 10 monclient: dump: 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.307 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 4 set_mon_vals no callback set 2024-09-16T06:38:30.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 10 monclient: shutdown 2024-09-16T06:38:30.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- 172.21.15.76:0/3532711504 >> v1:172.21.15.76:6790/0 conn(0x7fd084070ea0 legacy=0x7fd0840712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- 172.21.15.76:0/3532711504 shutdown_connections 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- 172.21.15.76:0/3532711504 >> 172.21.15.76:0/3532711504 conn(0x7fd0840a0030 msgr2=0x7fd0840a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: monmap: 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.264502+0000 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.264502+0000 2024-09-16T06:38:30.310 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714583756 0 0) 0x7f5e5809a880 con 0x7f5e5809ad50 2024-09-16T06:38:30.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): my global_id is 4271 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e400035d0 con 0x7f5e5809ad50 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481740749 0 0) 0x7f5e4c001110 con 0x7f5e5809d590 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): my global_id is 4275 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5809a880 con 0x7f5e5809d590 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 10 monclient: shutdown 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 >> v1:172.21.15.76:6789/0 conn(0x7f4b9009ed60 legacy=0x7f4b901315f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786850881 0 0) 0x7f5e5809c1a0 con 0x7f5e5809b130 2024-09-16T06:38:30.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient(hunting): my global_id is 4273 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4c001110 con 0x7f5e5809b130 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- 172.21.15.76:0/3532711504 shutdown_connections 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 1 Processor -- start 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 1 -- start start 2024-09-16T06:38:30.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: init 2024-09-16T06:38:30.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687725239 0 0) 0x7f5e400035d0 con 0x7f5e5809ad50 2024-09-16T06:38:30.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 >> v1:172.21.15.189:6789/0 conn(0x7f5e5809b130 legacy=0x7f5e5813a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 >> v1:172.21.15.76:6789/0 conn(0x7f5e5809d590 legacy=0x7f5e5809a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5809fc40 con 0x7f5e5809ad50 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e48001540 con 0x7f5e5809ad50 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: dump: 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.316 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 5 monclient: authenticate success, global_id 4271 2024-09-16T06:38:30.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 shutdown_connections 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 >> 172.21.15.76:0/2876890965 conn(0x7f4b900a0030 msgr2=0x7f4b900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 shutdown_connections 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2876890965 wait complete. 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e480017b0 con 0x7f5e5809ad50 2024-09-16T06:38:30.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 1 -- 172.21.15.76:0/4112669491 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e48001b40 con 0x7f5e5809ad50 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 4 set_mon_vals no callback set 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e577fe700 10 monclient: dump: 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.319 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.320 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.320 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-a con 0x7fa63409d180 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-b con 0x7fa63409d560 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-c con 0x7fa634138610 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63412f240 con 0x7fa63409d180 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa634099d40 con 0x7fa634138610 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63409a300 con 0x7fa63409d560 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa63ab6e700 10 monclient(hunting): authenticate will time out at 857.398757s 2024-09-16T06:38:30.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa639b6c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa63409d180 0x7fa63412eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60482/0 (socket says 172.21.15.76:60482) 2024-09-16T06:38:30.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa639b6c700 1 -- 172.21.15.76:0/1403685028 learned_addr learned my addr 172.21.15.76:0/1403685028 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 10 monclient: shutdown 2024-09-16T06:38:30.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- 172.21.15.76:0/4112669491 >> v1:172.21.15.76:6790/0 conn(0x7f5e5809ad50 legacy=0x7f5e58099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- 172.21.15.76:0/3532711504 wait complete. 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- 172.21.15.76:0/4112669491 shutdown_connections 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- 172.21.15.76:0/4112669491 >> 172.21.15.76:0/4112669491 conn(0x7f5e580a3e50 msgr2=0x7f5e580a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720507608 0 0) 0x7fa63412f240 con 0x7fa63409d180 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 10 monclient(hunting): my global_id is 4278 2024-09-16T06:38:30.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6200035d0 con 0x7fa63409d180 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 10 monclient: monmap: 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- 172.21.15.76:0/4112669491 shutdown_connections 2024-09-16T06:38:30.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7f5e5f247700 1 -- 172.21.15.76:0/4112669491 wait complete. 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 Processor -- start 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 1 -- start start 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.261+0000 7fd08359e700 10 monclient: init 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: monmap: 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient: monmap: 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.328 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:30.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 Processor -- start 2024-09-16T06:38:30.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd084070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd084070ea0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd08413b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084135400 con 0x7fd084070ac0 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08409a540 con 0x7fd084070ea0 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd084134f40 con 0x7fd08413b980 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd084070ac0 0x7fd08413a030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60498/0 (socket says 172.21.15.76:60498) 2024-09-16T06:38:30.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 -- start start 2024-09-16T06:38:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08259c700 1 -- 172.21.15.76:0/1863888218 learned_addr learned my addr 172.21.15.76:0/1863888218 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: init 2024-09-16T06:38:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient(hunting): authenticate will time out at 857.400832s 2024-09-16T06:38:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 1 Processor -- start 2024-09-16T06:38:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076165706 0 0) 0x7fa6200035d0 con 0x7fa63409d180 2024-09-16T06:38:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 >> v1:172.21.15.76:6790/0 conn(0x7fa63409d560 legacy=0x7fa634136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980622112 0 0) 0x7fd084134f40 con 0x7fd08413b980 2024-09-16T06:38:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): my global_id is 4277 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 >> v1:172.21.15.189:6789/0 conn(0x7fa634138610 legacy=0x7fa63413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 monclient: found mon.noname-a 2024-09-16T06:38:30.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0640035d0 con 0x7fd08413b980 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165290656 0 0) 0x7fd084072330 con 0x7fd084070ac0 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): my global_id is 4281 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd084134f40 con 0x7fd084070ac0 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63409a020 con 0x7fa63409d180 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa63ab6e700 5 monclient: authenticate success, global_id 4278 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa63ab6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa628001610 con 0x7fa63409d180 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: dump: 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa63ab6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f4b900984d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f4b9009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f4b901388c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8dd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4b9009ed60 0x7f4b9006e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60514/0 (socket says 172.21.15.76:60514) 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8dd9b700 1 -- 172.21.15.76:0/2613802241 learned_addr learned my addr 172.21.15.76:0/2613802241 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b90070a30 con 0x7f4b9009ed60 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9009b6d0 con 0x7f4b900984d0 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b90071940 con 0x7f4b901388c0 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient(hunting): authenticate will time out at 857.401939s 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 --> v1:172.21.15.76:6804/779009499 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9780a3450 con 0x7ff978060db0 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 360835370 0 0) 0x7fd084134f40 con 0x7fd084070ac0 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd084072330 con 0x7fd084070ac0 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd074001770 con 0x7fd084070ac0 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): dump: 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.338 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3710323513 0 0) 0x7fd0640035d0 con 0x7fd08413b980 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd084134f40 con 0x7fd08413b980 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2550151375 0 0) 0x7fd084072330 con 0x7fd084070ac0 2024-09-16T06:38:30.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 >> v1:172.21.15.76:6790/0 conn(0x7fd08413b980 legacy=0x7fd0841d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 >> v1:172.21.15.189:6789/0 conn(0x7fd084070ea0 legacy=0x7fd0841cee10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 monclient: found mon.a 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd084138820 con 0x7fd084070ac0 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 1 -- start start 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient: init 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 5 monclient: authenticate success, global_id 4281 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _renew_subs 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd084138b00 con 0x7fd084070ac0 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _renew_subs 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08413a740 con 0x7fd084070ac0 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff96e7fc700 1 -- 172.21.15.76:0/113579970 <== osd.2 v1:172.21.15.76:6804/779009499 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 839772338) 0x7ff970001b00 con 0x7ff978060db0 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd074001ce0 con 0x7fd084070ac0 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd074004a30 con 0x7fd084070ac0 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485634784 0 0) 0x7f4b90070a30 con 0x7f4b9009ed60 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): my global_id is 4284 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78003a20 con 0x7f4b9009ed60 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881651719 0 0) 0x7f4b9009b6d0 con 0x7f4b900984d0 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): my global_id is 4279 2024-09-16T06:38:30.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b90070a30 con 0x7f4b900984d0 2024-09-16T06:38:30.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912222205 0 0) 0x7f4b90071940 con 0x7f4b901388c0 2024-09-16T06:38:30.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd073fff700 4 set_mon_vals no callback set 2024-09-16T06:38:30.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): my global_id is 4280 2024-09-16T06:38:30.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-c con 0x7f5e5809ad50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-a con 0x7f5e5809b130 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5809d590 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5813f3d0 con 0x7f5e5809b130 2024-09-16T06:38:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5f247700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e580705c0 con 0x7f5e5809d590 2024-09-16T06:38:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b9009b6d0 con 0x7f4b901388c0 2024-09-16T06:38:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 >> v1:172.21.15.76:6804/779009499 conn(0x7ff978060db0 legacy=0x7ff978061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff9640365a0 legacy=0x7ff964038a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 10 monclient: shutdown 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 >> v1:172.21.15.76:6789/0 conn(0x7ff97806d480 legacy=0x7ff9781c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5cfe3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e5809ad50 0x7f5e581c82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52884/0 (socket says 172.21.15.76:52884) 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f5e5cfe3700 1 -- 172.21.15.76:0/1027855162 learned_addr learned my addr 172.21.15.76:0/1027855162 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 10 monclient: dump: 2024-09-16T06:38:30.348 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.349 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97e470700 1 -- 172.21.15.76:0/113579970 reap_dead start 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 shutdown_connections 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 >> 172.21.15.76:0/113579970 conn(0x7ff9780a41c0 msgr2=0x7ff97809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 shutdown_connections 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153658458 0 0) 0x7f4b90070a30 con 0x7f4b900984d0 2024-09-16T06:38:30.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7ff97ec71700 1 -- 172.21.15.76:0/113579970 wait complete. 2024-09-16T06:38:30.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b90071940 con 0x7f4b900984d0 2024-09-16T06:38:30.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd074004b90 con 0x7fd084070ac0 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4b84001430 con 0x7f4b900984d0 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): dump: 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.352 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2000035296 0 0) 0x7f4b9009b6d0 con 0x7f4b901388c0 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b90070a30 con 0x7f4b901388c0 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4b88001b30 con 0x7f4b901388c0 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient(hunting): dump: 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 841021852 0 0) 0x7f4b90070a30 con 0x7f4b901388c0 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 >> v1:172.21.15.189:6789/0 conn(0x7f4b900984d0 legacy=0x7f4b9006d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 >> v1:172.21.15.76:6789/0 conn(0x7f4b9009ed60 legacy=0x7f4b9006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 monclient: found mon.c 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b900707c0 con 0x7f4b901388c0 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fd074042fb0 con 0x7fd084070ac0 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5813a0f0 con 0x7f5e5809ad50 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f5e5f247700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f5e5f247700 10 monclient(hunting): authenticate will time out at 857.404110s 2024-09-16T06:38:30.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd08359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd068000f10 con 0x7fd084070ac0 2024-09-16T06:38:30.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd081d9b700 10 monclient: get_auth_request con 0x7fd064036370 auth_method 0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 5 monclient: authenticate success, global_id 4280 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: _renew_subs 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b900a2fc0 con 0x7f4b901388c0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b8f59e700 10 monclient: _renew_subs 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b88002000 con 0x7f4b901388c0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b8f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b8f59e700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9006e9b0 con 0x7f4b901388c0 2024-09-16T06:38:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4b88003be0 con 0x7f4b901388c0 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7effd700 10 monclient: dump: 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7ffff700 4 set_mon_vals no callback set 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa628001830 con 0x7fa63409d180 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 1 -- 172.21.15.76:0/1403685028 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa628001c00 con 0x7fa63409d180 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa638b6a700 10 monclient: dump: 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa63ab6e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.265+0000 7fa63ab6e700 4 set_mon_vals no callback set 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: shutdown 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- 172.21.15.76:0/1403685028 >> v1:172.21.15.76:6789/0 conn(0x7fa63409d180 legacy=0x7fa63412eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- 172.21.15.76:0/1403685028 shutdown_connections 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- 172.21.15.76:0/1403685028 >> 172.21.15.76:0/1403685028 conn(0x7fa6340a0030 msgr2=0x7fa6340a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- 172.21.15.76:0/1403685028 shutdown_connections 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b900a15d0 con 0x7f4b901388c0 2024-09-16T06:38:30.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- 172.21.15.76:0/1403685028 wait complete. 2024-09-16T06:38:30.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.189:6808/3809009238 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd064040b70 con 0x7fd06403cec0 2024-09-16T06:38:30.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7fd074005390 con 0x7fd084070ac0 2024-09-16T06:38:30.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd072ffd700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T06:38:30.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: monmap: 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.363 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fd082d9d700 10 monclient: get_auth_request con 0x7fd06403cec0 auth_method 0 2024-09-16T06:38:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 Processor -- start 2024-09-16T06:38:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- start start 2024-09-16T06:38:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: init 2024-09-16T06:38:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-c con 0x7fa63409d180 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-b con 0x7fa63409d560 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient(hunting): picked mon.noname-a con 0x7fa634138610 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63412b140 con 0x7fa634138610 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63412c0a0 con 0x7fa63409d560 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6341381b0 con 0x7fa63409d180 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 10 monclient(hunting): authenticate will time out at 857.406902s 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa639b6c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa63409d180 0x7fa634129530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52900/0 (socket says 172.21.15.76:52900) 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa639b6c700 1 -- 172.21.15.76:0/3493699028 learned_addr learned my addr 172.21.15.76:0/3493699028 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465544079 0 0) 0x7fa6341381b0 con 0x7fa63409d180 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): my global_id is 4286 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6180035d0 con 0x7fa63409d180 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37142924 0 0) 0x7fa63412b140 con 0x7fa634138610 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): my global_id is 4290 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6341381b0 con 0x7fa634138610 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724589187 0 0) 0x7fa63412c0a0 con 0x7fa63409d560 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): my global_id is 4285 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa63412b140 con 0x7fa63409d560 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293475999 0 0) 0x7fa6341381b0 con 0x7fa634138610 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa63412c0a0 con 0x7fa634138610 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa630001830 con 0x7fa634138610 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923104045 0 0) 0x7fa6180035d0 con 0x7fa63409d180 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6341381b0 con 0x7fa63409d180 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa628001b40 con 0x7fa63409d180 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4233017631 0 0) 0x7fa63412c0a0 con 0x7fa634138610 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 >> v1:172.21.15.76:6790/0 conn(0x7fa63409d180 legacy=0x7fa634129530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 >> v1:172.21.15.189:6789/0 conn(0x7fa63409d560 legacy=0x7fa6341cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 monclient: found mon.a 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63409a540 con 0x7fa634138610 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa6267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.269+0000 7fa63ab6e700 5 monclient: authenticate success, global_id 4290 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 10 monclient: _renew_subs 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63412abf0 con 0x7fa634138610 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 10 monclient: _renew_subs 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6341370b0 con 0x7fa634138610 2024-09-16T06:38:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== osd.3 v1:172.21.15.189:6808/3809009238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fd064040b70 con 0x7fd06403cec0 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6300036c0 con 0x7fa634138610 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6300039d0 con 0x7fa634138610 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 10 monclient: dump: 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa627fff700 4 set_mon_vals no callback set 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa61c000f10 con 0x7fa634138610 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa63000e440 con 0x7fa634138610 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fa630042030 con 0x7fa634138610 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6808/3838562525 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa618040db0 con 0x7fa61803d100 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa639b6c700 10 monclient: get_auth_request con 0x7fa618036510 auth_method 0 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7fa630013410 con 0x7fa634138610 2024-09-16T06:38:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa63936b700 10 monclient: get_auth_request con 0x7fa61803d100 auth_method 0 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== osd.0 v1:172.21.15.76:6808/3838562525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fa618040db0 con 0x7fa61803d100 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391493943 0 0) 0x7f5e5813f3d0 con 0x7f5e5809b130 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 --> v1:172.21.15.189:6800/2128366135 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc96c006fd0 con 0x7fc96c0015c0 2024-09-16T06:38:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): my global_id is 4287 2024-09-16T06:38:30.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e440035d0 con 0x7f5e5809b130 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592475523 0 0) 0x7f5e580705c0 con 0x7f5e5809d590 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): my global_id is 4282 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5813f3d0 con 0x7f5e5809d590 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129090441 0 0) 0x7f5e5813a0f0 con 0x7f5e5809ad50 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 10 monclient(hunting): my global_id is 4283 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.273+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e580705c0 con 0x7f5e5809ad50 2024-09-16T06:38:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343355657 0 0) 0x7f5e580705c0 con 0x7f5e5809ad50 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e5813a0f0 con 0x7f5e5809ad50 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e48001500 con 0x7f5e5809ad50 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): dump: 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.380 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721126848 0 0) 0x7f5e440035d0 con 0x7f5e5809b130 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e580705c0 con 0x7f5e5809b130 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e4c0017a0 con 0x7f5e5809b130 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient(hunting): dump: 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.381 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 967437438 0 0) 0x7f5e5813f3d0 con 0x7f5e5809d590 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e440035d0 con 0x7f5e5809d590 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111410673 0 0) 0x7f5e5813a0f0 con 0x7f5e5809ad50 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 >> v1:172.21.15.189:6789/0 conn(0x7f5e5809d590 legacy=0x7f5e581d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 >> v1:172.21.15.76:6789/0 conn(0x7f5e5809b130 legacy=0x7f5e581cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 monclient: found mon.c 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5813a3d0 con 0x7f5e5809ad50 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 5 monclient: authenticate success, global_id 4283 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 10 monclient: _renew_subs 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e580a2d90 con 0x7f5e5809ad50 2024-09-16T06:38:30.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7fc9737fe700 1 -- 172.21.15.76:0/759686145 <== osd.5 v1:172.21.15.189:6800/2128366135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7fc96c006fd0 con 0x7fc96c0015c0 2024-09-16T06:38:30.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 10 monclient: _renew_subs 2024-09-16T06:38:30.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5813a6b0 con 0x7f5e5809ad50 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e48003990 con 0x7f5e5809ad50 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e48003ca0 con 0x7f5e5809ad50 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: dump: 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e56ffd700 4 set_mon_vals no callback set 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.76:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e38000f10 con 0x7f5e5809ad50 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5e48003e00 con 0x7f5e5809ad50 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f5e480431a0 con 0x7f5e5809ad50 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.189:6804/407912501 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e44040f80 con 0x7f5e4403d2d0 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f5e480435d0 con 0x7f5e5809ad50 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e557fa700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e57fff700 10 monclient: get_auth_request con 0x7f5e44036700 auth_method 0 2024-09-16T06:38:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f5e5d7e4700 10 monclient: get_auth_request con 0x7f5e4403d2d0 auth_method 0 2024-09-16T06:38:30.387 INFO:teuthology.orchestra.run.smithi076.stdout:51539607555 2024-09-16T06:38:30.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T06:38:30.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4b8800e650 con 0x7f4b901388c0 2024-09-16T06:38:30.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f4b88041f90 con 0x7f4b901388c0 2024-09-16T06:38:30.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f4b8e59c700 10 monclient: get_auth_request con 0x7f4b78036770 auth_method 0 2024-09-16T06:38:30.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.277+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6800/3502405955 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b78041010 con 0x7f4b7803d360 2024-09-16T06:38:30.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7f4b8dd9b700 10 monclient: get_auth_request con 0x7f4b7803d360 auth_method 0 2024-09-16T06:38:30.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f4b88013330 con 0x7f4b901388c0 2024-09-16T06:38:30.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7f4b7effd700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T06:38:30.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== osd.4 v1:172.21.15.189:6804/407912501 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f5e44040f80 con 0x7f5e4403d2d0 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 >> v1:172.21.15.189:6800/2128366135 conn(0x7fc96c0015c0 legacy=0x7fc96c003a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc9780362d0 legacy=0x7fc978038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 10 monclient: shutdown 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 >> v1:172.21.15.76:6789/0 conn(0x7fc98c1320e0 legacy=0x7fc98c12e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98b7fe700 1 -- 172.21.15.76:0/759686145 reap_dead start 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 shutdown_connections 2024-09-16T06:38:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 >> 172.21.15.76:0/759686145 conn(0x7fc98c0a0030 msgr2=0x7fc98c0a1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 shutdown_connections 2024-09-16T06:38:30.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7fc98bfff700 1 -- 172.21.15.76:0/759686145 wait complete. 2024-09-16T06:38:30.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.281+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== osd.1 v1:172.21.15.76:6800/3502405955 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f4b78041010 con 0x7f4b7803d360 2024-09-16T06:38:30.391 INFO:teuthology.orchestra.run.smithi076.stdout:51539607557 2024-09-16T06:38:30.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T06:38:30.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 --> v1:172.21.15.189:6804/407912501 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e38002bd0 con 0x7f5e4403d2d0 2024-09-16T06:38:30.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e557fa700 1 -- 172.21.15.76:0/1027855162 <== osd.4 v1:172.21.15.189:6804/407912501 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3371909001) 0x7f5e38002bd0 con 0x7f5e4403d2d0 2024-09-16T06:38:30.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 --> v1:172.21.15.76:6800/3502405955 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b9009ba90 con 0x7f4b7803d360 2024-09-16T06:38:30.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 >> v1:172.21.15.189:6804/407912501 conn(0x7f5e4403d2d0 legacy=0x7f5e4403f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7effd700 1 -- 172.21.15.76:0/2613802241 <== osd.1 v1:172.21.15.76:6800/3502405955 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3543570501) 0x7f4b9009ba90 con 0x7f4b7803d360 2024-09-16T06:38:30.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5e44036700 legacy=0x7f5e44038bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 10 monclient: shutdown 2024-09-16T06:38:30.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 >> v1:172.21.15.76:6790/0 conn(0x7f5e5809ad50 legacy=0x7f5e581c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5d7e4700 1 -- 172.21.15.76:0/1027855162 reap_dead start 2024-09-16T06:38:30.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 shutdown_connections 2024-09-16T06:38:30.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 >> 172.21.15.76:0/1027855162 conn(0x7f5e580a3e50 msgr2=0x7f5e580a1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 shutdown_connections 2024-09-16T06:38:30.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f5e5f247700 1 -- 172.21.15.76:0/1027855162 wait complete. 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 >> v1:172.21.15.76:6800/3502405955 conn(0x7f4b7803d360 legacy=0x7f4b7803f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4b78036770 legacy=0x7f4b78038c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 10 monclient: shutdown 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 >> v1:172.21.15.76:6790/0 conn(0x7f4b901388c0 legacy=0x7f4b901d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b8ed9d700 1 -- 172.21.15.76:0/2613802241 reap_dead start 2024-09-16T06:38:30.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 --> v1:172.21.15.189:6808/3809009238 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd068002bd0 con 0x7fd06403cec0 2024-09-16T06:38:30.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 shutdown_connections 2024-09-16T06:38:30.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 >> 172.21.15.76:0/2613802241 conn(0x7f4b900a0030 msgr2=0x7f4b900972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 shutdown_connections 2024-09-16T06:38:30.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7f4b7cff9700 1 -- 172.21.15.76:0/2613802241 wait complete. 2024-09-16T06:38:30.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.297+0000 7fd072ffd700 1 -- 172.21.15.76:0/1863888218 <== osd.3 v1:172.21.15.189:6808/3809009238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7fd068002bd0 con 0x7fd06403cec0 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 >> v1:172.21.15.189:6808/3809009238 conn(0x7fd06403cec0 legacy=0x7fd06403f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd064036370 legacy=0x7fd064038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 10 monclient: shutdown 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 >> v1:172.21.15.76:6789/0 conn(0x7fd084070ac0 legacy=0x7fd08413a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd082d9d700 1 -- 172.21.15.76:0/1863888218 reap_dead start 2024-09-16T06:38:30.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 shutdown_connections 2024-09-16T06:38:30.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 >> 172.21.15.76:0/1863888218 conn(0x7fd0840a0030 msgr2=0x7fd08413ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 shutdown_connections 2024-09-16T06:38:30.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fd08359e700 1 -- 172.21.15.76:0/1863888218 wait complete. 2024-09-16T06:38:30.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 --> v1:172.21.15.76:6808/3838562525 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa61c002bd0 con 0x7fa61803d100 2024-09-16T06:38:30.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa6267fc700 1 -- 172.21.15.76:0/3493699028 <== osd.0 v1:172.21.15.76:6808/3838562525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 982563978) 0x7fa61c002bd0 con 0x7fa61803d100 2024-09-16T06:38:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 >> v1:172.21.15.76:6808/3838562525 conn(0x7fa61803d100 legacy=0x7fa61803f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa618036510 legacy=0x7fa6180389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 10 monclient: shutdown 2024-09-16T06:38:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 >> v1:172.21.15.76:6789/0 conn(0x7fa634138610 legacy=0x7fa6341d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63a36d700 1 -- 172.21.15.76:0/3493699028 reap_dead start 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 shutdown_connections 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 >> 172.21.15.76:0/3493699028 conn(0x7fa6340a0030 msgr2=0x7fa63412dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 shutdown_connections 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.301+0000 7fa63ab6e700 1 -- 172.21.15.76:0/3493699028 wait complete. 2024-09-16T06:38:30.401 INFO:teuthology.orchestra.run.smithi076.stdout:55834574850 2024-09-16T06:38:30.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T06:38:30.402 INFO:teuthology.orchestra.run.smithi076.stdout:51539607557 2024-09-16T06:38:30.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T06:38:30.403 INFO:teuthology.orchestra.run.smithi076.stdout:47244640259 2024-09-16T06:38:30.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T06:38:30.404 INFO:teuthology.orchestra.run.smithi076.stdout:55834574851 2024-09-16T06:38:30.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: monmap: 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.538300+0000 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.538300+0000 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: monmap: 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.538401+0000 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.538401+0000 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 1 Processor -- start 2024-09-16T06:38:30.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 1 Processor -- start 2024-09-16T06:38:30.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 1 -- start start 2024-09-16T06:38:30.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: init 2024-09-16T06:38:30.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 1 -- start start 2024-09-16T06:38:30.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: init 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-a con 0x7fb07c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-b con 0x7fb07c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-c con 0x7fb07c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c06c280 con 0x7fb07c132330 2024-09-16T06:38:30.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c06bfa0 con 0x7fb07c06cf50 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c09ace0 con 0x7fb07c06d480 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7fb0811b7700 10 monclient(hunting): authenticate will time out at 857.671986s 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.533+0000 7f888a54d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-a con 0x7f888406a910 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888954b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f888406a910 0x7f888406acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60560/0 (socket says 172.21.15.76:60560) 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888954b700 1 -- 172.21.15.76:0/1436830656 learned_addr learned my addr 172.21.15.76:0/1436830656 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb07c132330 0x7fb07c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60556/0 (socket says 172.21.15.76:60556) 2024-09-16T06:38:30.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07bfff700 1 -- 172.21.15.76:0/440134894 learned_addr learned my addr 172.21.15.76:0/440134894 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-c con 0x7f888406b400 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-b con 0x7f888409c350 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840a1e10 con 0x7f888406a910 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888409e8d0 con 0x7f888406b400 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8884072090 con 0x7f888409c350 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient(hunting): authenticate will time out at 857.672751s 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663366246 0 0) 0x7f88840a1e10 con 0x7f888406a910 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): my global_id is 4296 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c0035d0 con 0x7f888406a910 2024-09-16T06:38:30.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669352566 0 0) 0x7f8884072090 con 0x7f888409c350 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient(hunting): my global_id is 4292 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88840a1e10 con 0x7f888409c350 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928670712 0 0) 0x7f886c0035d0 con 0x7f888406a910 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398992082 0 0) 0x7fb07c09ace0 con 0x7fb07c06d480 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): my global_id is 4289 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0640035d0 con 0x7fb07c06d480 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411175945 0 0) 0x7fb07c06c280 con 0x7fb07c132330 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): my global_id is 4293 2024-09-16T06:38:30.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c09ace0 con 0x7fb07c132330 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150772350 0 0) 0x7fb07c06bfa0 con 0x7fb07c06cf50 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient(hunting): my global_id is 4288 2024-09-16T06:38:30.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c06c280 con 0x7fb07c06cf50 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 >> v1:172.21.15.76:6790/0 conn(0x7f888409c350 legacy=0x7f888413dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 >> v1:172.21.15.189:6789/0 conn(0x7f888406b400 legacy=0x7f888413a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 monclient: found mon.noname-a 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888409c7e0 con 0x7f888406a910 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8878001610 con 0x7f888406a910 2024-09-16T06:38:30.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: dump: 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 5 monclient: authenticate success, global_id 4296 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8878001830 con 0x7f888406a910 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 1 -- 172.21.15.76:0/1436830656 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8878001bc0 con 0x7f888406a910 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f8883fff700 10 monclient: dump: 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.549 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 4 set_mon_vals no callback set 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173915034 0 0) 0x7fb07c06c280 con 0x7fb07c06cf50 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 >> v1:172.21.15.76:6790/0 conn(0x7fb07c06d480 legacy=0x7fb07c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 >> v1:172.21.15.76:6789/0 conn(0x7fb07c132330 legacy=0x7fb07c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 monclient: found mon.noname-c 2024-09-16T06:38:30.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c09ae70 con 0x7fb07c06cf50 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb074001970 con 0x7fb07c06cf50 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: dump: 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.551 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 5 monclient: authenticate success, global_id 4288 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb074001b90 con 0x7fb07c06cf50 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 1 -- 172.21.15.76:0/440134894 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb074001dc0 con 0x7fb07c06cf50 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb07affd700 10 monclient: dump: 2024-09-16T06:38:30.552 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 4 set_mon_vals no callback set 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 10 monclient: shutdown 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 >> v1:172.21.15.76:6789/0 conn(0x7f888406a910 legacy=0x7f888406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 10 monclient: shutdown 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.537+0000 7fb0811b7700 1 -- 172.21.15.76:0/440134894 >> v1:172.21.15.189:6789/0 conn(0x7fb07c06cf50 legacy=0x7fb07c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 shutdown_connections 2024-09-16T06:38:30.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- 172.21.15.76:0/440134894 shutdown_connections 2024-09-16T06:38:30.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- 172.21.15.76:0/440134894 >> 172.21.15.76:0/440134894 conn(0x7fb07c0a41c0 msgr2=0x7fb07c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 >> 172.21.15.76:0/1436830656 conn(0x7f88840a12f0 msgr2=0x7f88840a1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 shutdown_connections 2024-09-16T06:38:30.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- 172.21.15.76:0/440134894 shutdown_connections 2024-09-16T06:38:30.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- 172.21.15.76:0/1436830656 wait complete. 2024-09-16T06:38:30.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- 172.21.15.76:0/440134894 wait complete. 2024-09-16T06:38:30.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient: monmap: 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient: monmap: 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.558 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 Processor -- start 2024-09-16T06:38:30.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- start start 2024-09-16T06:38:30.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient: init 2024-09-16T06:38:30.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 Processor -- start 2024-09-16T06:38:30.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- start start 2024-09-16T06:38:30.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient: init 2024-09-16T06:38:30.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:30.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-a con 0x7f888406a910 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-b con 0x7f888406b400 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient(hunting): picked mon.noname-c con 0x7f888409c350 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888413cb10 con 0x7f888406a910 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888413d770 con 0x7f888406b400 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840a3400 con 0x7f888409c350 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888a54d700 10 monclient(hunting): authenticate will time out at 857.677851s 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888954b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f888406a910 0x7f888413bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60570/0 (socket says 172.21.15.76:60570) 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8889d4c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f888409c350 0x7f88841d2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52940/0 (socket says 172.21.15.76:52940) 2024-09-16T06:38:30.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f888954b700 1 -- 172.21.15.76:0/403955521 learned_addr learned my addr 172.21.15.76:0/403955521 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-b con 0x7fb07c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-a con 0x7fb07c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient(hunting): picked mon.noname-c con 0x7fb07c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c12e090 con 0x7fb07c06d480 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c12cf20 con 0x7fb07c06cf50 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c12cce0 con 0x7fb07c132330 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0811b7700 10 monclient(hunting): authenticate will time out at 857.678308s 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0809b6700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb07c132330 0x7fb07c1d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52944/0 (socket says 172.21.15.76:52944) 2024-09-16T06:38:30.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb0809b6700 1 -- 172.21.15.76:0/3413852396 learned_addr learned my addr 172.21.15.76:0/3413852396 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456644347 0 0) 0x7f88840a3400 con 0x7f888409c350 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): my global_id is 4295 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c0035d0 con 0x7f888409c350 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737323136 0 0) 0x7f888413d770 con 0x7f888406b400 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): my global_id is 4294 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88840a3400 con 0x7f888406b400 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712269941 0 0) 0x7fb07c12cce0 con 0x7fb07c132330 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): my global_id is 4298 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb064004840 con 0x7fb07c132330 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370208752 0 0) 0x7fb07c12cf20 con 0x7fb07c06cf50 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): my global_id is 4297 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c12cce0 con 0x7fb07c06cf50 2024-09-16T06:38:30.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2807459306 0 0) 0x7f886c0035d0 con 0x7f888409c350 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f888413d770 con 0x7f888409c350 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561984777 0 0) 0x7f888413cb10 con 0x7f888406a910 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): my global_id is 4299 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c0035d0 con 0x7f888406a910 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8874001830 con 0x7f888409c350 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.566 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418304105 0 0) 0x7f88840a3400 con 0x7f888406b400 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f888413cb10 con 0x7f888406b400 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f887c001b20 con 0x7f888406b400 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.568 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058638651 0 0) 0x7fb07c12e090 con 0x7fb07c06d480 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): my global_id is 4302 2024-09-16T06:38:30.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c12cf20 con 0x7fb07c06d480 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1421936724 0 0) 0x7fb064004840 con 0x7fb07c132330 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb07c12e090 con 0x7fb07c132330 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb074001d00 con 0x7fb07c132330 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2744587217 0 0) 0x7f886c0035d0 con 0x7f888406a910 2024-09-16T06:38:30.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88840a3400 con 0x7f888406a910 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.541+0000 7fb078ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.571 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941322706 0 0) 0x7fb07c12cce0 con 0x7fb07c06cf50 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb064004840 con 0x7fb07c06cf50 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb06c001760 con 0x7fb07c06cf50 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929504772 0 0) 0x7fb07c12cf20 con 0x7fb07c06d480 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb07c12cce0 con 0x7fb07c06d480 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb0700018d0 con 0x7fb07c06d480 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient(hunting): dump: 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2323475137 0 0) 0x7fb07c12e090 con 0x7fb07c132330 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 >> v1:172.21.15.189:6789/0 conn(0x7fb07c06cf50 legacy=0x7fb07c1308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 >> v1:172.21.15.76:6789/0 conn(0x7fb07c06d480 legacy=0x7fb07c1ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8878001c00 con 0x7f888406a910 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2790787819 0 0) 0x7f888413cb10 con 0x7f888406b400 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 >> v1:172.21.15.76:6790/0 conn(0x7f888409c350 legacy=0x7f88841d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 >> v1:172.21.15.76:6789/0 conn(0x7f888406a910 legacy=0x7f888413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 monclient: found mon.c 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c1ca340 con 0x7fb07c132330 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb078ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 5 monclient: authenticate success, global_id 4298 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _renew_subs 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c12dad0 con 0x7fb07c132330 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _renew_subs 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c1ca060 con 0x7fb07c132330 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 monclient: found mon.b 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888409b5c0 con 0x7f888406b400 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 5 monclient: authenticate success, global_id 4294 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 10 monclient: _renew_subs 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888412a500 con 0x7f888406b400 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 10 monclient: _renew_subs 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f888413dec0 con 0x7f888406b400 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f887c001c80 con 0x7f888406b400 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f887c003be0 con 0x7f888406b400 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 10 monclient: dump: 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.580 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c12dcc0 con 0x7fb07c132330 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f887c003d40 con 0x7f888406b400 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f887c042250 con 0x7f888406b400 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.545+0000 7f888954b700 10 monclient: get_auth_request con 0x7f886c03ef00 auth_method 0 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7f8882ffd700 4 set_mon_vals no callback set 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7f888a54d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7f888a54d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7f888a54d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8860005100 con 0x7f888406b400 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb074003cd0 con 0x7fb07c132330 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb074003fe0 con 0x7fb07c132330 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 10 monclient: dump: 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.583 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb07a7fc700 4 set_mon_vals no callback set 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb074004140 con 0x7fb07c132330 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fb0740433d0 con 0x7fb07c132330 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.549+0000 7fb07bfff700 10 monclient: get_auth_request con 0x7fb064036960 auth_method 0 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.553+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb074014810 con 0x7fb07c132330 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.553+0000 7fb078ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.553+0000 7fb078ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.561+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f887c0045b0 con 0x7f888406b400 2024-09-16T06:38:30.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.561+0000 7f8881ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.561+0000 7f8881ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: monmap: 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.574235+0000 2024-09-16T06:38:30.587 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.574235+0000 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 1 Processor -- start 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 1 -- start start 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: init 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-a con 0x7f2a90070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-b con 0x7f2a90070ea0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-c con 0x7f2a9013b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90072610 con 0x7f2a90070ac0 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9009f560 con 0x7f2a9013b980 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90072330 con 0x7f2a90070ea0 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a96a8f700 10 monclient(hunting): authenticate will time out at 857.707826s 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a9528c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2a90070ea0 0x7f2a900712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52956/0 (socket says 172.21.15.76:52956) 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.569+0000 7f2a9528c700 1 -- 172.21.15.76:0/558878893 learned_addr learned my addr 172.21.15.76:0/558878893 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526824306 0 0) 0x7f2a90072330 con 0x7f2a90070ea0 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): my global_id is 4301 2024-09-16T06:38:30.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c0035d0 con 0x7f2a90070ea0 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166417546 0 0) 0x7f2a90072610 con 0x7f2a90070ac0 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): my global_id is 4305 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a90072330 con 0x7f2a90070ac0 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895680390 0 0) 0x7f2a9009f560 con 0x7f2a9013b980 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient(hunting): my global_id is 4300 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a90072610 con 0x7f2a9013b980 2024-09-16T06:38:30.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507280410 0 0) 0x7f2a7c0035d0 con 0x7f2a90070ea0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 >> v1:172.21.15.189:6789/0 conn(0x7f2a9013b980 legacy=0x7f2a9013dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 >> v1:172.21.15.76:6789/0 conn(0x7f2a90070ac0 legacy=0x7f2a90097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a900974f0 con 0x7f2a90070ea0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a8c001610 con 0x7f2a90070ea0 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: dump: 2024-09-16T06:38:30.591 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 5 monclient: authenticate success, global_id 4301 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a8c001830 con 0x7f2a90070ea0 2024-09-16T06:38:30.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 1 -- 172.21.15.76:0/558878893 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a8c001c00 con 0x7f2a90070ea0 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 4 set_mon_vals no callback set 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a94a8b700 10 monclient: dump: 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.593 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: shutdown 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- 172.21.15.76:0/558878893 >> v1:172.21.15.76:6790/0 conn(0x7f2a90070ea0 legacy=0x7f2a900712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- 172.21.15.76:0/558878893 shutdown_connections 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- 172.21.15.76:0/558878893 >> 172.21.15.76:0/558878893 conn(0x7f2a900a0030 msgr2=0x7f2a900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- 172.21.15.76:0/558878893 shutdown_connections 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- 172.21.15.76:0/558878893 wait complete. 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: monmap: 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.594 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 Processor -- start 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- start start 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: init 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-a con 0x7f2a90070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-c con 0x7f2a90070ea0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient(hunting): picked mon.noname-b con 0x7f2a9013b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9013a620 con 0x7f2a90070ac0 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90137770 con 0x7f2a9013b980 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90135370 con 0x7f2a90070ea0 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a96a8f700 10 monclient(hunting): authenticate will time out at 857.711622s 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a95a8d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2a90070ac0 0x7f2a90136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60604/0 (socket says 172.21.15.76:60604) 2024-09-16T06:38:30.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.573+0000 7f2a95a8d700 1 -- 172.21.15.76:0/3081196643 learned_addr learned my addr 172.21.15.76:0/3081196643 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: monmap: 2024-09-16T06:38:30.597 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.581970+0000 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.581970+0000 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 1 Processor -- start 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 1 -- start start 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: init 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-b con 0x7f112009d180 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-c con 0x7f112009d560 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-a con 0x7f1120138610 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112012f240 con 0x7f1120138610 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1120099d40 con 0x7f112009d560 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112009a300 con 0x7f112009d180 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1124a56700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f112009d180 0x7f112012eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52964/0 (socket says 172.21.15.76:52964) 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1124a56700 1 -- 172.21.15.76:0/2932467016 learned_addr learned my addr 172.21.15.76:0/2932467016 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f88a4c82700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f88a4c82700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f88a4c82700 10 monclient: monmap: 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.582984+0000 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.582984+0000 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f1125a58700 10 monclient(hunting): authenticate will time out at 857.715550s 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298071748 0 0) 0x7f112009a300 con 0x7f112009d180 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 10 monclient(hunting): my global_id is 4307 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11100035d0 con 0x7f112009d180 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425607295 0 0) 0x7f112012f240 con 0x7f1120138610 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient(hunting): my global_id is 4311 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112009a300 con 0x7f1120138610 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 1 Processor -- start 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 1 -- start start 2024-09-16T06:38:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient: init 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185762698 0 0) 0x7f2a90135370 con 0x7f2a90070ea0 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): my global_id is 4304 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a740035d0 con 0x7f2a90070ea0 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710986253 0 0) 0x7f2a90137770 con 0x7f2a9013b980 2024-09-16T06:38:30.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): my global_id is 4303 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a90135370 con 0x7f2a9013b980 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048594838 0 0) 0x7f2a9013a620 con 0x7f2a90070ac0 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.577+0000 7f2a867fc700 10 monclient(hunting): my global_id is 4308 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a90137770 con 0x7f2a90070ac0 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384632091 0 0) 0x7f2a90137770 con 0x7f2a90070ac0 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a9013a620 con 0x7f2a90070ac0 2024-09-16T06:38:30.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a88001ce0 con 0x7f2a90070ac0 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604921885 0 0) 0x7f2a740035d0 con 0x7f2a90070ea0 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a90137770 con 0x7f2a90070ea0 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310464961 0 0) 0x7f2a90135370 con 0x7f2a9013b980 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a740035d0 con 0x7f2a9013b980 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a8c001b40 con 0x7f2a90070ea0 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.606 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a80001830 con 0x7f2a9013b980 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient(hunting): dump: 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.607 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 399669232 0 0) 0x7f2a9013a620 con 0x7f2a90070ac0 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351251460 0 0) 0x7f11100035d0 con 0x7f112009d180 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 >> v1:172.21.15.189:6789/0 conn(0x7f112009d560 legacy=0x7f1120136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 >> v1:172.21.15.76:6789/0 conn(0x7f1120138610 legacy=0x7f112013aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112009a020 con 0x7f112009d180 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1114001610 con 0x7f112009d180 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: dump: 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 5 monclient: authenticate success, global_id 4307 2024-09-16T06:38:30.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 >> v1:172.21.15.76:6790/0 conn(0x7f2a90070ea0 legacy=0x7f2a90137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 >> v1:172.21.15.189:6789/0 conn(0x7f2a9013b980 legacy=0x7f2a901d2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 monclient: found mon.a 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90134fb0 con 0x7f2a90070ac0 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 5 monclient: authenticate success, global_id 4308 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 10 monclient: _renew_subs 2024-09-16T06:38:30.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9013a340 con 0x7f2a90070ac0 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1114001830 con 0x7f112009d180 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 1 -- 172.21.15.76:0/2932467016 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1114001c00 con 0x7f112009d180 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f111f7fe700 10 monclient: dump: 2024-09-16T06:38:30.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 10 monclient: _renew_subs 2024-09-16T06:38:30.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a90133b60 con 0x7f2a90070ac0 2024-09-16T06:38:30.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 4 set_mon_vals no callback set 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-c con 0x7f88a01315f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-a con 0x7f88a01320e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-b con 0x7f88a00983d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00996a0 con 0x7f88a01320e0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a0098820 con 0x7f88a01315f0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a009fc30 con 0x7f88a00983d0 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f88a4c82700 10 monclient(hunting): authenticate will time out at 857.717565s 2024-09-16T06:38:30.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f889f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88a01315f0 0x7f88a01319d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:52944/0 (socket says 172.21.15.76:52944) 2024-09-16T06:38:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 10 monclient: shutdown 2024-09-16T06:38:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- 172.21.15.76:0/2932467016 >> v1:172.21.15.76:6790/0 conn(0x7f112009d180 legacy=0x7f112012eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f889f7fe700 1 -- 172.21.15.76:0/2638865526 learned_addr learned my addr 172.21.15.76:0/2638865526 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- 172.21.15.76:0/2932467016 shutdown_connections 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- 172.21.15.76:0/2932467016 >> 172.21.15.76:0/2932467016 conn(0x7f11200a0030 msgr2=0x7f11200a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- 172.21.15.76:0/2932467016 shutdown_connections 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- 172.21.15.76:0/2932467016 wait complete. 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a88003a20 con 0x7f2a90070ac0 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a88003db0 con 0x7f2a90070ac0 2024-09-16T06:38:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 10 monclient: dump: 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 10 monclient: monmap: 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.619 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2a88003f90 con 0x7f2a90070ac0 2024-09-16T06:38:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f2a88043520 con 0x7f2a90070ac0 2024-09-16T06:38:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 Processor -- start 2024-09-16T06:38:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 1 -- start start 2024-09-16T06:38:30.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f1125a58700 10 monclient: init 2024-09-16T06:38:30.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.581+0000 7f2a9528c700 10 monclient: get_auth_request con 0x7f2a740366e0 auth_method 0 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73827625 0 0) 0x7f88a009fc30 con 0x7f88a00983d0 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): my global_id is 4310 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c0035d0 con 0x7f88a00983d0 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729313927 0 0) 0x7f88a0098820 con 0x7f88a01315f0 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): my global_id is 4306 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a009fc30 con 0x7f88a01315f0 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392056957 0 0) 0x7f88a00996a0 con 0x7f88a01320e0 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient(hunting): my global_id is 4314 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0098820 con 0x7f88a01320e0 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644460013 0 0) 0x7f888c0035d0 con 0x7f88a00983d0 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 >> v1:172.21.15.189:6789/0 conn(0x7f88a01315f0 legacy=0x7f88a01319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 >> v1:172.21.15.76:6789/0 conn(0x7f88a01320e0 legacy=0x7f88a01368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00990e0 con 0x7f88a00983d0 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 5 monclient: authenticate success, global_id 4310 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8898001990 con 0x7f88a00983d0 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: dump: 2024-09-16T06:38:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-c con 0x7f112009d180 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-b con 0x7f112009d560 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient(hunting): picked mon.noname-a con 0x7f1120138610 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112012ac30 con 0x7f1120138610 2024-09-16T06:38:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112012bf50 con 0x7f112009d560 2024-09-16T06:38:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112009a540 con 0x7f112009d180 2024-09-16T06:38:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient(hunting): authenticate will time out at 857.720607s 2024-09-16T06:38:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f2a877fe700 4 set_mon_vals no callback set 2024-09-16T06:38:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 1 -- 172.21.15.76:0/2638865526 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8898001c00 con 0x7f88a00983d0 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f889e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 4 set_mon_vals no callback set 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 10 monclient: shutdown 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 1 -- 172.21.15.76:0/2638865526 >> v1:172.21.15.76:6790/0 conn(0x7f88a00983d0 legacy=0x7f88a0139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125257700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1120138610 0x7f11201d24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60628/0 (socket says 172.21.15.76:60628) 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125257700 1 -- 172.21.15.76:0/4069340725 learned_addr learned my addr 172.21.15.76:0/4069340725 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212903217 0 0) 0x7f112009a540 con 0x7f112009d180 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): my global_id is 4313 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040035d0 con 0x7f112009d180 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476320577 0 0) 0x7f112012ac30 con 0x7f1120138610 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): my global_id is 4317 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112009a540 con 0x7f1120138610 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2202132766 0 0) 0x7f112009a540 con 0x7f1120138610 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f112012ac30 con 0x7f1120138610 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11180019d0 con 0x7f1120138610 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): dump: 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732016645 0 0) 0x7f11040035d0 con 0x7f112009d180 2024-09-16T06:38:30.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f112009a540 con 0x7f112009d180 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1114001b40 con 0x7f112009d180 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): dump: 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.632 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747312095 0 0) 0x7f112012bf50 con 0x7f112009d560 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient(hunting): my global_id is 4309 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040035d0 con 0x7f112009d560 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2752494867 0 0) 0x7f112012ac30 con 0x7f1120138610 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 >> v1:172.21.15.76:6790/0 conn(0x7f112009d180 legacy=0x7f11201290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 >> v1:172.21.15.189:6789/0 conn(0x7f112009d560 legacy=0x7f11201297b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 monclient: found mon.a 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11201337e0 con 0x7f1120138610 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 5 monclient: authenticate success, global_id 4317 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _renew_subs 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1120129ec0 con 0x7f1120138610 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _renew_subs 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1120133500 con 0x7f1120138610 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1118003a60 con 0x7f1120138610 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1118003df0 con 0x7f1120138610 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 10 monclient: dump: 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.635 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 1 -- 172.21.15.76:0/2638865526 shutdown_connections 2024-09-16T06:38:30.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 1 -- 172.21.15.76:0/2638865526 >> 172.21.15.76:0/2638865526 conn(0x7f88a00a0030 msgr2=0x7f88a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1118003f50 con 0x7f1120138610 2024-09-16T06:38:30.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 1 -- 172.21.15.76:0/2638865526 shutdown_connections 2024-09-16T06:38:30.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 1 -- 172.21.15.76:0/2638865526 wait complete. 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112009f340 con 0x7f1120138610 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f1118043570 con 0x7f1120138610 2024-09-16T06:38:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f1124a56700 10 monclient: get_auth_request con 0x7f11040474d0 auth_method 0 2024-09-16T06:38:30.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f2a96a8f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f2a96a8f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f2a96a8f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a78005100 con 0x7f2a90070ac0 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f88a4c82700 10 monclient: monmap: 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.639 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.585+0000 7f111effd700 4 set_mon_vals no callback set 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 Processor -- start 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- start start 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: init 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-b con 0x7f88a00983d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-c con 0x7f88a01315f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient(hunting): picked mon.noname-a con 0x7f88a01320e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a0097e70 con 0x7f88a01320e0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00a2b80 con 0x7f88a00983d0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a009a540 con 0x7f88a01315f0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient(hunting): authenticate will time out at 857.724972s 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f889effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f88a01315f0 0x7f88a01cef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:52996/0 (socket says 172.21.15.76:52996) 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f889effd700 1 -- 172.21.15.76:0/1531925892 learned_addr learned my addr 172.21.15.76:0/1531925892 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f889ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f88a01320e0 0x7f88a01d2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60642/0 (socket says 172.21.15.76:60642) 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852351431 0 0) 0x7f88a009a540 con 0x7f88a01315f0 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): my global_id is 4316 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88800035d0 con 0x7f88a01315f0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074217367 0 0) 0x7f88a00a2b80 con 0x7f88a00983d0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): my global_id is 4312 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a009a540 con 0x7f88a00983d0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546955460 0 0) 0x7f88800035d0 con 0x7f88a01315f0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88a00a2b80 con 0x7f88a01315f0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8888001830 con 0x7f88a01315f0 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:30.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): dump: 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.645 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638153860 0 0) 0x7f88a009a540 con 0x7f88a00983d0 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88800035d0 con 0x7f88a00983d0 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88900016e0 con 0x7f88a00983d0 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient(hunting): dump: 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.646 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 915499356 0 0) 0x7f88a00a2b80 con 0x7f88a01315f0 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 >> v1:172.21.15.189:6789/0 conn(0x7f88a00983d0 legacy=0x7f88a0096760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 >> v1:172.21.15.76:6789/0 conn(0x7f88a01320e0 legacy=0x7f88a01d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 monclient: found mon.c 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a008b190 con 0x7f88a01315f0 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 5 monclient: authenticate success, global_id 4316 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _renew_subs 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a012f360 con 0x7f88a01315f0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _renew_subs 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0096e70 con 0x7f88a01315f0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88880036c0 con 0x7f88a01315f0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88880039d0 con 0x7f88a01315f0 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 10 monclient: dump: 2024-09-16T06:38:30.648 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8888003b70 con 0x7f88a01315f0 2024-09-16T06:38:30.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f889dffb700 4 set_mon_vals no callback set 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: get_monmap_and_config 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: monmap: 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.650 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:30.593020+0000 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:30.593020+0000 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 1 Processor -- start 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 1 -- start start 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: init 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-c con 0x7f561008a310 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-b con 0x7f561009c460 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-a con 0x7f5610148630 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561009ac80 con 0x7f5610148630 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561009def0 con 0x7f561008a310 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561008ae00 con 0x7f561009c460 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f56175e8700 10 monclient(hunting): authenticate will time out at 857.726779s 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5614b83700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f561009c460 0x7f5610146f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53004/0 (socket says 172.21.15.76:53004) 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5614b83700 1 -- 172.21.15.76:0/869467779 learned_addr learned my addr 172.21.15.76:0/869467779 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615861908 0 0) 0x7f561008ae00 con 0x7f561009c460 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 10 monclient(hunting): my global_id is 4319 2024-09-16T06:38:30.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc0035d0 con 0x7f561009c460 2024-09-16T06:38:30.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f88880430d0 con 0x7f88a01315f0 2024-09-16T06:38:30.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f889f7fe700 10 monclient: get_auth_request con 0x7f88800364d0 auth_method 0 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.589+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667578391 0 0) 0x7f561009def0 con 0x7f561008a310 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient(hunting): my global_id is 4315 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561008ae00 con 0x7f561008a310 2024-09-16T06:38:30.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194164549 0 0) 0x7f55fc0035d0 con 0x7f561009c460 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 >> v1:172.21.15.189:6789/0 conn(0x7f561008a310 legacy=0x7f561008a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 >> v1:172.21.15.76:6789/0 conn(0x7f5610148630 legacy=0x7f561014aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 monclient: found mon.noname-b 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561009a410 con 0x7f561009c460 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5600001610 con 0x7f561009c460 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 5 monclient: authenticate success, global_id 4319 2024-09-16T06:38:30.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: dump: 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.656 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5600001830 con 0x7f561009c460 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 1 -- 172.21.15.76:0/869467779 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5600001bc0 con 0x7f561009c460 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 4 set_mon_vals no callback set 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f5607fff700 10 monclient: dump: 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.657 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 10 monclient: shutdown 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f56175e8700 1 -- 172.21.15.76:0/869467779 >> v1:172.21.15.76:6790/0 conn(0x7f561009c460 legacy=0x7f5610146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2a88014a40 con 0x7f2a90070ac0 2024-09-16T06:38:30.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f2a867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.593+0000 7f2a867fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/869467779 shutdown_connections 2024-09-16T06:38:30.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/869467779 >> 172.21.15.76:0/869467779 conn(0x7f56100a3e50 msgr2=0x7f56100a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f88a4c82700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f88a4c82700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f88a4c82700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8884005100 con 0x7f88a01315f0 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1118014b80 con 0x7f1120138610 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f111d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f111d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/869467779 shutdown_connections 2024-09-16T06:38:30.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/869467779 wait complete. 2024-09-16T06:38:30.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: build_initial_monmap 2024-09-16T06:38:30.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: monmap: 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 Processor -- start 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- start start 2024-09-16T06:38:30.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: init 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-c con 0x7f561008a310 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-a con 0x7f561009c460 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient(hunting): picked mon.noname-b con 0x7f5610148630 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561009ec30 con 0x7f561009c460 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5610142d70 con 0x7f5610148630 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5610148420 con 0x7f561008a310 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient(hunting): authenticate will time out at 857.734503s 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5615384700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f561008a310 0x7f56101472d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53018/0 (socket says 172.21.15.76:53018) 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5615384700 1 -- 172.21.15.76:0/3955276923 learned_addr learned my addr 172.21.15.76:0/3955276923 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437856894 0 0) 0x7f561009a410 con 0x7f561009c460 2024-09-16T06:38:30.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): my global_id is 4320 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc0035d0 con 0x7f561009c460 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979368390 0 0) 0x7f5610142d70 con 0x7f5610148630 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): my global_id is 4318 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561009a410 con 0x7f5610148630 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 637419503 0 0) 0x7f55fc0035d0 con 0x7f561009c460 2024-09-16T06:38:30.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.605+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8888003ea0 con 0x7f88a01315f0 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.605+0000 7f8897fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5610142d70 con 0x7f561009c460 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5600001370 con 0x7f561009c460 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:30.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.605+0000 7f8897fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): dump: 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.667 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588220103 0 0) 0x7f5610142d70 con 0x7f561009c460 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 >> v1:172.21.15.76:6790/0 conn(0x7f561008a310 legacy=0x7f56101472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 >> v1:172.21.15.189:6789/0 conn(0x7f5610148630 legacy=0x7f56101df2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 monclient: found mon.a 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610072450 con 0x7f561009c460 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f5605ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:30.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 5 monclient: authenticate success, global_id 4320 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _renew_subs 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610141570 con 0x7f561009c460 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _renew_subs 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.597+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56101409a0 con 0x7f561009c460 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5600001600 con 0x7f561009c460 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5600004b40 con 0x7f561009c460 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:30.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 10 monclient: dump: 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5600004d20 con 0x7f561009c460 2024-09-16T06:38:30.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f56000430b0 con 0x7f561009c460 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.601+0000 7f5615384700 10 monclient: get_auth_request con 0x7f55fc07c810 auth_method 0 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.605+0000 7f56077fe700 4 set_mon_vals no callback set 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.609+0000 7f56175e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.609+0000 7f56175e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.609+0000 7f56175e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.609+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec005100 con 0x7f561009c460 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.613+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5600014880 con 0x7f561009c460 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.613+0000 7f5605ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:30.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.613+0000 7f5605ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:30.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a96a8f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a96a8f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a96a8f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2a780053b0 con 0x7f2a90070ac0 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a867fc700 1 -- 172.21.15.76:0/3081196643 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f2a88014630 con 0x7f2a90070ac0 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.765+0000 7f2a867fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.768 INFO:teuthology.orchestra.run.smithi076.stdout:47244640258 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2a740366e0 legacy=0x7f2a74038ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 10 monclient: shutdown 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 >> v1:172.21.15.76:6789/0 conn(0x7f2a90070ac0 legacy=0x7f2a90136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 shutdown_connections 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 >> 172.21.15.76:0/3081196643 conn(0x7f2a900a0030 msgr2=0x7f2a901355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 shutdown_connections 2024-09-16T06:38:30.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.769+0000 7f2a96a8f700 1 -- 172.21.15.76:0/3081196643 wait complete. 2024-09-16T06:38:30.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f56175e8700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f56175e8700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f56175e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f55ec0053b0 con 0x7f561009c460 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f5605ffb700 1 -- 172.21.15.76:0/3955276923 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f5600012cf0 con 0x7f561009c460 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f5605ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:30.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.781+0000 7f5605ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.785 INFO:teuthology.orchestra.run.smithi076.stdout:51539607555 2024-09-16T06:38:30.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 >> v1:172.21.15.189:6812/3098810674 conn(0x7f55fc07c810 legacy=0x7f55fc07ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 10 monclient: shutdown 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 >> v1:172.21.15.76:6789/0 conn(0x7f561009c460 legacy=0x7f56101dbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 shutdown_connections 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 >> 172.21.15.76:0/3955276923 conn(0x7f56100a3e50 msgr2=0x7f56100a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 shutdown_connections 2024-09-16T06:38:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.785+0000 7f56175e8700 1 -- 172.21.15.76:0/3955276923 wait complete. 2024-09-16T06:38:30.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-16T06:38:30.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb0811b7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:30.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb0811b7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb0811b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb07c0a5800 con 0x7fb07c132330 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb078ff9700 1 -- 172.21.15.76:0/3413852396 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 956180462) 0x7fb074012eb0 con 0x7fb07c132330 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb078ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.845+0000 7fb078ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.849 INFO:teuthology.orchestra.run.smithi076.stdout:51539607555 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb064036960 legacy=0x7fb064038e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 10 monclient: shutdown 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 >> v1:172.21.15.76:6790/0 conn(0x7fb07c132330 legacy=0x7fb07c1d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 shutdown_connections 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 >> 172.21.15.76:0/3413852396 conn(0x7fb07c0a41c0 msgr2=0x7fb07c09eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 shutdown_connections 2024-09-16T06:38:30.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.849+0000 7fb0811b7700 1 -- 172.21.15.76:0/3413852396 wait complete. 2024-09-16T06:38:30.862 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.1 2024-09-16T06:38:30.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f888a54d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:30.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f888a54d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:30.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f888a54d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:30.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f88600053b0 con 0x7f888406b400 2024-09-16T06:38:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f8881ffb700 1 -- 172.21.15.76:0/403955521 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 1362196188) 0x7f887c0133f0 con 0x7f888406b400 2024-09-16T06:38:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f8881ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.885+0000 7f8881ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.889 INFO:teuthology.orchestra.run.smithi076.stdout:51539607553 2024-09-16T06:38:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f88a4c82700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f88a4c82700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f88a4c82700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f88840053b0 con 0x7f88a01315f0 2024-09-16T06:38:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f8897fff700 1 -- 172.21.15.76:0/1531925892 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 3130653186) 0x7f8888012c30 con 0x7f88a01315f0 2024-09-16T06:38:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f8897fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f8897fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 >> v1:172.21.15.189:6812/3098810674 conn(0x7f886c03ef00 legacy=0x7f886c0413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 10 monclient: shutdown 2024-09-16T06:38:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 >> v1:172.21.15.189:6789/0 conn(0x7f888406b400 legacy=0x7f888413c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.893 INFO:teuthology.orchestra.run.smithi076.stdout:55834574849 2024-09-16T06:38:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 shutdown_connections 2024-09-16T06:38:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 >> 172.21.15.76:0/403955521 conn(0x7f88840a12f0 msgr2=0x7f888413ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 shutdown_connections 2024-09-16T06:38:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.889+0000 7f888a54d700 1 -- 172.21.15.76:0/403955521 wait complete. 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 >> v1:172.21.15.189:6812/3098810674 conn(0x7f88800364d0 legacy=0x7f8880038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 10 monclient: shutdown 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 >> v1:172.21.15.76:6790/0 conn(0x7f88a01315f0 legacy=0x7f88a01cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 shutdown_connections 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 >> 172.21.15.76:0/1531925892 conn(0x7f88a00a0030 msgr2=0x7f88a00a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 shutdown_connections 2024-09-16T06:38:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.893+0000 7f88a4c82700 1 -- 172.21.15.76:0/1531925892 wait complete. 2024-09-16T06:38:30.908 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.4 2024-09-16T06:38:30.908 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607553 for osd.2 2024-09-16T06:38:30.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.913+0000 7f1125a58700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:30.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.913+0000 7f1125a58700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:30.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.913+0000 7f1125a58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:30.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.913+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f11200672a0 con 0x7f1120138610 2024-09-16T06:38:30.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f111d7fa700 1 -- 172.21.15.76:0/4069340725 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 2439918515) 0x7f1118012ff0 con 0x7f1120138610 2024-09-16T06:38:30.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f111d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:30.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f111d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:30.918 INFO:teuthology.orchestra.run.smithi076.stdout:55834574850 2024-09-16T06:38:30.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 >> v1:172.21.15.189:6812/3098810674 conn(0x7f11040474d0 legacy=0x7f1104049990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 10 monclient: shutdown 2024-09-16T06:38:30.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:30.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 >> v1:172.21.15.76:6789/0 conn(0x7f1120138610 legacy=0x7f11201d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:30.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 shutdown_connections 2024-09-16T06:38:30.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 >> 172.21.15.76:0/4069340725 conn(0x7f11200a0030 msgr2=0x7f11200a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:30.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 shutdown_connections 2024-09-16T06:38:30.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:30.917+0000 7f1125a58700 1 -- 172.21.15.76:0/4069340725 wait complete. 2024-09-16T06:38:30.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2024-09-16T06:38:31.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T06:38:31.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T06:38:31.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: get_monmap_and_config 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: monmap: 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:31.897051+0000 2024-09-16T06:38:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:31.897051+0000 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:31.896 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 1 Processor -- start 2024-09-16T06:38:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 1 -- start start 2024-09-16T06:38:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: init 2024-09-16T06:38:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-a con 0x7f3144132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-b con 0x7f314406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-c con 0x7f314406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314406c280 con 0x7f3144132330 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314406bfa0 con 0x7f314406cf50 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314409ace0 con 0x7f314406d480 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient(hunting): authenticate will time out at 859.030799s 2024-09-16T06:38:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3149bf3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3144132330 0x7f3144132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60668/0 (socket says 172.21.15.76:60668) 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f31493f2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f314406d480 0x7f314413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53026/0 (socket says 172.21.15.76:53026) 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3149bf3700 1 -- 172.21.15.76:0/2547114732 learned_addr learned my addr 172.21.15.76:0/2547114732 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266689343 0 0) 0x7f314409ace0 con 0x7f314406d480 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): my global_id is 4322 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300035d0 con 0x7f314406d480 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465699915 0 0) 0x7f314406c280 con 0x7f3144132330 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient(hunting): my global_id is 4323 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314409ace0 con 0x7f3144132330 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2225944178 0 0) 0x7f31300035d0 con 0x7f314406d480 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 >> v1:172.21.15.189:6789/0 conn(0x7f314406cf50 legacy=0x7f314413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 >> v1:172.21.15.76:6789/0 conn(0x7f3144132330 legacy=0x7f3144132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 monclient: found mon.noname-b 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314409ae70 con 0x7f314406d480 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3134001610 con 0x7f314406d480 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: dump: 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 5 monclient: authenticate success, global_id 4322 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3134001830 con 0x7f314406d480 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f314abf5700 4 set_mon_vals no callback set 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 1 -- 172.21.15.76:0/2547114732 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3134001a60 con 0x7f314406d480 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.893+0000 7f3148bf1700 10 monclient: dump: 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: shutdown 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- 172.21.15.76:0/2547114732 >> v1:172.21.15.76:6790/0 conn(0x7f314406d480 legacy=0x7f314413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- 172.21.15.76:0/2547114732 shutdown_connections 2024-09-16T06:38:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- 172.21.15.76:0/2547114732 >> 172.21.15.76:0/2547114732 conn(0x7f31440a41c0 msgr2=0x7f31440a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- 172.21.15.76:0/2547114732 shutdown_connections 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- 172.21.15.76:0/2547114732 wait complete. 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: monmap: 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 Processor -- start 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- start start 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: init 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-c con 0x7f314406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-b con 0x7f314406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient(hunting): picked mon.noname-a con 0x7f3144132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314409e8b0 con 0x7f3144132330 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31441360a0 con 0x7f314406d480 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144131770 con 0x7f314406cf50 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314abf5700 10 monclient(hunting): authenticate will time out at 859.034848s 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314a3f4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3144132330 0x7f31441d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60674/0 (socket says 172.21.15.76:60674) 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f314a3f4700 1 -- 172.21.15.76:0/1017391307 learned_addr learned my addr 172.21.15.76:0/1017391307 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140936606 0 0) 0x7f314409e8b0 con 0x7f3144132330 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): my global_id is 4326 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31280035d0 con 0x7f3144132330 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907001527 0 0) 0x7f3144131770 con 0x7f314406cf50 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): my global_id is 4325 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314409e8b0 con 0x7f314406cf50 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562405426 0 0) 0x7f31441360a0 con 0x7f314406d480 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): my global_id is 4324 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3144131770 con 0x7f314406d480 2024-09-16T06:38:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545695562 0 0) 0x7f314409e8b0 con 0x7f314406cf50 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31441360a0 con 0x7f314406cf50 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3140001170 con 0x7f314406cf50 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.916 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 623812783 0 0) 0x7f31280035d0 con 0x7f3144132330 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f314409e8b0 con 0x7f3144132330 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f313c001ba0 con 0x7f3144132330 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.918 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.897+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3714071729 0 0) 0x7f31441360a0 con 0x7f314406cf50 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 >> v1:172.21.15.189:6789/0 conn(0x7f314406d480 legacy=0x7f3144135990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 >> v1:172.21.15.76:6789/0 conn(0x7f3144132330 legacy=0x7f31441d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 monclient: found mon.c 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3144137cc0 con 0x7f314406cf50 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 5 monclient: authenticate success, global_id 4325 2024-09-16T06:38:31.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _renew_subs 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144136ed0 con 0x7f314406cf50 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _renew_subs 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3144138c70 con 0x7f314406cf50 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3140004890 con 0x7f314406cf50 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313b7fe700 4 set_mon_vals no callback set 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3140004ba0 con 0x7f314406cf50 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 10 monclient: dump: 2024-09-16T06:38:31.921 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f314000f610 con 0x7f314406cf50 2024-09-16T06:38:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f31400430c0 con 0x7f314406cf50 2024-09-16T06:38:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c005100 con 0x7f314406cf50 2024-09-16T06:38:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.901+0000 7f31493f2700 10 monclient: get_auth_request con 0x7f3128036660 auth_method 0 2024-09-16T06:38:31.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T06:38:31.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T06:38:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.905+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f31400146d0 con 0x7f314406cf50 2024-09-16T06:38:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.905+0000 7f313a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.905+0000 7f313a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:31.934 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T06:38:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: get_monmap_and_config 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: monmap: 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:31.912395+0000 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:31.912395+0000 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 Processor -- start 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- start start 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: init 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-b con 0x7f2da0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-c con 0x7f2da006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-a con 0x7f2da006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da006c280 con 0x7f2da006cf50 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da006bfa0 con 0x7f2da006d480 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da009ace0 con 0x7f2da0132330 2024-09-16T06:38:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient(hunting): authenticate will time out at 859.045961s 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2da006cf50 0x7f2da013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60686/0 (socket says 172.21.15.76:60686) 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9ffff700 1 -- 172.21.15.76:0/935258364 learned_addr learned my addr 172.21.15.76:0/935258364 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523339134 0 0) 0x7f2da006c280 con 0x7f2da006cf50 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): my global_id is 4329 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8c0035d0 con 0x7f2da006cf50 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267138566 0 0) 0x7f2da009ace0 con 0x7f2da0132330 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient(hunting): my global_id is 4328 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da006c280 con 0x7f2da0132330 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217010506 0 0) 0x7f2d8c0035d0 con 0x7f2da006cf50 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 >> v1:172.21.15.76:6790/0 conn(0x7f2da0132330 legacy=0x7f2da0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 >> v1:172.21.15.189:6789/0 conn(0x7f2da006d480 legacy=0x7f2da013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 monclient: found mon.noname-a 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da009ae70 con 0x7f2da006cf50 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 5 monclient: authenticate success, global_id 4329 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d98001990 con 0x7f2da006cf50 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: dump: 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d98001bb0 con 0x7f2da006cf50 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 1 -- 172.21.15.76:0/935258364 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d98001da0 con 0x7f2da006cf50 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 4 set_mon_vals no callback set 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2d9effd700 10 monclient: dump: 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 10 monclient: shutdown 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- 172.21.15.76:0/935258364 >> v1:172.21.15.76:6789/0 conn(0x7f2da006cf50 legacy=0x7f2da013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- 172.21.15.76:0/935258364 shutdown_connections 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- 172.21.15.76:0/935258364 >> 172.21.15.76:0/935258364 conn(0x7f2da00a41c0 msgr2=0x7f2da00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- 172.21.15.76:0/935258364 shutdown_connections 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.909+0000 7f2da48bf700 1 -- 172.21.15.76:0/935258364 wait complete. 2024-09-16T06:38:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: monmap: 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 Processor -- start 2024-09-16T06:38:31.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- start start 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: init 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-c con 0x7f2da006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-a con 0x7f2da006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient(hunting): picked mon.noname-b con 0x7f2da0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da011e490 con 0x7f2da006d480 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da011dfa0 con 0x7f2da0132330 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da01310d0 con 0x7f2da006cf50 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient(hunting): authenticate will time out at 859.049613s 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2da006cf50 0x7f2da009cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53068/0 (socket says 172.21.15.76:53068) 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9f7fe700 1 -- 172.21.15.76:0/1525173697 learned_addr learned my addr 172.21.15.76:0/1525173697 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244247441 0 0) 0x7f2da01310d0 con 0x7f2da006cf50 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): my global_id is 4331 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d800035d0 con 0x7f2da006cf50 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298781192 0 0) 0x7f2da011e490 con 0x7f2da006d480 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): my global_id is 4332 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da01310d0 con 0x7f2da006d480 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2446614269 0 0) 0x7f2d800035d0 con 0x7f2da006cf50 2024-09-16T06:38:31.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da011e490 con 0x7f2da006cf50 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d90001990 con 0x7f2da006cf50 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): dump: 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563062674 0 0) 0x7f2da011dfa0 con 0x7f2da0132330 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): my global_id is 4330 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d800035d0 con 0x7f2da0132330 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4149572384 0 0) 0x7f2da01310d0 con 0x7f2da006d480 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2da011dfa0 con 0x7f2da006d480 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d88001770 con 0x7f2da006d480 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:31.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient(hunting): dump: 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3110655760 0 0) 0x7f2da011e490 con 0x7f2da006cf50 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 >> v1:172.21.15.189:6789/0 conn(0x7f2da0132330 legacy=0x7f2da01cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 >> v1:172.21.15.76:6789/0 conn(0x7f2da006d480 legacy=0x7f2da009d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 monclient: found mon.c 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0131620 con 0x7f2da006cf50 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 5 monclient: authenticate success, global_id 4331 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _renew_subs 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da01202e0 con 0x7f2da006cf50 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d90001ce0 con 0x7f2da006cf50 2024-09-16T06:38:31.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _renew_subs 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d90003b10 con 0x7f2da006cf50 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da009dd80 con 0x7f2da006cf50 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9dffb700 4 set_mon_vals no callback set 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.913+0000 7f2d9cff9700 10 monclient: dump: 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2da48bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2da48bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2da48bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da00a0260 con 0x7f2da006cf50 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2d90003cf0 con 0x7f2da006cf50 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f2d900432c0 con 0x7f2da006cf50 2024-09-16T06:38:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.917+0000 7f2d96ffd700 10 monclient: get_auth_request con 0x7f2d80036450 auth_method 0 2024-09-16T06:38:31.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.921+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2d900146a0 con 0x7f2da006cf50 2024-09-16T06:38:31.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.921+0000 7f2d9cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.921+0000 7f2d9cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: get_monmap_and_config 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: monmap: 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:31.972903+0000 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:31.972903+0000 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.972 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 1 Processor -- start 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 1 -- start start 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: init 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-a con 0x7f0bb4139a00 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-c con 0x7f0bb4099ec0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-b con 0x7f0bb413fe80 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb409f7e0 con 0x7f0bb4139a00 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb409a2a0 con 0x7f0bb4099ec0 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb4098790 con 0x7f0bb413fe80 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient(hunting): authenticate will time out at 859.106403s 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb37fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0bb4139a00 0x7f0bb4139de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60710/0 (socket says 172.21.15.76:60710) 2024-09-16T06:38:31.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb37fe700 1 -- 172.21.15.76:0/129709879 learned_addr learned my addr 172.21.15.76:0/129709879 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327246292 0 0) 0x7f0bb409f7e0 con 0x7f0bb4139a00 2024-09-16T06:38:31.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): my global_id is 4335 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b9c0035d0 con 0x7f0bb4139a00 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307928491 0 0) 0x7f0bb409a2a0 con 0x7f0bb4099ec0 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): my global_id is 4333 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb409f7e0 con 0x7f0bb4099ec0 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473286060 0 0) 0x7f0bb4098790 con 0x7f0bb413fe80 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient(hunting): my global_id is 4334 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb409a2a0 con 0x7f0bb413fe80 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711212627 0 0) 0x7f0b9c0035d0 con 0x7f0bb4139a00 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 >> v1:172.21.15.76:6790/0 conn(0x7f0bb413fe80 legacy=0x7f0bb4142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 >> v1:172.21.15.189:6789/0 conn(0x7f0bb4099ec0 legacy=0x7f0bb413e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 monclient: found mon.noname-a 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4098c30 con 0x7f0bb4139a00 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ba4001590 con 0x7f0bb4139a00 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: dump: 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.981 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 5 monclient: authenticate success, global_id 4335 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba40017b0 con 0x7f0bb4139a00 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 1 -- 172.21.15.76:0/129709879 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ba4001b40 con 0x7f0bb4139a00 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb8c2f700 4 set_mon_vals no callback set 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.969+0000 7f0bb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb27fc700 10 monclient: dump: 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.982 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: shutdown 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/129709879 >> v1:172.21.15.76:6789/0 conn(0x7f0bb4139a00 legacy=0x7f0bb4139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/129709879 shutdown_connections 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/129709879 >> 172.21.15.76:0/129709879 conn(0x7f0bb40a0030 msgr2=0x7f0bb40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:31.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/129709879 shutdown_connections 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/129709879 wait complete. 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: build_initial_monmap 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: monmap: 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:31.984 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 Processor -- start 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- start start 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: init 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-b con 0x7f0bb4099ec0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-a con 0x7f0bb4139a00 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient(hunting): picked mon.noname-c con 0x7f0bb413fe80 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb41392a0 con 0x7f0bb4139a00 2024-09-16T06:38:31.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb413ac00 con 0x7f0bb4099ec0 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40a14a0 con 0x7f0bb413fe80 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient(hunting): authenticate will time out at 859.110327s 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0bb4139a00 0x7f0bb413a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60720/0 (socket says 172.21.15.76:60720) 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb2ffd700 1 -- 172.21.15.76:0/1244668516 learned_addr learned my addr 172.21.15.76:0/1244668516 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352088297 0 0) 0x7f0bb41392a0 con 0x7f0bb4139a00 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): my global_id is 4338 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba00035d0 con 0x7f0bb4139a00 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963965426 0 0) 0x7f0bb40a14a0 con 0x7f0bb413fe80 2024-09-16T06:38:31.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): my global_id is 4337 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb41392a0 con 0x7f0bb413fe80 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501525186 0 0) 0x7f0bb413ac00 con 0x7f0bb4099ec0 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): my global_id is 4336 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb40a14a0 con 0x7f0bb4099ec0 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039615961 0 0) 0x7f0ba00035d0 con 0x7f0bb4139a00 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb413ac00 con 0x7f0bb4139a00 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613037189 0 0) 0x7f0bb41392a0 con 0x7f0bb413fe80 2024-09-16T06:38:31.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba00035d0 con 0x7f0bb413fe80 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ba80018a0 con 0x7f0bb4139a00 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): dump: 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.988 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0bac001af0 con 0x7f0bb413fe80 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient(hunting): dump: 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.989 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1362836553 0 0) 0x7f0ba00035d0 con 0x7f0bb413fe80 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 >> v1:172.21.15.189:6789/0 conn(0x7f0bb4099ec0 legacy=0x7f0bb4139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 >> v1:172.21.15.76:6789/0 conn(0x7f0bb4139a00 legacy=0x7f0bb413a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 monclient: found mon.c 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb409a540 con 0x7f0bb413fe80 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient: _finish_auth 0 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:31.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0b9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 5 monclient: authenticate success, global_id 4337 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _renew_subs 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb413fa60 con 0x7f0bb413fe80 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _renew_subs 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.973+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb413bf70 con 0x7f0bb413fe80 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bac001c90 con 0x7f0bb413fe80 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0bac002100 con 0x7f0bb413fe80 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0bb1ffb700 4 set_mon_vals no callback set 2024-09-16T06:38:31.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 10 monclient: dump: 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0bac00e4f0 con 0x7f0bb413fe80 2024-09-16T06:38:31.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f0bac042000 con 0x7f0bb413fe80 2024-09-16T06:38:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.977+0000 7f0bb37fe700 10 monclient: get_auth_request con 0x7f0ba00365b0 auth_method 0 2024-09-16T06:38:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.981+0000 7f0bb8c2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.981+0000 7f0bb8c2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.981+0000 7f0bb8c2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:31.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.981+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b94005100 con 0x7f0bb413fe80 2024-09-16T06:38:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.985+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0bac013520 con 0x7f0bb413fe80 2024-09-16T06:38:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.985+0000 7f0b9bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:31.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:31.985+0000 7f0b9bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: get_monmap_and_config 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: monmap: 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:32.061661+0000 2024-09-16T06:38:32.060 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:32.061661+0000 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 1 Processor -- start 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 1 -- start start 2024-09-16T06:38:32.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: init 2024-09-16T06:38:32.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-c con 0x7ff5640999f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-b con 0x7ff564132f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-a con 0x7ff564133340 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564097cb0 con 0x7ff564133340 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564097f00 con 0x7ff5640999f0 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640981e0 con 0x7ff564132f40 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff56aa00700 10 monclient(hunting): authenticate will time out at 859.195291s 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff5691fd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff564132f40 0x7ff564138020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53080/0 (socket says 172.21.15.76:53080) 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff5691fd700 1 -- 172.21.15.76:0/2776325644 learned_addr learned my addr 172.21.15.76:0/2776325644 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.057+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623336804 0 0) 0x7ff5640981e0 con 0x7ff564132f40 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): my global_id is 4340 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5500035d0 con 0x7ff564132f40 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571720275 0 0) 0x7ff564097cb0 con 0x7ff564133340 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): my global_id is 4341 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5640981e0 con 0x7ff564133340 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866954765 0 0) 0x7ff564097f00 con 0x7ff5640999f0 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient(hunting): my global_id is 4339 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564097cb0 con 0x7ff5640999f0 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146687727 0 0) 0x7ff5500035d0 con 0x7ff564132f40 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 >> v1:172.21.15.189:6789/0 conn(0x7ff5640999f0 legacy=0x7ff564099dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 >> v1:172.21.15.76:6789/0 conn(0x7ff564133340 legacy=0x7ff56413b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 monclient: found mon.noname-b 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56409ccd0 con 0x7ff564132f40 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff560001610 con 0x7ff564132f40 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: dump: 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.066 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 5 monclient: authenticate success, global_id 4340 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff560001830 con 0x7ff564132f40 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 1 -- 172.21.15.76:0/2776325644 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff560001bc0 con 0x7ff564132f40 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff5689fc700 10 monclient: dump: 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:32.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 4 set_mon_vals no callback set 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 10 monclient: shutdown 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- 172.21.15.76:0/2776325644 >> v1:172.21.15.76:6790/0 conn(0x7ff564132f40 legacy=0x7ff564138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- 172.21.15.76:0/2776325644 shutdown_connections 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- 172.21.15.76:0/2776325644 >> 172.21.15.76:0/2776325644 conn(0x7ff5640a0030 msgr2=0x7ff5640a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: get_monmap_and_config 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: monmap: 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:32.064931+0000 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:32.064931+0000 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:32.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- 172.21.15.76:0/2776325644 shutdown_connections 2024-09-16T06:38:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- 172.21.15.76:0/2776325644 wait complete. 2024-09-16T06:38:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 1 Processor -- start 2024-09-16T06:38:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 1 -- start start 2024-09-16T06:38:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: init 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 10 monclient: monmap: 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:32.072 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:32.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient(hunting): picked mon.noname-c con 0x7ff89009d180 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient(hunting): picked mon.noname-b con 0x7ff89009d560 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient(hunting): picked mon.noname-a con 0x7ff890138610 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89012f240 con 0x7ff890138610 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890099d40 con 0x7ff89009d180 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89009a300 con 0x7ff89009d560 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff89009d560 0x7ff890136ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53090/0 (socket says 172.21.15.76:53090) 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88e59c700 1 -- 172.21.15.76:0/2072926980 learned_addr learned my addr 172.21.15.76:0/2072926980 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff894966700 10 monclient(hunting): authenticate will time out at 859.199024s 2024-09-16T06:38:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 Processor -- start 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46194733 0 0) 0x7ff89009a300 con 0x7ff89009d560 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): my global_id is 4343 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8780035d0 con 0x7ff89009d560 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250742218 0 0) 0x7ff89012f240 con 0x7ff890138610 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): my global_id is 4344 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89009a300 con 0x7ff890138610 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820288884 0 0) 0x7ff890099d40 con 0x7ff89009d180 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient(hunting): my global_id is 4342 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89012f240 con 0x7ff89009d180 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491815532 0 0) 0x7ff89009a300 con 0x7ff890138610 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 >> v1:172.21.15.76:6790/0 conn(0x7ff89009d560 legacy=0x7ff890136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 >> v1:172.21.15.189:6789/0 conn(0x7ff89009d180 legacy=0x7ff89012eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 monclient: found mon.noname-a 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89009a020 con 0x7ff890138610 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff888001970 con 0x7ff890138610 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff88dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: dump: 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.078 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 5 monclient: authenticate success, global_id 4344 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff888001c90 con 0x7ff890138610 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 1 -- 172.21.15.76:0/2072926980 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff888002020 con 0x7ff890138610 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 4 set_mon_vals no callback set 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88dd9b700 10 monclient: dump: 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 1 -- start start 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.061+0000 7ff56aa00700 10 monclient: init 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-c con 0x7ff5640999f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-a con 0x7ff564132f40 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient(hunting): picked mon.noname-b con 0x7ff564133340 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56406ead0 con 0x7ff564132f40 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56406edb0 con 0x7ff564133340 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564132040 con 0x7ff5640999f0 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff56aa00700 10 monclient(hunting): authenticate will time out at 859.200386s 2024-09-16T06:38:32.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff5691fd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff564132f40 0x7ff56412c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60752/0 (socket says 172.21.15.76:60752) 2024-09-16T06:38:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff5691fd700 1 -- 172.21.15.76:0/2666621671 learned_addr learned my addr 172.21.15.76:0/2666621671 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff5699fe700 1 --1- 172.21.15.76:0/2666621671 >> v1:172.21.15.76:6790/0 conn(0x7ff5640999f0 0x7ff56412bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53092/0 (socket says 172.21.15.76:53092) 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: shutdown 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- 172.21.15.76:0/2072926980 >> v1:172.21.15.76:6789/0 conn(0x7ff890138610 legacy=0x7ff89013aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- 172.21.15.76:0/2072926980 shutdown_connections 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- 172.21.15.76:0/2072926980 >> 172.21.15.76:0/2072926980 conn(0x7ff8900a0030 msgr2=0x7ff8900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- 172.21.15.76:0/2072926980 shutdown_connections 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- 172.21.15.76:0/2072926980 wait complete. 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: monmap: 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 Processor -- start 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- start start 2024-09-16T06:38:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: init 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient(hunting): picked mon.noname-c con 0x7ff89009d180 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient(hunting): picked mon.noname-a con 0x7ff89009d560 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient(hunting): picked mon.noname-b con 0x7ff890138610 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890129860 con 0x7ff89009d560 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890137d80 con 0x7ff890138610 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8901c5170 con 0x7ff89009d180 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff894966700 10 monclient(hunting): authenticate will time out at 859.203215s 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff89009d560 0x7ff8901c4a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60762/0 (socket says 172.21.15.76:60762) 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff88e59c700 1 -- 172.21.15.76:0/2739324493 learned_addr learned my addr 172.21.15.76:0/2739324493 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29369879 0 0) 0x7ff890129860 con 0x7ff89009d560 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): my global_id is 4350 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878004da0 con 0x7ff89009d560 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522868705 0 0) 0x7ff8901c5170 con 0x7ff89009d180 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 10 monclient(hunting): my global_id is 4349 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.065+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890129860 con 0x7ff89009d180 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450729847 0 0) 0x7ff890137d80 con 0x7ff890138610 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): my global_id is 4348 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8901c5170 con 0x7ff890138610 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222124789 0 0) 0x7ff8780035d0 con 0x7ff89009d560 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff890137d80 con 0x7ff89009d560 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff880001730 con 0x7ff89009d560 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): dump: 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2480450031 0 0) 0x7ff890137d80 con 0x7ff89009d560 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 >> v1:172.21.15.76:6790/0 conn(0x7ff89009d180 legacy=0x7ff8901c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 >> v1:172.21.15.189:6789/0 conn(0x7ff890138610 legacy=0x7ff8901cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 monclient: found mon.a 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89013bc40 con 0x7ff89009d560 2024-09-16T06:38:32.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 5 monclient: authenticate success, global_id 4350 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 10 monclient: _renew_subs 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89009a5a0 con 0x7ff89009d560 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff880001c00 con 0x7ff89009d560 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff880004a30 con 0x7ff89009d560 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 10 monclient: dump: 2024-09-16T06:38:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 10 monclient: _renew_subs 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff890131d10 con 0x7ff89009d560 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff880004b90 con 0x7ff89009d560 2024-09-16T06:38:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7ff880014e90 con 0x7ff89009d560 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.069+0000 7ff88ed9d700 10 monclient: get_auth_request con 0x7ff8780366e0 auth_method 0 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff88d59a700 4 set_mon_vals no callback set 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953842736 0 0) 0x7ff56406ead0 con 0x7ff564132f40 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): my global_id is 4347 2024-09-16T06:38:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5480035d0 con 0x7ff564132f40 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff894966700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff894966700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff894966700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137391468 0 0) 0x7ff56406edb0 con 0x7ff564133340 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): my global_id is 4345 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff56406ead0 con 0x7ff564133340 2024-09-16T06:38:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220701896 0 0) 0x7ff564132040 con 0x7ff5640999f0 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): my global_id is 4346 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff56406edb0 con 0x7ff5640999f0 2024-09-16T06:38:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff870005100 con 0x7ff89009d560 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 862960341 0 0) 0x7ff5480035d0 con 0x7ff564132f40 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff564132040 con 0x7ff564132f40 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff560001b40 con 0x7ff564132f40 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593354300 0 0) 0x7ff56406edb0 con 0x7ff5640999f0 2024-09-16T06:38:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5480035d0 con 0x7ff5640999f0 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff55c001bb0 con 0x7ff5640999f0 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.073+0000 7ff55a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.100 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010689015 0 0) 0x7ff56406ead0 con 0x7ff564133340 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff56406edb0 con 0x7ff564133340 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff554001830 con 0x7ff564133340 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.101 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1716596209 0 0) 0x7ff5480035d0 con 0x7ff5640999f0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 >> v1:172.21.15.189:6789/0 conn(0x7ff564133340 legacy=0x7ff5641d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 >> v1:172.21.15.76:6789/0 conn(0x7ff564132f40 legacy=0x7ff56412c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 monclient: found mon.c 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56409e6b0 con 0x7ff5640999f0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff55c0020a0 con 0x7ff5640999f0 2024-09-16T06:38:32.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff55c003be0 con 0x7ff5640999f0 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 10 monclient: dump: 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 5 monclient: authenticate success, global_id 4346 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _renew_subs 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56409a540 con 0x7ff5640999f0 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _renew_subs 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5640724c0 con 0x7ff5640999f0 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55bfff700 4 set_mon_vals no callback set 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff55c003d40 con 0x7ff5640999f0 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c005100 con 0x7ff5640999f0 2024-09-16T06:38:32.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7ff55c0432a0 con 0x7ff5640999f0 2024-09-16T06:38:32.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.077+0000 7ff5691fd700 10 monclient: get_auth_request con 0x7ff54803aa80 auth_method 0 2024-09-16T06:38:32.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff88001cb50 con 0x7ff89009d560 2024-09-16T06:38:32.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff8777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff8777fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:32.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff55c014620 con 0x7ff5640999f0 2024-09-16T06:38:32.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff55a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.085+0000 7ff55a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:32.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:32.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: monmap: 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:32.117832+0000 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:32.117832+0000 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.121 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 1 Processor -- start 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 1 -- start start 2024-09-16T06:38:32.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: init 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f38d8070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f38d8070ea0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f38d813b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8072610 con 0x7f38d8070ac0 2024-09-16T06:38:32.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d809f560 con 0x7f38d8070ea0 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8072330 con 0x7f38d813b980 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d7fff700 10 monclient(hunting): authenticate will time out at 859.251792s 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d6ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f38d8070ac0 0x7f38d8097bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60778/0 (socket says 172.21.15.76:60778) 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.113+0000 7f38d77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f38d813b980 0x7f38d813dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53106/0 (socket says 172.21.15.76:53106) 2024-09-16T06:38:32.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d77fe700 1 -- 172.21.15.76:0/3456789611 learned_addr learned my addr 172.21.15.76:0/3456789611 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698323463 0 0) 0x7f38d8072330 con 0x7f38d813b980 2024-09-16T06:38:32.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): my global_id is 4352 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c00035d0 con 0x7f38d813b980 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699802736 0 0) 0x7f38d8072610 con 0x7f38d8070ac0 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient(hunting): my global_id is 4353 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d8072330 con 0x7f38d8070ac0 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693548053 0 0) 0x7f38c00035d0 con 0x7f38d813b980 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 >> v1:172.21.15.189:6789/0 conn(0x7f38d8070ea0 legacy=0x7f38d80712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 >> v1:172.21.15.76:6789/0 conn(0x7f38d8070ac0 legacy=0x7f38d8097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 monclient: found mon.noname-b 2024-09-16T06:38:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80974f0 con 0x7f38d813b980 2024-09-16T06:38:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38cc001610 con 0x7f38d813b980 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: dump: 2024-09-16T06:38:32.132 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 5 monclient: authenticate success, global_id 4352 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38cc0018b0 con 0x7f38d813b980 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 1 -- 172.21.15.76:0/3456789611 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38cc001c40 con 0x7f38d813b980 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 4 set_mon_vals no callback set 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d5ffb700 10 monclient: dump: 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 10 monclient: shutdown 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- 172.21.15.76:0/3456789611 >> v1:172.21.15.76:6790/0 conn(0x7f38d813b980 legacy=0x7f38d813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- 172.21.15.76:0/3456789611 shutdown_connections 2024-09-16T06:38:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- 172.21.15.76:0/3456789611 >> 172.21.15.76:0/3456789611 conn(0x7f38d80a0030 msgr2=0x7f38d80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- 172.21.15.76:0/3456789611 shutdown_connections 2024-09-16T06:38:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- 172.21.15.76:0/3456789611 wait complete. 2024-09-16T06:38:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 10 monclient: monmap: 2024-09-16T06:38:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:32.145 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:32.145 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.145 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:32.145 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.145 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 Processor -- start 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 1 -- start start 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.117+0000 7f38d7fff700 10 monclient: init 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f38d8070ac0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f38d8070ea0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f38d813b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8138b60 con 0x7f38d813b980 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8137960 con 0x7f38d8070ac0 2024-09-16T06:38:32.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d8137f90 con 0x7f38d8070ea0 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d7fff700 10 monclient(hunting): authenticate will time out at 859.256508s 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f38d813b980 0x7f38d81d2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60784/0 (socket says 172.21.15.76:60784) 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.121+0000 7f38d77fe700 1 -- 172.21.15.76:0/3249749666 learned_addr learned my addr 172.21.15.76:0/3249749666 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088861407 0 0) 0x7f38d8138b60 con 0x7f38d813b980 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): my global_id is 4356 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c40035d0 con 0x7f38d813b980 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230950529 0 0) 0x7f38d8137960 con 0x7f38d8070ac0 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): my global_id is 4354 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d8138b60 con 0x7f38d8070ac0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365171872 0 0) 0x7f38d8137f90 con 0x7f38d8070ea0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): my global_id is 4355 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d8137960 con 0x7f38d8070ea0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439581421 0 0) 0x7f38d8137960 con 0x7f38d8070ea0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d8137f90 con 0x7f38d8070ea0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38d0001af0 con 0x7f38d8070ea0 2024-09-16T06:38:32.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.149 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 914466724 0 0) 0x7f38c40035d0 con 0x7f38d813b980 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d8137960 con 0x7f38d813b980 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38cc001bc0 con 0x7f38d813b980 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.151 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1148805298 0 0) 0x7f38d8137f90 con 0x7f38d8070ea0 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 >> v1:172.21.15.189:6789/0 conn(0x7f38d8070ac0 legacy=0x7f38d81ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 >> v1:172.21.15.76:6789/0 conn(0x7f38d813b980 legacy=0x7f38d81d2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 monclient: found mon.c 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d81cb4a0 con 0x7f38d8070ea0 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 5 monclient: authenticate success, global_id 4355 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _renew_subs 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d813b570 con 0x7f38d8070ea0 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38d0002080 con 0x7f38d8070ea0 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _renew_subs 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d8139ca0 con 0x7f38d8070ea0 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38d0003bc0 con 0x7f38d8070ea0 2024-09-16T06:38:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d4ff9700 4 set_mon_vals no callback set 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 10 monclient: dump: 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f38d0003de0 con 0x7f38d8070ea0 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f38d0042130 con 0x7f38d8070ea0 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.133+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d813e530 con 0x7f38d8070ea0 2024-09-16T06:38:32.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.137+0000 7f38d6ffd700 10 monclient: get_auth_request con 0x7f38c40366f0 auth_method 0 2024-09-16T06:38:32.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.145+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f38d00137f0 con 0x7f38d8070ea0 2024-09-16T06:38:32.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.145+0000 7f38bf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.145+0000 7f38bf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.169+0000 7f2da48bf700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.169+0000 7f2da48bf700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.169+0000 7f2da48bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.169+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2da011fbd0 con 0x7f2da006cf50 2024-09-16T06:38:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2d9cff9700 1 -- 172.21.15.76:0/1525173697 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7f2d900144c0 con 0x7f2da006cf50 2024-09-16T06:38:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2d9cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T06:38:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2d9cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.175 INFO:teuthology.orchestra.run.smithi076.stdout:51539607557 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2d80036450 legacy=0x7f2d80038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 10 monclient: shutdown 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 >> v1:172.21.15.76:6790/0 conn(0x7f2da006cf50 legacy=0x7f2da009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 shutdown_connections 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 >> 172.21.15.76:0/1525173697 conn(0x7f2da00a41c0 msgr2=0x7f2da0070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 shutdown_connections 2024-09-16T06:38:32.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.173+0000 7f2da48bf700 1 -- 172.21.15.76:0/1525173697 wait complete. 2024-09-16T06:38:32.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-16T06:38:32.186 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f312c0053b0 con 0x7f314406cf50 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f313a7fc700 1 -- 172.21.15.76:0/1017391307 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 847495621) 0x7f3140012b40 con 0x7f314406cf50 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f313a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f313a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.194 INFO:teuthology.orchestra.run.smithi076.stdout:47244640259 2024-09-16T06:38:32.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3128036660 legacy=0x7f3128038b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 10 monclient: shutdown 2024-09-16T06:38:32.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 >> v1:172.21.15.76:6790/0 conn(0x7f314406cf50 legacy=0x7f3144135280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.193+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 shutdown_connections 2024-09-16T06:38:32.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.197+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 >> 172.21.15.76:0/1017391307 conn(0x7f31440a41c0 msgr2=0x7f31440a1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.197+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 shutdown_connections 2024-09-16T06:38:32.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.197+0000 7f314abf5700 1 -- 172.21.15.76:0/1017391307 wait complete. 2024-09-16T06:38:32.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-16T06:38:32.212 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0bb8c2f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0bb8c2f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0bb8c2f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0b940053b0 con 0x7f0bb413fe80 2024-09-16T06:38:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0b9bfff700 1 -- 172.21.15.76:0/1244668516 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 532354816) 0x7f0bac011bc0 con 0x7f0bb413fe80 2024-09-16T06:38:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0b9bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T06:38:32.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7f0b9bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.263 INFO:teuthology.orchestra.run.smithi076.stdout:51539607557 2024-09-16T06:38:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff894966700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff894966700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff894966700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff8700053b0 con 0x7ff89009d560 2024-09-16T06:38:32.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff8777fe700 1 -- 172.21.15.76:0/2739324493 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 3056272221) 0x7ff88001cb50 con 0x7ff89009d560 2024-09-16T06:38:32.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff8777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T06:38:32.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.261+0000 7ff8777fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.265 INFO:teuthology.orchestra.run.smithi076.stdout:55834574852 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff8780366e0 legacy=0x7ff878038ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 10 monclient: shutdown 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 >> v1:172.21.15.76:6789/0 conn(0x7ff89009d560 legacy=0x7ff8901c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 shutdown_connections 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 >> 172.21.15.76:0/2739324493 conn(0x7ff8900a0030 msgr2=0x7ff89009c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 shutdown_connections 2024-09-16T06:38:32.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7ff894966700 1 -- 172.21.15.76:0/2739324493 wait complete. 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ba00365b0 legacy=0x7f0ba0038a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 10 monclient: shutdown 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 >> v1:172.21.15.76:6790/0 conn(0x7f0bb413fe80 legacy=0x7f0bb41d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 shutdown_connections 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 >> 172.21.15.76:0/1244668516 conn(0x7f0bb40a0030 msgr2=0x7f0bb4142970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 shutdown_connections 2024-09-16T06:38:32.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.265+0000 7f0bb8c2f700 1 -- 172.21.15.76:0/1244668516 wait complete. 2024-09-16T06:38:32.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.1 2024-09-16T06:38:32.279 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.279 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.0 2024-09-16T06:38:32.279 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff56aa00700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:32.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff56aa00700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:32.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff56aa00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff54c0053b0 con 0x7ff5640999f0 2024-09-16T06:38:32.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff55a7fc700 1 -- 172.21.15.76:0/2666621671 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 2194451396) 0x7ff55c014440 con 0x7ff5640999f0 2024-09-16T06:38:32.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff55a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T06:38:32.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.341+0000 7ff55a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.347 INFO:teuthology.orchestra.run.smithi076.stdout:55834574851 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff54803aa80 legacy=0x7ff54803cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 10 monclient: shutdown 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 >> v1:172.21.15.76:6790/0 conn(0x7ff5640999f0 legacy=0x7ff56412bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 shutdown_connections 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 >> 172.21.15.76:0/2666621671 conn(0x7ff5640a0030 msgr2=0x7ff564130e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 shutdown_connections 2024-09-16T06:38:32.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.345+0000 7ff56aa00700 1 -- 172.21.15.76:0/2666621671 wait complete. 2024-09-16T06:38:32.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574851 for osd.4 2024-09-16T06:38:32.358 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38d7fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38d7fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f38d8134060 con 0x7f38d8070ea0 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38bf7fe700 1 -- 172.21.15.76:0/3249749666 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f38d00133e0 con 0x7f38d8070ea0 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38bf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T06:38:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.389+0000 7f38bf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:32.394 INFO:teuthology.orchestra.run.smithi076.stdout:51539607555 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 >> v1:172.21.15.189:6812/3098810674 conn(0x7f38c40366f0 legacy=0x7f38c4038bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 10 monclient: shutdown 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 >> v1:172.21.15.76:6790/0 conn(0x7f38d8070ea0 legacy=0x7f38d81cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 shutdown_connections 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 >> 172.21.15.76:0/3249749666 conn(0x7f38d80a0030 msgr2=0x7f38d8135da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 shutdown_connections 2024-09-16T06:38:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.393+0000 7f38d7fff700 1 -- 172.21.15.76:0/3249749666 wait complete. 2024-09-16T06:38:32.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.2 2024-09-16T06:38:32.415 DEBUG:teuthology.parallel:result is None 2024-09-16T06:38:32.415 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T06:38:32.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:38:32.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:32.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: monmap: 2024-09-16T06:38:32.519 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.519 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:32.521492+0000 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:32.521492+0000 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:32.520 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 1 Processor -- start 2024-09-16T06:38:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 1 -- start start 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: init 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fc13801f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fc138025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fc138025860 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13801f8a0 con 0x7fc138025410 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1380233f0 con 0x7fc138025860 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc138020050 con 0x7fc13801f4c0 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient(hunting): authenticate will time out at 859.654763s 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc138025410 0x7fc1380bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60800/0 (socket says 172.21.15.76:60800) 2024-09-16T06:38:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13e7fc700 1 -- 172.21.15.76:0/1285695067 learned_addr learned my addr 172.21.15.76:0/1285695067 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821716021 0 0) 0x7fc13801f8a0 con 0x7fc138025410 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): my global_id is 4359 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1280035d0 con 0x7fc138025410 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235107206 0 0) 0x7fc138020050 con 0x7fc13801f4c0 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): my global_id is 4358 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc13801f8a0 con 0x7fc13801f4c0 2024-09-16T06:38:32.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408430069 0 0) 0x7fc1380233f0 con 0x7fc138025860 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient(hunting): my global_id is 4357 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc138020050 con 0x7fc138025860 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169025406 0 0) 0x7fc1280035d0 con 0x7fc138025410 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 >> v1:172.21.15.76:6790/0 conn(0x7fc13801f4c0 legacy=0x7fc138024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 >> v1:172.21.15.189:6789/0 conn(0x7fc138025860 legacy=0x7fc1380c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 monclient: found mon.noname-a 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138021a90 con 0x7fc138025410 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 5 monclient: authenticate success, global_id 4359 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc130001610 con 0x7fc138025410 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: dump: 2024-09-16T06:38:32.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc130001880 con 0x7fc138025410 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 1 -- 172.21.15.76:0/1285695067 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc130001c10 con 0x7fc138025410 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13ffff700 4 set_mon_vals no callback set 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.517+0000 7fc13dffb700 10 monclient: dump: 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: shutdown 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/1285695067 >> v1:172.21.15.76:6789/0 conn(0x7fc138025410 legacy=0x7fc1380bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/1285695067 shutdown_connections 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/1285695067 >> 172.21.15.76:0/1285695067 conn(0x7fc138005b40 msgr2=0x7fc138005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/1285695067 shutdown_connections 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/1285695067 wait complete. 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: monmap: 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:32.530 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 Processor -- start 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- start start 2024-09-16T06:38:32.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: init 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fc13801f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fc138025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fc138025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13801d2b0 con 0x7fc138025410 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13812f410 con 0x7fc13801f4c0 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1380204a0 con 0x7fc138025860 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient(hunting): authenticate will time out at 859.658717s 2024-09-16T06:38:32.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc138025410 0x7fc138155120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60804/0 (socket says 172.21.15.76:60804) 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13e7fc700 1 -- 172.21.15.76:0/299797753 learned_addr learned my addr 172.21.15.76:0/299797753 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349725354 0 0) 0x7fc1380204a0 con 0x7fc138025860 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): my global_id is 4361 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c0035d0 con 0x7fc138025860 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21320444 0 0) 0x7fc13801d2b0 con 0x7fc138025410 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): my global_id is 4362 2024-09-16T06:38:32.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1380204a0 con 0x7fc138025410 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962289734 0 0) 0x7fc13812f410 con 0x7fc13801f4c0 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): my global_id is 4360 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc13801d2b0 con 0x7fc13801f4c0 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383162916 0 0) 0x7fc1380204a0 con 0x7fc138025410 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc13812f410 con 0x7fc138025410 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc130001c50 con 0x7fc138025410 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): dump: 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024765655 0 0) 0x7fc12c0035d0 con 0x7fc138025860 2024-09-16T06:38:32.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1380204a0 con 0x7fc138025860 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc140069400 con 0x7fc138025860 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient(hunting): dump: 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.542 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2826554142 0 0) 0x7fc13812f410 con 0x7fc138025410 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 >> v1:172.21.15.76:6790/0 conn(0x7fc138025860 legacy=0x7fc138158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 >> v1:172.21.15.189:6789/0 conn(0x7fc13801f4c0 legacy=0x7fc13814be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 monclient: found mon.a 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138130f00 con 0x7fc138025410 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc1277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 5 monclient: authenticate success, global_id 4362 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _renew_subs 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13814c570 con 0x7fc138025410 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _renew_subs 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.521+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138130200 con 0x7fc138025410 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1300011b0 con 0x7fc138025410 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc130004df0 con 0x7fc138025410 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 10 monclient: dump: 2024-09-16T06:38:32.546 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.546 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.546 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.546 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13cff9700 4 set_mon_vals no callback set 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc130004fd0 con 0x7fc138025410 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fc1300433c0 con 0x7fc138025410 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc120005100 con 0x7fc138025410 2024-09-16T06:38:32.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.525+0000 7fc13effd700 10 monclient: get_auth_request con 0x7fc12c0363e0 auth_method 0 2024-09-16T06:38:32.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.529+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc130006640 con 0x7fc138025410 2024-09-16T06:38:32.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.529+0000 7fc1277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.529+0000 7fc1277fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:32.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.689+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc120000b90 con 0x7fc12c0363e0 2024-09-16T06:38:32.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc1277fe700 1 -- 172.21.15.76:0/299797753 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41036 (unknown 2965378022 0 2244622403) 0x7fc120000b90 con 0x7fc12c0363e0 2024-09-16T06:38:32.695 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc12c0363e0 legacy=0x7fc12c0388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 10 monclient: shutdown 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 >> v1:172.21.15.76:6789/0 conn(0x7fc138025410 legacy=0x7fc138155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 shutdown_connections 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 >> 172.21.15.76:0/299797753 conn(0x7fc138005b40 msgr2=0x7fc13801b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 shutdown_connections 2024-09-16T06:38:32.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.693+0000 7fc13ffff700 1 -- 172.21.15.76:0/299797753 wait complete. 2024-09-16T06:38:32.698 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:38:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-16T06:38:31.644439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167264,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063776,"statfs":{"total":579820584960,"available":579649306624,"internally_reserved":0,"allocated":995328,"data_stored":1327349,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111899+0000","last_change":"2024-09-16T06:38:29.111899+0000","last_active":"2024-09-16T06:38:29.111899+0000","last_peered":"2024-09-16T06:38:29.111899+0000","last_clean":"2024-09-16T06:38:29.111899+0000","last_became_active":"2024-09-16T06:38:27.099196+0000","last_became_peered":"2024-09-16T06:38:27.099196+0000","last_unstale":"2024-09-16T06:38:29.111899+0000","last_undegraded":"2024-09-16T06:38:29.111899+0000","last_fullsized":"2024-09-16T06:38:29.111899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:04:40.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.120221+0000","last_change":"2024-09-16T06:38:29.120221+0000","last_active":"2024-09-16T06:38:29.120221+0000","last_peered":"2024-09-16T06:38:29.120221+0000","last_clean":"2024-09-16T06:38:29.120221+0000","last_became_active":"2024-09-16T06:38:27.099931+0000","last_became_peered":"2024-09-16T06:38:27.099931+0000","last_unstale":"2024-09-16T06:38:29.120221+0000","last_undegraded":"2024-09-16T06:38:29.120221+0000","last_fullsized":"2024-09-16T06:38:29.120221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:50.958236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111921+0000","last_change":"2024-09-16T06:38:29.111921+0000","last_active":"2024-09-16T06:38:29.111921+0000","last_peered":"2024-09-16T06:38:29.111921+0000","last_clean":"2024-09-16T06:38:29.111921+0000","last_became_active":"2024-09-16T06:38:27.098425+0000","last_became_peered":"2024-09-16T06:38:27.098425+0000","last_unstale":"2024-09-16T06:38:29.111921+0000","last_undegraded":"2024-09-16T06:38:29.111921+0000","last_fullsized":"2024-09-16T06:38:29.111921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:17.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.480183+0000","last_change":"2024-09-16T06:38:29.480183+0000","last_active":"2024-09-16T06:38:29.480183+0000","last_peered":"2024-09-16T06:38:29.480183+0000","last_clean":"2024-09-16T06:38:29.480183+0000","last_became_active":"2024-09-16T06:38:27.101585+0000","last_became_peered":"2024-09-16T06:38:27.101585+0000","last_unstale":"2024-09-16T06:38:29.480183+0000","last_undegraded":"2024-09-16T06:38:29.480183+0000","last_fullsized":"2024-09-16T06:38:29.480183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:55.916580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.117904+0000","last_change":"2024-09-16T06:38:29.114236+0000","last_active":"2024-09-16T06:38:29.117904+0000","last_peered":"2024-09-16T06:38:29.117904+0000","last_clean":"2024-09-16T06:38:29.117904+0000","last_became_active":"2024-09-16T06:38:27.101165+0000","last_became_peered":"2024-09-16T06:38:27.101165+0000","last_unstale":"2024-09-16T06:38:29.117904+0000","last_undegraded":"2024-09-16T06:38:29.117904+0000","last_fullsized":"2024-09-16T06:38:29.117904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:57.229189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.478940+0000","last_change":"2024-09-16T06:38:29.479525+0000","last_active":"2024-09-16T06:38:29.478940+0000","last_peered":"2024-09-16T06:38:29.478940+0000","last_clean":"2024-09-16T06:38:29.478940+0000","last_became_active":"2024-09-16T06:38:27.099768+0000","last_became_peered":"2024-09-16T06:38:27.099768+0000","last_unstale":"2024-09-16T06:38:29.478940+0000","last_undegraded":"2024-09-16T06:38:29.478940+0000","last_fullsized":"2024-09-16T06:38:29.478940+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:39.212825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.113909+0000","last_change":"2024-09-16T06:38:29.113909+0000","last_active":"2024-09-16T06:38:29.113909+0000","last_peered":"2024-09-16T06:38:29.113909+0000","last_clean":"2024-09-16T06:38:29.113909+0000","last_became_active":"2024-09-16T06:38:27.099002+0000","last_became_peered":"2024-09-16T06:38:27.099002+0000","last_unstale":"2024-09-16T06:38:29.113909+0000","last_undegraded":"2024-09-16T06:38:29.113909+0000","last_fullsized":"2024-09-16T06:38:29.113909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:20.128665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112636+0000","last_change":"2024-09-16T06:38:29.112636+0000","last_active":"2024-09-16T06:38:29.112636+0000","last_peered":"2024-09-16T06:38:29.112636+0000","last_clean":"2024-09-16T06:38:29.112636+0000","last_became_active":"2024-09-16T06:38:27.097289+0000","last_became_peered":"2024-09-16T06:38:27.097289+0000","last_unstale":"2024-09-16T06:38:29.112636+0000","last_undegraded":"2024-09-16T06:38:29.112636+0000","last_fullsized":"2024-09-16T06:38:29.112636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:03.935631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112531+0000","last_change":"2024-09-16T06:38:27.100613+0000","last_active":"2024-09-16T06:38:29.112531+0000","last_peered":"2024-09-16T06:38:29.112531+0000","last_clean":"2024-09-16T06:38:29.112531+0000","last_became_active":"2024-09-16T06:38:27.100231+0000","last_became_peered":"2024-09-16T06:38:27.100231+0000","last_unstale":"2024-09-16T06:38:29.112531+0000","last_undegraded":"2024-09-16T06:38:29.112531+0000","last_fullsized":"2024-09-16T06:38:29.112531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:56:17.957577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":45105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":44044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:38:32.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:38:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 10 monclient: get_monmap_and_config 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 10 monclient: monmap: 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:32.822916+0000 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:32.822916+0000 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:32.821 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 1 Processor -- start 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 1 -- start start 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.817+0000 7f0ab6e86700 10 monclient: init 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-a con 0x7f0ab00a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-b con 0x7f0ab00a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-c con 0x7f0ab013b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab009b3e0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00a3450 con 0x7f0ab013b6b0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab009b1a0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient(hunting): authenticate will time out at 859.956224s 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab5683700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ab00a24a0 0x7f0ab00a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53146/0 (socket says 172.21.15.76:53146) 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab5683700 1 -- 172.21.15.76:0/1149647227 learned_addr learned my addr 172.21.15.76:0/1149647227 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268198453 0 0) 0x7f0ab009b1a0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): my global_id is 4364 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c0035d0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019259652 0 0) 0x7f0ab009b3e0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): my global_id is 4365 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab009b1a0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607442803 0 0) 0x7f0ab00a3450 con 0x7f0ab013b6b0 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient(hunting): my global_id is 4363 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab009b3e0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717558914 0 0) 0x7f0a9c0035d0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 >> v1:172.21.15.189:6789/0 conn(0x7f0ab013b6b0 legacy=0x7f0ab013daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 >> v1:172.21.15.76:6789/0 conn(0x7f0ab00a20c0 legacy=0x7f0ab006cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 monclient: found mon.noname-b 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab006bd30 con 0x7f0ab00a24a0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aac001610 con 0x7f0ab00a24a0 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: dump: 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 5 monclient: authenticate success, global_id 4364 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 1 -- 172.21.15.76:0/1149647227 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0aac001880 con 0x7f0ab00a24a0 2024-09-16T06:38:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab4e82700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 4 set_mon_vals no callback set 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: shutdown 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- 172.21.15.76:0/1149647227 >> v1:172.21.15.76:6790/0 conn(0x7f0ab00a24a0 legacy=0x7f0ab00a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- 172.21.15.76:0/1149647227 shutdown_connections 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- 172.21.15.76:0/1149647227 >> 172.21.15.76:0/1149647227 conn(0x7f0ab00a41c0 msgr2=0x7f0ab00a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- 172.21.15.76:0/1149647227 shutdown_connections 2024-09-16T06:38:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 1 -- 172.21.15.76:0/1149647227 wait complete. 2024-09-16T06:38:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: build_initial_monmap 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.821+0000 7f0ab6e86700 10 monclient: monmap: 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 Processor -- start 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- start start 2024-09-16T06:38:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: init 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-c con 0x7f0ab00a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-b con 0x7f0ab00a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient(hunting): picked mon.noname-a con 0x7f0ab013b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00a71a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab01396a0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab01ca6a0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient(hunting): authenticate will time out at 859.961793s 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6685700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0ab013b6b0 0x7f0ab01d1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60814/0 (socket says 172.21.15.76:60814) 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6685700 1 -- 172.21.15.76:0/2364692046 learned_addr learned my addr 172.21.15.76:0/2364692046 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007239244 0 0) 0x7f0ab01ca6a0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): my global_id is 4367 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a940035d0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506489206 0 0) 0x7f0aa8001630 con 0x7f0ab013b6b0 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): my global_id is 4368 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab01ca6a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405772771 0 0) 0x7f0ab01396a0 con 0x7f0ab00a24a0 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): my global_id is 4366 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa8001630 con 0x7f0ab00a24a0 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3340756672 0 0) 0x7f0ab01ca6a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ab01396a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa8001ca0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): dump: 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259266948 0 0) 0x7f0a940035d0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ab01ca6a0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa00018b0 con 0x7f0ab00a20c0 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient(hunting): dump: 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3943655600 0 0) 0x7f0ab01396a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 >> v1:172.21.15.76:6790/0 conn(0x7f0ab00a20c0 legacy=0x7f0ab01c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 >> v1:172.21.15.189:6789/0 conn(0x7f0ab00a24a0 legacy=0x7f0ab01c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 monclient: found mon.a 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab009ebe0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 5 monclient: authenticate success, global_id 4368 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _renew_subs 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab009e8b0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _renew_subs 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab0138690 con 0x7f0ab013b6b0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0aa8002030 con 0x7f0ab013b6b0 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa8003e90 con 0x7f0ab013b6b0 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa67fc700 10 monclient: dump: 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.825+0000 7f0aa7fff700 4 set_mon_vals no callback set 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0aa8004070 con 0x7f0ab013b6b0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0ab6e86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0ab6e86700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0ab6e86700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab013a630 con 0x7f0ab013b6b0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f0aa80436a0 con 0x7f0ab013b6b0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.829+0000 7f0ab5e84700 10 monclient: get_auth_request con 0x7f0a94036450 auth_method 0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.837+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0aa8014a20 con 0x7f0ab013b6b0 2024-09-16T06:38:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.837+0000 7f0aa67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:32.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:32.837+0000 7f0aa67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:33.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.001+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ab013a1c0 con 0x7f0a94036450 2024-09-16T06:38:33.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.005+0000 7f0aa67fc700 1 -- 172.21.15.76:0/2364692046 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41036 (unknown 2965378022 0 2244622403) 0x7f0ab013a1c0 con 0x7f0a94036450 2024-09-16T06:38:33.009 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:33.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0a94036450 legacy=0x7f0a94038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 10 monclient: shutdown 2024-09-16T06:38:33.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 >> v1:172.21.15.76:6789/0 conn(0x7f0ab013b6b0 legacy=0x7f0ab01d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 shutdown_connections 2024-09-16T06:38:33.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 >> 172.21.15.76:0/2364692046 conn(0x7f0ab00a41c0 msgr2=0x7f0ab00a5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 shutdown_connections 2024-09-16T06:38:33.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.009+0000 7f0ab6e86700 1 -- 172.21.15.76:0/2364692046 wait complete. 2024-09-16T06:38:33.012 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:38:33.024 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-16T06:38:31.644439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167264,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063776,"statfs":{"total":579820584960,"available":579649306624,"internally_reserved":0,"allocated":995328,"data_stored":1327349,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111899+0000","last_change":"2024-09-16T06:38:29.111899+0000","last_active":"2024-09-16T06:38:29.111899+0000","last_peered":"2024-09-16T06:38:29.111899+0000","last_clean":"2024-09-16T06:38:29.111899+0000","last_became_active":"2024-09-16T06:38:27.099196+0000","last_became_peered":"2024-09-16T06:38:27.099196+0000","last_unstale":"2024-09-16T06:38:29.111899+0000","last_undegraded":"2024-09-16T06:38:29.111899+0000","last_fullsized":"2024-09-16T06:38:29.111899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:04:40.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.120221+0000","last_change":"2024-09-16T06:38:29.120221+0000","last_active":"2024-09-16T06:38:29.120221+0000","last_peered":"2024-09-16T06:38:29.120221+0000","last_clean":"2024-09-16T06:38:29.120221+0000","last_became_active":"2024-09-16T06:38:27.099931+0000","last_became_peered":"2024-09-16T06:38:27.099931+0000","last_unstale":"2024-09-16T06:38:29.120221+0000","last_undegraded":"2024-09-16T06:38:29.120221+0000","last_fullsized":"2024-09-16T06:38:29.120221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:50.958236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111921+0000","last_change":"2024-09-16T06:38:29.111921+0000","last_active":"2024-09-16T06:38:29.111921+0000","last_peered":"2024-09-16T06:38:29.111921+0000","last_clean":"2024-09-16T06:38:29.111921+0000","last_became_active":"2024-09-16T06:38:27.098425+0000","last_became_peered":"2024-09-16T06:38:27.098425+0000","last_unstale":"2024-09-16T06:38:29.111921+0000","last_undegraded":"2024-09-16T06:38:29.111921+0000","last_fullsized":"2024-09-16T06:38:29.111921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:17.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.480183+0000","last_change":"2024-09-16T06:38:29.480183+0000","last_active":"2024-09-16T06:38:29.480183+0000","last_peered":"2024-09-16T06:38:29.480183+0000","last_clean":"2024-09-16T06:38:29.480183+0000","last_became_active":"2024-09-16T06:38:27.101585+0000","last_became_peered":"2024-09-16T06:38:27.101585+0000","last_unstale":"2024-09-16T06:38:29.480183+0000","last_undegraded":"2024-09-16T06:38:29.480183+0000","last_fullsized":"2024-09-16T06:38:29.480183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:55.916580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.117904+0000","last_change":"2024-09-16T06:38:29.114236+0000","last_active":"2024-09-16T06:38:29.117904+0000","last_peered":"2024-09-16T06:38:29.117904+0000","last_clean":"2024-09-16T06:38:29.117904+0000","last_became_active":"2024-09-16T06:38:27.101165+0000","last_became_peered":"2024-09-16T06:38:27.101165+0000","last_unstale":"2024-09-16T06:38:29.117904+0000","last_undegraded":"2024-09-16T06:38:29.117904+0000","last_fullsized":"2024-09-16T06:38:29.117904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:57.229189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.478940+0000","last_change":"2024-09-16T06:38:29.479525+0000","last_active":"2024-09-16T06:38:29.478940+0000","last_peered":"2024-09-16T06:38:29.478940+0000","last_clean":"2024-09-16T06:38:29.478940+0000","last_became_active":"2024-09-16T06:38:27.099768+0000","last_became_peered":"2024-09-16T06:38:27.099768+0000","last_unstale":"2024-09-16T06:38:29.478940+0000","last_undegraded":"2024-09-16T06:38:29.478940+0000","last_fullsized":"2024-09-16T06:38:29.478940+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:39.212825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.113909+0000","last_change":"2024-09-16T06:38:29.113909+0000","last_active":"2024-09-16T06:38:29.113909+0000","last_peered":"2024-09-16T06:38:29.113909+0000","last_clean":"2024-09-16T06:38:29.113909+0000","last_became_active":"2024-09-16T06:38:27.099002+0000","last_became_peered":"2024-09-16T06:38:27.099002+0000","last_unstale":"2024-09-16T06:38:29.113909+0000","last_undegraded":"2024-09-16T06:38:29.113909+0000","last_fullsized":"2024-09-16T06:38:29.113909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:20.128665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112636+0000","last_change":"2024-09-16T06:38:29.112636+0000","last_active":"2024-09-16T06:38:29.112636+0000","last_peered":"2024-09-16T06:38:29.112636+0000","last_clean":"2024-09-16T06:38:29.112636+0000","last_became_active":"2024-09-16T06:38:27.097289+0000","last_became_peered":"2024-09-16T06:38:27.097289+0000","last_unstale":"2024-09-16T06:38:29.112636+0000","last_undegraded":"2024-09-16T06:38:29.112636+0000","last_fullsized":"2024-09-16T06:38:29.112636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:03.935631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112531+0000","last_change":"2024-09-16T06:38:27.100613+0000","last_active":"2024-09-16T06:38:29.112531+0000","last_peered":"2024-09-16T06:38:29.112531+0000","last_clean":"2024-09-16T06:38:29.112531+0000","last_became_active":"2024-09-16T06:38:27.100231+0000","last_became_peered":"2024-09-16T06:38:27.100231+0000","last_unstale":"2024-09-16T06:38:29.112531+0000","last_undegraded":"2024-09-16T06:38:29.112531+0000","last_fullsized":"2024-09-16T06:38:29.112531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:56:17.957577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":45105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":44044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:38:33.025 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T06:38:33.025 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T06:38:33.026 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T06:38:33.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.129+0000 7fb5c7fff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.129+0000 7fb5c7fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.129+0000 7fb5c7fff700 10 monclient: monmap: 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:33.135129+0000 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:33.135129+0000 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:33.134 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 Processor -- start 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- start start 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: init 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb5c8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb5c806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb5c806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c806c280 con 0x7fb5c8132330 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c806bfa0 con 0x7fb5c806d480 2024-09-16T06:38:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c809ace0 con 0x7fb5c806cf50 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient(hunting): authenticate will time out at 860.268912s 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb5c806cf50 0x7fb5c813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53168/0 (socket says 172.21.15.76:53168) 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c77fe700 1 -- 172.21.15.76:0/2860464303 learned_addr learned my addr 172.21.15.76:0/2860464303 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002672296 0 0) 0x7fb5c806c280 con 0x7fb5c8132330 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): my global_id is 4371 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b40035d0 con 0x7fb5c8132330 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839655614 0 0) 0x7fb5c809ace0 con 0x7fb5c806cf50 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient(hunting): my global_id is 4370 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c806c280 con 0x7fb5c806cf50 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520181017 0 0) 0x7fb5b40035d0 con 0x7fb5c8132330 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 >> v1:172.21.15.76:6790/0 conn(0x7fb5c806cf50 legacy=0x7fb5c813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 >> v1:172.21.15.189:6789/0 conn(0x7fb5c806d480 legacy=0x7fb5c813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 monclient: found mon.noname-a 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c809ae70 con 0x7fb5c8132330 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 5 monclient: authenticate success, global_id 4371 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb5b8001610 con 0x7fb5c8132330 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: dump: 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.139 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5b80018b0 con 0x7fb5c8132330 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 1 -- 172.21.15.76:0/2860464303 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb5b8001aa0 con 0x7fb5c8132330 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 4 set_mon_vals no callback set 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c67fc700 10 monclient: dump: 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.141 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: shutdown 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- 172.21.15.76:0/2860464303 >> v1:172.21.15.76:6789/0 conn(0x7fb5c8132330 legacy=0x7fb5c8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- 172.21.15.76:0/2860464303 shutdown_connections 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- 172.21.15.76:0/2860464303 >> 172.21.15.76:0/2860464303 conn(0x7fb5c80a41c0 msgr2=0x7fb5c80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- 172.21.15.76:0/2860464303 shutdown_connections 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 1 -- 172.21.15.76:0/2860464303 wait complete. 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.133+0000 7fb5c7fff700 10 monclient: monmap: 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 Processor -- start 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- start start 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: init 2024-09-16T06:38:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb5c806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb5c806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb5c8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c809e920 con 0x7fb5c8132330 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c809e2f0 con 0x7fb5c806cf50 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c812ce00 con 0x7fb5c806d480 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient(hunting): authenticate will time out at 860.273063s 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5be7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb5c806d480 0x7fb5c81ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53172/0 (socket says 172.21.15.76:53172) 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb5c8132330 0x7fb5c81d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60824/0 (socket says 172.21.15.76:60824) 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5be7fc700 1 -- 172.21.15.76:0/1588392582 learned_addr learned my addr 172.21.15.76:0/1588392582 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701726793 0 0) 0x7fb5c812ce00 con 0x7fb5c806d480 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): my global_id is 4373 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a80035d0 con 0x7fb5c806d480 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334281368 0 0) 0x7fb5c806c280 con 0x7fb5c8132330 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): my global_id is 4374 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c812ce00 con 0x7fb5c8132330 2024-09-16T06:38:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936945990 0 0) 0x7fb5c809e2f0 con 0x7fb5c806cf50 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): my global_id is 4372 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c806c280 con 0x7fb5c806cf50 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122314938 0 0) 0x7fb5a80035d0 con 0x7fb5c806d480 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5c809e2f0 con 0x7fb5c806d480 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb5b00019c0 con 0x7fb5c806d480 2024-09-16T06:38:33.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): dump: 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:33.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180524912 0 0) 0x7fb5c812ce00 con 0x7fb5c8132330 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a80035d0 con 0x7fb5c8132330 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb5c0001a30 con 0x7fb5c8132330 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient(hunting): dump: 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.154 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 426755342 0 0) 0x7fb5c809e2f0 con 0x7fb5c806d480 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 >> v1:172.21.15.189:6789/0 conn(0x7fb5c806cf50 legacy=0x7fb5c81c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 >> v1:172.21.15.76:6789/0 conn(0x7fb5c8132330 legacy=0x7fb5c81d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 monclient: found mon.c 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c812d9b0 con 0x7fb5c806d480 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 5 monclient: authenticate success, global_id 4373 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _renew_subs 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c8130dc0 con 0x7fb5c806d480 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _renew_subs 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c81c7120 con 0x7fb5c806d480 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5b0001d30 con 0x7fb5c806d480 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb5b0003b90 con 0x7fb5c806d480 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c5ffb700 4 set_mon_vals no callback set 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 10 monclient: dump: 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb5b0003cf0 con 0x7fb5c806d480 2024-09-16T06:38:33.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac005100 con 0x7fb5c806d480 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fb5b0043280 con 0x7fb5c806d480 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.137+0000 7fb5c6ffd700 10 monclient: get_auth_request con 0x7fb5a8036450 auth_method 0 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.145+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb5b0014830 con 0x7fb5c806d480 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.145+0000 7fb5bffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.145+0000 7fb5bffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:33.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5c7fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T06:38:33.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5c7fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T06:38:33.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5ac005930 con 0x7fb5c806d480 2024-09-16T06:38:33.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5bffff700 1 -- 172.21.15.76:0/1588392582 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fb5b0012ca0 con 0x7fb5c806d480 2024-09-16T06:38:33.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5bffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T06:38:33.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.333+0000 7fb5bffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:33.337 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:33.337 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T06:38:33.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb5a8036450 legacy=0x7fb5a8038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 10 monclient: shutdown 2024-09-16T06:38:33.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 >> v1:172.21.15.76:6790/0 conn(0x7fb5c806d480 legacy=0x7fb5c81ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 shutdown_connections 2024-09-16T06:38:33.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 >> 172.21.15.76:0/1588392582 conn(0x7fb5c80a41c0 msgr2=0x7fb5c8129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 shutdown_connections 2024-09-16T06:38:33.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.337+0000 7fb5c7fff700 1 -- 172.21.15.76:0/1588392582 wait complete. 2024-09-16T06:38:33.357 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T06:38:33.360 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T06:38:33.414 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T06:38:33.414 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: monmap: 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:33.529073+0000 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:33.529073+0000 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:33.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 1 Processor -- start 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 1 -- start start 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: init 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd7c40a20c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd7c40a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd7c413b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c409b3e0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40a3450 con 0x7fd7c40a20c0 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c409b1a0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c359e700 10 monclient(hunting): authenticate will time out at 860.663292s 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7bbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd7c40a24a0 0x7fd7c40a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53176/0 (socket says 172.21.15.76:53176) 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7bbfff700 1 -- 172.21.15.76:0/934942805 learned_addr learned my addr 172.21.15.76:0/934942805 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859361375 0 0) 0x7fd7c409b1a0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 10 monclient(hunting): my global_id is 4376 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.525+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b00035d0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253951058 0 0) 0x7fd7c409b3e0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): my global_id is 4377 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c409b1a0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874973767 0 0) 0x7fd7c40a3450 con 0x7fd7c40a20c0 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient(hunting): my global_id is 4375 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c409b3e0 con 0x7fd7c40a20c0 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240204027 0 0) 0x7fd7b00035d0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 >> v1:172.21.15.189:6789/0 conn(0x7fd7c40a20c0 legacy=0x7fd7c406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 >> v1:172.21.15.76:6789/0 conn(0x7fd7c413b6b0 legacy=0x7fd7c413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 monclient: found mon.noname-b 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c406bd30 con 0x7fd7c40a24a0 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd7ac001610 con 0x7fd7c40a24a0 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: dump: 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 5 monclient: authenticate success, global_id 4376 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7ac001830 con 0x7fd7c40a24a0 2024-09-16T06:38:33.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 1 -- 172.21.15.76:0/934942805 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd7ac001b80 con 0x7fd7c40a24a0 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 4 set_mon_vals no callback set 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c1d9b700 10 monclient: dump: 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: shutdown 2024-09-16T06:38:33.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- 172.21.15.76:0/934942805 >> v1:172.21.15.76:6790/0 conn(0x7fd7c40a24a0 legacy=0x7fd7c40a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- 172.21.15.76:0/934942805 shutdown_connections 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- 172.21.15.76:0/934942805 >> 172.21.15.76:0/934942805 conn(0x7fd7c40a41c0 msgr2=0x7fd7c40a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- 172.21.15.76:0/934942805 shutdown_connections 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- 172.21.15.76:0/934942805 wait complete. 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: monmap: 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.543 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 Processor -- start 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 1 -- start start 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: init 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd7c40a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd7c40a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.529+0000 7fd7c359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd7c413b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c413a470 con 0x7fd7c413b6b0 2024-09-16T06:38:33.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c41374a0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c41391c0 con 0x7fd7c40a20c0 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient(hunting): authenticate will time out at 860.668030s 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c2d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd7c413b6b0 0x7fd7c41d1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60836/0 (socket says 172.21.15.76:60836) 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd7c40a20c0 0x7fd7c4136680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53178/0 (socket says 172.21.15.76:53178) 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c2d9d700 1 -- 172.21.15.76:0/1685979777 learned_addr learned my addr 172.21.15.76:0/1685979777 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910011535 0 0) 0x7fd7c413a470 con 0x7fd7c413b6b0 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): my global_id is 4380 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a40035d0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349290624 0 0) 0x7fd7c41391c0 con 0x7fd7c40a20c0 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): my global_id is 4379 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c413a470 con 0x7fd7c40a20c0 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057985063 0 0) 0x7fd7c41374a0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): my global_id is 4378 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c41391c0 con 0x7fd7c40a24a0 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1167363144 0 0) 0x7fd7a40035d0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c41374a0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd7bc001d50 con 0x7fd7c413b6b0 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): dump: 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.547 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1977653250 0 0) 0x7fd7c413a470 con 0x7fd7c40a20c0 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a40035d0 con 0x7fd7c40a20c0 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd7b40018d0 con 0x7fd7c40a20c0 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient(hunting): dump: 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3571658636 0 0) 0x7fd7c41374a0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 >> v1:172.21.15.76:6790/0 conn(0x7fd7c40a20c0 legacy=0x7fd7c4136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 >> v1:172.21.15.189:6789/0 conn(0x7fd7c40a24a0 legacy=0x7fd7c4136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 monclient: found mon.a 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c41394a0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 5 monclient: authenticate success, global_id 4380 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _renew_subs 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c409e8b0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _renew_subs 2024-09-16T06:38:33.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c409dcc0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7bc002070 con 0x7fd7c413b6b0 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd7bc003e70 con 0x7fd7c413b6b0 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 10 monclient: dump: 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.552 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c0d99700 4 set_mon_vals no callback set 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd7bc0040d0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fd7bc0434f0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c41399c0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.533+0000 7fd7c259c700 10 monclient: get_auth_request con 0x7fd7a4036440 auth_method 0 2024-09-16T06:38:33.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.541+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd7bc0149b0 con 0x7fd7c413b6b0 2024-09-16T06:38:33.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.541+0000 7fd7baffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.541+0000 7fd7baffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:33.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7c359e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:33.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7c359e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:33.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:33.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7c4139690 con 0x7fd7c413b6b0 2024-09-16T06:38:33.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7baffd700 1 -- 172.21.15.76:0/1685979777 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+10832 (unknown 3225238715 0 4229994956) 0x7fd7bc013050 con 0x7fd7c413b6b0 2024-09-16T06:38:33.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7baffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:38:33.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.713+0000 7fd7baffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:33.717 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:33.717 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:38:29.102711+0000","last_up_change":"2024-09-16T06:38:25.048660+0000","last_in_change":"2024-09-16T06:38:17.703691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T06:38:25.702868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T06:38:25.996213+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6808","nonce":3838562525}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6809","nonce":3838562525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6811","nonce":3838562525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6810","nonce":3838562525}]},"public_addr":"172.21.15.76:6808/3838562525","cluster_addr":"172.21.15.76:6809/3838562525","heartbeat_back_addr":"172.21.15.76:6811/3838562525","heartbeat_front_addr":"172.21.15.76:6810/3838562525","state":["exists","up"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6800","nonce":3502405955}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6801","nonce":3502405955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6803","nonce":3502405955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6802","nonce":3502405955}]},"public_addr":"172.21.15.76:6800/3502405955","cluster_addr":"172.21.15.76:6801/3502405955","heartbeat_back_addr":"172.21.15.76:6803/3502405955","heartbeat_front_addr":"172.21.15.76:6802/3502405955","state":["exists","up"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6804","nonce":779009499}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6805","nonce":779009499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6807","nonce":779009499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6806","nonce":779009499}]},"public_addr":"172.21.15.76:6804/779009499","cluster_addr":"172.21.15.76:6805/779009499","heartbeat_back_addr":"172.21.15.76:6807/779009499","heartbeat_front_addr":"172.21.15.76:6806/779009499","state":["exists","up"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3809009238}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3809009238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3809009238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3809009238}]},"public_addr":"172.21.15.189:6808/3809009238","cluster_addr":"172.21.15.189:6809/3809009238","heartbeat_back_addr":"172.21.15.189:6811/3809009238","heartbeat_front_addr":"172.21.15.189:6810/3809009238","state":["exists","up"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":407912501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":407912501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":407912501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":407912501}]},"public_addr":"172.21.15.189:6804/407912501","cluster_addr":"172.21.15.189:6805/407912501","heartbeat_back_addr":"172.21.15.189:6807/407912501","heartbeat_front_addr":"172.21.15.189:6806/407912501","state":["exists","up"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":2128366135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":2128366135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":2128366135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":2128366135}]},"public_addr":"172.21.15.189:6800/2128366135","cluster_addr":"172.21.15.189:6801/2128366135","heartbeat_back_addr":"172.21.15.189:6803/2128366135","heartbeat_front_addr":"172.21.15.189:6802/2128366135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.209148+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.408945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.850780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:21.901722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.166044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.780816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:38:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd7a4036440 legacy=0x7fd7a4038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 10 monclient: shutdown 2024-09-16T06:38:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 >> v1:172.21.15.76:6789/0 conn(0x7fd7c413b6b0 legacy=0x7fd7c41d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 shutdown_connections 2024-09-16T06:38:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 >> 172.21.15.76:0/1685979777 conn(0x7fd7c40a41c0 msgr2=0x7fd7c413abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 shutdown_connections 2024-09-16T06:38:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.717+0000 7fd7c359e700 1 -- 172.21.15.76:0/1685979777 wait complete. 2024-09-16T06:38:33.731 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T06:38:25.702868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T06:38:25.996213+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 17, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T06:38:33.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: get_monmap_and_config 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: monmap: 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:33.846352+0000 2024-09-16T06:38:33.844 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:33.846352+0000 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:33.845 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 1 Processor -- start 2024-09-16T06:38:33.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 1 -- start start 2024-09-16T06:38:33.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: init 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-c con 0x7fe424097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-a con 0x7fe424099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-b con 0x7fe424139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe424098990 con 0x7fe424099090 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe424097f40 con 0x7fe424097b60 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42409d610 con 0x7fe424139500 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42af7f700 10 monclient(hunting): authenticate will time out at 860.979627s 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42977c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe424099090 0x7fe42409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60848/0 (socket says 172.21.15.76:60848) 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.841+0000 7fe42977c700 1 -- 172.21.15.76:0/1892977285 learned_addr learned my addr 172.21.15.76:0/1892977285 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309670351 0 0) 0x7fe42409d610 con 0x7fe424139500 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): my global_id is 4382 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4100035d0 con 0x7fe424139500 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828783765 0 0) 0x7fe424098990 con 0x7fe424099090 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): my global_id is 4383 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42409d610 con 0x7fe424099090 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280159729 0 0) 0x7fe424097f40 con 0x7fe424097b60 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient(hunting): my global_id is 4381 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe424098990 con 0x7fe424097b60 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996995732 0 0) 0x7fe4100035d0 con 0x7fe424139500 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 >> v1:172.21.15.189:6789/0 conn(0x7fe424097b60 legacy=0x7fe42409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 >> v1:172.21.15.76:6789/0 conn(0x7fe424099090 legacy=0x7fe42409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 monclient: found mon.noname-b 2024-09-16T06:38:33.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240986b0 con 0x7fe424139500 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 5 monclient: authenticate success, global_id 4382 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe41c001990 con 0x7fe424139500 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: dump: 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.856 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe41c001c70 con 0x7fe424139500 2024-09-16T06:38:33.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 4 set_mon_vals no callback set 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 1 -- 172.21.15.76:0/1892977285 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe41c002000 con 0x7fe424139500 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe428f7b700 10 monclient: dump: 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.858 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: shutdown 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- 172.21.15.76:0/1892977285 >> v1:172.21.15.76:6790/0 conn(0x7fe424139500 legacy=0x7fe42413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- 172.21.15.76:0/1892977285 shutdown_connections 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- 172.21.15.76:0/1892977285 >> 172.21.15.76:0/1892977285 conn(0x7fe4240a0030 msgr2=0x7fe4240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- 172.21.15.76:0/1892977285 shutdown_connections 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- 172.21.15.76:0/1892977285 wait complete. 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: build_initial_monmap 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: monmap: 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 Processor -- start 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- start start 2024-09-16T06:38:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: init 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-b con 0x7fe424097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-c con 0x7fe424099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient(hunting): picked mon.noname-a con 0x7fe424139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240906a0 con 0x7fe424139500 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240a2c30 con 0x7fe424097b60 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42409a540 con 0x7fe424099090 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42af7f700 10 monclient(hunting): authenticate will time out at 860.983142s 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42977c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe424099090 0x7fe4241ca690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53194/0 (socket says 172.21.15.76:53194) 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42977c700 1 -- 172.21.15.76:0/1732315592 learned_addr learned my addr 172.21.15.76:0/1732315592 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe42a77e700 1 --1- 172.21.15.76:0/1732315592 >> v1:172.21.15.76:6789/0 conn(0x7fe424139500 0x7fe4241cddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60860/0 (socket says 172.21.15.76:60860) 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323177610 0 0) 0x7fe4240986b0 con 0x7fe424139500 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): my global_id is 4386 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4080035d0 con 0x7fe424139500 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958350487 0 0) 0x7fe42409a540 con 0x7fe424099090 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 10 monclient(hunting): my global_id is 4385 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.845+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4240986b0 con 0x7fe424099090 2024-09-16T06:38:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 205576329 0 0) 0x7fe4240986b0 con 0x7fe424099090 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe42409a540 con 0x7fe424099090 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe420001870 con 0x7fe424099090 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1581218363 0 0) 0x7fe4080035d0 con 0x7fe424139500 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4240986b0 con 0x7fe424139500 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe41c001de0 con 0x7fe424139500 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient(hunting): dump: 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1660136349 0 0) 0x7fe42409a540 con 0x7fe424099090 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 >> v1:172.21.15.189:6789/0 conn(0x7fe424097b60 legacy=0x7fe424096140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 >> v1:172.21.15.76:6789/0 conn(0x7fe424139500 legacy=0x7fe4241cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 monclient: found mon.c 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42409c3e0 con 0x7fe424099090 2024-09-16T06:38:33.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 5 monclient: authenticate success, global_id 4385 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _renew_subs 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe424090bf0 con 0x7fe424099090 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _renew_subs 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42408e890 con 0x7fe424099090 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe420004890 con 0x7fe424099090 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe420004c20 con 0x7fe424099090 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41bfff700 4 set_mon_vals no callback set 2024-09-16T06:38:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 10 monclient: dump: 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe420004d80 con 0x7fe424099090 2024-09-16T06:38:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fe420043260 con 0x7fe424099090 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe429f7d700 10 monclient: get_auth_request con 0x7fe408036310 auth_method 0 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.849+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c005100 con 0x7fe424099090 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.857+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe420005550 con 0x7fe424099090 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.857+0000 7fe41a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:33.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:33.857+0000 7fe41a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe42af7f700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe42af7f700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe42af7f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe40c005930 con 0x7fe424099090 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe41a7fc700 1 -- 172.21.15.76:0/1732315592 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (unknown 3971575918 0 2170607528) 0x7fe420012c80 con 0x7fe424099090 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe41a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T06:38:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.025+0000 7fe41a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:34.029 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-09-16T06:38:34.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe408036310 legacy=0x7fe4080387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 10 monclient: shutdown 2024-09-16T06:38:34.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 >> v1:172.21.15.76:6790/0 conn(0x7fe424099090 legacy=0x7fe4241ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 shutdown_connections 2024-09-16T06:38:34.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 >> 172.21.15.76:0/1732315592 conn(0x7fe4240a0030 msgr2=0x7fe4240a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 shutdown_connections 2024-09-16T06:38:34.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.029+0000 7fe42af7f700 1 -- 172.21.15.76:0/1732315592 wait complete. 2024-09-16T06:38:34.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T06:38:34.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: get_monmap_and_config 2024-09-16T06:38:34.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: monmap: 2024-09-16T06:38:34.151 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.151 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:34.153518+0000 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:34.153518+0000 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:34.152 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 1 Processor -- start 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 1 -- start start 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: init 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-c con 0x7fecb0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-a con 0x7fecb006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-b con 0x7fecb006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb006c280 con 0x7fecb006d480 2024-09-16T06:38:34.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb006bfa0 con 0x7fecb0132330 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb009ace0 con 0x7fecb006cf50 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb5770700 10 monclient(hunting): authenticate will time out at 861.287399s 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb4f6f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fecb006cf50 0x7fecb013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41870/0 (socket says 172.21.15.76:41870) 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecb4f6f700 1 -- 172.21.15.76:0/722024042 learned_addr learned my addr 172.21.15.76:0/722024042 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935093257 0 0) 0x7fecb009ace0 con 0x7fecb006cf50 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 10 monclient(hunting): my global_id is 4388 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.149+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0035d0 con 0x7fecb006cf50 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934236546 0 0) 0x7fecb006c280 con 0x7fecb006d480 2024-09-16T06:38:34.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): my global_id is 4389 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb009ace0 con 0x7fecb006d480 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654606749 0 0) 0x7fecb006bfa0 con 0x7fecb0132330 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient(hunting): my global_id is 4387 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb006c280 con 0x7fecb0132330 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2513941332 0 0) 0x7fec9c0035d0 con 0x7fecb006cf50 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 >> v1:172.21.15.189:6789/0 conn(0x7fecb0132330 legacy=0x7fecb0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 >> v1:172.21.15.76:6789/0 conn(0x7fecb006d480 legacy=0x7fecb013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 monclient: found mon.noname-b 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb009ae70 con 0x7fecb006cf50 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feca8001990 con 0x7fecb006cf50 2024-09-16T06:38:34.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: dump: 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 5 monclient: authenticate success, global_id 4388 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feca8001bb0 con 0x7fecb006cf50 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 1 -- 172.21.15.76:0/722024042 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feca8001da0 con 0x7fecb006cf50 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 4 set_mon_vals no callback set 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaeffd700 10 monclient: dump: 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: shutdown 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- 172.21.15.76:0/722024042 >> v1:172.21.15.76:6790/0 conn(0x7fecb006cf50 legacy=0x7fecb013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- 172.21.15.76:0/722024042 shutdown_connections 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- 172.21.15.76:0/722024042 >> 172.21.15.76:0/722024042 conn(0x7fecb00a41c0 msgr2=0x7fecb00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- 172.21.15.76:0/722024042 shutdown_connections 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- 172.21.15.76:0/722024042 wait complete. 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: monmap: 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:34.164 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 Processor -- start 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- start start 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: init 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-a con 0x7fecb006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-b con 0x7fecb006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient(hunting): picked mon.noname-c con 0x7fecb0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb0136ea0 con 0x7fecb006cf50 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb01378a0 con 0x7fecb006d480 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb009e8b0 con 0x7fecb0132330 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fecb006cf50 0x7fecb0135200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39268/0 (socket says 172.21.15.76:39268) 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecaffff700 1 -- 172.21.15.76:0/2431735079 learned_addr learned my addr 172.21.15.76:0/2431735079 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.153+0000 7fecb5770700 10 monclient(hunting): authenticate will time out at 861.291886s 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860558852 0 0) 0x7fecb0136ea0 con 0x7fecb006cf50 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): my global_id is 4392 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec900035d0 con 0x7fecb006cf50 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277655189 0 0) 0x7fecb009e8b0 con 0x7fecb0132330 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): my global_id is 4391 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0136ea0 con 0x7fecb0132330 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879408114 0 0) 0x7fecb01378a0 con 0x7fecb006d480 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): my global_id is 4390 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb009e8b0 con 0x7fecb006d480 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562179263 0 0) 0x7fec900035d0 con 0x7fecb006cf50 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb01378a0 con 0x7fecb006cf50 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feca0001990 con 0x7fecb006cf50 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): dump: 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.171 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3581261626 0 0) 0x7fecb0136ea0 con 0x7fecb0132330 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec900035d0 con 0x7fecb0132330 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feca8001d20 con 0x7fecb0132330 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient(hunting): dump: 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3431514056 0 0) 0x7fecb01378a0 con 0x7fecb006cf50 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 >> v1:172.21.15.76:6790/0 conn(0x7fecb0132330 legacy=0x7fecb01d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 >> v1:172.21.15.189:6789/0 conn(0x7fecb006d480 legacy=0x7fecb0135910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 monclient: found mon.a 2024-09-16T06:38:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb0139200 con 0x7fecb006cf50 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 5 monclient: authenticate success, global_id 4392 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _renew_subs 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb0136020 con 0x7fecb006cf50 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _renew_subs 2024-09-16T06:38:34.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb0131260 con 0x7fecb006cf50 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feca0001d40 con 0x7fecb006cf50 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feca0003b10 con 0x7fecb006cf50 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecae7fc700 4 set_mon_vals no callback set 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 10 monclient: dump: 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec94005100 con 0x7fecb006cf50 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7feca0003cf0 con 0x7fecb006cf50 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7feca00431c0 con 0x7fecb006cf50 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.157+0000 7fecaf7fe700 10 monclient: get_auth_request con 0x7fec90036410 auth_method 0 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.165+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feca00146a0 con 0x7fecb006cf50 2024-09-16T06:38:34.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.165+0000 7fecacff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.165+0000 7fecacff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:34.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecb5770700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T06:38:34.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecb5770700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T06:38:34.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecb5770700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fec94005930 con 0x7fecb006cf50 2024-09-16T06:38:34.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecacff9700 1 -- 172.21.15.76:0/2431735079 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (unknown 898784134 0 265718887) 0x7feca00144c0 con 0x7fecb006cf50 2024-09-16T06:38:34.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecacff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T06:38:34.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.333+0000 7fecacff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:34.337 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 8 2024-09-16T06:38:34.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 >> v1:172.21.15.189:6812/3098810674 conn(0x7fec90036410 legacy=0x7fec900388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 10 monclient: shutdown 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 >> v1:172.21.15.76:6789/0 conn(0x7fecb006cf50 legacy=0x7fecb0135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 shutdown_connections 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 >> 172.21.15.76:0/2431735079 conn(0x7fecb00a41c0 msgr2=0x7fecb00a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 shutdown_connections 2024-09-16T06:38:34.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.337+0000 7fecb5770700 1 -- 172.21.15.76:0/2431735079 wait complete. 2024-09-16T06:38:34.350 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T06:38:34.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:38:34.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: get_monmap_and_config 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: monmap: 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:34.464635+0000 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:34.464635+0000 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.463 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:34.464 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:34.464 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 Processor -- start 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- start start 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: init 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-a con 0x7f4ae4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-b con 0x7f4ae4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-c con 0x7f4ae4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4098990 con 0x7f4ae4097b60 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4097f40 con 0x7f4ae4139500 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae409d610 con 0x7f4ae4099090 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient(hunting): authenticate will time out at 861.598161s 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae91ea700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4ae4099090 0x7f4ae409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41888/0 (socket says 172.21.15.76:41888) 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae91ea700 1 -- 172.21.15.76:0/2799639979 learned_addr learned my addr 172.21.15.76:0/2799639979 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311001843 0 0) 0x7f4ae409d610 con 0x7f4ae4099090 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient(hunting): my global_id is 4394 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4acc0035d0 con 0x7f4ae4099090 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549040478 0 0) 0x7f4acc0035d0 con 0x7f4ae4099090 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 >> v1:172.21.15.189:6789/0 conn(0x7f4ae4139500 legacy=0x7f4ae413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 >> v1:172.21.15.76:6789/0 conn(0x7f4ae4097b60 legacy=0x7f4ae409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 monclient: found mon.noname-b 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae40986b0 con 0x7f4ae4099090 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 5 monclient: authenticate success, global_id 4394 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4ad4001990 con 0x7f4ae4099090 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: dump: 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ad4001cb0 con 0x7f4ae4099090 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 4 set_mon_vals no callback set 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 1 -- 172.21.15.76:0/2799639979 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4ad4002040 con 0x7f4ae4099090 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4ae89e9700 10 monclient: dump: 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 10 monclient: shutdown 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- 172.21.15.76:0/2799639979 >> v1:172.21.15.76:6790/0 conn(0x7f4ae4099090 legacy=0x7f4ae409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- 172.21.15.76:0/2799639979 shutdown_connections 2024-09-16T06:38:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- 172.21.15.76:0/2799639979 >> 172.21.15.76:0/2799639979 conn(0x7f4ae40a0030 msgr2=0x7f4ae40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.461+0000 7f4aea9ed700 1 -- 172.21.15.76:0/2799639979 shutdown_connections 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- 172.21.15.76:0/2799639979 wait complete. 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: monmap: 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:34.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 Processor -- start 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- start start 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: init 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-a con 0x7f4ae4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-c con 0x7f4ae4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient(hunting): picked mon.noname-b con 0x7f4ae4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae409a540 con 0x7f4ae4097b60 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae409c3e0 con 0x7f4ae4139500 2024-09-16T06:38:34.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae40a2c40 con 0x7f4ae4099090 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient(hunting): authenticate will time out at 861.602637s 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae91ea700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4ae4099090 0x7f4ae41ca690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41904/0 (socket says 172.21.15.76:41904) 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae91ea700 1 -- 172.21.15.76:0/3879080975 learned_addr learned my addr 172.21.15.76:0/3879080975 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366457409 0 0) 0x7f4ae40a2c40 con 0x7f4ae4099090 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): my global_id is 4397 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad00035d0 con 0x7f4ae4099090 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130470114 0 0) 0x7f4ae409a540 con 0x7f4ae4097b60 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): my global_id is 4398 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae40a2c40 con 0x7f4ae4097b60 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703694965 0 0) 0x7f4ad00035d0 con 0x7f4ae4099090 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae409a540 con 0x7f4ae4099090 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4ad4001950 con 0x7f4ae4099090 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): dump: 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.479 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2737094713 0 0) 0x7f4ae409a540 con 0x7f4ae4099090 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 >> v1:172.21.15.189:6789/0 conn(0x7f4ae4139500 legacy=0x7f4ae41cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 >> v1:172.21.15.76:6789/0 conn(0x7f4ae4097b60 legacy=0x7f4ae4096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 monclient: found mon.c 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae4096c20 con 0x7f4ae4099090 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 5 monclient: authenticate success, global_id 4397 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _renew_subs 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae40904d0 con 0x7f4ae4099090 2024-09-16T06:38:34.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _renew_subs 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae408f800 con 0x7f4ae4099090 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ad4005070 con 0x7f4ae4099090 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4ad40051e0 con 0x7f4ae4099090 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.465+0000 7f4ae27fc700 10 monclient: dump: 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4ae3fff700 4 set_mon_vals no callback set 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4ad400fc50 con 0x7f4ae4099090 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7f4ad4043790 con 0x7f4ae4099090 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4ae99eb700 10 monclient: get_auth_request con 0x7f4ad00361b0 auth_method 0 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4aea9ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4aea9ed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4aea9ed700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.469+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac4005100 con 0x7f4ae4099090 2024-09-16T06:38:34.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.477+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4ad4014bd0 con 0x7f4ae4099090 2024-09-16T06:38:34.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.477+0000 7f4ae27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.477+0000 7f4ae27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.641+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ac4000b90 con 0x7f4ad00361b0 2024-09-16T06:38:34.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.645+0000 7f4ae27fc700 1 -- 172.21.15.76:0/3879080975 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41036 (unknown 2965378022 0 2766754894) 0x7f4ac4000b90 con 0x7f4ad00361b0 2024-09-16T06:38:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4ad00361b0 legacy=0x7f4ad0038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 10 monclient: shutdown 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 >> v1:172.21.15.76:6790/0 conn(0x7f4ae4099090 legacy=0x7f4ae41ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 shutdown_connections 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 >> 172.21.15.76:0/3879080975 conn(0x7f4ae40a0030 msgr2=0x7f4ae40a11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 shutdown_connections 2024-09-16T06:38:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.649+0000 7f4aea9ed700 1 -- 172.21.15.76:0/3879080975 wait complete. 2024-09-16T06:38:34.653 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:38:34.665 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T06:38:33.645038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167264,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063776,"statfs":{"total":579820584960,"available":579649306624,"internally_reserved":0,"allocated":995328,"data_stored":1327349,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000598"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111899+0000","last_change":"2024-09-16T06:38:29.111899+0000","last_active":"2024-09-16T06:38:29.111899+0000","last_peered":"2024-09-16T06:38:29.111899+0000","last_clean":"2024-09-16T06:38:29.111899+0000","last_became_active":"2024-09-16T06:38:27.099196+0000","last_became_peered":"2024-09-16T06:38:27.099196+0000","last_unstale":"2024-09-16T06:38:29.111899+0000","last_undegraded":"2024-09-16T06:38:29.111899+0000","last_fullsized":"2024-09-16T06:38:29.111899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:04:40.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.120221+0000","last_change":"2024-09-16T06:38:29.120221+0000","last_active":"2024-09-16T06:38:29.120221+0000","last_peered":"2024-09-16T06:38:29.120221+0000","last_clean":"2024-09-16T06:38:29.120221+0000","last_became_active":"2024-09-16T06:38:27.099931+0000","last_became_peered":"2024-09-16T06:38:27.099931+0000","last_unstale":"2024-09-16T06:38:29.120221+0000","last_undegraded":"2024-09-16T06:38:29.120221+0000","last_fullsized":"2024-09-16T06:38:29.120221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:50.958236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111921+0000","last_change":"2024-09-16T06:38:29.111921+0000","last_active":"2024-09-16T06:38:29.111921+0000","last_peered":"2024-09-16T06:38:29.111921+0000","last_clean":"2024-09-16T06:38:29.111921+0000","last_became_active":"2024-09-16T06:38:27.098425+0000","last_became_peered":"2024-09-16T06:38:27.098425+0000","last_unstale":"2024-09-16T06:38:29.111921+0000","last_undegraded":"2024-09-16T06:38:29.111921+0000","last_fullsized":"2024-09-16T06:38:29.111921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:17.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.480183+0000","last_change":"2024-09-16T06:38:29.480183+0000","last_active":"2024-09-16T06:38:29.480183+0000","last_peered":"2024-09-16T06:38:29.480183+0000","last_clean":"2024-09-16T06:38:29.480183+0000","last_became_active":"2024-09-16T06:38:27.101585+0000","last_became_peered":"2024-09-16T06:38:27.101585+0000","last_unstale":"2024-09-16T06:38:29.480183+0000","last_undegraded":"2024-09-16T06:38:29.480183+0000","last_fullsized":"2024-09-16T06:38:29.480183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:55.916580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.117904+0000","last_change":"2024-09-16T06:38:29.114236+0000","last_active":"2024-09-16T06:38:29.117904+0000","last_peered":"2024-09-16T06:38:29.117904+0000","last_clean":"2024-09-16T06:38:29.117904+0000","last_became_active":"2024-09-16T06:38:27.101165+0000","last_became_peered":"2024-09-16T06:38:27.101165+0000","last_unstale":"2024-09-16T06:38:29.117904+0000","last_undegraded":"2024-09-16T06:38:29.117904+0000","last_fullsized":"2024-09-16T06:38:29.117904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:57.229189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.478940+0000","last_change":"2024-09-16T06:38:29.479525+0000","last_active":"2024-09-16T06:38:29.478940+0000","last_peered":"2024-09-16T06:38:29.478940+0000","last_clean":"2024-09-16T06:38:29.478940+0000","last_became_active":"2024-09-16T06:38:27.099768+0000","last_became_peered":"2024-09-16T06:38:27.099768+0000","last_unstale":"2024-09-16T06:38:29.478940+0000","last_undegraded":"2024-09-16T06:38:29.478940+0000","last_fullsized":"2024-09-16T06:38:29.478940+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:39.212825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.113909+0000","last_change":"2024-09-16T06:38:29.113909+0000","last_active":"2024-09-16T06:38:29.113909+0000","last_peered":"2024-09-16T06:38:29.113909+0000","last_clean":"2024-09-16T06:38:29.113909+0000","last_became_active":"2024-09-16T06:38:27.099002+0000","last_became_peered":"2024-09-16T06:38:27.099002+0000","last_unstale":"2024-09-16T06:38:29.113909+0000","last_undegraded":"2024-09-16T06:38:29.113909+0000","last_fullsized":"2024-09-16T06:38:29.113909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:20.128665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112636+0000","last_change":"2024-09-16T06:38:29.112636+0000","last_active":"2024-09-16T06:38:29.112636+0000","last_peered":"2024-09-16T06:38:29.112636+0000","last_clean":"2024-09-16T06:38:29.112636+0000","last_became_active":"2024-09-16T06:38:27.097289+0000","last_became_peered":"2024-09-16T06:38:27.097289+0000","last_unstale":"2024-09-16T06:38:29.112636+0000","last_undegraded":"2024-09-16T06:38:29.112636+0000","last_fullsized":"2024-09-16T06:38:29.112636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:03.935631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112531+0000","last_change":"2024-09-16T06:38:27.100613+0000","last_active":"2024-09-16T06:38:29.112531+0000","last_peered":"2024-09-16T06:38:29.112531+0000","last_clean":"2024-09-16T06:38:29.112531+0000","last_became_active":"2024-09-16T06:38:27.100231+0000","last_became_peered":"2024-09-16T06:38:27.100231+0000","last_unstale":"2024-09-16T06:38:29.112531+0000","last_undegraded":"2024-09-16T06:38:29.112531+0000","last_fullsized":"2024-09-16T06:38:29.112531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:56:17.957577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":45105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":44044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:38:34.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:38:34.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:34.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: monmap: 2024-09-16T06:38:34.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:34.782522+0000 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:34.782522+0000 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:34.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 1 Processor -- start 2024-09-16T06:38:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 1 -- start start 2024-09-16T06:38:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: init 2024-09-16T06:38:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc84409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc844098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc844099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84409d560 con 0x7fc844098be0 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84409f140 con 0x7fc84409ed60 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84409d790 con 0x7fc844099030 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.777+0000 7fc843fff700 10 monclient(hunting): authenticate will time out at 861.915879s 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc8427fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc844098be0 0x7fc8441374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39296/0 (socket says 172.21.15.76:39296) 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc8427fc700 1 -- 172.21.15.76:0/2687054861 learned_addr learned my addr 172.21.15.76:0/2687054861 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512581955 0 0) 0x7fc84409d790 con 0x7fc844099030 2024-09-16T06:38:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): my global_id is 4400 2024-09-16T06:38:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c0035d0 con 0x7fc844099030 2024-09-16T06:38:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013211509 0 0) 0x7fc84409d560 con 0x7fc844098be0 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): my global_id is 4401 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84409d790 con 0x7fc844098be0 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332197539 0 0) 0x7fc84409f140 con 0x7fc84409ed60 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient(hunting): my global_id is 4399 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84409d560 con 0x7fc84409ed60 2024-09-16T06:38:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849758258 0 0) 0x7fc84409d790 con 0x7fc844098be0 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 >> v1:172.21.15.76:6790/0 conn(0x7fc844099030 legacy=0x7fc84413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 >> v1:172.21.15.189:6789/0 conn(0x7fc84409ed60 legacy=0x7fc8440984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 monclient: found mon.noname-a 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84409f620 con 0x7fc844098be0 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 5 monclient: authenticate success, global_id 4401 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc834001520 con 0x7fc844098be0 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: dump: 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8340017c0 con 0x7fc844098be0 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 4 set_mon_vals no callback set 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 1 -- 172.21.15.76:0/2687054861 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc834001b50 con 0x7fc844098be0 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc841ffb700 10 monclient: dump: 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: shutdown 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- 172.21.15.76:0/2687054861 >> v1:172.21.15.76:6789/0 conn(0x7fc844098be0 legacy=0x7fc8441374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- 172.21.15.76:0/2687054861 shutdown_connections 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- 172.21.15.76:0/2687054861 >> 172.21.15.76:0/2687054861 conn(0x7fc8440a0030 msgr2=0x7fc8440a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- 172.21.15.76:0/2687054861 shutdown_connections 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- 172.21.15.76:0/2687054861 wait complete. 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: monmap: 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:34.798 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 Processor -- start 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- start start 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: init 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc844098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc844099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc84409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc844090bb0 con 0x7fc844098be0 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440a1250 con 0x7fc844099030 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8440986a0 con 0x7fc84409ed60 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.781+0000 7fc843fff700 10 monclient(hunting): authenticate will time out at 861.919990s 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc8437fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc84409ed60 0x7fc8441cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41916/0 (socket says 172.21.15.76:41916) 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc8437fe700 1 -- 172.21.15.76:0/3581047934 learned_addr learned my addr 172.21.15.76:0/3581047934 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774145046 0 0) 0x7fc8440986a0 con 0x7fc84409ed60 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): my global_id is 4403 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8300035d0 con 0x7fc84409ed60 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443369408 0 0) 0x7fc844090bb0 con 0x7fc844098be0 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): my global_id is 4404 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8440986a0 con 0x7fc844098be0 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158232101 0 0) 0x7fc8300035d0 con 0x7fc84409ed60 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc844090bb0 con 0x7fc84409ed60 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc83c001d70 con 0x7fc84409ed60 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352507647 0 0) 0x7fc8440a1250 con 0x7fc844099030 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient(hunting): my global_id is 4402 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8300035d0 con 0x7fc844099030 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759170091 0 0) 0x7fc844090bb0 con 0x7fc84409ed60 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 >> v1:172.21.15.189:6789/0 conn(0x7fc844099030 legacy=0x7fc8441ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 >> v1:172.21.15.76:6789/0 conn(0x7fc844098be0 legacy=0x7fc844096400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 monclient: found mon.c 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844090e90 con 0x7fc84409ed60 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 5 monclient: authenticate success, global_id 4403 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _renew_subs 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84408eac0 con 0x7fc84409ed60 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _renew_subs 2024-09-16T06:38:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8441c5790 con 0x7fc84409ed60 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc83c002000 con 0x7fc84409ed60 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc83c003ef0 con 0x7fc84409ed60 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc840ff9700 4 set_mon_vals no callback set 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 10 monclient: dump: 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc83c004050 con 0x7fc84409ed60 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824005100 con 0x7fc84409ed60 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3562+0+0 (unknown 409209912 0 0) 0x7fc83c0435a0 con 0x7fc84409ed60 2024-09-16T06:38:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.785+0000 7fc842ffd700 10 monclient: get_auth_request con 0x7fc8300363e0 auth_method 0 2024-09-16T06:38:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.789+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc83c014b50 con 0x7fc84409ed60 2024-09-16T06:38:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.789+0000 7fc82b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.789+0000 7fc82b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.957+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc824000b90 con 0x7fc8300363e0 2024-09-16T06:38:34.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.961+0000 7fc82b7fe700 1 -- 172.21.15.76:0/3581047934 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41013 (unknown 2965378022 0 3808327254) 0x7fc824000b90 con 0x7fc8300363e0 2024-09-16T06:38:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc8300363e0 legacy=0x7fc8300388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 10 monclient: shutdown 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 >> v1:172.21.15.76:6790/0 conn(0x7fc84409ed60 legacy=0x7fc8441cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 shutdown_connections 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 >> 172.21.15.76:0/3581047934 conn(0x7fc8440a0030 msgr2=0x7fc8440972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 shutdown_connections 2024-09-16T06:38:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:34.965+0000 7fc843fff700 1 -- 172.21.15.76:0/3581047934 wait complete. 2024-09-16T06:38:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:38:34.981 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T06:38:34.892016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167264,"kb_used_data":972,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566063776,"statfs":{"total":579820584960,"available":579649306624,"internally_reserved":0,"allocated":995328,"data_stored":1327349,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.247576"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111899+0000","last_change":"2024-09-16T06:38:29.111899+0000","last_active":"2024-09-16T06:38:29.111899+0000","last_peered":"2024-09-16T06:38:29.111899+0000","last_clean":"2024-09-16T06:38:29.111899+0000","last_became_active":"2024-09-16T06:38:27.099196+0000","last_became_peered":"2024-09-16T06:38:27.099196+0000","last_unstale":"2024-09-16T06:38:29.111899+0000","last_undegraded":"2024-09-16T06:38:29.111899+0000","last_fullsized":"2024-09-16T06:38:29.111899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:04:40.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.120221+0000","last_change":"2024-09-16T06:38:29.120221+0000","last_active":"2024-09-16T06:38:29.120221+0000","last_peered":"2024-09-16T06:38:29.120221+0000","last_clean":"2024-09-16T06:38:29.120221+0000","last_became_active":"2024-09-16T06:38:27.099931+0000","last_became_peered":"2024-09-16T06:38:27.099931+0000","last_unstale":"2024-09-16T06:38:29.120221+0000","last_undegraded":"2024-09-16T06:38:29.120221+0000","last_fullsized":"2024-09-16T06:38:29.120221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:50.958236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.111921+0000","last_change":"2024-09-16T06:38:29.111921+0000","last_active":"2024-09-16T06:38:29.111921+0000","last_peered":"2024-09-16T06:38:29.111921+0000","last_clean":"2024-09-16T06:38:29.111921+0000","last_became_active":"2024-09-16T06:38:27.098425+0000","last_became_peered":"2024-09-16T06:38:27.098425+0000","last_unstale":"2024-09-16T06:38:29.111921+0000","last_undegraded":"2024-09-16T06:38:29.111921+0000","last_fullsized":"2024-09-16T06:38:29.111921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:17.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.480183+0000","last_change":"2024-09-16T06:38:29.480183+0000","last_active":"2024-09-16T06:38:29.480183+0000","last_peered":"2024-09-16T06:38:29.480183+0000","last_clean":"2024-09-16T06:38:29.480183+0000","last_became_active":"2024-09-16T06:38:27.101585+0000","last_became_peered":"2024-09-16T06:38:27.101585+0000","last_unstale":"2024-09-16T06:38:29.480183+0000","last_undegraded":"2024-09-16T06:38:29.480183+0000","last_fullsized":"2024-09-16T06:38:29.480183+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:01:55.916580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.117904+0000","last_change":"2024-09-16T06:38:29.114236+0000","last_active":"2024-09-16T06:38:29.117904+0000","last_peered":"2024-09-16T06:38:29.117904+0000","last_clean":"2024-09-16T06:38:29.117904+0000","last_became_active":"2024-09-16T06:38:27.101165+0000","last_became_peered":"2024-09-16T06:38:27.101165+0000","last_unstale":"2024-09-16T06:38:29.117904+0000","last_undegraded":"2024-09-16T06:38:29.117904+0000","last_fullsized":"2024-09-16T06:38:29.117904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:57.229189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.478940+0000","last_change":"2024-09-16T06:38:29.479525+0000","last_active":"2024-09-16T06:38:29.478940+0000","last_peered":"2024-09-16T06:38:29.478940+0000","last_clean":"2024-09-16T06:38:29.478940+0000","last_became_active":"2024-09-16T06:38:27.099768+0000","last_became_peered":"2024-09-16T06:38:27.099768+0000","last_unstale":"2024-09-16T06:38:29.478940+0000","last_undegraded":"2024-09-16T06:38:29.478940+0000","last_fullsized":"2024-09-16T06:38:29.478940+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:39.212825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.113909+0000","last_change":"2024-09-16T06:38:29.113909+0000","last_active":"2024-09-16T06:38:29.113909+0000","last_peered":"2024-09-16T06:38:29.113909+0000","last_clean":"2024-09-16T06:38:29.113909+0000","last_became_active":"2024-09-16T06:38:27.099002+0000","last_became_peered":"2024-09-16T06:38:27.099002+0000","last_unstale":"2024-09-16T06:38:29.113909+0000","last_undegraded":"2024-09-16T06:38:29.113909+0000","last_fullsized":"2024-09-16T06:38:29.113909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:20.128665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112636+0000","last_change":"2024-09-16T06:38:29.112636+0000","last_active":"2024-09-16T06:38:29.112636+0000","last_peered":"2024-09-16T06:38:29.112636+0000","last_clean":"2024-09-16T06:38:29.112636+0000","last_became_active":"2024-09-16T06:38:27.097289+0000","last_became_peered":"2024-09-16T06:38:27.097289+0000","last_unstale":"2024-09-16T06:38:29.112636+0000","last_undegraded":"2024-09-16T06:38:29.112636+0000","last_fullsized":"2024-09-16T06:38:29.112636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:03.935631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-16T06:38:29.112531+0000","last_change":"2024-09-16T06:38:27.100613+0000","last_active":"2024-09-16T06:38:29.112531+0000","last_peered":"2024-09-16T06:38:29.112531+0000","last_clean":"2024-09-16T06:38:29.112531+0000","last_became_active":"2024-09-16T06:38:27.100231+0000","last_became_peered":"2024-09-16T06:38:27.100231+0000","last_unstale":"2024-09-16T06:38:29.112531+0000","last_undegraded":"2024-09-16T06:38:29.112531+0000","last_fullsized":"2024-09-16T06:38:29.112531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:56:17.957577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10546,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":151552,"data_stored":45105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":44044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":200704,"data_stored":574014,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":45086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5273,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:38:34.982 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T06:38:34.983 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T06:38:34.983 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T06:38:35.000 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T06:38:35.000 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T06:38:35.000 DEBUG:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T06:38:35.001 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T06:38:35.001 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726468714, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-09-16T06:38:35.001 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:38:35.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:38:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:38:35.009 INFO:teuthology.orchestra.run.smithi189.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T06:38:35.009 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T06:38:35.056 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T06:38:35.056 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T06:38:35.063 INFO:tasks.workunit:timeout=3h 2024-09-16T06:38:35.063 INFO:tasks.workunit:cleanup=True 2024-09-16T06:38:35.064 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T06:38:35.073 INFO:tasks.workunit.client.0.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T06:38:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: get_monmap_and_config 2024-09-16T06:38:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: monmap: 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:35.112521+0000 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:35.112521+0000 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:35.111 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 1 Processor -- start 2024-09-16T06:38:35.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 1 -- start start 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: init 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient(hunting): picked mon.noname-b con 0x7f067812fdf0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient(hunting): picked mon.noname-c con 0x7f067812f820 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient(hunting): picked mon.noname-a con 0x7f0678139760 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780a14c0 con 0x7f0678139760 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067809e1e0 con 0x7f067812f820 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067809faf0 con 0x7f067812fdf0 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient(hunting): authenticate will time out at 862.246383s 2024-09-16T06:38:35.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067dc42700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0678139760 0x7f067813bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39308/0 (socket says 172.21.15.76:39308) 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067d441700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f067812fdf0 0x7f06781301d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41924/0 (socket says 172.21.15.76:41924) 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067dc42700 1 -- 172.21.15.76:0/3210820144 learned_addr learned my addr 172.21.15.76:0/3210820144 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297976038 0 0) 0x7f067809faf0 con 0x7f067812fdf0 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): my global_id is 4406 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06640035d0 con 0x7f067812fdf0 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222079025 0 0) 0x7f06780a14c0 con 0x7f0678139760 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): my global_id is 4407 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067809faf0 con 0x7f0678139760 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451913797 0 0) 0x7f067809e1e0 con 0x7f067812f820 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient(hunting): my global_id is 4405 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780a14c0 con 0x7f067812f820 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 626521882 0 0) 0x7f06640035d0 con 0x7f067812fdf0 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 >> v1:172.21.15.189:6789/0 conn(0x7f067812f820 legacy=0x7f0678138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 >> v1:172.21.15.76:6789/0 conn(0x7f0678139760 legacy=0x7f067813bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 monclient: found mon.noname-b 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067809fdd0 con 0x7f067812fdf0 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0674001540 con 0x7f067812fdf0 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 5 monclient: authenticate success, global_id 4406 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: dump: 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.123 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06740017b0 con 0x7f067812fdf0 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f067e443700 4 set_mon_vals no callback set 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 1 -- 172.21.15.76:0/3210820144 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0674001b40 con 0x7f067812fdf0 2024-09-16T06:38:35.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.109+0000 7f066ffff700 10 monclient: dump: 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: shutdown 2024-09-16T06:38:35.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- 172.21.15.76:0/3210820144 >> v1:172.21.15.76:6790/0 conn(0x7f067812fdf0 legacy=0x7f06781301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- 172.21.15.76:0/3210820144 shutdown_connections 2024-09-16T06:38:35.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- 172.21.15.76:0/3210820144 >> 172.21.15.76:0/3210820144 conn(0x7f06780a4430 msgr2=0x7f06780a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- 172.21.15.76:0/3210820144 shutdown_connections 2024-09-16T06:38:35.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- 172.21.15.76:0/3210820144 wait complete. 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: monmap: 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:35.127 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 Processor -- start 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- start start 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: init 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient(hunting): picked mon.noname-a con 0x7f067812f820 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient(hunting): picked mon.noname-c con 0x7f067812fdf0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient(hunting): picked mon.noname-b con 0x7f0678139760 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0678139050 con 0x7f067812f820 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067813cf40 con 0x7f0678139760 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06781367f0 con 0x7f067812fdf0 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067e443700 10 monclient(hunting): authenticate will time out at 862.250441s 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067cc40700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f067812fdf0 0x7f067809b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41938/0 (socket says 172.21.15.76:41938) 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f067cc40700 1 -- 172.21.15.76:0/2374611052 learned_addr learned my addr 172.21.15.76:0/2374611052 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159791162 0 0) 0x7f06781367f0 con 0x7f067812fdf0 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): my global_id is 4409 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f065c0035d0 con 0x7f067812fdf0 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478637149 0 0) 0x7f0678139050 con 0x7f067812f820 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): my global_id is 4410 2024-09-16T06:38:35.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06781367f0 con 0x7f067812f820 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750181527 0 0) 0x7f067813cf40 con 0x7f0678139760 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): my global_id is 4408 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0678139050 con 0x7f0678139760 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3432966212 0 0) 0x7f065c0035d0 con 0x7f067812fdf0 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f067813cf40 con 0x7f067812fdf0 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0668001990 con 0x7f067812fdf0 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): dump: 2024-09-16T06:38:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:35.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448157926 0 0) 0x7f06781367f0 con 0x7f067812f820 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f065c0035d0 con 0x7f067812f820 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0674001500 con 0x7f067812f820 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): dump: 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.132 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665421883 0 0) 0x7f0678139050 con 0x7f0678139760 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06781367f0 con 0x7f0678139760 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0670001bf0 con 0x7f0678139760 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 10 monclient(hunting): dump: 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.113+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1340347813 0 0) 0x7f067813cf40 con 0x7f067812fdf0 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 >> v1:172.21.15.189:6789/0 conn(0x7f0678139760 legacy=0x7f06781d20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 >> v1:172.21.15.76:6789/0 conn(0x7f067812f820 legacy=0x7f067809b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 monclient: found mon.c 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678136ad0 con 0x7f067812fdf0 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 5 monclient: authenticate success, global_id 4409 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _renew_subs 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067809ca40 con 0x7f067812fdf0 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _renew_subs 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f067809bf10 con 0x7f067812fdf0 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0668001d40 con 0x7f067812fdf0 2024-09-16T06:38:35.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0668003b10 con 0x7f067812fdf0 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066effd700 4 set_mon_vals no callback set 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 10 monclient: dump: 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.136 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0668003c70 con 0x7f067812fdf0 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0658005100 con 0x7f067812fdf0 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f06680431d0 con 0x7f067812fdf0 2024-09-16T06:38:35.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.117+0000 7f067d441700 10 monclient: get_auth_request con 0x7f065c036500 auth_method 0 2024-09-16T06:38:35.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.125+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0668014800 con 0x7f067812fdf0 2024-09-16T06:38:35.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.125+0000 7f066dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.125+0000 7f066dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f067e443700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f067e443700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f067e443700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0658005930 con 0x7f067812fdf0 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f066dffb700 1 -- 172.21.15.76:0/2374611052 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1282972391) 0x7f06680143f0 con 0x7f067812fdf0 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f066dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:35.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.353+0000 7f066dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:35.356 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 >> v1:172.21.15.189:6812/3098810674 conn(0x7f065c036500 legacy=0x7f065c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 10 monclient: shutdown 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 >> v1:172.21.15.76:6790/0 conn(0x7f067812fdf0 legacy=0x7f067809b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 shutdown_connections 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 >> 172.21.15.76:0/2374611052 conn(0x7f06780a4430 msgr2=0x7f067813c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 shutdown_connections 2024-09-16T06:38:35.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.357+0000 7f067e443700 1 -- 172.21.15.76:0/2374611052 wait complete. 2024-09-16T06:38:35.371 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:38:35.371 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:38:35.371 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:38:35.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:38:35.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:35.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: monmap: 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:35.485507+0000 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:35.485507+0000 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:35.484 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 1 Processor -- start 2024-09-16T06:38:35.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 1 -- start start 2024-09-16T06:38:35.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: init 2024-09-16T06:38:35.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7faf90097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7faf90099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7faf90139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf90098990 con 0x7faf90099090 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf90097f40 con 0x7faf90139500 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9009d610 con 0x7faf90097b60 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 10 monclient(hunting): authenticate will time out at 862.618768s 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faf90097b60 0x7faf9009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41954/0 (socket says 172.21.15.76:41954) 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8e59c700 1 -- 172.21.15.76:0/3356622823 learned_addr learned my addr 172.21.15.76:0/3356622823 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716295209 0 0) 0x7faf9009d610 con 0x7faf90097b60 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): my global_id is 4412 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7c0035d0 con 0x7faf90097b60 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354637318 0 0) 0x7faf90098990 con 0x7faf90099090 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): my global_id is 4413 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9009d610 con 0x7faf90099090 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431236824 0 0) 0x7faf90097f40 con 0x7faf90139500 2024-09-16T06:38:35.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient(hunting): my global_id is 4411 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf90098990 con 0x7faf90139500 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397695675 0 0) 0x7faf7c0035d0 con 0x7faf90097b60 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 >> v1:172.21.15.189:6789/0 conn(0x7faf90139500 legacy=0x7faf9013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 >> v1:172.21.15.76:6789/0 conn(0x7faf90099090 legacy=0x7faf9009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 monclient: found mon.noname-b 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900986b0 con 0x7faf90097b60 2024-09-16T06:38:35.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf78001590 con 0x7faf90097b60 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 5 monclient: authenticate success, global_id 4412 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8d59a700 10 monclient: dump: 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.481+0000 7faf8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf780017b0 con 0x7faf90097b60 2024-09-16T06:38:35.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 1 -- 172.21.15.76:0/3356622823 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf78001b80 con 0x7faf90097b60 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 4 set_mon_vals no callback set 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8d59a700 10 monclient: dump: 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.499 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: shutdown 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/3356622823 >> v1:172.21.15.76:6790/0 conn(0x7faf90097b60 legacy=0x7faf9009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/3356622823 shutdown_connections 2024-09-16T06:38:35.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/3356622823 >> 172.21.15.76:0/3356622823 conn(0x7faf900a0030 msgr2=0x7faf900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/3356622823 shutdown_connections 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/3356622823 wait complete. 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: monmap: 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:35.501 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 Processor -- start 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- start start 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: init 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7faf90097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7faf90099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7faf90139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9008f200 con 0x7faf90099090 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9009a540 con 0x7faf90097b60 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9008e220 con 0x7faf90139500 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient(hunting): authenticate will time out at 862.622482s 2024-09-16T06:38:35.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8ed9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faf90139500 0x7faf9008c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41966/0 (socket says 172.21.15.76:41966) 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8dd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faf90099090 0x7faf9008bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39322/0 (socket says 172.21.15.76:39322) 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8ed9d700 1 -- 172.21.15.76:0/2986571583 learned_addr learned my addr 172.21.15.76:0/2986571583 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622013408 0 0) 0x7faf9008e220 con 0x7faf90139500 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): my global_id is 4415 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf700035d0 con 0x7faf90139500 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485339134 0 0) 0x7faf9008f200 con 0x7faf90099090 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): my global_id is 4416 2024-09-16T06:38:35.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9008e220 con 0x7faf90099090 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552167926 0 0) 0x7faf9009a540 con 0x7faf90097b60 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): my global_id is 4414 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9008f200 con 0x7faf90097b60 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2054101002 0 0) 0x7faf700035d0 con 0x7faf90139500 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf9009a540 con 0x7faf90139500 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf800018b0 con 0x7faf90139500 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): dump: 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:35.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1374666872 0 0) 0x7faf9008e220 con 0x7faf90099090 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf700035d0 con 0x7faf90099090 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf88001d50 con 0x7faf90099090 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient(hunting): dump: 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3917327421 0 0) 0x7faf9009a540 con 0x7faf90139500 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.189:6789/0 conn(0x7faf90097b60 legacy=0x7faf9008b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6789/0 conn(0x7faf90099090 legacy=0x7faf9008bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 monclient: found mon.c 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9008cdc0 con 0x7faf90139500 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 5 monclient: authenticate success, global_id 4415 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _renew_subs 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9008e8f0 con 0x7faf90139500 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _renew_subs 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.485+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf90138e70 con 0x7faf90139500 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf800036c0 con 0x7faf90139500 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf800039d0 con 0x7faf90139500 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf8cd99700 4 set_mon_vals no callback set 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: dump: 2024-09-16T06:38:35.508 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf8f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faf74006590 con 0x7faf74002830 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faf80003bb0 con 0x7faf90139500 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7faf80043120 con 0x7faf90139500 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936505737 0 0) 0x7faf74006590 con 0x7faf74002830 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient(hunting): my global_id is 4419 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9009a540 con 0x7faf74002830 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf8dd9b700 10 monclient: get_auth_request con 0x7faf700364c0 auth_method 0 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1210456551 0 0) 0x7faf9009a540 con 0x7faf74002830 2024-09-16T06:38:35.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf74006590 con 0x7faf74002830 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf780013d0 con 0x7faf74002830 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: dump: 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.511 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3648954909 0 0) 0x7faf74006590 con 0x7faf74002830 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf7400a190 con 0x7faf74002830 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7faf78004890 con 0x7faf74002830 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6789/0 conn(0x7faf74002830 legacy=0x7faf74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.489+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6789/0 conn(0x7faf74002830 legacy=0x7faf74004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf8f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faf74009e50 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998875739 0 0) 0x7faf74009e50 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 10 monclient(hunting): my global_id is 4422 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.501+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf74006590 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2766422843 0 0) 0x7faf74006590 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf74009e50 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faf78004e10 con 0x7faf74009170 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 10 monclient: dump: 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3069572986 0 0) 0x7faf74009e50 con 0x7faf74009170 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faf7400c340 con 0x7faf74009170 2024-09-16T06:38:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 4255510111) 0x7faf78005020 con 0x7faf74009170 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6789/0 conn(0x7faf74009170 legacy=0x7faf74009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf86ffd700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6789/0 conn(0x7faf74009170 legacy=0x7faf74009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 12, 2024-09-16T06:38:35.516 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 19, 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.517 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.518 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.519 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.520 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:35.521 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.522 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.523 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:38:35.524 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:38:35.525 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:38:35.527 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:38:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.189:6812/3098810674 conn(0x7faf700364c0 legacy=0x7faf70038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 10 monclient: shutdown 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8ed9d700 1 -- 172.21.15.76:0/2986571583 reap_dead start 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 >> v1:172.21.15.76:6790/0 conn(0x7faf90139500 legacy=0x7faf9008c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 shutdown_connections 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 >> 172.21.15.76:0/2986571583 conn(0x7faf900a0030 msgr2=0x7faf9013c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 shutdown_connections 2024-09-16T06:38:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.505+0000 7faf8f59e700 1 -- 172.21.15.76:0/2986571583 wait complete. 2024-09-16T06:38:35.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:38:35.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:38:35.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: get_monmap_and_config 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: monmap: 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:35.647965+0000 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:35.647965+0000 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.646 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:35.647 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:35.647 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 Processor -- start 2024-09-16T06:38:35.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- start start 2024-09-16T06:38:35.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: init 2024-09-16T06:38:35.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-b con 0x7feba009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-c con 0x7feba009c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-a con 0x7feba009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba009a040 con 0x7feba009fd20 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba009ac80 con 0x7feba009c460 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba009def0 con 0x7feba009f230 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient(hunting): authenticate will time out at 862.781202s 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feba009fd20 0x7feba0142160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39348/0 (socket says 172.21.15.76:39348) 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9ffff700 1 -- 172.21.15.76:0/3077042722 learned_addr learned my addr 172.21.15.76:0/3077042722 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725589192 0 0) 0x7feba009def0 con 0x7feba009f230 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): my global_id is 4418 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb880035d0 con 0x7feba009f230 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122000227 0 0) 0x7feb90001490 con 0x7feba009c460 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): my global_id is 4417 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba009def0 con 0x7feba009c460 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556181830 0 0) 0x7feba009a040 con 0x7feba009fd20 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient(hunting): my global_id is 4425 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90001490 con 0x7feba009fd20 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203846723 0 0) 0x7feb880035d0 con 0x7feba009f230 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 >> v1:172.21.15.189:6789/0 conn(0x7feba009c460 legacy=0x7feba013ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 >> v1:172.21.15.76:6789/0 conn(0x7feba009fd20 legacy=0x7feba0142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:38:35.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba009a410 con 0x7feba009f230 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 5 monclient: authenticate success, global_id 4418 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb94001610 con 0x7feba009f230 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: dump: 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb940018f0 con 0x7feba009f230 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 1 -- 172.21.15.76:0/3077042722 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb94001c80 con 0x7feba009f230 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 4 set_mon_vals no callback set 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feb9e7fc700 10 monclient: dump: 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: shutdown 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- 172.21.15.76:0/3077042722 >> v1:172.21.15.76:6790/0 conn(0x7feba009f230 legacy=0x7feba009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- 172.21.15.76:0/3077042722 shutdown_connections 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- 172.21.15.76:0/3077042722 >> 172.21.15.76:0/3077042722 conn(0x7feba00a3e50 msgr2=0x7feba00a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- 172.21.15.76:0/3077042722 shutdown_connections 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 -- 172.21.15.76:0/3077042722 wait complete. 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 10 monclient: monmap: 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:35.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.645+0000 7feba5e1f700 1 Processor -- start 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- start start 2024-09-16T06:38:35.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: init 2024-09-16T06:38:35.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-b con 0x7feba009c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-c con 0x7feba009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient(hunting): picked mon.noname-a con 0x7feba009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0135ca0 con 0x7feba009fd20 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0137d30 con 0x7feba009c460 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0138400 con 0x7feba009f230 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient(hunting): authenticate will time out at 862.784470s 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb9effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7feba009f230 0x7feba0135590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41980/0 (socket says 172.21.15.76:41980) 2024-09-16T06:38:35.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb9effd700 1 -- 172.21.15.76:0/113888614 learned_addr learned my addr 172.21.15.76:0/113888614 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029862779 0 0) 0x7feba0137d30 con 0x7feba009c460 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): my global_id is 4420 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c0035d0 con 0x7feba009c460 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297794892 0 0) 0x7feba0135ca0 con 0x7feba009fd20 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): my global_id is 4428 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0137d30 con 0x7feba009fd20 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753890546 0 0) 0x7feba0138400 con 0x7feba009f230 2024-09-16T06:38:35.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): my global_id is 4421 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0135ca0 con 0x7feba009f230 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569741441 0 0) 0x7feb8c0035d0 con 0x7feba009c460 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feba0138400 con 0x7feba009c460 2024-09-16T06:38:35.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb94001c00 con 0x7feba009c460 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): dump: 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 984025375 0 0) 0x7feba0137d30 con 0x7feba009fd20 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb8c0035d0 con 0x7feba009fd20 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb98001830 con 0x7feba009fd20 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): dump: 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.666 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.667 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.667 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.667 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.667 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536713485 0 0) 0x7feba0138400 con 0x7feba009c460 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6790/0 conn(0x7feba009f230 legacy=0x7feba0135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6789/0 conn(0x7feba009fd20 legacy=0x7feba01d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 monclient: found mon.b 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba01386e0 con 0x7feba009c460 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 5 monclient: authenticate success, global_id 4420 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _renew_subs 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba013a400 con 0x7feba009c460 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _renew_subs 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feba01395f0 con 0x7feba009c460 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb940039f0 con 0x7feba009c460 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb94003d00 con 0x7feba009c460 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb9dffb700 4 set_mon_vals no callback set 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient: dump: 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7feb9400e770 con 0x7feba009c460 2024-09-16T06:38:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7feb94042100 con 0x7feba009c460 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb80006590 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb9effd700 10 monclient: get_auth_request con 0x7feb8c036170 auth_method 0 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557728804 0 0) 0x7feb80006590 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 10 monclient(hunting): my global_id is 4424 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.649+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0138400 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1072036057 0 0) 0x7feba0138400 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb80006590 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb98004d20 con 0x7feb80002830 2024-09-16T06:38:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 10 monclient: dump: 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2527668982 0 0) 0x7feb80006590 con 0x7feb80002830 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb8000a190 con 0x7feb80002830 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7feb98005980 con 0x7feb80002830 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6790/0 conn(0x7feb80002830 legacy=0x7feb80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.653+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6790/0 conn(0x7feb80002830 legacy=0x7feb80004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb80009e50 con 0x7feb80009170 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244523563 0 0) 0x7feb80009e50 con 0x7feb80009170 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient(hunting): my global_id is 4427 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb80006590 con 0x7feb80009170 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1034647422 0 0) 0x7feb80006590 con 0x7feb80009170 2024-09-16T06:38:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb80009e50 con 0x7feb80009170 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feb98005390 con 0x7feb80009170 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient: dump: 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.676 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1730851440 0 0) 0x7feb80009e50 con 0x7feb80009170 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb8000c340 con 0x7feb80009170 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 317242967) 0x7feb98005700 con 0x7feb80009170 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6790/0 conn(0x7feb80009170 legacy=0x7feb80009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb87fff700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.76:6790/0 conn(0x7feb80009170 legacy=0x7feb80009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:38:35.677 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 12, 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 19, 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:38:35.678 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:38:35.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.680 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:38:35.681 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.682 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.684 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.686 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.687 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.688 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:38:35.689 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:38:35.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.189:6812/3098810674 conn(0x7feb8c036170 legacy=0x7feb8c038630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 10 monclient: shutdown 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feb9ffff700 1 -- 172.21.15.76:0/113888614 reap_dead start 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.661+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 >> v1:172.21.15.189:6789/0 conn(0x7feba009c460 legacy=0x7feba0134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.665+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 shutdown_connections 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.665+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 >> 172.21.15.76:0/113888614 conn(0x7feba00a3e50 msgr2=0x7feba009c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.665+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 shutdown_connections 2024-09-16T06:38:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.665+0000 7feba5e1f700 1 -- 172.21.15.76:0/113888614 wait complete. 2024-09-16T06:38:35.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:38:35.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:38:35.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:38:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: get_monmap_and_config 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: monmap: 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:35.809790+0000 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:35.809790+0000 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:35.809 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 1 Processor -- start 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 1 -- start start 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: init 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-c con 0x7f5ee00964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-b con 0x7f5ee009b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-a con 0x7f5ee0137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0098560 con 0x7f5ee0137e70 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0099ec0 con 0x7f5ee00964f0 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee009b5f0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5ee48ec700 10 monclient(hunting): authenticate will time out at 862.943676s 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5edf59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5ee0137e70 0x7f5ee013a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39372/0 (socket says 172.21.15.76:39372) 2024-09-16T06:38:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.805+0000 7f5edf59e700 1 -- 172.21.15.76:0/1200397511 learned_addr learned my addr 172.21.15.76:0/1200397511 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847912102 0 0) 0x7f5ee0098560 con 0x7f5ee0137e70 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): my global_id is 4431 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec80034f0 con 0x7f5ee0137e70 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827500515 0 0) 0x7f5ee009b5f0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): my global_id is 4430 2024-09-16T06:38:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0098560 con 0x7f5ee009b1c0 2024-09-16T06:38:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405686946 0 0) 0x7f5ee0099ec0 con 0x7f5ee00964f0 2024-09-16T06:38:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient(hunting): my global_id is 4423 2024-09-16T06:38:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee009b5f0 con 0x7f5ee00964f0 2024-09-16T06:38:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087672674 0 0) 0x7f5ee0098560 con 0x7f5ee009b1c0 2024-09-16T06:38:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 >> v1:172.21.15.189:6789/0 conn(0x7f5ee00964f0 legacy=0x7f5ee00968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 >> v1:172.21.15.76:6789/0 conn(0x7f5ee0137e70 legacy=0x7f5ee013a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 monclient: found mon.noname-b 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0099be0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 5 monclient: authenticate success, global_id 4430 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed40015f0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: dump: 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.824 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed4001890 con 0x7f5ee009b1c0 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 4 set_mon_vals no callback set 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 1 -- 172.21.15.76:0/1200397511 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed4001c60 con 0x7f5ee009b1c0 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5eddd9b700 10 monclient: dump: 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.825 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 10 monclient: shutdown 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 -- 172.21.15.76:0/1200397511 >> v1:172.21.15.76:6790/0 conn(0x7f5ee009b1c0 legacy=0x7f5ee0136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 -- 172.21.15.76:0/1200397511 shutdown_connections 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 -- 172.21.15.76:0/1200397511 >> 172.21.15.76:0/1200397511 conn(0x7f5ee00a0030 msgr2=0x7f5ee00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 -- 172.21.15.76:0/1200397511 shutdown_connections 2024-09-16T06:38:35.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 -- 172.21.15.76:0/1200397511 wait complete. 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 10 monclient: build_initial_monmap 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 10 monclient: monmap: 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:35.827 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.809+0000 7f5ee48ec700 1 Processor -- start 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- start start 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: init 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-b con 0x7f5ee00964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-c con 0x7f5ee009b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient(hunting): picked mon.noname-a con 0x7f5ee0137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00936b0 con 0x7f5ee0137e70 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee009a540 con 0x7f5ee00964f0 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0095b40 con 0x7f5ee009b1c0 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient(hunting): authenticate will time out at 862.948929s 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5edf59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5ee0137e70 0x7f5ee01d2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39384/0 (socket says 172.21.15.76:39384) 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5edf59e700 1 -- 172.21.15.76:0/2755746224 learned_addr learned my addr 172.21.15.76:0/2755746224 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126954016 0 0) 0x7f5ee00936b0 con 0x7f5ee0137e70 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): my global_id is 4434 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc0035d0 con 0x7f5ee0137e70 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362074494 0 0) 0x7f5ee0095b40 con 0x7f5ee009b1c0 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): my global_id is 4433 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee00936b0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427452032 0 0) 0x7f5ee009a540 con 0x7f5ee00964f0 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): my global_id is 4426 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0095b40 con 0x7f5ee00964f0 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882158296 0 0) 0x7f5ecc0035d0 con 0x7f5ee0137e70 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee009a540 con 0x7f5ee0137e70 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed80016e0 con 0x7f5ee0137e70 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): dump: 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.831 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3207133152 0 0) 0x7f5ee00936b0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ecc0035d0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed4001ba0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): dump: 2024-09-16T06:38:35.832 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2799057570 0 0) 0x7f5ee0095b40 con 0x7f5ee00964f0 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee00936b0 con 0x7f5ee00964f0 2024-09-16T06:38:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed0001750 con 0x7f5ee00964f0 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient(hunting): dump: 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2478966078 0 0) 0x7f5ecc0035d0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6789/0 conn(0x7f5ee00964f0 legacy=0x7f5ee0090450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.76:6789/0 conn(0x7f5ee0137e70 legacy=0x7f5ee01d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 monclient: found mon.c 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0091270 con 0x7f5ee009b1c0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 5 monclient: authenticate success, global_id 4433 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _renew_subs 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee00a2ec0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _renew_subs 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee0091f80 con 0x7f5ee009b1c0 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed40010c0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed4003bb0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5edd59a700 4 set_mon_vals no callback set 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 10 monclient: dump: 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5ec0006590 con 0x7f5ec0002830 2024-09-16T06:38:35.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.813+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5ed400e620 con 0x7f5ee009b1c0 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f5ed40420a0 con 0x7f5ee009b1c0 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834118577 0 0) 0x7f5ec0006590 con 0x7f5ec0002830 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient(hunting): my global_id is 4429 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc0035d0 con 0x7f5ec0002830 2024-09-16T06:38:35.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5edf59e700 10 monclient: get_auth_request con 0x7f5ecc0365b0 auth_method 0 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1284185094 0 0) 0x7f5ecc0035d0 con 0x7f5ec0002830 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec0006590 con 0x7f5ec0002830 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed0001d40 con 0x7f5ec0002830 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient: dump: 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3401843634 0 0) 0x7f5ec0006590 con 0x7f5ec0002830 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ec000a190 con 0x7f5ec0002830 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5ed0003820 con 0x7f5ec0002830 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6789/0 conn(0x7f5ec0002830 legacy=0x7f5ec0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.817+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6789/0 conn(0x7f5ec0002830 legacy=0x7f5ec0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.825+0000 7f5ee48ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.825+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5ec0009e50 con 0x7f5ec0009170 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218782999 0 0) 0x7f5ec0009e50 con 0x7f5ec0009170 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:35.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient(hunting): my global_id is 4432 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec0006590 con 0x7f5ec0009170 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3327539826 0 0) 0x7f5ec0006590 con 0x7f5ec0009170 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec0009e50 con 0x7f5ec0009170 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ed0003de0 con 0x7f5ec0009170 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient: dump: 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2972162143 0 0) 0x7f5ec0009e50 con 0x7f5ec0009170 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5ec000c340 con 0x7f5ec0009170 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 1182457808) 0x7f5ed0006150 con 0x7f5ec0009170 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6789/0 conn(0x7f5ec0009170 legacy=0x7f5ec0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ec77fe700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6789/0 conn(0x7f5ec0009170 legacy=0x7f5ec0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 12, 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:38:35.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 19, 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.846 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.847 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.848 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:35.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:35.850 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.851 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:35.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:38:35.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:38:35.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:35.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:35.857 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:38:35.857 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:38:35.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5ecc0365b0 legacy=0x7f5ecc038a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 10 monclient: shutdown 2024-09-16T06:38:35.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:35.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 >> v1:172.21.15.76:6790/0 conn(0x7f5ee009b1c0 legacy=0x7f5ee0090b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:35.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5edf59e700 1 -- 172.21.15.76:0/2755746224 reap_dead start 2024-09-16T06:38:35.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5edf59e700 1 -- 172.21.15.76:0/2755746224 reap_dead start 2024-09-16T06:38:35.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 shutdown_connections 2024-09-16T06:38:35.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 >> 172.21.15.76:0/2755746224 conn(0x7f5ee00a0030 msgr2=0x7f5ee009dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:35.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 shutdown_connections 2024-09-16T06:38:35.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:35.829+0000 7f5ee48ec700 1 -- 172.21.15.76:0/2755746224 wait complete. 2024-09-16T06:38:35.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:38:35.859 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:38:35.860 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:38:35.860 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:38:35.860 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:38:35.860 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:38:35.860 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:38:35.890 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:38:35.890 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:38:35.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: build_initial_monmap 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: monmap: 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:36.008208+0000 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:36.008208+0000 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:36.006 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:36.007 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:36.007 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:36.007 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:36.007 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 Processor -- start 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- start start 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: init 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f50cc0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f50cc097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f50cc1400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc099350 con 0x7f50cc1400d0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc09f690 con 0x7f50cc097eb0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc099bf0 con 0x7f50cc0973c0 2024-09-16T06:38:36.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient(hunting): authenticate will time out at 863.141464s 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50ca7fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f50cc097eb0 legacy=0x7f50cc13e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50caffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f50cc0973c0 0x7f50cc0977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42032/0 (socket says 172.21.15.76:42032) 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50caffd700 1 -- 172.21.15.76:0/2400545616 learned_addr learned my addr 172.21.15.76:0/2400545616 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352447532 0 0) 0x7f50cc099350 con 0x7f50cc1400d0 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): my global_id is 4437 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b40035d0 con 0x7f50cc1400d0 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450521619 0 0) 0x7f50cc099bf0 con 0x7f50cc0973c0 2024-09-16T06:38:36.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient(hunting): my global_id is 4436 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc099350 con 0x7f50cc0973c0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1954671184 0 0) 0x7f50b40035d0 con 0x7f50cc1400d0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 >> v1:172.21.15.76:6790/0 conn(0x7f50cc0973c0 legacy=0x7f50cc0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 >> v1:172.21.15.189:6789/0 conn(0x7f50cc097eb0 legacy=0x7f50cc13e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 monclient: found mon.noname-a 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc098360 con 0x7f50cc1400d0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:36.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 5 monclient: authenticate success, global_id 4437 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50c4001990 con 0x7f50cc1400d0 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: dump: 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:36.014 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c4001c70 con 0x7f50cc1400d0 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 1 -- 172.21.15.76:0/2400545616 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50c4002000 con 0x7f50cc1400d0 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 4 set_mon_vals no callback set 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:36.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50c9ffb700 10 monclient: dump: 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 10 monclient: shutdown 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:36.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- 172.21.15.76:0/2400545616 >> v1:172.21.15.76:6789/0 conn(0x7f50cc1400d0 legacy=0x7f50cc1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- 172.21.15.76:0/2400545616 shutdown_connections 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- 172.21.15.76:0/2400545616 >> 172.21.15.76:0/2400545616 conn(0x7f50cc0a0030 msgr2=0x7f50cc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- 172.21.15.76:0/2400545616 shutdown_connections 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.005+0000 7f50cbfff700 1 -- 172.21.15.76:0/2400545616 wait complete. 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: build_initial_monmap 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: monmap: 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:36.017 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:36.018 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:36.018 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 Processor -- start 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- start start 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: init 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f50cc0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:36.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f50cc097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f50cc1400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc072660 con 0x7f50cc097eb0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc06c370 con 0x7f50cc1400d0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc13fda0 con 0x7f50cc0973c0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient(hunting): authenticate will time out at 863.146389s 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50ca7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f50cc097eb0 0x7f50cc1d3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39428/0 (socket says 172.21.15.76:39428) 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50ca7fc700 1 -- 172.21.15.76:0/2239997256 learned_addr learned my addr 172.21.15.76:0/2239997256 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cb7fe700 1 -- 172.21.15.76:0/2239997256 >> v1:172.21.15.189:6789/0 conn(0x7f50cc1400d0 legacy=0x7f50cc1d6bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540910917 0 0) 0x7f50cc072660 con 0x7f50cc097eb0 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:36.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): my global_id is 4440 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b80035d0 con 0x7f50cc097eb0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999237494 0 0) 0x7f50cc13fda0 con 0x7f50cc0973c0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): my global_id is 4439 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc072660 con 0x7f50cc0973c0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453248557 0 0) 0x7f50b80035d0 con 0x7f50cc097eb0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50cc13fda0 con 0x7f50cc097eb0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50bc004230 con 0x7f50cc097eb0 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:36.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): dump: 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:36.027 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1636118683 0 0) 0x7f50cc13fda0 con 0x7f50cc097eb0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 >> v1:172.21.15.76:6790/0 conn(0x7f50cc0973c0 legacy=0x7f50cc1ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 >> v1:172.21.15.189:6789/0 conn(0x7f50cc1400d0 legacy=0x7f50cc1d6bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 monclient: found mon.a 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc09a540 con 0x7f50cc097eb0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 5 monclient: authenticate success, global_id 4440 2024-09-16T06:38:36.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _renew_subs 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc13ed50 con 0x7f50cc097eb0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _renew_subs 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.009+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc06c910 con 0x7f50cc097eb0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50bc004510 con 0x7f50cc097eb0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50bc0048a0 con 0x7f50cc097eb0 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50c97fa700 4 set_mon_vals no callback set 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 10 monclient: dump: 2024-09-16T06:38:36.029 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:36.030 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50cbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f50bc004a80 con 0x7f50cc097eb0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50cbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50cbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac005100 con 0x7f50cc097eb0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f50bc0431b0 con 0x7f50cc097eb0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.013+0000 7f50caffd700 10 monclient: get_auth_request con 0x7f50b80361b0 auth_method 0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.017+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f50bc005250 con 0x7f50cc097eb0 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.017+0000 7f50b37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:36.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.017+0000 7f50b37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.253+0000 7f50cbfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:38:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.253+0000 7f50cbfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.253+0000 7f50cbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.253+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f50ac005930 con 0x7f50cc097eb0 2024-09-16T06:38:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50b37fe700 1 -- 172.21.15.76:0/2239997256 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2475105355) 0x7f50bc012c30 con 0x7f50cc097eb0 2024-09-16T06:38:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50b37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50b37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:36.258 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 >> v1:172.21.15.189:6812/3098810674 conn(0x7f50b80361b0 legacy=0x7f50b8038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 10 monclient: shutdown 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 >> v1:172.21.15.76:6789/0 conn(0x7f50cc097eb0 legacy=0x7f50cc1d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 shutdown_connections 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 >> 172.21.15.76:0/2239997256 conn(0x7f50cc0a0030 msgr2=0x7f50cc09c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 shutdown_connections 2024-09-16T06:38:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:36.257+0000 7f50cbfff700 1 -- 172.21.15.76:0/2239997256 wait complete. 2024-09-16T06:38:39.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:38:39.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:38:39.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:39.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: monmap: 2024-09-16T06:38:39.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:39.390500+0000 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:39.390500+0000 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:39.389 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 1 Processor -- start 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 1 -- start start 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: init 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f37c412fbf0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f37c412f7e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f37c4139720 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:39.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c409db90 con 0x7f37c412f7e0 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c409d0f0 con 0x7f37c4139720 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c41306e0 con 0x7f37c412fbf0 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c3fff700 10 monclient(hunting): authenticate will time out at 866.523746s 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c27fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f37c412f7e0 0x7f37c4138000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39444/0 (socket says 172.21.15.76:39444) 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c27fc700 1 -- 172.21.15.76:0/3752224067 learned_addr learned my addr 172.21.15.76:0/3752224067 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.385+0000 7f37c37fe700 1 -- 172.21.15.76:0/3752224067 >> v1:172.21.15.189:6789/0 conn(0x7f37c4139720 legacy=0x7f37c413bb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813046654 0 0) 0x7f37c409db90 con 0x7f37c412f7e0 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient(hunting): my global_id is 4443 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac0035d0 con 0x7f37c412f7e0 2024-09-16T06:38:39.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898382059 0 0) 0x7f37ac0035d0 con 0x7f37c412f7e0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 >> v1:172.21.15.76:6790/0 conn(0x7f37c412fbf0 legacy=0x7f37c412ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 >> v1:172.21.15.189:6789/0 conn(0x7f37c4139720 legacy=0x7f37c413bb10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 monclient: found mon.noname-a 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c409cc70 con 0x7f37c412f7e0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37b4001990 con 0x7f37c412f7e0 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 5 monclient: authenticate success, global_id 4443 2024-09-16T06:38:39.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: dump: 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:39.400 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37b4001cb0 con 0x7f37c412f7e0 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 4 set_mon_vals no callback set 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 1 -- 172.21.15.76:0/3752224067 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37b4002040 con 0x7f37c412f7e0 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c1ffb700 10 monclient: dump: 2024-09-16T06:38:39.401 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: shutdown 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- 172.21.15.76:0/3752224067 >> v1:172.21.15.76:6789/0 conn(0x7f37c412f7e0 legacy=0x7f37c4138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:39.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- 172.21.15.76:0/3752224067 shutdown_connections 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- 172.21.15.76:0/3752224067 >> 172.21.15.76:0/3752224067 conn(0x7f37c40a5870 msgr2=0x7f37c40a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- 172.21.15.76:0/3752224067 shutdown_connections 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- 172.21.15.76:0/3752224067 wait complete. 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: monmap: 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:39.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 Processor -- start 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- start start 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: init 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f37c412f7e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f37c412fbf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f37c4139720 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.389+0000 7f37c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c4138eb0 con 0x7f37c412fbf0 2024-09-16T06:38:39.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c406fc10 con 0x7f37c4139720 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c409c640 con 0x7f37c412f7e0 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f37c412f7e0 0x7f37c4098330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42064/0 (socket says 172.21.15.76:42064) 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient(hunting): authenticate will time out at 866.528099s 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c2ffd700 1 -- 172.21.15.76:0/1775275294 learned_addr learned my addr 172.21.15.76:0/1775275294 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c37fe700 1 -- 172.21.15.76:0/1775275294 >> v1:172.21.15.189:6789/0 conn(0x7f37c4139720 legacy=0x7f37c41d0500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382207555 0 0) 0x7f37c4138eb0 con 0x7f37c412fbf0 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): my global_id is 4446 2024-09-16T06:38:39.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b00035d0 con 0x7f37c412fbf0 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2624914784 0 0) 0x7f37b00035d0 con 0x7f37c412fbf0 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37c4138eb0 con 0x7f37c412fbf0 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37b4001950 con 0x7f37c412fbf0 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): dump: 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:39.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3851430325 0 0) 0x7f37c4138eb0 con 0x7f37c412fbf0 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 >> v1:172.21.15.76:6790/0 conn(0x7f37c412f7e0 legacy=0x7f37c4098330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 >> v1:172.21.15.189:6789/0 conn(0x7f37c4139720 legacy=0x7f37c41d0500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 monclient: found mon.a 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40704b0 con 0x7f37c412fbf0 2024-09-16T06:38:39.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: _finish_auth 0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 5 monclient: authenticate success, global_id 4446 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _renew_subs 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c406eed0 con 0x7f37c412fbf0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _renew_subs 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c412c570 con 0x7f37c412fbf0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37b4005070 con 0x7f37c412fbf0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37b4005380 con 0x7f37c412fbf0 2024-09-16T06:38:39.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 10 monclient: dump: 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:39.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c17fa700 4 set_mon_vals no callback set 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c406fe50 con 0x7f37c412fbf0 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f37b400fdf0 con 0x7f37c412fbf0 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f37b40436e0 con 0x7f37c412fbf0 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.393+0000 7f37c2ffd700 10 monclient: get_auth_request con 0x7f37b00360b0 auth_method 0 2024-09-16T06:38:39.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.401+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f37b4014ba0 con 0x7f37c412fbf0 2024-09-16T06:38:39.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.401+0000 7f37ab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:39.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.401+0000 7f37ab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:39.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37c3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:38:39.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37c3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:39.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:39.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f37c40672a0 con 0x7f37c412fbf0 2024-09-16T06:38:39.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37ab7fe700 1 -- 172.21.15.76:0/1775275294 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1601735675) 0x7f37b40149c0 con 0x7f37c412fbf0 2024-09-16T06:38:39.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37ab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:39.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.633+0000 7f37ab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:39.637 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 >> v1:172.21.15.189:6812/3098810674 conn(0x7f37b00360b0 legacy=0x7f37b0038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 10 monclient: shutdown 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 >> v1:172.21.15.76:6789/0 conn(0x7f37c412fbf0 legacy=0x7f37c41ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 shutdown_connections 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 >> 172.21.15.76:0/1775275294 conn(0x7f37c40a5870 msgr2=0x7f37c41328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 shutdown_connections 2024-09-16T06:38:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:39.637+0000 7f37c3fff700 1 -- 172.21.15.76:0/1775275294 wait complete. 2024-09-16T06:38:42.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.765+0000 7f68952c8700 10 monclient: get_monmap_and_config 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.765+0000 7f68952c8700 10 monclient: build_initial_monmap 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.765+0000 7f68952c8700 10 monclient: monmap: 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:42.771873+0000 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:42.771873+0000 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:42.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 1 Processor -- start 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 1 -- start start 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient: init 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-a con 0x7f6890132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-c con 0x7f689006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-b con 0x7f689006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689006c280 con 0x7f6890132330 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689006bfa0 con 0x7f689006d480 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689009ace0 con 0x7f689006cf50 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:42.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f68952c8700 10 monclient(hunting): authenticate will time out at 869.905725s 2024-09-16T06:38:42.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f688f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f689013a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:42.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f6894ac7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f689006cf50 0x7f689013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42070/0 (socket says 172.21.15.76:42070) 2024-09-16T06:38:42.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.769+0000 7f6894ac7700 1 -- 172.21.15.76:0/1336368853 learned_addr learned my addr 172.21.15.76:0/1336368853 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:42.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:42.973+0000 7f688f7fe700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f689013a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:43.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:43.377+0000 7f688f7fe700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f689013a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:44.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:44.181+0000 7f688f7fe700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f689013a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): tick 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6790/0 conn(0x7f689006cf50 legacy=0x7f689013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f689013a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6789/0 conn(0x7f6890132330 legacy=0x7f6890132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-c con 0x7f689006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-a con 0x7f6874006ed0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-b con 0x7f689006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689006bfa0 con 0x7f6874006ed0 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687400f360 con 0x7f689006d480 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6874010570 con 0x7f689006cf50 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688dffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.769+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:45.973+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:46.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:46.377+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:47.181+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:48.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): tick 2024-09-16T06:38:48.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6790/0 conn(0x7f689006cf50 legacy=0x7f687400cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6789/0 conn(0x7f6874006ed0 legacy=0x7f6874009360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:38:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-c con 0x7f689006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-b con 0x7f689006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): picked mon.noname-a con 0x7f6874010830 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6874011780 con 0x7f6874010830 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687400f360 con 0x7f689006d480 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68740114b0 con 0x7f689006cf50 2024-09-16T06:38:48.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.773+0000 7f688dffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:48.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:48.977+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:49.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:49.381+0000 7f688ffff700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215062844 0 0) 0x7f68740114b0 con 0x7f689006cf50 2024-09-16T06:38:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient(hunting): my global_id is 4442 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68700035d0 con 0x7f689006cf50 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124470828 0 0) 0x7f68700035d0 con 0x7f689006cf50 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f6874005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6789/0 conn(0x7f6874010830 legacy=0x7f6874010c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 monclient: found mon.noname-b 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874009370 con 0x7f689006cf50 2024-09-16T06:38:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68880055d0 con 0x7f689006cf50 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f68952c8700 5 monclient: authenticate success, global_id 4442 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: dump: 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f68952c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6888005870 con 0x7f689006cf50 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f68952c8700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 1 -- 172.21.15.76:0/1336368853 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6888005c40 con 0x7f689006cf50 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f68952c8700 4 set_mon_vals no callback set 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.161+0000 7f688effd700 10 monclient: dump: 2024-09-16T06:38:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 10 monclient: shutdown 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- 172.21.15.76:0/1336368853 >> v1:172.21.15.76:6790/0 conn(0x7f689006cf50 legacy=0x7f687400cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- 172.21.15.76:0/1336368853 shutdown_connections 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- 172.21.15.76:0/1336368853 >> 172.21.15.76:0/1336368853 conn(0x7f68900a41c0 msgr2=0x7f68900a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- 172.21.15.76:0/1336368853 shutdown_connections 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- 172.21.15.76:0/1336368853 wait complete. 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 10 monclient: build_initial_monmap 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 10 monclient: monmap: 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 Processor -- start 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 1 -- start start 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.165+0000 7f68952c8700 10 monclient: init 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-a con 0x7f6874010830 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-c con 0x7f689006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient(hunting): picked mon.noname-b con 0x7f689006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6890122fd0 con 0x7f6874010830 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6890124410 con 0x7f689006d480 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68901246f0 con 0x7f689006cf50 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient(hunting): authenticate will time out at 877.305069s 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6874010830 0x7f68901c9f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:46776/0 (socket says 172.21.15.76:46776) 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f6894ac7700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f68901d2810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688ffff700 1 -- 172.21.15.76:0/448480627 learned_addr learned my addr 172.21.15.76:0/448480627 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407304061 0 0) 0x7f68901246f0 con 0x7f689006cf50 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): my global_id is 4445 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6874011460 con 0x7f689006cf50 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235735636 0 0) 0x7f6874011460 con 0x7f689006cf50 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68901246f0 con 0x7f689006cf50 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6888003e50 con 0x7f689006cf50 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): dump: 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.176 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 806376994 0 0) 0x7f68901246f0 con 0x7f689006cf50 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 >> v1:172.21.15.189:6789/0 conn(0x7f689006d480 legacy=0x7f68901d2810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 >> v1:172.21.15.76:6789/0 conn(0x7f6874010830 legacy=0x7f68901c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 monclient: found mon.c 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6890131a40 con 0x7f689006cf50 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 5 monclient: authenticate success, global_id 4445 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _renew_subs 2024-09-16T06:38:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6890122390 con 0x7f689006cf50 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _renew_subs 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6890122ce0 con 0x7f689006cf50 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68880043c0 con 0x7f689006cf50 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68880046d0 con 0x7f689006cf50 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688e7fc700 4 set_mon_vals no callback set 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f688cff9700 10 monclient: dump: 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.169+0000 7f68952c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f68952c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f68952c8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868005100 con 0x7f689006cf50 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f68880048b0 con 0x7f689006cf50 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f6888043620 con 0x7f689006cf50 2024-09-16T06:38:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.173+0000 7f688ffff700 10 monclient: get_auth_request con 0x7f6874037070 auth_method 0 2024-09-16T06:38:50.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.181+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6888014a80 con 0x7f689006cf50 2024-09-16T06:38:50.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.181+0000 7f688cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.181+0000 7f688cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:38:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6868005930 con 0x7f689006cf50 2024-09-16T06:38:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f688cff9700 1 -- 172.21.15.76:0/448480627 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2359101255) 0x7f6888013120 con 0x7f689006cf50 2024-09-16T06:38:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f688cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:38:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f688cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:50.417 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:38:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6874037070 legacy=0x7f6874039530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 10 monclient: shutdown 2024-09-16T06:38:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 >> v1:172.21.15.76:6790/0 conn(0x7f689006cf50 legacy=0x7f68901cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 shutdown_connections 2024-09-16T06:38:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.413+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 >> 172.21.15.76:0/448480627 conn(0x7f68900a41c0 msgr2=0x7f689009eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.417+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 shutdown_connections 2024-09-16T06:38:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.417+0000 7f68952c8700 1 -- 172.21.15.76:0/448480627 wait complete. 2024-09-16T06:38:50.430 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:38:50.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:38:50.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: monmap: 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.546 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:50.548519+0000 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:50.548519+0000 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:50.547 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 Processor -- start 2024-09-16T06:38:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- start start 2024-09-16T06:38:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: init 2024-09-16T06:38:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f800c09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f800c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f800c099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c09d560 con 0x7f800c09ed60 2024-09-16T06:38:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c09f140 con 0x7f800c099030 2024-09-16T06:38:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c09d790 con 0x7f800c098be0 2024-09-16T06:38:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient(hunting): authenticate will time out at 877.681999s 2024-09-16T06:38:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f8009d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f800c098be0 0x7f800c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43440/0 (socket says 172.21.15.76:43440) 2024-09-16T06:38:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f8009d9b700 1 -- 172.21.15.76:0/3501821335 learned_addr learned my addr 172.21.15.76:0/3501821335 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800ad9d700 1 -- 172.21.15.76:0/3501821335 >> v1:172.21.15.189:6789/0 conn(0x7f800c099030 legacy=0x7f800c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308914796 0 0) 0x7f800c09d790 con 0x7f800c098be0 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): my global_id is 4448 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff40057e0 con 0x7f800c098be0 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995862240 0 0) 0x7f800c09d560 con 0x7f800c09ed60 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient(hunting): my global_id is 4449 2024-09-16T06:38:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c09d790 con 0x7f800c09ed60 2024-09-16T06:38:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 831293804 0 0) 0x7f800c09d790 con 0x7f800c09ed60 2024-09-16T06:38:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 >> v1:172.21.15.76:6790/0 conn(0x7f800c098be0 legacy=0x7f800c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 >> v1:172.21.15.189:6789/0 conn(0x7f800c099030 legacy=0x7f800c13ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 monclient: found mon.noname-a 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c09f620 con 0x7f800c09ed60 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 5 monclient: authenticate success, global_id 4449 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80000015f0 con 0x7f800c09ed60 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: dump: 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.570 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8000001890 con 0x7f800c09ed60 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 1 -- 172.21.15.76:0/3501821335 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8000001c60 con 0x7f800c09ed60 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 4 set_mon_vals no callback set 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800959a700 10 monclient: dump: 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 10 monclient: shutdown 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- 172.21.15.76:0/3501821335 >> v1:172.21.15.76:6789/0 conn(0x7f800c09ed60 legacy=0x7f800c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.545+0000 7f800b59e700 1 -- 172.21.15.76:0/3501821335 shutdown_connections 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- 172.21.15.76:0/3501821335 >> 172.21.15.76:0/3501821335 conn(0x7f800c0a0030 msgr2=0x7f800c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- 172.21.15.76:0/3501821335 shutdown_connections 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- 172.21.15.76:0/3501821335 wait complete. 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: build_initial_monmap 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: monmap: 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 Processor -- start 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- start start 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: init 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f800c098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f800c099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:50.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f800c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0a2cc0 con 0x7f800c098be0 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0984d0 con 0x7f800c09ed60 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c090d50 con 0x7f800c099030 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient(hunting): authenticate will time out at 877.686179s 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800ad9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f800c09ed60 legacy=0x7f800c1cdd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f800c098be0 0x7f800c0962c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:46794/0 (socket says 172.21.15.76:46794) 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800a59c700 1 -- 172.21.15.76:0/3928979617 learned_addr learned my addr 172.21.15.76:0/3928979617 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017688140 0 0) 0x7f800c0a2cc0 con 0x7f800c098be0 2024-09-16T06:38:50.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): my global_id is 4452 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fec0035d0 con 0x7f800c098be0 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862155015 0 0) 0x7f800c090d50 con 0x7f800c099030 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): my global_id is 4451 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c0a2cc0 con 0x7f800c099030 2024-09-16T06:38:50.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747011916 0 0) 0x7f7fec0035d0 con 0x7f800c098be0 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f800c090d50 con 0x7f800c098be0 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8000001c60 con 0x7f800c098be0 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): dump: 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.577 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768288417 0 0) 0x7f800c0a2cc0 con 0x7f800c099030 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fec0035d0 con 0x7f800c099030 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ffc0016a0 con 0x7f800c099030 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient(hunting): dump: 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.578 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3398277172 0 0) 0x7f800c090d50 con 0x7f800c098be0 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6790/0 conn(0x7f800c099030 legacy=0x7f800c1ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.189:6789/0 conn(0x7f800c09ed60 legacy=0x7f800c1cdd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 monclient: found mon.a 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c08f340 con 0x7f800c098be0 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 5 monclient: authenticate success, global_id 4452 2024-09-16T06:38:50.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _renew_subs 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c090a70 con 0x7f800c098be0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _renew_subs 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c096bb0 con 0x7f800c098be0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8000003ab0 con 0x7f800c098be0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8000003e40 con 0x7f800c098be0 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f8008d99700 4 set_mon_vals no callback set 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.549+0000 7f7ffaffd700 10 monclient: dump: 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f800b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ff0006590 con 0x7f7ff0002830 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8000004020 con 0x7f800c098be0 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f8000043480 con 0x7f800c098be0 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394938257 0 0) 0x7f7ff0006590 con 0x7f7ff0002830 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient(hunting): my global_id is 4455 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c090d50 con 0x7f7ff0002830 2024-09-16T06:38:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f800ad9d700 10 monclient: get_auth_request con 0x7f7fec0362c0 auth_method 0 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 902517985 0 0) 0x7f800c090d50 con 0x7f7ff0002830 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff0006590 con 0x7f7ff0002830 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ffc001d60 con 0x7f7ff0002830 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient: dump: 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 282538686 0 0) 0x7f7ff0006590 con 0x7f7ff0002830 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ff000a190 con 0x7f7ff0002830 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7ffc006ac0 con 0x7f7ff0002830 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6789/0 conn(0x7f7ff0002830 legacy=0x7f7ff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.553+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6789/0 conn(0x7f7ff0002830 legacy=0x7f7ff0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:38:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ff0009e30 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674961317 0 0) 0x7f7ff0009e30 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient(hunting): my global_id is 4458 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0006590 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2914391277 0 0) 0x7f7ff0006590 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff0009e30 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ffc004f00 con 0x7f7ff0009150 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient: dump: 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1485086244 0 0) 0x7f7ff0009e30 con 0x7f7ff0009150 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7ff000c340 con 0x7f7ff0009150 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 3375071680) 0x7f7ffc0063b0 con 0x7f7ff0009150 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6789/0 conn(0x7f7ff0009150 legacy=0x7f7ff00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f7ffaffd700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6789/0 conn(0x7f7ff0009150 legacy=0x7f7ff00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:38:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 16, 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:38:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:38:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:38:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:38:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:38:50.605 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7fec0362c0 legacy=0x7f7fec038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 10 monclient: shutdown 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 >> v1:172.21.15.76:6789/0 conn(0x7f800c098be0 legacy=0x7f800c0962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800ad9d700 1 -- 172.21.15.76:0/3928979617 reap_dead start 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 shutdown_connections 2024-09-16T06:38:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 >> 172.21.15.76:0/3928979617 conn(0x7f800c0a0030 msgr2=0x7f800c0972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 shutdown_connections 2024-09-16T06:38:50.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.565+0000 7f800b59e700 1 -- 172.21.15.76:0/3928979617 wait complete. 2024-09-16T06:38:50.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:38:50.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:38:50.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:38:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: get_monmap_and_config 2024-09-16T06:38:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: build_initial_monmap 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: monmap: 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:38:50.728789+0000 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:38:50.728789+0000 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:38:50.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 Processor -- start 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 -- start start 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: init 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-a con 0x7fda18132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-b con 0x7fda1806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-c con 0x7fda1806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1806c280 con 0x7fda18132330 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1806bfa0 con 0x7fda1806cf50 2024-09-16T06:38:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1809ace0 con 0x7fda1806d480 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient(hunting): authenticate will time out at 877.862199s 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda15d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fda1806d480 0x7fda1813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43448/0 (socket says 172.21.15.76:43448) 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda15d9b700 1 -- 172.21.15.76:0/3216762051 learned_addr learned my addr 172.21.15.76:0/3216762051 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda16d9d700 1 -- 172.21.15.76:0/3216762051 >> v1:172.21.15.189:6789/0 conn(0x7fda1806cf50 legacy=0x7fda1813da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470637456 0 0) 0x7fda1806c280 con 0x7fda18132330 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): my global_id is 4461 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda040035d0 con 0x7fda18132330 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082429752 0 0) 0x7fda1809ace0 con 0x7fda1806d480 2024-09-16T06:38:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient(hunting): my global_id is 4454 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda1806c280 con 0x7fda1806d480 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024717273 0 0) 0x7fda040035d0 con 0x7fda18132330 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 >> v1:172.21.15.76:6790/0 conn(0x7fda1806d480 legacy=0x7fda1813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 >> v1:172.21.15.189:6789/0 conn(0x7fda1806cf50 legacy=0x7fda1813da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 monclient: found mon.noname-a 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:38:50.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1809ae70 con 0x7fda18132330 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda08001610 con 0x7fda18132330 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: dump: 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 5 monclient: authenticate success, global_id 4461 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda080018f0 con 0x7fda18132330 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 1 -- 172.21.15.76:0/3216762051 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda08001ae0 con 0x7fda18132330 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 4 set_mon_vals no callback set 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:38:50.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1559a700 10 monclient: dump: 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 10 monclient: shutdown 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.725+0000 7fda1759e700 1 -- 172.21.15.76:0/3216762051 >> v1:172.21.15.76:6789/0 conn(0x7fda18132330 legacy=0x7fda18132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3216762051 shutdown_connections 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3216762051 >> 172.21.15.76:0/3216762051 conn(0x7fda180a41c0 msgr2=0x7fda180a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3216762051 shutdown_connections 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3216762051 wait complete. 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: build_initial_monmap 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: monmap: 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:38:50.739 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 Processor -- start 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- start start 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: init 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-c con 0x7fda1806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-b con 0x7fda1806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient(hunting): picked mon.noname-a con 0x7fda18132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda181317e0 con 0x7fda18132330 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda181306f0 con 0x7fda1806d480 2024-09-16T06:38:50.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1809e8b0 con 0x7fda1806cf50 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient(hunting): authenticate will time out at 877.865539s 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fda1806cf50 0x7fda1813e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43464/0 (socket says 172.21.15.76:43464) 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1659c700 1 -- 172.21.15.76:0/3658871777 learned_addr learned my addr 172.21.15.76:0/3658871777 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda15d9b700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.189:6789/0 conn(0x7fda1806d480 legacy=0x7fda181ceb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690382079 0 0) 0x7fda1809e8b0 con 0x7fda1806cf50 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): my global_id is 4457 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f80035d0 con 0x7fda1806cf50 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991771380 0 0) 0x7fda10003df0 con 0x7fda18132330 2024-09-16T06:38:50.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): my global_id is 4464 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda1809e8b0 con 0x7fda18132330 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325204439 0 0) 0x7fd9f80035d0 con 0x7fda1806cf50 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda10003df0 con 0x7fda1806cf50 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda080036a0 con 0x7fda1806cf50 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): dump: 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.742 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3009106602 0 0) 0x7fda10003df0 con 0x7fda1806cf50 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.189:6789/0 conn(0x7fda1806d480 legacy=0x7fda181ceb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6789/0 conn(0x7fda18132330 legacy=0x7fda181d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: _finish_hunting 0 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 monclient: found mon.c 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:38:50.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18131260 con 0x7fda1806cf50 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: _finish_auth 0 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: _check_auth_tickets 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 5 monclient: authenticate success, global_id 4457 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _renew_subs 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda18130cb0 con 0x7fda1806cf50 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _renew_subs 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:38:50.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1812f890 con 0x7fda1806cf50 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda08003bc0 con 0x7fda1806cf50 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda08003f50 con 0x7fda1806cf50 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda14d99700 4 set_mon_vals no callback set 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda0effd700 10 monclient: dump: 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.745 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.729+0000 7fda1759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd9fc006590 con 0x7fd9fc002830 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fda080041b0 con 0x7fda1806cf50 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fda08043350 con 0x7fda1806cf50 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377856420 0 0) 0x7fd9fc006590 con 0x7fd9fc002830 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient(hunting): my global_id is 4460 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda10003df0 con 0x7fd9fc002830 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2097238308 0 0) 0x7fda10003df0 con 0x7fd9fc002830 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9fc006590 con 0x7fd9fc002830 2024-09-16T06:38:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda00001740 con 0x7fd9fc002830 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient: dump: 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda16d9d700 10 monclient: get_auth_request con 0x7fd9f8036180 auth_method 0 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1330461327 0 0) 0x7fd9fc006590 con 0x7fd9fc002830 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9fc00a190 con 0x7fd9fc002830 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fda00001a20 con 0x7fd9fc002830 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6790/0 conn(0x7fd9fc002830 legacy=0x7fd9fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.733+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6790/0 conn(0x7fd9fc002830 legacy=0x7fd9fc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd9fc009e30 con 0x7fd9fc009150 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101293093 0 0) 0x7fd9fc009e30 con 0x7fd9fc009150 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient(hunting): _negotiate 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:38:50.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient(hunting): my global_id is 4463 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9fc006590 con 0x7fd9fc009150 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2735203411 0 0) 0x7fd9fc006590 con 0x7fd9fc009150 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9fc009e30 con 0x7fd9fc009150 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fda00004a30 con 0x7fd9fc009150 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient: dump: 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.752 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3005335851 0 0) 0x7fd9fc009e30 con 0x7fd9fc009150 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9fc00c340 con 0x7fd9fc009150 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 614150294) 0x7fda00004d70 con 0x7fd9fc009150 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6790/0 conn(0x7fd9fc009150 legacy=0x7fd9fc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda0effd700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6790/0 conn(0x7fd9fc009150 legacy=0x7fd9fc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 16, 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:38:50.754 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.755 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.756 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:38:50.757 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:38:50.758 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.759 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:38:50.760 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:38:50.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:38:50.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.764 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:38:50.765 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:38:50.766 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd9f8036180 legacy=0x7fd9f8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 10 monclient: shutdown 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 >> v1:172.21.15.76:6790/0 conn(0x7fda1806cf50 legacy=0x7fda1813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda16d9d700 1 -- 172.21.15.76:0/3658871777 reap_dead start 2024-09-16T06:38:50.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda16d9d700 1 -- 172.21.15.76:0/3658871777 reap_dead start 2024-09-16T06:38:50.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 shutdown_connections 2024-09-16T06:38:50.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 >> 172.21.15.76:0/3658871777 conn(0x7fda180a41c0 msgr2=0x7fda1809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:38:50.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 shutdown_connections 2024-09-16T06:38:50.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:38:50.741+0000 7fda1759e700 1 -- 172.21.15.76:0/3658871777 wait complete. 2024-09-16T06:38:50.768 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:38:50.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:39:10.769 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:39:10.770 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:39:10.770 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:39:10.770 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:39:10.773 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:39:10.773 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:39:10.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 10 monclient: get_monmap_and_config 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 10 monclient: build_initial_monmap 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 10 monclient: monmap: 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:10.897237+0000 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:10.897237+0000 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:10.895 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 1 Processor -- start 2024-09-16T06:39:10.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 1 -- start start 2024-09-16T06:39:10.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 10 monclient: init 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.892+0000 7fe743c31700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-c con 0x7fe73c09ea50 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-a con 0x7fe73c09ee30 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-b con 0x7fe73c137c20 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c09b9d0 con 0x7fe73c09ee30 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c0a0970 con 0x7fe73c09ea50 2024-09-16T06:39:10.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c09c7e0 con 0x7fe73c137c20 2024-09-16T06:39:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe743c31700 10 monclient(hunting): authenticate will time out at 898.032230s 2024-09-16T06:39:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe7419cd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe73c09ea50 0x7fe73c09c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:60622/0 (socket says 172.21.15.76:60622) 2024-09-16T06:39:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe7419cd700 1 -- 172.21.15.76:0/1359208729 learned_addr learned my addr 172.21.15.76:0/1359208729 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:10.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.896+0000 7fe7411cc700 1 --1- 172.21.15.76:0/1359208729 >> v1:172.21.15.76:6789/0 conn(0x7fe73c09ee30 0x7fe73c09f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42250/0 (socket says 172.21.15.76:42250) 2024-09-16T06:39:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262478604 0 0) 0x7fe73c09c7e0 con 0x7fe73c137c20 2024-09-16T06:39:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:10.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient(hunting): my global_id is 4466 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7280035d0 con 0x7fe73c137c20 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598255678 0 0) 0x7fe7280035d0 con 0x7fe73c137c20 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 >> v1:172.21.15.189:6789/0 conn(0x7fe73c09ea50 legacy=0x7fe73c09c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 >> v1:172.21.15.76:6789/0 conn(0x7fe73c09ee30 legacy=0x7fe73c09f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 monclient: found mon.noname-b 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c09b6f0 con 0x7fe73c137c20 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: _finish_auth 0 2024-09-16T06:39:10.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 5 monclient: authenticate success, global_id 4466 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe734001990 con 0x7fe73c137c20 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: dump: 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:10.921 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe734001c40 con 0x7fe73c137c20 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 1 -- 172.21.15.76:0/1359208729 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe734002010 con 0x7fe73c137c20 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 4 set_mon_vals no callback set 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:10.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe7409cb700 10 monclient: dump: 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:10.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 10 monclient: shutdown 2024-09-16T06:39:10.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 1 -- 172.21.15.76:0/1359208729 >> v1:172.21.15.76:6790/0 conn(0x7fe73c137c20 legacy=0x7fe73c13a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 1 -- 172.21.15.76:0/1359208729 shutdown_connections 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 1 -- 172.21.15.76:0/1359208729 >> 172.21.15.76:0/1359208729 conn(0x7fe73c0a0d30 msgr2=0x7fe73c0a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 1 -- 172.21.15.76:0/1359208729 shutdown_connections 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.916+0000 7fe743c31700 1 -- 172.21.15.76:0/1359208729 wait complete. 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient: build_initial_monmap 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient: monmap: 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:10.926 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 1 Processor -- start 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 1 -- start start 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient: init 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-c con 0x7fe73c09ea50 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-a con 0x7fe73c09ee30 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient(hunting): picked mon.noname-b con 0x7fe73c137c20 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c11ea20 con 0x7fe73c09ee30 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c131150 con 0x7fe73c137c20 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c09b240 con 0x7fe73c09ea50 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe743c31700 10 monclient(hunting): authenticate will time out at 898.058494s 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe7419cd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe73c09ea50 0x7fe73c130a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60422/0 (socket says 172.21.15.76:60422) 2024-09-16T06:39:10.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.920+0000 7fe7419cd700 1 -- 172.21.15.76:0/618826201 learned_addr learned my addr 172.21.15.76:0/618826201 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641688299 0 0) 0x7fe73c11ea20 con 0x7fe73c09ee30 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): my global_id is 4467 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7200035d0 con 0x7fe73c09ee30 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577763236 0 0) 0x7fe7200035d0 con 0x7fe73c09ee30 2024-09-16T06:39:10.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe73c11ea20 con 0x7fe73c09ee30 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe72c0017e0 con 0x7fe73c09ee30 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): dump: 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:10.932 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4194300200 0 0) 0x7fe73c11ea20 con 0x7fe73c09ee30 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 >> v1:172.21.15.76:6790/0 conn(0x7fe73c09ea50 legacy=0x7fe73c130a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 >> v1:172.21.15.189:6789/0 conn(0x7fe73c137c20 legacy=0x7fe73c1d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 monclient: found mon.a 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c131670 con 0x7fe73c09ee30 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 5 monclient: authenticate success, global_id 4467 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 10 monclient: _renew_subs 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe72c004890 con 0x7fe73c09ee30 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73c1321e0 con 0x7fe73c09ee30 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe72c004c20 con 0x7fe73c09ee30 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 10 monclient: dump: 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:10.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe733fff700 4 set_mon_vals no callback set 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 10 monclient: _renew_subs 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c11de50 con 0x7fe73c09ee30 2024-09-16T06:39:10.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.924+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe72c004e50 con 0x7fe73c09ee30 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe743c31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe743c31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe743c31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c1363c0 con 0x7fe73c09ee30 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fe72c0431b0 con 0x7fe73c09ee30 2024-09-16T06:39:10.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.928+0000 7fe7419cd700 10 monclient: get_auth_request con 0x7fe7200360d0 auth_method 0 2024-09-16T06:39:10.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.936+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe72c0147e0 con 0x7fe73c09ee30 2024-09-16T06:39:10.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.936+0000 7fe7327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:10.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:10.936+0000 7fe7327fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe743c31700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe743c31700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe743c31700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe73c06b920 con 0x7fe73c09ee30 2024-09-16T06:39:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe7327fc700 1 -- 172.21.15.76:0/618826201 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4271381147) 0x7fe72c012c50 con 0x7fe73c09ee30 2024-09-16T06:39:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe7327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.168+0000 7fe7327fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:11.173 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe7200360d0 legacy=0x7fe720038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 10 monclient: shutdown 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 >> v1:172.21.15.76:6789/0 conn(0x7fe73c09ee30 legacy=0x7fe73c1cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 shutdown_connections 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 >> 172.21.15.76:0/618826201 conn(0x7fe73c0a0d30 msgr2=0x7fe73c071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 shutdown_connections 2024-09-16T06:39:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.172+0000 7fe743c31700 1 -- 172.21.15.76:0/618826201 wait complete. 2024-09-16T06:39:11.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:39:11.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:39:11.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:39:11.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: get_monmap_and_config 2024-09-16T06:39:11.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: monmap: 2024-09-16T06:39:11.296 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.296 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:11.298377+0000 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:11.298377+0000 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:11.297 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 1 Processor -- start 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 1 -- start start 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: init 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:11.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-c con 0x7fade0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-a con 0x7fade0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-b con 0x7fade0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade0098990 con 0x7fade0099090 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade0097f40 con 0x7fade0097b60 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade009d610 con 0x7fade0139500 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7faddf59e700 10 monclient(hunting): authenticate will time out at 898.431593s 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7fadded9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fade0139500 0x7fade013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60426/0 (socket says 172.21.15.76:60426) 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.292+0000 7fadded9d700 1 -- 172.21.15.76:0/1953993850 learned_addr learned my addr 172.21.15.76:0/1953993850 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952139691 0 0) 0x7fade009d610 con 0x7fade0139500 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): my global_id is 4469 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc80035d0 con 0x7fade0139500 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843759760 0 0) 0x7fade0098990 con 0x7fade0099090 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): my global_id is 4470 2024-09-16T06:39:11.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade009d610 con 0x7fade0099090 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803058439 0 0) 0x7fade0097f40 con 0x7fade0097b60 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient(hunting): my global_id is 94099 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade0098990 con 0x7fade0097b60 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346224380 0 0) 0x7fade009d610 con 0x7fade0099090 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 >> v1:172.21.15.76:6790/0 conn(0x7fade0139500 legacy=0x7fade013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 >> v1:172.21.15.189:6789/0 conn(0x7fade0097b60 legacy=0x7fade009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 monclient: found mon.noname-a 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00986b0 con 0x7fade0099090 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 5 monclient: authenticate success, global_id 4470 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd00015f0 con 0x7fade0099090 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: dump: 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.313 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd0001810 con 0x7fade0099090 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 4 set_mon_vals no callback set 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 1 -- 172.21.15.76:0/1953993850 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd0001ba0 con 0x7fade0099090 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddd59a700 10 monclient: dump: 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: shutdown 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- 172.21.15.76:0/1953993850 >> v1:172.21.15.76:6789/0 conn(0x7fade0099090 legacy=0x7fade009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- 172.21.15.76:0/1953993850 shutdown_connections 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- 172.21.15.76:0/1953993850 >> 172.21.15.76:0/1953993850 conn(0x7fade00a0030 msgr2=0x7fade00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- 172.21.15.76:0/1953993850 shutdown_connections 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- 172.21.15.76:0/1953993850 wait complete. 2024-09-16T06:39:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: monmap: 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 Processor -- start 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- start start 2024-09-16T06:39:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: init 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-a con 0x7fade0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-b con 0x7fade0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient(hunting): picked mon.noname-c con 0x7fade0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade006ed70 con 0x7fade0097b60 2024-09-16T06:39:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade006cc90 con 0x7fade0099090 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00a2d10 con 0x7fade0139500 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7faddf59e700 10 monclient(hunting): authenticate will time out at 898.435582s 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7fadded9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fade0139500 0x7fade01cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60440/0 (socket says 172.21.15.76:60440) 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7fadded9d700 1 -- 172.21.15.76:0/2234049155 learned_addr learned my addr 172.21.15.76:0/2234049155 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.296+0000 7fadde59c700 1 --1- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fade0097b60 0x7fade006be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42294/0 (socket says 172.21.15.76:42294) 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844110787 0 0) 0x7fade006ed70 con 0x7fade0097b60 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): my global_id is 4473 2024-09-16T06:39:11.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc00035d0 con 0x7fade0097b60 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725764488 0 0) 0x7fade006cc90 con 0x7fade0099090 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): my global_id is 94102 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade006ed70 con 0x7fade0099090 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679446907 0 0) 0x7fade00a2d10 con 0x7fade0139500 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): my global_id is 4472 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade006cc90 con 0x7fade0139500 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668649884 0 0) 0x7fade006cc90 con 0x7fade0139500 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade00a2d10 con 0x7fade0139500 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd8001b00 con 0x7fade0139500 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): dump: 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970036946 0 0) 0x7fadc00035d0 con 0x7fade0097b60 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade006cc90 con 0x7fade0097b60 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 960267321 0 0) 0x7fade006ed70 con 0x7fade0099090 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc00035d0 con 0x7fade0099090 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd4001850 con 0x7fade0097b60 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): dump: 2024-09-16T06:39:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd00014f0 con 0x7fade0099090 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): dump: 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 15249583 0 0) 0x7fade006cc90 con 0x7fade0097b60 2024-09-16T06:39:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6790/0 conn(0x7fade0139500 legacy=0x7fade01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.189:6789/0 conn(0x7fade0099090 legacy=0x7fade006c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 monclient: found mon.a 2024-09-16T06:39:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade009a540 con 0x7fade0097b60 2024-09-16T06:39:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 5 monclient: authenticate success, global_id 4473 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 10 monclient: _renew_subs 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade006f330 con 0x7fade0097b60 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 10 monclient: _renew_subs 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade006dd30 con 0x7fade0097b60 2024-09-16T06:39:11.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd40036c0 con 0x7fade0097b60 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd40039d0 con 0x7fade0097b60 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddcd99700 4 set_mon_vals no callback set 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient: dump: 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.329 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadc4006590 con 0x7fadc4002830 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fadd4003b30 con 0x7fade0097b60 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fadd4043170 con 0x7fade0097b60 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadded9d700 10 monclient: get_auth_request con 0x7fadc0036610 auth_method 0 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657110253 0 0) 0x7fadc4006590 con 0x7fadc4002830 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 10 monclient(hunting): my global_id is 4476 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.300+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade006cc90 con 0x7fadc4002830 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1671849951 0 0) 0x7fade006cc90 con 0x7fadc4002830 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc4006590 con 0x7fadc4002830 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd0003a30 con 0x7fadc4002830 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 10 monclient: dump: 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.331 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 775635643 0 0) 0x7fadc4006590 con 0x7fadc4002830 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadc400a190 con 0x7fadc4002830 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fadd0003b90 con 0x7fadc4002830 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fadc4002830 legacy=0x7fadc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.304+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fadc4002830 legacy=0x7fadc4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7faddf59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fadc4009e50 con 0x7fadc4009170 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223614164 0 0) 0x7fadc4009e50 con 0x7fadc4009170 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient(hunting): my global_id is 4479 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4006590 con 0x7fadc4009170 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3976404612 0 0) 0x7fadc4006590 con 0x7fadc4009170 2024-09-16T06:39:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc4009e50 con 0x7fadc4009170 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd0004260 con 0x7fadc4009170 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient: dump: 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4219713194 0 0) 0x7fadc4009e50 con 0x7fadc4009170 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fadc400c340 con 0x7fadc4009170 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2840783069) 0x7fadd00065e0 con 0x7fadc4009170 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fadc4009170 legacy=0x7fadc4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.312+0000 7fadceffd700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fadc4009170 legacy=0x7fadc4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.336 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:11.337 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:11.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.339 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:11.340 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.341 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.342 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.344 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:11.345 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.346 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.347 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:39:11.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:11.349 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:11.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.189:6812/3098810674 conn(0x7fadc0036610 legacy=0x7fadc0038ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 10 monclient: shutdown 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 >> v1:172.21.15.76:6789/0 conn(0x7fade0097b60 legacy=0x7fade006be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7fadded9d700 1 -- 172.21.15.76:0/2234049155 reap_dead start 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7fadded9d700 1 -- 172.21.15.76:0/2234049155 reap_dead start 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 shutdown_connections 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 >> 172.21.15.76:0/2234049155 conn(0x7fade00a0030 msgr2=0x7fade00a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 shutdown_connections 2024-09-16T06:39:11.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.316+0000 7faddf59e700 1 -- 172.21.15.76:0/2234049155 wait complete. 2024-09-16T06:39:11.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:11.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:39:11.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:39:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: get_monmap_and_config 2024-09-16T06:39:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: monmap: 2024-09-16T06:39:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.483 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:11.472384+0000 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:11.472384+0000 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:11.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 Processor -- start 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- start start 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: init 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-c con 0x7f9f64097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-b con 0x7f9f64099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-a con 0x7f9f64139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f64098990 con 0x7f9f64139500 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f64097f40 con 0x7f9f64097b60 2024-09-16T06:39:11.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6409d610 con 0x7f9f64099090 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient(hunting): authenticate will time out at 898.605628s 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f69e36700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9f64139500 0x7f9f6413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42322/0 (socket says 172.21.15.76:42322) 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f68e34700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9f64099090 0x7f9f6409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60444/0 (socket says 172.21.15.76:60444) 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f69e36700 1 -- 172.21.15.76:0/394518505 learned_addr learned my addr 172.21.15.76:0/394518505 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550045159 0 0) 0x7f9f64098990 con 0x7f9f64139500 2024-09-16T06:39:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): my global_id is 4482 2024-09-16T06:39:11.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f500035d0 con 0x7f9f64139500 2024-09-16T06:39:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333794366 0 0) 0x7f9f6409d610 con 0x7f9f64099090 2024-09-16T06:39:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): my global_id is 4475 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64098990 con 0x7f9f64099090 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847878848 0 0) 0x7f9f64097f40 con 0x7f9f64097b60 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient(hunting): my global_id is 94105 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6409d610 con 0x7f9f64097b60 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204393695 0 0) 0x7f9f500035d0 con 0x7f9f64139500 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 >> v1:172.21.15.76:6790/0 conn(0x7f9f64099090 legacy=0x7f9f6409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 >> v1:172.21.15.189:6789/0 conn(0x7f9f64097b60 legacy=0x7f9f6409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 monclient: found mon.noname-a 2024-09-16T06:39:11.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f640986b0 con 0x7f9f64139500 2024-09-16T06:39:11.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 5 monclient: authenticate success, global_id 4482 2024-09-16T06:39:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f5c001990 con 0x7f9f64139500 2024-09-16T06:39:11.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:11.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: dump: 2024-09-16T06:39:11.495 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.495 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.495 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f5c001c70 con 0x7f9f64139500 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:11.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 4 set_mon_vals no callback set 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 1 -- 172.21.15.76:0/394518505 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f5c002040 con 0x7f9f64139500 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f5bfff700 10 monclient: dump: 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.498 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 10 monclient: shutdown 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- 172.21.15.76:0/394518505 >> v1:172.21.15.76:6789/0 conn(0x7f9f64139500 legacy=0x7f9f6413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- 172.21.15.76:0/394518505 shutdown_connections 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- 172.21.15.76:0/394518505 >> 172.21.15.76:0/394518505 conn(0x7f9f640a0030 msgr2=0x7f9f640a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- 172.21.15.76:0/394518505 shutdown_connections 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.468+0000 7f9f6a637700 1 -- 172.21.15.76:0/394518505 wait complete. 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: monmap: 2024-09-16T06:39:11.500 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:11.501 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 Processor -- start 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- start start 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: init 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-c con 0x7f9f64097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-b con 0x7f9f64099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient(hunting): picked mon.noname-a con 0x7f9f64139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f64070280 con 0x7f9f64139500 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f640a2be0 con 0x7f9f64099090 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6409a540 con 0x7f9f64097b60 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient(hunting): authenticate will time out at 898.609942s 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f69635700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9f64097b60 0x7f9f6406cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60456/0 (socket says 172.21.15.76:60456) 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f69635700 1 -- 172.21.15.76:0/3326013861 learned_addr learned my addr 172.21.15.76:0/3326013861 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984002769 0 0) 0x7f9f6409a540 con 0x7f9f64097b60 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): my global_id is 4478 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f480035d0 con 0x7f9f64097b60 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641579869 0 0) 0x7f9f640a2be0 con 0x7f9f64099090 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): my global_id is 94108 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6409a540 con 0x7f9f64099090 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428819507 0 0) 0x7f9f64070280 con 0x7f9f64139500 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): my global_id is 4485 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f640a2be0 con 0x7f9f64139500 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495052445 0 0) 0x7f9f480035d0 con 0x7f9f64097b60 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f64070280 con 0x7f9f64097b60 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f54001910 con 0x7f9f64097b60 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512484666 0 0) 0x7f9f64070280 con 0x7f9f64097b60 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.189:6789/0 conn(0x7f9f64099090 legacy=0x7f9f6406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6789/0 conn(0x7f9f64139500 legacy=0x7f9f641cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 monclient: found mon.c 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6406f9e0 con 0x7f9f64097b60 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 5 monclient: authenticate success, global_id 4478 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _renew_subs 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f64070df0 con 0x7f9f64097b60 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _renew_subs 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6406dee0 con 0x7f9f64097b60 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f54001ce0 con 0x7f9f64097b60 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f54003af0 con 0x7f9f64097b60 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f5affd700 4 set_mon_vals no callback set 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.472+0000 7f9f59ffb700 10 monclient: dump: 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f6a637700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9f54003c50 con 0x7f9f64097b60 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f9f54043270 con 0x7f9f64097b60 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662562226 0 0) 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient(hunting): my global_id is 4481 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64070280 con 0x7f9f44002830 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f69e36700 10 monclient: get_auth_request con 0x7f9f48036110 auth_method 0 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2778126305 0 0) 0x7f9f64070280 con 0x7f9f44002830 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f60001900 con 0x7f9f44002830 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient: dump: 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1842678500 0 0) 0x7f9f44006590 con 0x7f9f44002830 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f4400a190 con 0x7f9f44002830 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9f60001bb0 con 0x7f9f44002830 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6790/0 conn(0x7f9f44002830 legacy=0x7f9f44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.476+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6790/0 conn(0x7f9f44002830 legacy=0x7f9f44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.484+0000 7f9f6a637700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.484+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f44009e30 con 0x7f9f44009150 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934522799 0 0) 0x7f9f44009e30 con 0x7f9f44009150 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient(hunting): my global_id is 4484 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44006590 con 0x7f9f44009150 2024-09-16T06:39:11.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1478252226 0 0) 0x7f9f44006590 con 0x7f9f44009150 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44009e30 con 0x7f9f44009150 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f60004a80 con 0x7f9f44009150 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient: dump: 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.514 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3548073829 0 0) 0x7f9f44009e30 con 0x7f9f44009150 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f4400c340 con 0x7f9f44009150 2024-09-16T06:39:11.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 974484734) 0x7f9f60004d80 con 0x7f9f44009150 2024-09-16T06:39:11.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:11.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:11.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6790/0 conn(0x7f9f44009150 legacy=0x7f9f440095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f59ffb700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6790/0 conn(0x7f9f44009150 legacy=0x7f9f440095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.516 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:11.517 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.518 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.519 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.520 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:11.521 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:11.522 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.523 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.524 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:11.525 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.526 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:11.527 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.531 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.533 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:11.533 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:11.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9f48036110 legacy=0x7f9f480385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 10 monclient: shutdown 2024-09-16T06:39:11.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 >> v1:172.21.15.76:6790/0 conn(0x7f9f64097b60 legacy=0x7f9f6406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f69e36700 1 -- 172.21.15.76:0/3326013861 reap_dead start 2024-09-16T06:39:11.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f69e36700 1 -- 172.21.15.76:0/3326013861 reap_dead start 2024-09-16T06:39:11.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 shutdown_connections 2024-09-16T06:39:11.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 >> 172.21.15.76:0/3326013861 conn(0x7f9f640a0030 msgr2=0x7f9f640a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 shutdown_connections 2024-09-16T06:39:11.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.488+0000 7f9f6a637700 1 -- 172.21.15.76:0/3326013861 wait complete. 2024-09-16T06:39:11.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:11.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:39:11.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: get_monmap_and_config 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: monmap: 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:11.656085+0000 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:11.656085+0000 2024-09-16T06:39:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:11.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 1 Processor -- start 2024-09-16T06:39:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 1 -- start start 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: init 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc12ec10 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc12eff0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc0a0220 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc071b30 con 0x7fb9dc12ec10 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09f2a0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09f580 con 0x7fb9dc0a0220 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient(hunting): authenticate will time out at 898.790370s 2024-09-16T06:39:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e1db2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb9dc12ec10 0x7fb9dc070d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42336/0 (socket says 172.21.15.76:42336) 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e25b3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb9dc0a0220 0x7fb9dc13e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60498/0 (socket says 172.21.15.76:60498) 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e1db2700 1 -- 172.21.15.76:0/2671146050 learned_addr learned my addr 172.21.15.76:0/2671146050 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636189035 0 0) 0x7fb9dc071b30 con 0x7fb9dc12ec10 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): my global_id is 4488 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c80035d0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698110620 0 0) 0x7fb9dc09f580 con 0x7fb9dc0a0220 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): my global_id is 4487 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc071b30 con 0x7fb9dc0a0220 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139061517 0 0) 0x7fb9dc09f2a0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient(hunting): my global_id is 94111 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc09f580 con 0x7fb9dc12eff0 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094132080 0 0) 0x7fb9c80035d0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 >> v1:172.21.15.76:6790/0 conn(0x7fb9dc0a0220 legacy=0x7fb9dc13e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 >> v1:172.21.15.189:6789/0 conn(0x7fb9dc12eff0 legacy=0x7fb9dc071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 monclient: found mon.noname-a 2024-09-16T06:39:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc09f860 con 0x7fb9dc12ec10 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d8001990 con 0x7fb9dc12ec10 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 5 monclient: authenticate success, global_id 4488 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: dump: 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.661 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d8001c70 con 0x7fb9dc12ec10 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 1 -- 172.21.15.76:0/2671146050 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d8002000 con 0x7fb9dc12ec10 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e2db4700 4 set_mon_vals no callback set 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.652+0000 7fb9e0db0700 10 monclient: dump: 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.663 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.664 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: shutdown 2024-09-16T06:39:11.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- 172.21.15.76:0/2671146050 >> v1:172.21.15.76:6789/0 conn(0x7fb9dc12ec10 legacy=0x7fb9dc070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- 172.21.15.76:0/2671146050 shutdown_connections 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- 172.21.15.76:0/2671146050 >> 172.21.15.76:0/2671146050 conn(0x7fb9dc0a4430 msgr2=0x7fb9dc0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- 172.21.15.76:0/2671146050 shutdown_connections 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- 172.21.15.76:0/2671146050 wait complete. 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: monmap: 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:11.672 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 Processor -- start 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- start start 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: init 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc0a0220 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc12ec10 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc12eff0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0a3120 con 0x7fb9dc0a0220 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc12c8a0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09b0f0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e2db4700 10 monclient(hunting): authenticate will time out at 898.795292s 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e1db2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb9dc0a0220 0x7fb9dc1ca050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42344/0 (socket says 172.21.15.76:42344) 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9e1db2700 1 -- 172.21.15.76:0/968521751 learned_addr learned my addr 172.21.15.76:0/968521751 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142224810 0 0) 0x7fb9dc0a3120 con 0x7fb9dc0a0220 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): my global_id is 4491 2024-09-16T06:39:11.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c00035d0 con 0x7fb9dc0a0220 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990920668 0 0) 0x7fb9dc09b0f0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.656+0000 7fb9ce7fc700 10 monclient(hunting): my global_id is 4490 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc0a3120 con 0x7fb9dc12ec10 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053308639 0 0) 0x7fb9dc12c8a0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): my global_id is 94114 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc09b0f0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529147993 0 0) 0x7fb9dc0a3120 con 0x7fb9dc12ec10 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9dc12c8a0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d00018b0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): dump: 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770533702 0 0) 0x7fb9c00035d0 con 0x7fb9dc0a0220 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9dc0a3120 con 0x7fb9dc0a0220 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596852081 0 0) 0x7fb9dc09b0f0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9c00035d0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d8001950 con 0x7fb9dc0a0220 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): dump: 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.678 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d40017f0 con 0x7fb9dc12eff0 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): dump: 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3840673785 0 0) 0x7fb9dc12c8a0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6789/0 conn(0x7fb9dc12eff0 legacy=0x7fb9dc1d2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.76:6789/0 conn(0x7fb9dc0a0220 legacy=0x7fb9dc1ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 monclient: found mon.c 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc11d0e0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 5 monclient: authenticate success, global_id 4490 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 10 monclient: _renew_subs 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc11c840 con 0x7fb9dc12ec10 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 10 monclient: _renew_subs 2024-09-16T06:39:11.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc11b460 con 0x7fb9dc12ec10 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d0001c40 con 0x7fb9dc12ec10 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d0003b10 con 0x7fb9dc12ec10 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9cffff700 4 set_mon_vals no callback set 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient: dump: 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9c4006590 con 0x7fb9c4002830 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb9d0003c70 con 0x7fb9dc12ec10 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fb9d0041fb0 con 0x7fb9dc12ec10 2024-09-16T06:39:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9e25b3700 10 monclient: get_auth_request con 0x7fb9c00365c0 auth_method 0 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501968108 0 0) 0x7fb9c4006590 con 0x7fb9c4002830 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 10 monclient(hunting): my global_id is 94117 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc12c8a0 con 0x7fb9c4002830 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3774729487 0 0) 0x7fb9dc12c8a0 con 0x7fb9c4002830 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9c4006590 con 0x7fb9c4002830 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.660+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d8004fa0 con 0x7fb9c4002830 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 10 monclient: dump: 2024-09-16T06:39:11.684 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3632169925 0 0) 0x7fb9c4006590 con 0x7fb9c4002830 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9c400a190 con 0x7fb9c4002830 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb9d8007180 con 0x7fb9c4002830 2024-09-16T06:39:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:11.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6789/0 conn(0x7fb9c4002830 legacy=0x7fb9c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.664+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6789/0 conn(0x7fb9c4002830 legacy=0x7fb9c4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9e2db4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:11.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9c4009e30 con 0x7fb9c4009150 2024-09-16T06:39:11.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77084002 0 0) 0x7fb9c4009e30 con 0x7fb9c4009150 2024-09-16T06:39:11.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient(hunting): my global_id is 94120 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c4006590 con 0x7fb9c4009150 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 997371397 0 0) 0x7fb9c4006590 con 0x7fb9c4009150 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9c4009e30 con 0x7fb9c4009150 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9d8005650 con 0x7fb9c4009150 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 10 monclient: dump: 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4177018091 0 0) 0x7fb9c4009e30 con 0x7fb9c4009150 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.672+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb9c400c340 con 0x7fb9c4009150 2024-09-16T06:39:11.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 926317350) 0x7fb9d8006a20 con 0x7fb9c4009150 2024-09-16T06:39:11.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9ce7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:11.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9ce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:11.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6789/0 conn(0x7fb9c4009150 legacy=0x7fb9c40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9ce7fc700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6789/0 conn(0x7fb9c4009150 legacy=0x7fb9c40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:11.690 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.691 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.692 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.693 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:11.694 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:11.695 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:11.696 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:11.699 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:11.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:11.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:11.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:11.703 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:11.703 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb9c00365c0 legacy=0x7fb9c0038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 10 monclient: shutdown 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 >> v1:172.21.15.76:6790/0 conn(0x7fb9dc12ec10 legacy=0x7fb9dc1cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e25b3700 1 -- 172.21.15.76:0/968521751 reap_dead start 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 shutdown_connections 2024-09-16T06:39:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 >> 172.21.15.76:0/968521751 conn(0x7fb9dc0a4430 msgr2=0x7fb9dc1319a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 shutdown_connections 2024-09-16T06:39:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.676+0000 7fb9e2db4700 1 -- 172.21.15.76:0/968521751 wait complete. 2024-09-16T06:39:11.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:11.705 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:39:11.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:39:11.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.816+0000 7f76d10a2700 10 monclient: get_monmap_and_config 2024-09-16T06:39:11.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.816+0000 7f76d10a2700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.816+0000 7f76d10a2700 10 monclient: monmap: 2024-09-16T06:39:11.821 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:11.823632+0000 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:11.823632+0000 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:11.822 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 Processor -- start 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- start start 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient: init 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-c con 0x7f76cc09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-b con 0x7f76cc09d520 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-a con 0x7f76cc09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc09b520 con 0x7f76cc09fd20 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc071f30 con 0x7f76cc09f230 2024-09-16T06:39:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc09b0e0 con 0x7f76cc09d520 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient(hunting): authenticate will time out at 898.956906s 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76c3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f76cc09d520 0x7f76cc137440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60520/0 (socket says 172.21.15.76:60520) 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76cb59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f76cc09fd20 0x7f76cc13ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42358/0 (socket says 172.21.15.76:42358) 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76c3fff700 1 -- 172.21.15.76:0/3435324716 learned_addr learned my addr 172.21.15.76:0/3435324716 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583244097 0 0) 0x7f76cc09b0e0 con 0x7f76cc09d520 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): my global_id is 4493 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b80035d0 con 0x7f76cc09d520 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511002691 0 0) 0x7f76cc09b520 con 0x7f76cc09fd20 2024-09-16T06:39:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): my global_id is 4494 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc09b0e0 con 0x7f76cc09fd20 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650161449 0 0) 0x7f76cc071f30 con 0x7f76cc09f230 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient(hunting): my global_id is 94123 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc09b520 con 0x7f76cc09f230 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456029911 0 0) 0x7f76b80035d0 con 0x7f76cc09d520 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 >> v1:172.21.15.189:6789/0 conn(0x7f76cc09f230 legacy=0x7f76cc09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 >> v1:172.21.15.76:6789/0 conn(0x7f76cc09fd20 legacy=0x7f76cc13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 monclient: found mon.noname-b 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc09c7d0 con 0x7f76cc09d520 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76b4001610 con 0x7f76cc09d520 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: dump: 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.831 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.831 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 5 monclient: authenticate success, global_id 4493 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b4001880 con 0x7f76cc09d520 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 1 -- 172.21.15.76:0/3435324716 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76b4001c10 con 0x7f76cc09d520 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76ca59c700 10 monclient: dump: 2024-09-16T06:39:11.832 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 4 set_mon_vals no callback set 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 10 monclient: shutdown 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:11.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- 172.21.15.76:0/3435324716 >> v1:172.21.15.76:6790/0 conn(0x7f76cc09d520 legacy=0x7f76cc137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- 172.21.15.76:0/3435324716 shutdown_connections 2024-09-16T06:39:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- 172.21.15.76:0/3435324716 >> 172.21.15.76:0/3435324716 conn(0x7f76cc0a3e50 msgr2=0x7f76cc0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- 172.21.15.76:0/3435324716 shutdown_connections 2024-09-16T06:39:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.820+0000 7f76d10a2700 1 -- 172.21.15.76:0/3435324716 wait complete. 2024-09-16T06:39:11.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient: build_initial_monmap 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient: monmap: 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 1 Processor -- start 2024-09-16T06:39:11.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 1 -- start start 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient: init 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-a con 0x7f76cc09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-c con 0x7f76cc09fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient(hunting): picked mon.noname-b con 0x7f76cc121510 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc1243e0 con 0x7f76cc09f230 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc131070 con 0x7f76cc121510 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0a2c90 con 0x7f76cc09fd20 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76d10a2700 10 monclient(hunting): authenticate will time out at 898.962876s 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76cad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f76cc09f230 0x7f76cc1206f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42360/0 (socket says 172.21.15.76:42360) 2024-09-16T06:39:11.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76cad9d700 1 -- 172.21.15.76:0/2790153588 learned_addr learned my addr 172.21.15.76:0/2790153588 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742681084 0 0) 0x7f76cc1243e0 con 0x7f76cc09f230 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): my global_id is 4497 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ac0035d0 con 0x7f76cc09f230 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907446622 0 0) 0x7f76cc0a2c90 con 0x7f76cc09fd20 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): my global_id is 4496 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc1243e0 con 0x7f76cc09fd20 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377323028 0 0) 0x7f76cc131070 con 0x7f76cc121510 2024-09-16T06:39:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 10 monclient(hunting): my global_id is 94126 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc0a2c90 con 0x7f76cc121510 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275308653 0 0) 0x7f76ac0035d0 con 0x7f76cc09f230 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc131070 con 0x7f76cc09f230 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.824+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76bc001990 con 0x7f76cc09f230 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): dump: 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.843 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 146450502 0 0) 0x7f76cc1243e0 con 0x7f76cc09fd20 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ac0035d0 con 0x7f76cc09fd20 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76b40015d0 con 0x7f76cc09fd20 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:11.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): dump: 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269730334 0 0) 0x7f76cc0a2c90 con 0x7f76cc121510 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc1243e0 con 0x7f76cc121510 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76c4001bf0 con 0x7f76cc121510 2024-09-16T06:39:11.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient(hunting): dump: 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896511685 0 0) 0x7f76cc131070 con 0x7f76cc09f230 2024-09-16T06:39:11.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 >> v1:172.21.15.76:6790/0 conn(0x7f76cc09fd20 legacy=0x7f76cc120e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 >> v1:172.21.15.189:6789/0 conn(0x7f76cc121510 legacy=0x7f76cc1d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 monclient: found mon.a 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc072450 con 0x7f76cc09f230 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 5 monclient: authenticate success, global_id 4497 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _renew_subs 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc124100 con 0x7f76cc09f230 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _renew_subs 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc122080 con 0x7f76cc09f230 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76bc001d40 con 0x7f76cc09f230 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f76bc003b10 con 0x7f76cc09f230 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 10 monclient: dump: 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.848 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c9d9b700 4 set_mon_vals no callback set 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b0005100 con 0x7f76cc09f230 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f76bc003c70 con 0x7f76cc09f230 2024-09-16T06:39:11.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f76bc043190 con 0x7f76cc09f230 2024-09-16T06:39:11.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.828+0000 7f76c3fff700 10 monclient: get_auth_request con 0x7f76ac036520 auth_method 0 2024-09-16T06:39:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.836+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f76bc0147c0 con 0x7f76cc09f230 2024-09-16T06:39:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.836+0000 7f76c37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:11.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:11.836+0000 7f76c37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:12.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.072+0000 7f76d10a2700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:39:12.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.072+0000 7f76d10a2700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:39:12.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.072+0000 7f76d10a2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.072+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f76b0005bc0 con 0x7f76cc09f230 2024-09-16T06:39:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76c37fe700 1 -- 172.21.15.76:0/2790153588 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f76bc012c30 con 0x7f76cc09f230 2024-09-16T06:39:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76c37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:39:12.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76c37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:12.079 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.079 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=1387096827,config=1097430493,health=305525474,kv=1001081567,logm=228446712}) 2024-09-16T06:39:12.079 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=2599526064,config=1097430493,health=861809967,kv=1588170259,logm=2412514528}) 2024-09-16T06:39:12.080 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.080 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=1387096827,config=1097430493,health=305525474,kv=1001081567,logm=228446712}) 2024-09-16T06:39:12.080 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=1,health=11,kv=12,logm=62} crc {auth=753816030,config=1228700967,health=3172947059,kv=2335881212,logm=3279780782}) 2024-09-16T06:39:12.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 >> v1:172.21.15.189:6812/3098810674 conn(0x7f76ac036520 legacy=0x7f76ac0389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:12.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 10 monclient: shutdown 2024-09-16T06:39:12.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 >> v1:172.21.15.76:6789/0 conn(0x7f76cc09f230 legacy=0x7f76cc1206f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 shutdown_connections 2024-09-16T06:39:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 >> 172.21.15.76:0/2790153588 conn(0x7f76cc0a3e50 msgr2=0x7f76cc09e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 shutdown_connections 2024-09-16T06:39:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f76d10a2700 1 -- 172.21.15.76:0/2790153588 wait complete. 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822720959}) 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280304974}) 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.076+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822720959}) 2024-09-16T06:39:12.083 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692786323}) 2024-09-16T06:39:12.084 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.085 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326099988}) 2024-09-16T06:39:12.085 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197831666}) 2024-09-16T06:39:12.085 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.085 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326099988}) 2024-09-16T06:39:12.085 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.080+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054042712}) 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=522365864,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1852485998,mon_config_key=2058934144,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=3511010134}) 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=2062736830,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=663397256,mon_config_key=4093611796,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=4223076834}) 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=522365864,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1852485998,mon_config_key=2058934144,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=3511010134}) 2024-09-16T06:39:12.086 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=20} crc {logm=3243294208,mdsmap=1531151339,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3290627745,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=2284879832}) 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=1423084688}) 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=20} crc {osdmap=235960805}) 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=1423084688}) 2024-09-16T06:39:12.089 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:39:12.084+0000 7f39719dd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=4255646943}) 2024-09-16T06:39:12.094 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:39:13.095 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:39:13.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: get_monmap_and_config 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: monmap: 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:13.217552+0000 2024-09-16T06:39:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:13.217552+0000 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:13.217 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 1 Processor -- start 2024-09-16T06:39:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 1 -- start start 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: init 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-a con 0x7f6638097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-c con 0x7f6638099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-b con 0x7f6638139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638098990 con 0x7f6638097b60 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638097f40 con 0x7f6638099090 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663809d610 con 0x7f6638139500 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 10 monclient(hunting): authenticate will time out at 900.350856s 2024-09-16T06:39:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663d427700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6638139500 0x7f663813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60538/0 (socket says 172.21.15.76:60538) 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663d427700 1 -- 172.21.15.76:0/1002034475 learned_addr learned my addr 172.21.15.76:0/1002034475 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648082484 0 0) 0x7f6638098990 con 0x7f6638097b60 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): my global_id is 4500 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66180035d0 con 0x7f6638097b60 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826703243 0 0) 0x7f663809d610 con 0x7f6638139500 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): my global_id is 4499 2024-09-16T06:39:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638098990 con 0x7f6638139500 2024-09-16T06:39:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902878283 0 0) 0x7f6638097f40 con 0x7f6638099090 2024-09-16T06:39:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient(hunting): my global_id is 94129 2024-09-16T06:39:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663809d610 con 0x7f6638099090 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746568374 0 0) 0x7f6638098990 con 0x7f6638139500 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 >> v1:172.21.15.189:6789/0 conn(0x7f6638099090 legacy=0x7f663809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 >> v1:172.21.15.76:6789/0 conn(0x7f6638097b60 legacy=0x7f663809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 monclient: found mon.noname-b 2024-09-16T06:39:13.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380986b0 con 0x7f6638139500 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f66200015f0 con 0x7f6638139500 2024-09-16T06:39:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f66377fe700 10 monclient: dump: 2024-09-16T06:39:13.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.244 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 5 monclient: authenticate success, global_id 4499 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.212+0000 7f663dc28700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6620001810 con 0x7f6638139500 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 1 -- 172.21.15.76:0/1002034475 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6620001be0 con 0x7f6638139500 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 4 set_mon_vals no callback set 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f66377fe700 10 monclient: dump: 2024-09-16T06:39:13.245 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 10 monclient: shutdown 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- 172.21.15.76:0/1002034475 >> v1:172.21.15.76:6790/0 conn(0x7f6638139500 legacy=0x7f663813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- 172.21.15.76:0/1002034475 shutdown_connections 2024-09-16T06:39:13.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- 172.21.15.76:0/1002034475 >> 172.21.15.76:0/1002034475 conn(0x7f66380a0030 msgr2=0x7f66380a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- 172.21.15.76:0/1002034475 shutdown_connections 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- 172.21.15.76:0/1002034475 wait complete. 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 10 monclient: monmap: 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:13.247 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 Processor -- start 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 1 -- start start 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.216+0000 7f663dc28700 10 monclient: init 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-b con 0x7f6638097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-c con 0x7f6638099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient(hunting): picked mon.noname-a con 0x7f6638139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663806ff80 con 0x7f6638139500 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66381c8cc0 con 0x7f6638097b60 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663806def0 con 0x7f6638099090 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient(hunting): authenticate will time out at 900.356451s 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663d427700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6638139500 0x7f66381d0100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42382/0 (socket says 172.21.15.76:42382) 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663d427700 1 -- 172.21.15.76:0/1803055048 learned_addr learned my addr 172.21.15.76:0/1803055048 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621841070 0 0) 0x7f663806ff80 con 0x7f6638139500 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): my global_id is 4503 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c0035d0 con 0x7f6638139500 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726096670 0 0) 0x7f663806def0 con 0x7f6638099090 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): my global_id is 4502 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663806ff80 con 0x7f6638099090 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549565978 0 0) 0x7f66381c8cc0 con 0x7f6638097b60 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): my global_id is 94132 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663806def0 con 0x7f6638097b60 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2112050801 0 0) 0x7f663806ff80 con 0x7f6638099090 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66381c8cc0 con 0x7f6638099090 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1088415400 0 0) 0x7f663806def0 con 0x7f6638097b60 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f663806ff80 con 0x7f6638097b60 2024-09-16T06:39:13.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6628001c30 con 0x7f6638099090 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): dump: 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1961838530 0 0) 0x7f661c0035d0 con 0x7f6638139500 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f663806def0 con 0x7f6638139500 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6630001850 con 0x7f6638097b60 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): dump: 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6620001b20 con 0x7f6638139500 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient(hunting): dump: 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.253 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1345919902 0 0) 0x7f66381c8cc0 con 0x7f6638099090 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 >> v1:172.21.15.189:6789/0 conn(0x7f6638097b60 legacy=0x7f66381c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 >> v1:172.21.15.76:6789/0 conn(0x7f6638139500 legacy=0x7f66381d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 monclient: found mon.c 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380a2bb0 con 0x7f6638099090 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 5 monclient: authenticate success, global_id 4502 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _renew_subs 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663809a540 con 0x7f6638099090 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _renew_subs 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6638070540 con 0x7f6638099090 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6628001d90 con 0x7f6638099090 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6628003c60 con 0x7f6638099090 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f6636ffd700 4 set_mon_vals no callback set 2024-09-16T06:39:13.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 10 monclient: dump: 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6610005100 con 0x7f6638099090 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.220+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f662800e750 con 0x7f6638099090 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.224+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f66280422e0 con 0x7f6638099090 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.224+0000 7f663cc26700 10 monclient: get_auth_request con 0x7f661c03a890 auth_method 0 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.232+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6628013690 con 0x7f6638099090 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.232+0000 7f66357fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.232+0000 7f66357fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:13.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.468+0000 7f663dc28700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:13.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.468+0000 7f663dc28700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:13.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.468+0000 7f663dc28700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.468+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6610005930 con 0x7f6638099090 2024-09-16T06:39:13.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f66357fa700 1 -- 172.21.15.76:0/1803055048 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2325139988) 0x7f6628011d30 con 0x7f6638099090 2024-09-16T06:39:13.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f66357fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:13.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f66357fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:13.474 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 >> v1:172.21.15.189:6812/3098810674 conn(0x7f661c03a890 legacy=0x7f661c03cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 10 monclient: shutdown 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 >> v1:172.21.15.76:6790/0 conn(0x7f6638099090 legacy=0x7f66381c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 shutdown_connections 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 >> 172.21.15.76:0/1803055048 conn(0x7f66380a0030 msgr2=0x7f66380a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 shutdown_connections 2024-09-16T06:39:13.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.472+0000 7f663dc28700 1 -- 172.21.15.76:0/1803055048 wait complete. 2024-09-16T06:39:13.488 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:39:13.488 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:39:13.488 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:39:13.488 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: get_monmap_and_config 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: monmap: 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:13.597340+0000 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:13.597340+0000 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:13.596 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 1 Processor -- start 2024-09-16T06:39:13.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 1 -- start start 2024-09-16T06:39:13.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: init 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient(hunting): picked mon.noname-c con 0x7feea8132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient(hunting): picked mon.noname-a con 0x7feea806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient(hunting): picked mon.noname-b con 0x7feea806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea806c280 con 0x7feea806d480 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea806bfa0 con 0x7feea8132330 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea809ace0 con 0x7feea806cf50 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feeac885700 10 monclient(hunting): authenticate will time out at 900.731491s 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feea659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feea806d480 0x7feea813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42396/0 (socket says 172.21.15.76:42396) 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feea659c700 1 -- 172.21.15.76:0/2160962455 learned_addr learned my addr 172.21.15.76:0/2160962455 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.592+0000 7feea759e700 1 --1- 172.21.15.76:0/2160962455 >> v1:172.21.15.76:6790/0 conn(0x7feea806cf50 0x7feea813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60544/0 (socket says 172.21.15.76:60544) 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283758229 0 0) 0x7feea806c280 con 0x7feea806d480 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): my global_id is 4506 2024-09-16T06:39:13.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee940035d0 con 0x7feea806d480 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426141099 0 0) 0x7feea809ace0 con 0x7feea806cf50 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): my global_id is 4505 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea806c280 con 0x7feea806cf50 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802220708 0 0) 0x7feea806bfa0 con 0x7feea8132330 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient(hunting): my global_id is 94135 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea809ace0 con 0x7feea8132330 2024-09-16T06:39:13.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2631074850 0 0) 0x7fee940035d0 con 0x7feea806d480 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 >> v1:172.21.15.76:6790/0 conn(0x7feea806cf50 legacy=0x7feea813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 >> v1:172.21.15.189:6789/0 conn(0x7feea8132330 legacy=0x7feea8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 monclient: found mon.noname-a 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea809ae70 con 0x7feea806d480 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee98001610 con 0x7feea806d480 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:13.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: dump: 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.603 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 5 monclient: authenticate success, global_id 4506 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee980018f0 con 0x7feea806d480 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 1 -- 172.21.15.76:0/2160962455 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee98001b20 con 0x7feea806d480 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 4 set_mon_vals no callback set 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feea5d9b700 10 monclient: dump: 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.605 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 10 monclient: shutdown 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- 172.21.15.76:0/2160962455 >> v1:172.21.15.76:6789/0 conn(0x7feea806d480 legacy=0x7feea813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- 172.21.15.76:0/2160962455 shutdown_connections 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- 172.21.15.76:0/2160962455 >> 172.21.15.76:0/2160962455 conn(0x7feea80a41c0 msgr2=0x7feea80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- 172.21.15.76:0/2160962455 shutdown_connections 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- 172.21.15.76:0/2160962455 wait complete. 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 10 monclient: monmap: 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 Processor -- start 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 1 -- start start 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.596+0000 7feeac885700 10 monclient: init 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient(hunting): picked mon.noname-a con 0x7feea806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient(hunting): picked mon.noname-b con 0x7feea806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient(hunting): picked mon.noname-c con 0x7feea8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea812f000 con 0x7feea806cf50 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea812e790 con 0x7feea806d480 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea809e8b0 con 0x7feea8132330 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feea759e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7feea8132330 0x7feea81d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60556/0 (socket says 172.21.15.76:60556) 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feea6d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feea806cf50 0x7feea813e560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42412/0 (socket says 172.21.15.76:42412) 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient(hunting): authenticate will time out at 900.737110s 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feea759e700 1 -- 172.21.15.76:0/3233233711 learned_addr learned my addr 172.21.15.76:0/3233233711 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362283521 0 0) 0x7feea809e8b0 con 0x7feea8132330 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): my global_id is 4508 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee880035d0 con 0x7feea8132330 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278668443 0 0) 0x7feea812f000 con 0x7feea806cf50 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): my global_id is 4509 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea809e8b0 con 0x7feea806cf50 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87665455 0 0) 0x7feea812e790 con 0x7feea806d480 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): my global_id is 94138 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea812f000 con 0x7feea806d480 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926405221 0 0) 0x7fee880035d0 con 0x7feea8132330 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea812e790 con 0x7feea8132330 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feea0001c30 con 0x7feea8132330 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997905114 0 0) 0x7feea809e8b0 con 0x7feea806cf50 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee880035d0 con 0x7feea806cf50 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840091326 0 0) 0x7feea812f000 con 0x7feea806d480 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea809e8b0 con 0x7feea806d480 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee90001910 con 0x7feea806cf50 2024-09-16T06:39:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee980010c0 con 0x7feea806d480 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1521613815 0 0) 0x7fee880035d0 con 0x7feea806cf50 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6790/0 conn(0x7feea8132330 legacy=0x7feea81d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.189:6789/0 conn(0x7feea806d480 legacy=0x7feea81cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 monclient: found mon.a 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8131650 con 0x7feea806cf50 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 5 monclient: authenticate success, global_id 4509 2024-09-16T06:39:13.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _renew_subs 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea812f650 con 0x7feea806cf50 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _renew_subs 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee900014f0 con 0x7feea806cf50 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feea813ec70 con 0x7feea806cf50 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee90003840 con 0x7feea806cf50 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feea559a700 4 set_mon_vals no callback set 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 10 monclient: dump: 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.622 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feea80a3430 con 0x7feea8130ee0 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.600+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fee900039a0 con 0x7feea806cf50 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fee90042ea0 con 0x7feea806cf50 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601403418 0 0) 0x7feea809e8b0 con 0x7feea8130ee0 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient(hunting): my global_id is 4512 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee880035d0 con 0x7feea8130ee0 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7feea759e700 10 monclient: get_auth_request con 0x7fee88036570 auth_method 0 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2054783160 0 0) 0x7fee880035d0 con 0x7feea8130ee0 2024-09-16T06:39:13.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea809e8b0 con 0x7feea8130ee0 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee98004a80 con 0x7feea8130ee0 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient: dump: 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.624 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3933155199 0 0) 0x7feea809e8b0 con 0x7feea8130ee0 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feea81309c0 con 0x7feea8130ee0 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fee98007480 con 0x7feea8130ee0 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6789/0 conn(0x7feea8130ee0 legacy=0x7feea809fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.604+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6789/0 conn(0x7feea8130ee0 legacy=0x7feea809fad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feea81c6840 con 0x7feea81c6430 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042007140 0 0) 0x7feea81c6840 con 0x7feea81c6430 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient(hunting): my global_id is 4515 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea809e8b0 con 0x7feea81c6430 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3350222530 0 0) 0x7feea809e8b0 con 0x7feea81c6430 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea81c6840 con 0x7feea81c6430 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee98005530 con 0x7feea81c6430 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient: dump: 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 596020287 0 0) 0x7feea81c6840 con 0x7feea81c6430 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feea80a3430 con 0x7feea81c6430 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 4243314543) 0x7fee980075f0 con 0x7feea81c6430 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6789/0 conn(0x7feea81c6430 legacy=0x7feea802c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7fee9f7fe700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6789/0 conn(0x7feea81c6430 legacy=0x7feea802c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:13.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:13.634 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.635 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.640 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.641 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.642 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:13.643 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.189:6812/3098810674 conn(0x7fee88036570 legacy=0x7fee88038a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 10 monclient: shutdown 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 >> v1:172.21.15.76:6789/0 conn(0x7feea806cf50 legacy=0x7feea813e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feea759e700 1 -- 172.21.15.76:0/3233233711 reap_dead start 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feea759e700 1 -- 172.21.15.76:0/3233233711 reap_dead start 2024-09-16T06:39:13.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 shutdown_connections 2024-09-16T06:39:13.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 >> 172.21.15.76:0/3233233711 conn(0x7feea80a41c0 msgr2=0x7feea809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 shutdown_connections 2024-09-16T06:39:13.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.616+0000 7feeac885700 1 -- 172.21.15.76:0/3233233711 wait complete. 2024-09-16T06:39:13.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:13.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:39:13.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:39:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.756+0000 7fcd27fff700 10 monclient: get_monmap_and_config 2024-09-16T06:39:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.756+0000 7fcd27fff700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.756+0000 7fcd27fff700 10 monclient: monmap: 2024-09-16T06:39:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:13.763434+0000 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:13.763434+0000 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:13.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 Processor -- start 2024-09-16T06:39:13.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- start start 2024-09-16T06:39:13.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient: init 2024-09-16T06:39:13.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-c con 0x7fcd28097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-a con 0x7fcd28099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-b con 0x7fcd28139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd28098990 con 0x7fcd28099090 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd28097f40 con 0x7fcd28097b60 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2809d610 con 0x7fcd28139500 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient(hunting): authenticate will time out at 900.896725s 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd267fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcd28099090 0x7fcd2809f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42426/0 (socket says 172.21.15.76:42426) 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd267fc700 1 -- 172.21.15.76:0/317035950 learned_addr learned my addr 172.21.15.76:0/317035950 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395507797 0 0) 0x7fcd2809d610 con 0x7fcd28139500 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): my global_id is 4511 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd100035d0 con 0x7fcd28139500 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885965291 0 0) 0x7fcd28097f40 con 0x7fcd28097b60 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): my global_id is 94141 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd2809d610 con 0x7fcd28097b60 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718223879 0 0) 0x7fcd28098990 con 0x7fcd28099090 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient(hunting): my global_id is 4518 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28097f40 con 0x7fcd28099090 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485434626 0 0) 0x7fcd28097f40 con 0x7fcd28099090 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 >> v1:172.21.15.76:6790/0 conn(0x7fcd28139500 legacy=0x7fcd2813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 >> v1:172.21.15.189:6789/0 conn(0x7fcd28097b60 legacy=0x7fcd2809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 monclient: found mon.noname-a 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd280986b0 con 0x7fcd28099090 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd1c0015b0 con 0x7fcd28099090 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 5 monclient: authenticate success, global_id 4518 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: dump: 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd1c001890 con 0x7fcd28099090 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 4 set_mon_vals no callback set 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 1 -- 172.21.15.76:0/317035950 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd1c001c60 con 0x7fcd28099090 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd25ffb700 10 monclient: dump: 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 10 monclient: shutdown 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- 172.21.15.76:0/317035950 >> v1:172.21.15.76:6789/0 conn(0x7fcd28099090 legacy=0x7fcd2809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- 172.21.15.76:0/317035950 shutdown_connections 2024-09-16T06:39:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- 172.21.15.76:0/317035950 >> 172.21.15.76:0/317035950 conn(0x7fcd280a0030 msgr2=0x7fcd280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- 172.21.15.76:0/317035950 shutdown_connections 2024-09-16T06:39:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.760+0000 7fcd27fff700 1 -- 172.21.15.76:0/317035950 wait complete. 2024-09-16T06:39:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient: monmap: 2024-09-16T06:39:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 1 Processor -- start 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 1 -- start start 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient: init 2024-09-16T06:39:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-a con 0x7fcd28097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-c con 0x7fcd28099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient(hunting): picked mon.noname-b con 0x7fcd28139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd267fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcd28099090 0x7fcd28133ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60568/0 (socket says 172.21.15.76:60568) 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd267fc700 1 -- 172.21.15.76:0/3465021800 learned_addr learned my addr 172.21.15.76:0/3465021800 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd26ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcd28097b60 0x7fcd281337b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42434/0 (socket says 172.21.15.76:42434) 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2809a5b0 con 0x7fcd28097b60 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd281384e0 con 0x7fcd28139500 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd28071d70 con 0x7fcd28099090 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd27fff700 10 monclient(hunting): authenticate will time out at 900.902964s 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467488406 0 0) 0x7fcd28071d70 con 0x7fcd28099090 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): my global_id is 4514 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd140035d0 con 0x7fcd28099090 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657582175 0 0) 0x7fcd18001130 con 0x7fcd28097b60 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): my global_id is 4521 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28071d70 con 0x7fcd28097b60 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569877992 0 0) 0x7fcd281384e0 con 0x7fcd28139500 2024-09-16T06:39:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): my global_id is 94144 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd18001130 con 0x7fcd28139500 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 215273061 0 0) 0x7fcd140035d0 con 0x7fcd28099090 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd281384e0 con 0x7fcd28099090 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068946707 0 0) 0x7fcd18001130 con 0x7fcd28139500 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd140035d0 con 0x7fcd28139500 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 146023532 0 0) 0x7fcd28071d70 con 0x7fcd28097b60 2024-09-16T06:39:13.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd18001130 con 0x7fcd28097b60 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd1c001630 con 0x7fcd28099090 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.764+0000 7fcd0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd20001bd0 con 0x7fcd28139500 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd18001830 con 0x7fcd28097b60 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 338130322 0 0) 0x7fcd281384e0 con 0x7fcd28099090 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.189:6789/0 conn(0x7fcd28139500 legacy=0x7fcd281cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6789/0 conn(0x7fcd28097b60 legacy=0x7fcd281337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 monclient: found mon.c 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28135fe0 con 0x7fcd28099090 2024-09-16T06:39:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 5 monclient: authenticate success, global_id 4514 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 10 monclient: _renew_subs 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd281362c0 con 0x7fcd28099090 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 10 monclient: _renew_subs 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd28135210 con 0x7fcd28099090 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd1c004b90 con 0x7fcd28099090 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd1c004ea0 con 0x7fcd28099090 2024-09-16T06:39:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: dump: 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd257fa700 4 set_mon_vals no callback set 2024-09-16T06:39:13.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcd1c005080 con 0x7fcd28099090 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd08006590 con 0x7fcd08002830 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fcd1c043420 con 0x7fcd28099090 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd277fe700 10 monclient: get_auth_request con 0x7fcd140366b0 auth_method 0 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939724670 0 0) 0x7fcd08006590 con 0x7fcd08002830 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient(hunting): my global_id is 4517 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd281384e0 con 0x7fcd08002830 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2328055025 0 0) 0x7fcd281384e0 con 0x7fcd08002830 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd08006590 con 0x7fcd08002830 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd18003ab0 con 0x7fcd08002830 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: dump: 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1761765950 0 0) 0x7fcd08006590 con 0x7fcd08002830 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd0800a190 con 0x7fcd08002830 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fcd180038f0 con 0x7fcd08002830 2024-09-16T06:39:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:13.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6790/0 conn(0x7fcd08002830 legacy=0x7fcd08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.768+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6790/0 conn(0x7fcd08002830 legacy=0x7fcd08004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd27fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcd08009e50 con 0x7fcd08009170 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673849630 0 0) 0x7fcd08009e50 con 0x7fcd08009170 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient(hunting): my global_id is 4520 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08006590 con 0x7fcd08009170 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 186006724 0 0) 0x7fcd08006590 con 0x7fcd08009170 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd08009e50 con 0x7fcd08009170 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcd18004240 con 0x7fcd08009170 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient: dump: 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1697791391 0 0) 0x7fcd08009e50 con 0x7fcd08009170 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcd0800c340 con 0x7fcd08009170 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:13.800 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:13.801 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.802 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:13.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:13.804 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.807 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.808 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.809 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:13.809 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:13.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2939052497) 0x7fcd180044f0 con 0x7fcd08009170 2024-09-16T06:39:13.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6790/0 conn(0x7fcd08009170 legacy=0x7fcd08009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd0f7fe700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6790/0 conn(0x7fcd08009170 legacy=0x7fcd08009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.780+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcd140366b0 legacy=0x7fcd14038b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 10 monclient: shutdown 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 >> v1:172.21.15.76:6790/0 conn(0x7fcd28099090 legacy=0x7fcd28133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd277fe700 1 -- 172.21.15.76:0/3465021800 reap_dead start 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd277fe700 1 -- 172.21.15.76:0/3465021800 reap_dead start 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 shutdown_connections 2024-09-16T06:39:13.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 >> 172.21.15.76:0/3465021800 conn(0x7fcd280a0030 msgr2=0x7fcd2806be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 shutdown_connections 2024-09-16T06:39:13.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.784+0000 7fcd27fff700 1 -- 172.21.15.76:0/3465021800 wait complete. 2024-09-16T06:39:13.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:13.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:39:13.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:39:13.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: monmap: 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:13.930732+0000 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:13.930732+0000 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:13.929 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 1 Processor -- start 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 1 -- start start 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: init 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fb48c0973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fb48c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fb48c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c099350 con 0x7fb48c0973c0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c09f690 con 0x7fb48c097eb0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c099bf0 con 0x7fb48c1400d0 2024-09-16T06:39:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.924+0000 7fb48bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient(hunting): authenticate will time out at 901.064007s 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb48c1400d0 0x7fb48c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35300/0 (socket says 172.21.15.76:35300) 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb48c0973c0 0x7fb48c0977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:32856/0 (socket says 172.21.15.76:32856) 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48b7fe700 1 -- 172.21.15.76:0/1040911087 learned_addr learned my addr 172.21.15.76:0/1040911087 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777473495 0 0) 0x7fb48c09f690 con 0x7fb48c097eb0 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): my global_id is 94147 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4780035d0 con 0x7fb48c097eb0 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396047749 0 0) 0x7fb48c099bf0 con 0x7fb48c1400d0 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): my global_id is 4523 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c09f690 con 0x7fb48c1400d0 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910484125 0 0) 0x7fb48c099350 con 0x7fb48c0973c0 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient(hunting): my global_id is 4524 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c099bf0 con 0x7fb48c0973c0 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819929538 0 0) 0x7fb4780035d0 con 0x7fb48c097eb0 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 >> v1:172.21.15.76:6790/0 conn(0x7fb48c1400d0 legacy=0x7fb48c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 >> v1:172.21.15.76:6789/0 conn(0x7fb48c0973c0 legacy=0x7fb48c0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 monclient: found mon.noname-c 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c098360 con 0x7fb48c097eb0 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 5 monclient: authenticate success, global_id 94147 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb47c001610 con 0x7fb48c097eb0 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: dump: 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb47c001830 con 0x7fb48c097eb0 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 4 set_mon_vals no callback set 2024-09-16T06:39:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 1 -- 172.21.15.76:0/1040911087 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb47c001bc0 con 0x7fb48c097eb0 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48a7fc700 10 monclient: dump: 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient: shutdown 2024-09-16T06:39:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- 172.21.15.76:0/1040911087 >> v1:172.21.15.189:6789/0 conn(0x7fb48c097eb0 legacy=0x7fb48c13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- 172.21.15.76:0/1040911087 shutdown_connections 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- 172.21.15.76:0/1040911087 >> 172.21.15.76:0/1040911087 conn(0x7fb48c0a0030 msgr2=0x7fb48c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- 172.21.15.76:0/1040911087 shutdown_connections 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- 172.21.15.76:0/1040911087 wait complete. 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient: build_initial_monmap 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient: monmap: 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:13.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 Processor -- start 2024-09-16T06:39:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 1 -- start start 2024-09-16T06:39:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.928+0000 7fb48bfff700 10 monclient: init 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fb48c0973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fb48c097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fb48c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c072660 con 0x7fb48c097eb0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c06c780 con 0x7fb48c0973c0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c06bee0 con 0x7fb48c1400d0 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient(hunting): authenticate will time out at 901.068542s 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb48c1400d0 0x7fb48c1d62a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35310/0 (socket says 172.21.15.76:35310) 2024-09-16T06:39:13.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48b7fe700 1 -- 172.21.15.76:0/1102543752 learned_addr learned my addr 172.21.15.76:0/1102543752 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146362502 0 0) 0x7fb48c06c780 con 0x7fb48c0973c0 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): my global_id is 94150 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c0035d0 con 0x7fb48c0973c0 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852727613 0 0) 0x7fb48c06bee0 con 0x7fb48c1400d0 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): my global_id is 4526 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c06c780 con 0x7fb48c1400d0 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378152000 0 0) 0x7fb46c0035d0 con 0x7fb48c0973c0 2024-09-16T06:39:13.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c06bee0 con 0x7fb48c0973c0 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493884559 0 0) 0x7fb48c072660 con 0x7fb48c097eb0 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): my global_id is 4527 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c0035d0 con 0x7fb48c097eb0 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb474001990 con 0x7fb48c0973c0 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028672289 0 0) 0x7fb48c06c780 con 0x7fb48c1400d0 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c072660 con 0x7fb48c1400d0 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb484001c30 con 0x7fb48c1400d0 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2467690525 0 0) 0x7fb48c06bee0 con 0x7fb48c0973c0 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.76:6790/0 conn(0x7fb48c1400d0 legacy=0x7fb48c1d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.76:6789/0 conn(0x7fb48c097eb0 legacy=0x7fb48c13cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 monclient: found mon.b 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c06c1c0 con 0x7fb48c0973c0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 5 monclient: authenticate success, global_id 94150 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _renew_subs 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c13dff0 con 0x7fb48c0973c0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb474001570 con 0x7fb48c0973c0 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:13.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb474004b40 con 0x7fb48c0973c0 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _renew_subs 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c13d4d0 con 0x7fb48c0973c0 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 10 monclient: dump: 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb489ffb700 4 set_mon_vals no callback set 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb48c0008d0 con 0x7fb48c06cba0 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb47400f630 con 0x7fb48c0973c0 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.932+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7fb474043180 con 0x7fb48c0973c0 2024-09-16T06:39:13.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817613635 0 0) 0x7fb48c0008d0 con 0x7fb48c06cba0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient(hunting): my global_id is 94153 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c06bee0 con 0x7fb48c06cba0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb48b7fe700 10 monclient: get_auth_request con 0x7fb46c036430 auth_method 0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3114606841 0 0) 0x7fb48c06bee0 con 0x7fb48c06cba0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c0008d0 con 0x7fb48c06cba0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb47c0011a0 con 0x7fb48c06cba0 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient: dump: 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2398525893 0 0) 0x7fb48c0008d0 con 0x7fb48c06cba0 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c06cf80 con 0x7fb48c06cba0 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb47c001540 con 0x7fb48c06cba0 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:13.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6789/0 conn(0x7fb48c06cba0 legacy=0x7fb48c061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.936+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6789/0 conn(0x7fb48c06cba0 legacy=0x7fb48c061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb48bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb48c02cc30 con 0x7fb48c02c820 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214231443 0 0) 0x7fb48c02cc30 con 0x7fb48c02c820 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient(hunting): my global_id is 94156 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c0008d0 con 0x7fb48c02c820 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1644249161 0 0) 0x7fb48c0008d0 con 0x7fb48c02c820 2024-09-16T06:39:13.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c02cc30 con 0x7fb48c02c820 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb47c003a30 con 0x7fb48c02c820 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient: dump: 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:13.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1014997885 0 0) 0x7fb48c02cc30 con 0x7fb48c02c820 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb48c0724e0 con 0x7fb48c02c820 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1071171102) 0x7fb47c004f30 con 0x7fb48c02c820 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6789/0 conn(0x7fb48c02c820 legacy=0x7fb48c094810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.944+0000 7fb488ff9700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6789/0 conn(0x7fb48c02c820 legacy=0x7fb48c094810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:13.959 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 18, 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.961 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:13.962 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:13.963 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.964 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.965 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:13.966 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:13.967 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:13.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:13.969 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:13.970 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:13.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb46c036430 legacy=0x7fb46c0388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 10 monclient: shutdown 2024-09-16T06:39:13.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:13.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 >> v1:172.21.15.189:6789/0 conn(0x7fb48c0973c0 legacy=0x7fb48c13c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:13.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48b7fe700 1 -- 172.21.15.76:0/1102543752 reap_dead start 2024-09-16T06:39:13.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 shutdown_connections 2024-09-16T06:39:13.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 >> 172.21.15.76:0/1102543752 conn(0x7fb48c0a0030 msgr2=0x7fb48c09d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:13.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 shutdown_connections 2024-09-16T06:39:13.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:13.948+0000 7fb48bfff700 1 -- 172.21.15.76:0/1102543752 wait complete. 2024-09-16T06:39:13.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:13.972 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:39:13.972 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:39:13.973 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:39:13.973 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:39:13.973 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:39:13.973 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:39:14.017 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:39:14.018 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:39:14.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:14.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 10 monclient: get_monmap_and_config 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 10 monclient: build_initial_monmap 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 10 monclient: monmap: 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:14.134742+0000 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:14.134742+0000 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:14.133 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 1 Processor -- start 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 1 -- start start 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.128+0000 7f05518f0700 10 monclient: init 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-a con 0x7f054c0973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-c con 0x7f054c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:14.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-b con 0x7f054c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c099350 con 0x7f054c0973c0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c09f690 con 0x7f054c097eb0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c099bf0 con 0x7f054c1400d0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient(hunting): authenticate will time out at 901.268319s 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05508ee700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f054c0973c0 legacy=0x7f054c0977a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05510ef700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f054c1400d0 0x7f054c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35342/0 (socket says 172.21.15.76:35342) 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05510ef700 1 -- 172.21.15.76:0/2009531593 learned_addr learned my addr 172.21.15.76:0/2009531593 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021120074 0 0) 0x7f054c099bf0 con 0x7f054c1400d0 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:14.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): my global_id is 4529 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05340035d0 con 0x7f054c1400d0 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145168553 0 0) 0x7f054c09f690 con 0x7f054c097eb0 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient(hunting): my global_id is 94159 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f054c099bf0 con 0x7f054c097eb0 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978529616 0 0) 0x7f05340035d0 con 0x7f054c1400d0 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 >> v1:172.21.15.189:6789/0 conn(0x7f054c097eb0 legacy=0x7f054c13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 >> v1:172.21.15.76:6789/0 conn(0x7f054c0973c0 legacy=0x7f054c0977a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:39:14.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c098360 con 0x7f054c1400d0 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 5 monclient: authenticate success, global_id 4529 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0544001610 con 0x7f054c1400d0 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: dump: 2024-09-16T06:39:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05440018f0 con 0x7f054c1400d0 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 4 set_mon_vals no callback set 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 1 -- 172.21.15.76:0/2009531593 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0544001c80 con 0x7f054c1400d0 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f054b7fe700 10 monclient: dump: 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: shutdown 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- 172.21.15.76:0/2009531593 >> v1:172.21.15.76:6790/0 conn(0x7f054c1400d0 legacy=0x7f054c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- 172.21.15.76:0/2009531593 shutdown_connections 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- 172.21.15.76:0/2009531593 >> 172.21.15.76:0/2009531593 conn(0x7f054c0a0030 msgr2=0x7f054c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- 172.21.15.76:0/2009531593 shutdown_connections 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 -- 172.21.15.76:0/2009531593 wait complete. 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: build_initial_monmap 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 10 monclient: monmap: 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.132+0000 7f05518f0700 1 Processor -- start 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- start start 2024-09-16T06:39:14.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: init 2024-09-16T06:39:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-c con 0x7f054c0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-b con 0x7f054c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient(hunting): picked mon.noname-a con 0x7f054c1400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c072660 con 0x7f054c1400d0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c06cc90 con 0x7f054c097eb0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c13fda0 con 0x7f054c0973c0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient(hunting): authenticate will time out at 901.272976s 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05508ee700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f054c0973c0 0x7f054c137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35358/0 (socket says 172.21.15.76:35358) 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05508ee700 1 -- 172.21.15.76:0/3174420710 learned_addr learned my addr 172.21.15.76:0/3174420710 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05510ef700 1 -- 172.21.15.76:0/3174420710 >> v1:172.21.15.76:6789/0 conn(0x7f054c1400d0 legacy=0x7f054c1d6dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376344670 0 0) 0x7f054c06cc90 con 0x7f054c097eb0 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): my global_id is 94162 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05380035d0 con 0x7f054c097eb0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276188619 0 0) 0x7f054c13fda0 con 0x7f054c0973c0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): my global_id is 4532 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f054c06cc90 con 0x7f054c0973c0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3928379875 0 0) 0x7f05380035d0 con 0x7f054c097eb0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f054c13fda0 con 0x7f054c097eb0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f053c001c30 con 0x7f054c097eb0 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): dump: 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:14.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687444702 0 0) 0x7f054c06cc90 con 0x7f054c0973c0 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05380035d0 con 0x7f054c0973c0 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f05400041d0 con 0x7f054c0973c0 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient(hunting): dump: 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:14.153 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 537109372 0 0) 0x7f054c13fda0 con 0x7f054c097eb0 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 >> v1:172.21.15.76:6790/0 conn(0x7f054c0973c0 legacy=0x7f054c137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 >> v1:172.21.15.76:6789/0 conn(0x7f054c1400d0 legacy=0x7f054c1d6dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 monclient: found mon.b 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c09a540 con 0x7f054c097eb0 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 5 monclient: authenticate success, global_id 94162 2024-09-16T06:39:14.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _renew_subs 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c13ee30 con 0x7f054c097eb0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _renew_subs 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c13f860 con 0x7f054c097eb0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f053c003a40 con 0x7f054c097eb0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f053c003d50 con 0x7f054c097eb0 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 10 monclient: dump: 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:14.155 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f054affd700 4 set_mon_vals no callback set 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.136+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f053c00e840 con 0x7f054c097eb0 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05518f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05518f0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05518f0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:14.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c0008d0 con 0x7f054c097eb0 2024-09-16T06:39:14.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f053c0423c0 con 0x7f054c097eb0 2024-09-16T06:39:14.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.140+0000 7f05508ee700 10 monclient: get_auth_request con 0x7f05380362b0 auth_method 0 2024-09-16T06:39:14.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.148+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f053c013760 con 0x7f054c097eb0 2024-09-16T06:39:14.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.148+0000 7f05497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:14.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.148+0000 7f05497fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:14.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f054c06c8a0 con 0x7f054c097eb0 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05497fa700 1 -- 172.21.15.76:0/3174420710 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 845647147) 0x7f053c011e00 con 0x7f054c097eb0 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:14.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05497fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:14.380 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:14.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 >> v1:172.21.15.189:6812/3098810674 conn(0x7f05380362b0 legacy=0x7f0538038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:14.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 10 monclient: shutdown 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 >> v1:172.21.15.189:6789/0 conn(0x7f054c097eb0 legacy=0x7f054c1d36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.376+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 shutdown_connections 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.380+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 >> 172.21.15.76:0/3174420710 conn(0x7f054c0a0030 msgr2=0x7f054c0a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.380+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 shutdown_connections 2024-09-16T06:39:14.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:14.380+0000 7f05518f0700 1 -- 172.21.15.76:0/3174420710 wait complete. 2024-09-16T06:39:17.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: get_monmap_and_config 2024-09-16T06:39:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: build_initial_monmap 2024-09-16T06:39:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: monmap: 2024-09-16T06:39:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:17.516248+0000 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:17.516248+0000 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:17.515 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 1 Processor -- start 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 1 -- start start 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: init 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3144132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient(hunting): picked mon.noname-c con 0x7f314406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient(hunting): picked mon.noname-a con 0x7f314406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314406c280 con 0x7f314406cf50 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314406bfa0 con 0x7f314406d480 2024-09-16T06:39:17.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314409ace0 con 0x7f3144132330 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient(hunting): authenticate will time out at 904.649698s 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3142d9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f314406cf50 legacy=0x7f314413da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3144132330 0x7f3144132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35372/0 (socket says 172.21.15.76:35372) 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314259c700 1 -- 172.21.15.76:0/3609295933 learned_addr learned my addr 172.21.15.76:0/3609295933 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474989965 0 0) 0x7f314409ace0 con 0x7f3144132330 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): my global_id is 4535 2024-09-16T06:39:17.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31300035d0 con 0x7f3144132330 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689389182 0 0) 0x7f314406bfa0 con 0x7f314406d480 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient(hunting): my global_id is 94165 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314409ace0 con 0x7f314406d480 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409449030 0 0) 0x7f314409ace0 con 0x7f314406d480 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 >> v1:172.21.15.76:6790/0 conn(0x7f3144132330 legacy=0x7f3144132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 >> v1:172.21.15.76:6789/0 conn(0x7f314406cf50 legacy=0x7f314413da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:17.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 monclient: found mon.noname-c 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314409ae70 con 0x7f314406d480 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: _finish_auth 0 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 5 monclient: authenticate success, global_id 94165 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31340015f0 con 0x7f314406d480 2024-09-16T06:39:17.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: dump: 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3134001910 con 0x7f314406d480 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 1 -- 172.21.15.76:0/3609295933 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3134001b00 con 0x7f314406d480 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f314359e700 4 set_mon_vals no callback set 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.512+0000 7f3141d9b700 10 monclient: dump: 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.524 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: shutdown 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- 172.21.15.76:0/3609295933 >> v1:172.21.15.189:6789/0 conn(0x7f314406d480 legacy=0x7f314413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- 172.21.15.76:0/3609295933 shutdown_connections 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- 172.21.15.76:0/3609295933 >> 172.21.15.76:0/3609295933 conn(0x7f31440a41c0 msgr2=0x7f31440a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:17.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- 172.21.15.76:0/3609295933 shutdown_connections 2024-09-16T06:39:17.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- 172.21.15.76:0/3609295933 wait complete. 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: build_initial_monmap 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: monmap: 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:17.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 Processor -- start 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- start start 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: init 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient(hunting): picked mon.noname-a con 0x7f314406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient(hunting): picked mon.noname-c con 0x7f314406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3144132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314409e8b0 con 0x7f314406cf50 2024-09-16T06:39:17.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31441308e0 con 0x7f3144132330 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144130bc0 con 0x7f314406d480 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314259c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f314406cf50 legacy=0x7f31441c9850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f314359e700 10 monclient(hunting): authenticate will time out at 904.655373s 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f313bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f314406d480 0x7f314413e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35384/0 (socket says 172.21.15.76:35384) 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f313bfff700 1 -- 172.21.15.76:0/4220342062 learned_addr learned my addr 172.21.15.76:0/4220342062 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.516+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733929025 0 0) 0x7f31441308e0 con 0x7f3144132330 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:17.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): my global_id is 94168 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31240035d0 con 0x7f3144132330 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366775388 0 0) 0x7f3144130bc0 con 0x7f314406d480 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): my global_id is 4538 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31441308e0 con 0x7f314406d480 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1182373591 0 0) 0x7f31441308e0 con 0x7f314406d480 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3144130bc0 con 0x7f314406d480 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31340036a0 con 0x7f314406d480 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:17.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): dump: 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166119860 0 0) 0x7f31240035d0 con 0x7f3144132330 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31441308e0 con 0x7f3144132330 2024-09-16T06:39:17.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f313c004390 con 0x7f3144132330 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient(hunting): dump: 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1827383579 0 0) 0x7f3144130bc0 con 0x7f314406d480 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 >> v1:172.21.15.189:6789/0 conn(0x7f3144132330 legacy=0x7f31441d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 >> v1:172.21.15.76:6789/0 conn(0x7f314406cf50 legacy=0x7f31441c9850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 monclient: found mon.c 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314412fb80 con 0x7f314406d480 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: _finish_auth 0 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 5 monclient: authenticate success, global_id 4538 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _renew_subs 2024-09-16T06:39:17.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31441ca6e0 con 0x7f314406d480 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _renew_subs 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f314413edc0 con 0x7f314406d480 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3134003b70 con 0x7f314406d480 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3134003e80 con 0x7f314406d480 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f3140d99700 4 set_mon_vals no callback set 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 10 monclient: dump: 2024-09-16T06:39:17.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:17.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3128005100 con 0x7f314406d480 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f31340040e0 con 0x7f314406d480 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f31340433d0 con 0x7f314406d480 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.520+0000 7f314259c700 10 monclient: get_auth_request con 0x7f31240362c0 auth_method 0 2024-09-16T06:39:17.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.528+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3134014ac0 con 0x7f314406d480 2024-09-16T06:39:17.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.528+0000 7f313affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:17.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.528+0000 7f313affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:17.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.760+0000 7f314359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:17.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.760+0000 7f314359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:17.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.760+0000 7f314359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:17.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.760+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3128005930 con 0x7f314406d480 2024-09-16T06:39:17.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f313affd700 1 -- 172.21.15.76:0/4220342062 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1653293834) 0x7f31340146b0 con 0x7f314406d480 2024-09-16T06:39:17.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f313affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:17.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f313affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:17.767 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:17.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 >> v1:172.21.15.189:6812/3098810674 conn(0x7f31240362c0 legacy=0x7f3124038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:17.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 10 monclient: shutdown 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 >> v1:172.21.15.76:6790/0 conn(0x7f314406d480 legacy=0x7f314413e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 shutdown_connections 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 >> 172.21.15.76:0/4220342062 conn(0x7f31440a41c0 msgr2=0x7f314409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 shutdown_connections 2024-09-16T06:39:17.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:17.764+0000 7f314359e700 1 -- 172.21.15.76:0/4220342062 wait complete. 2024-09-16T06:39:20.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: get_monmap_and_config 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: build_initial_monmap 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: monmap: 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:20.900909+0000 2024-09-16T06:39:20.899 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:20.900909+0000 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:20.900 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:20.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 1 Processor -- start 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 1 -- start start 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: init 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-b con 0x7f9f240973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-c con 0x7f9f24097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-a con 0x7f9f241400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24099350 con 0x7f9f241400d0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2409f690 con 0x7f9f24097eb0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24099bf0 con 0x7f9f240973c0 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f23fff700 10 monclient(hunting): authenticate will time out at 908.034633s 2024-09-16T06:39:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f237fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f22ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9f240973c0 0x7f9f240977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35386/0 (socket says 172.21.15.76:35386) 2024-09-16T06:39:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:20.896+0000 7f9f22ffd700 1 -- 172.21.15.76:0/860289839 learned_addr learned my addr 172.21.15.76:0/860289839 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:21.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:21.100+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:21.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:21.504+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:22.308+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:23.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): tick 2024-09-16T06:39:23.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:39:23.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 monclient(hunting): continuing hunt 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6790/0 conn(0x7f9f240973c0 legacy=0x7f9f240977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.189:6789/0 conn(0x7f9f24097eb0 legacy=0x7f9f2413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241424c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-c con 0x7f9f10003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-b con 0x7f9f240973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f9f241400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24098360 con 0x7f9f241400d0 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24099350 con 0x7f9f10003740 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f10010420 con 0x7f9f240973c0 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f20ff9700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:23.896+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:24.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:24.100+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:24.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:24.504+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:25.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:25.308+0000 7f9f237fe700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): tick 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 monclient(hunting): continuing hunt 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6790/0 conn(0x7f9f240973c0 legacy=0x7f9f10009450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.189:6789/0 conn(0x7f9f10003740 legacy=0x7f9f10005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:26.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-b con 0x7f9f240973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f9f241400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): picked mon.noname-c con 0x7f9f10003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f10011630 con 0x7f9f241400d0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24098360 con 0x7f9f10003740 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f227fc700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f100094d0 con 0x7f9f240973c0 2024-09-16T06:39:26.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:26.899+0000 7f9f20ff9700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:27.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:27.103+0000 7f9f227fc700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:27.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:27.507+0000 7f9f227fc700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528163726 0 0) 0x7f9f24098360 con 0x7f9f10003740 2024-09-16T06:39:28.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient(hunting): my global_id is 94171 2024-09-16T06:39:28.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c0035d0 con 0x7f9f10003740 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859105100 0 0) 0x7f9f0c0035d0 con 0x7f9f10003740 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6790/0 conn(0x7f9f240973c0 legacy=0x7f9f100106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f1000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 monclient: found mon.noname-c 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009680 con 0x7f9f10003740 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f1c0042d0 con 0x7f9f10003740 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f23fff700 5 monclient: authenticate success, global_id 94171 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f23fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: dump: 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f23fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f1c0044c0 con 0x7f9f10003740 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f23fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 1 -- 172.21.15.76:0/860289839 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f1c0048d0 con 0x7f9f10003740 2024-09-16T06:39:28.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f23fff700 4 set_mon_vals no callback set 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.035+0000 7f9f21ffb700 10 monclient: dump: 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.044 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 10 monclient: shutdown 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- 172.21.15.76:0/860289839 >> v1:172.21.15.189:6789/0 conn(0x7f9f10003740 legacy=0x7f9f10005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- 172.21.15.76:0/860289839 shutdown_connections 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- 172.21.15.76:0/860289839 >> 172.21.15.76:0/860289839 conn(0x7f9f240a0030 msgr2=0x7f9f240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- 172.21.15.76:0/860289839 shutdown_connections 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- 172.21.15.76:0/860289839 wait complete. 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 10 monclient: build_initial_monmap 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 10 monclient: monmap: 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:28.045 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:28.046 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:28.046 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:28.046 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:28.046 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:28.046 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 Processor -- start 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 1 -- start start 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.039+0000 7f9f23fff700 10 monclient: init 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-c con 0x7f9f10003740 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-b con 0x7f9f240973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient(hunting): picked mon.noname-a con 0x7f9f241400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f24072660 con 0x7f9f241400d0 2024-09-16T06:39:28.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2406c650 con 0x7f9f240973c0 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f237fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241d62a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2413fe30 con 0x7f9f10003740 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient(hunting): authenticate will time out at 915.177658s 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f22ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9f10003740 0x7f9f2413c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60794/0 (socket says 172.21.15.76:60794) 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f22ffd700 1 -- 172.21.15.76:0/3006979088 learned_addr learned my addr 172.21.15.76:0/3006979088 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064477044 0 0) 0x7f9f2406c650 con 0x7f9f240973c0 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): my global_id is 94174 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f10005340 con 0x7f9f240973c0 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392130679 0 0) 0x7f9f10005340 con 0x7f9f240973c0 2024-09-16T06:39:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f2406c650 con 0x7f9f240973c0 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f14001b80 con 0x7f9f240973c0 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): dump: 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2406165586 0 0) 0x7f9f2406c650 con 0x7f9f240973c0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 >> v1:172.21.15.76:6790/0 conn(0x7f9f10003740 legacy=0x7f9f2413c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 >> v1:172.21.15.76:6789/0 conn(0x7f9f241400d0 legacy=0x7f9f241d62a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 monclient: found mon.b 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2409a540 con 0x7f9f240973c0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 5 monclient: authenticate success, global_id 94174 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _renew_subs 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2413f9e0 con 0x7f9f240973c0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _renew_subs 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2413f480 con 0x7f9f240973c0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f14003a20 con 0x7f9f240973c0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f14003db0 con 0x7f9f240973c0 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f0bfff700 10 monclient: dump: 2024-09-16T06:39:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.043+0000 7f9f217fa700 4 set_mon_vals no callback set 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9f14003f90 con 0x7f9f240973c0 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f23fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f23fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f23fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f24094fc0 con 0x7f9f240973c0 2024-09-16T06:39:28.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f9f14043520 con 0x7f9f240973c0 2024-09-16T06:39:28.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.047+0000 7f9f22ffd700 10 monclient: get_auth_request con 0x7f9f10036d60 auth_method 0 2024-09-16T06:39:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.059+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9f140047b0 con 0x7f9f240973c0 2024-09-16T06:39:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.059+0000 7f9f0bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.059+0000 7f9f0bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:28.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.291+0000 7f9f23fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:28.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.291+0000 7f9f23fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:28.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.291+0000 7f9f23fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:28.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.291+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9f240672a0 con 0x7f9f240973c0 2024-09-16T06:39:28.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f0bfff700 1 -- 172.21.15.76:0/3006979088 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1418651499) 0x7f9f14012f60 con 0x7f9f240973c0 2024-09-16T06:39:28.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f0bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:28.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f0bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:28.295 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:28.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9f10036d60 legacy=0x7f9f10039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 10 monclient: shutdown 2024-09-16T06:39:28.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 >> v1:172.21.15.189:6789/0 conn(0x7f9f240973c0 legacy=0x7f9f2413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 shutdown_connections 2024-09-16T06:39:28.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 >> 172.21.15.76:0/3006979088 conn(0x7f9f240a0030 msgr2=0x7f9f2409d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 shutdown_connections 2024-09-16T06:39:28.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.295+0000 7f9f23fff700 1 -- 172.21.15.76:0/3006979088 wait complete. 2024-09-16T06:39:28.309 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:39:28.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:39:28.310 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: get_monmap_and_config 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: build_initial_monmap 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: monmap: 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:28.420931+0000 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:28.420931+0000 2024-09-16T06:39:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:28.420 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.420 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:28.420 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:28.420 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:28.420 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 1 Processor -- start 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 1 -- start start 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: init 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-a con 0x7f380009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:28.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-b con 0x7f3800098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-c con 0x7f3800099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380009d560 con 0x7f380009ed60 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380009f140 con 0x7f3800099030 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380009d790 con 0x7f3800098be0 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient(hunting): authenticate will time out at 915.554281s 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3804b2a700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f380009ed60 legacy=0x7f38000984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37fffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3800098be0 0x7f38001374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60806/0 (socket says 172.21.15.76:60806) 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37fffff700 1 -- 172.21.15.76:0/2440818938 learned_addr learned my addr 172.21.15.76:0/2440818938 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681213212 0 0) 0x7f380009d790 con 0x7f3800098be0 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient(hunting): my global_id is 4541 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e80057e0 con 0x7f3800098be0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2692382210 0 0) 0x7f37e80057e0 con 0x7f3800098be0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 >> v1:172.21.15.189:6789/0 conn(0x7f3800099030 legacy=0x7f380013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 >> v1:172.21.15.76:6789/0 conn(0x7f380009ed60 legacy=0x7f38000984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 monclient: found mon.noname-b 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380009f620 con 0x7f3800098be0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f0001610 con 0x7f3800098be0 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 5 monclient: authenticate success, global_id 4541 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: dump: 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.432 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f0001930 con 0x7f3800098be0 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 1 -- 172.21.15.76:0/2440818938 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f0001cc0 con 0x7f3800098be0 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f3805b2c700 4 set_mon_vals no callback set 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.419+0000 7f37ff7fe700 10 monclient: dump: 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.433 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: shutdown 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/2440818938 >> v1:172.21.15.76:6790/0 conn(0x7f3800098be0 legacy=0x7f38001374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/2440818938 shutdown_connections 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/2440818938 >> 172.21.15.76:0/2440818938 conn(0x7f38000a0030 msgr2=0x7f38000a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/2440818938 shutdown_connections 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/2440818938 wait complete. 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: build_initial_monmap 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: monmap: 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 Processor -- start 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- start start 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: init 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-b con 0x7f3800098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-a con 0x7f3800099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient(hunting): picked mon.noname-c con 0x7f380009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380008f3a0 con 0x7f3800099030 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3800098540 con 0x7f3800098be0 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3800099030 legacy=0x7f38001ca840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38000a2d50 con 0x7f380009ed60 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:28.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient(hunting): authenticate will time out at 915.558476s 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f380532b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f380009ed60 0x7f38001cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60820/0 (socket says 172.21.15.76:60820) 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f380532b700 1 -- 172.21.15.76:0/3770058496 learned_addr learned my addr 172.21.15.76:0/3770058496 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730754636 0 0) 0x7f38000a2d50 con 0x7f380009ed60 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): my global_id is 4544 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec0035d0 con 0x7f380009ed60 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870349234 0 0) 0x7f3800098540 con 0x7f3800098be0 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): my global_id is 94180 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38000a2d50 con 0x7f3800098be0 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606282975 0 0) 0x7f37ec0035d0 con 0x7f380009ed60 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3800098540 con 0x7f380009ed60 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f8001cc0 con 0x7f380009ed60 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.438 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928963629 0 0) 0x7f38000a2d50 con 0x7f3800098be0 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec0035d0 con 0x7f3800098be0 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f4004170 con 0x7f3800098be0 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.439 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1154058259 0 0) 0x7f3800098540 con 0x7f380009ed60 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.189:6789/0 conn(0x7f3800098be0 legacy=0x7f38001c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6789/0 conn(0x7f3800099030 legacy=0x7f38001ca840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 monclient: found mon.c 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000907b0 con 0x7f380009ed60 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 5 monclient: authenticate success, global_id 4544 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _renew_subs 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3800090a90 con 0x7f380009ed60 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _renew_subs 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f380008df00 con 0x7f380009ed60 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f8002030 con 0x7f380009ed60 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f8003e90 con 0x7f380009ed60 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37feffd700 4 set_mon_vals no callback set 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.423+0000 7f37fd7fa700 10 monclient: dump: 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.442 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f37f8004070 con 0x7f380009ed60 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f3805b2c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f37e0006590 con 0x7f37e0002830 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f37f8043610 con 0x7f380009ed60 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fffff700 10 monclient: get_auth_request con 0x7f37ec0362a0 auth_method 0 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434417086 0 0) 0x7f37e0006590 con 0x7f37e0002830 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient(hunting): my global_id is 4547 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3800098540 con 0x7f37e0002830 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1062646108 0 0) 0x7f3800098540 con 0x7f37e0002830 2024-09-16T06:39:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37e0006590 con 0x7f37e0002830 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f40048c0 con 0x7f37e0002830 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient: dump: 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.444 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2085409103 0 0) 0x7f37e0006590 con 0x7f37e0002830 2024-09-16T06:39:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37e000a190 con 0x7f37e0002830 2024-09-16T06:39:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f37f4004c50 con 0x7f37e0002830 2024-09-16T06:39:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:28.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6790/0 conn(0x7f37e0002830 legacy=0x7f37e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.427+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6790/0 conn(0x7f37e0002830 legacy=0x7f37e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f37e0009e50 con 0x7f37e0009170 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8170869 0 0) 0x7f37e0009e50 con 0x7f37e0009170 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient(hunting): my global_id is 4550 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e0006590 con 0x7f37e0009170 2024-09-16T06:39:28.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3346979242 0 0) 0x7f37e0006590 con 0x7f37e0009170 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37e0009e50 con 0x7f37e0009170 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37f4005260 con 0x7f37e0009170 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient: dump: 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3556441277 0 0) 0x7f37e0009e50 con 0x7f37e0009170 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37e000c340 con 0x7f37e0009170 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 2410485102) 0x7f37f4006680 con 0x7f37e0009170 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6790/0 conn(0x7f37e0009170 legacy=0x7f37e0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f37fd7fa700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6790/0 conn(0x7f37e0009170 legacy=0x7f37e0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:28.452 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 22, 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.453 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:28.455 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:28.456 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:28.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.458 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.459 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:28.460 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.461 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.462 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.463 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:28.464 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.189:6812/3098810674 conn(0x7f37ec0362a0 legacy=0x7f37ec038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 10 monclient: shutdown 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 >> v1:172.21.15.76:6790/0 conn(0x7f380009ed60 legacy=0x7f38001cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f380532b700 1 -- 172.21.15.76:0/3770058496 reap_dead start 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f380532b700 1 -- 172.21.15.76:0/3770058496 reap_dead start 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 shutdown_connections 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 >> 172.21.15.76:0/3770058496 conn(0x7f38000a0030 msgr2=0x7f380006ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 shutdown_connections 2024-09-16T06:39:28.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.439+0000 7f3805b2c700 1 -- 172.21.15.76:0/3770058496 wait complete. 2024-09-16T06:39:28.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:39:28.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:39:28.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:39:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.583+0000 7f4d094b2700 10 monclient: get_monmap_and_config 2024-09-16T06:39:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.583+0000 7f4d094b2700 10 monclient: build_initial_monmap 2024-09-16T06:39:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.583+0000 7f4d094b2700 10 monclient: monmap: 2024-09-16T06:39:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:28.587048+0000 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:28.587048+0000 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:28.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.583+0000 7f4d094b2700 1 Processor -- start 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- start start 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient: init 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-a con 0x7f4d0409ee40 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-c con 0x7f4d04138570 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-b con 0x7f4d0413c080 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0409f930 con 0x7f4d0409ee40 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0406d9b0 con 0x7f4d04138570 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0406cb30 con 0x7f4d0413c080 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient(hunting): authenticate will time out at 915.720871s 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d03fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4d0409ee40 legacy=0x7f4d0409f220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d08cb1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d0413c080 0x7f4d0413e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60836/0 (socket says 172.21.15.76:60836) 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d08cb1700 1 -- 172.21.15.76:0/2146870552 learned_addr learned my addr 172.21.15.76:0/2146870552 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710320233 0 0) 0x7f4d0406cb30 con 0x7f4d0413c080 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): my global_id is 4553 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf00035d0 con 0x7f4d0413c080 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422029603 0 0) 0x7f4d0406d9b0 con 0x7f4d04138570 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient(hunting): my global_id is 94183 2024-09-16T06:39:28.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0406cb30 con 0x7f4d04138570 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1037438352 0 0) 0x7f4cf00035d0 con 0x7f4d0413c080 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 >> v1:172.21.15.189:6789/0 conn(0x7f4d04138570 legacy=0x7f4d0413a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 >> v1:172.21.15.76:6789/0 conn(0x7f4d0409ee40 legacy=0x7f4d0409f220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 monclient: found mon.noname-b 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0409d7d0 con 0x7f4d0413c080 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cfc001990 con 0x7f4d0413c080 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 5 monclient: authenticate success, global_id 4553 2024-09-16T06:39:28.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: dump: 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cfc001c70 con 0x7f4d0413c080 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 1 -- 172.21.15.76:0/2146870552 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cfc002000 con 0x7f4d0413c080 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 4 set_mon_vals no callback set 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d02ffd700 10 monclient: dump: 2024-09-16T06:39:28.595 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 10 monclient: shutdown 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- 172.21.15.76:0/2146870552 >> v1:172.21.15.76:6790/0 conn(0x7f4d0413c080 legacy=0x7f4d0413e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- 172.21.15.76:0/2146870552 shutdown_connections 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- 172.21.15.76:0/2146870552 >> 172.21.15.76:0/2146870552 conn(0x7f4d040a2300 msgr2=0x7f4d040a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- 172.21.15.76:0/2146870552 shutdown_connections 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.587+0000 7f4d094b2700 1 -- 172.21.15.76:0/2146870552 wait complete. 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: build_initial_monmap 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: monmap: 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:28.597 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 Processor -- start 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- start start 2024-09-16T06:39:28.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: init 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-b con 0x7f4d0409ee40 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-c con 0x7f4d04138570 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient(hunting): picked mon.noname-a con 0x7f4d0413c080 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:28.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0409b750 con 0x7f4d0413c080 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0413a930 con 0x7f4d0409ee40 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0413b560 con 0x7f4d04138570 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d08cb1700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4d0413c080 legacy=0x7f4d041d2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient(hunting): authenticate will time out at 915.725667s 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d037fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d04138570 0x7f4d041ca910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60848/0 (socket says 172.21.15.76:60848) 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d037fe700 1 -- 172.21.15.76:0/3742863021 learned_addr learned my addr 172.21.15.76:0/3742863021 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608151902 0 0) 0x7f4d0413b560 con 0x7f4d04138570 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): my global_id is 4556 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce40035d0 con 0x7f4d04138570 2024-09-16T06:39:28.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152030205 0 0) 0x7f4d0413a930 con 0x7f4d0409ee40 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): my global_id is 94186 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0413b560 con 0x7f4d0409ee40 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228121001 0 0) 0x7f4ce40035d0 con 0x7f4d04138570 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d0413a930 con 0x7f4d04138570 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cf4001970 con 0x7f4d04138570 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 97305884 0 0) 0x7f4d0413a930 con 0x7f4d04138570 2024-09-16T06:39:28.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6789/0 conn(0x7f4d0409ee40 legacy=0x7f4d041ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.76:6789/0 conn(0x7f4d0413c080 legacy=0x7f4d041d2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 monclient: found mon.c 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d041cb020 con 0x7f4d04138570 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 5 monclient: authenticate success, global_id 4556 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _renew_subs 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0413ae60 con 0x7f4d04138570 2024-09-16T06:39:28.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _renew_subs 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d04139960 con 0x7f4d04138570 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf4001c80 con 0x7f4d04138570 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cf4003b10 con 0x7f4d04138570 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:28.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d00ff9700 10 monclient: dump: 2024-09-16T06:39:28.608 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.608 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.608 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.608 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d027fc700 4 set_mon_vals no callback set 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.591+0000 7f4d094b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d040678b0 con 0x7f4d0409aeb0 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4cf4003c70 con 0x7f4d04138570 2024-09-16T06:39:28.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f4cf40431f0 con 0x7f4d04138570 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013918148 0 0) 0x7f4d040678b0 con 0x7f4d0409aeb0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient(hunting): my global_id is 94189 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0413a930 con 0x7f4d0409aeb0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d08cb1700 10 monclient: get_auth_request con 0x7f4ce40361d0 auth_method 0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1840559725 0 0) 0x7f4d0413a930 con 0x7f4d0409aeb0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d040678b0 con 0x7f4d0409aeb0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cec004150 con 0x7f4d0409aeb0 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:28.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient: dump: 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 315506181 0 0) 0x7f4d040678b0 con 0x7f4d0409aeb0 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d0413b880 con 0x7f4d0409aeb0 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4cec004540 con 0x7f4d0409aeb0 2024-09-16T06:39:28.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:28.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:28.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6789/0 conn(0x7f4d0409aeb0 legacy=0x7f4d04135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.595+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6789/0 conn(0x7f4d0409aeb0 legacy=0x7f4d04135dd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d094b2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d0409b290 con 0x7f4d0402c6b0 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856570543 0 0) 0x7f4d0409b290 con 0x7f4d0402c6b0 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient(hunting): my global_id is 94192 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d040678b0 con 0x7f4d0402c6b0 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1238960385 0 0) 0x7f4d040678b0 con 0x7f4d0402c6b0 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d0409b290 con 0x7f4d0402c6b0 2024-09-16T06:39:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4cec004b50 con 0x7f4d0402c6b0 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient: dump: 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2412033672 0 0) 0x7f4d0409b290 con 0x7f4d0402c6b0 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d040a18e0 con 0x7f4d0402c6b0 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 3669613002) 0x7f4cec004e20 con 0x7f4d0402c6b0 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6789/0 conn(0x7f4d0402c6b0 legacy=0x7f4d041364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.607+0000 7f4d00ff9700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6789/0 conn(0x7f4d0402c6b0 legacy=0x7f4d041364e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:28.615 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 22, 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.616 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:28.618 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:28.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:28.621 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.622 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:28.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.624 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.625 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:28.626 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:28.627 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:28.627 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4ce40361d0 legacy=0x7f4ce4038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 10 monclient: shutdown 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 >> v1:172.21.15.76:6790/0 conn(0x7f4d04138570 legacy=0x7f4d041ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d08cb1700 1 -- 172.21.15.76:0/3742863021 reap_dead start 2024-09-16T06:39:28.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d08cb1700 1 -- 172.21.15.76:0/3742863021 reap_dead start 2024-09-16T06:39:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 shutdown_connections 2024-09-16T06:39:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 >> 172.21.15.76:0/3742863021 conn(0x7f4d040a2300 msgr2=0x7f4d0409dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 shutdown_connections 2024-09-16T06:39:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:28.611+0000 7f4d094b2700 1 -- 172.21.15.76:0/3742863021 wait complete. 2024-09-16T06:39:28.629 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:39:28.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:39:32.667 INFO:tasks.workunit.client.0.smithi189.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T06:39:32.667 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr: git switch -c 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.668 INFO:tasks.workunit.client.0.smithi189.stderr:Or undo this operation with: 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr: git switch - 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:32.669 INFO:tasks.workunit.client.0.smithi189.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T06:39:32.676 DEBUG:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T06:39:32.704 INFO:tasks.workunit.client.0.smithi189.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T06:39:32.707 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T06:39:32.707 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T06:39:33.577 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T06:39:33.690 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T06:39:33.753 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T06:39:33.755 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T06:39:33.755 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T06:39:33.817 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T06:39:33.824 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-16T06:39:33.824 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T06:39:33.830 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-16T06:39:33.832 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-16T06:39:33.832 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-16T06:39:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:+ TEST_POOL=testpool1234 2024-09-16T06:39:33.889 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.008+0000 7f65f49a9700 10 monclient: get_monmap_and_config 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.008+0000 7f65f49a9700 10 monclient: build_initial_monmap 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.008+0000 7f65f49a9700 10 monclient: monmap: 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:34.011710+0000 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:34.011710+0000 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:34.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:34.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:34.011 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:34.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.008+0000 7f65f49a9700 1 Processor -- start 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- start start 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: init 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-b con 0x7f65f0099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-a con 0x7f65f0099e40 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-c con 0x7f65f009a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f009c460 con 0x7f65f0099e40 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f009c1e0 con 0x7f65f009a250 2024-09-16T06:39:34.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f009bd60 con 0x7f65f0099a60 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient(hunting): authenticate will time out at 926.524924s 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65eed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65f009a250 0x7f65f013a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34938/0 (socket says 172.21.15.189:34938) 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65eed9d700 1 -- 172.21.15.189:0/2734592959 learned_addr learned my addr 172.21.15.189:0/2734592959 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65edd9b700 1 -- 172.21.15.189:0/2734592959 >> v1:172.21.15.76:6789/0 conn(0x7f65f0099e40 legacy=0x7f65f0136960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382053466 0 0) 0x7f65f009c1e0 con 0x7f65f009a250 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient(hunting): _negotiate 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient(hunting): my global_id is 94195 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d80046e0 con 0x7f65f009a250 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3894775279 0 0) 0x7f65d80046e0 con 0x7f65f009a250 2024-09-16T06:39:34.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 >> v1:172.21.15.76:6790/0 conn(0x7f65f0099a60 legacy=0x7f65f0098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 >> v1:172.21.15.76:6789/0 conn(0x7f65f0099e40 legacy=0x7f65f0136960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: _finish_hunting 0 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 monclient: found mon.noname-c 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f009d490 con 0x7f65f009a250 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: _finish_auth 0 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: _check_auth_tickets 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 5 monclient: authenticate success, global_id 94195 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65e8001990 con 0x7f65f009a250 2024-09-16T06:39:34.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: dump: 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:34.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65e8001c30 con 0x7f65f009a250 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 4 set_mon_vals no callback set 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 1 -- 172.21.15.189:0/2734592959 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65e8001fc0 con 0x7f65f009a250 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65ed59a700 10 monclient: dump: 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:34.022 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: shutdown 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- 172.21.15.189:0/2734592959 >> v1:172.21.15.189:6789/0 conn(0x7f65f009a250 legacy=0x7f65f013a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- 172.21.15.189:0/2734592959 shutdown_connections 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- 172.21.15.189:0/2734592959 >> 172.21.15.189:0/2734592959 conn(0x7f65f00a03e0 msgr2=0x7f65f00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- 172.21.15.189:0/2734592959 shutdown_connections 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- 172.21.15.189:0/2734592959 wait complete. 2024-09-16T06:39:34.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: build_initial_monmap 2024-09-16T06:39:34.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: monmap: 2024-09-16T06:39:34.024 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:34.024 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:34.024 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 Processor -- start 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 1 -- start start 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.012+0000 7f65f49a9700 10 monclient: init 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-a con 0x7f65f0099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-b con 0x7f65f0099e40 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient(hunting): picked mon.noname-c con 0x7f65f009a250 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f009e800 con 0x7f65f0099a60 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f009a6e0 con 0x7f65f0099e40 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f0092de0 con 0x7f65f009a250 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient(hunting): authenticate will time out at 926.528394s 2024-09-16T06:39:34.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65ee59c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f65f0099a60 legacy=0x7f65f01c9950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65edd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f65f0099e40 0x7f65f0096670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34942/0 (socket says 172.21.15.189:34942) 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65edd9b700 1 -- 172.21.15.189:0/3892575484 learned_addr learned my addr 172.21.15.189:0/3892575484 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700898648 0 0) 0x7f65f009a6e0 con 0x7f65f0099e40 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): my global_id is 94198 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d00035d0 con 0x7f65f0099e40 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140657789 0 0) 0x7f65f0092de0 con 0x7f65f009a250 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): my global_id is 4562 2024-09-16T06:39:34.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f009a6e0 con 0x7f65f009a250 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2737736439 0 0) 0x7f65d00035d0 con 0x7f65f0099e40 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f0092de0 con 0x7f65f0099e40 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65e0004230 con 0x7f65f0099e40 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): dump: 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:34.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838721928 0 0) 0x7f65f0092de0 con 0x7f65f0099e40 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 >> v1:172.21.15.76:6790/0 conn(0x7f65f009a250 legacy=0x7f65f01d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 >> v1:172.21.15.76:6789/0 conn(0x7f65f0099a60 legacy=0x7f65f01c9950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 monclient: found mon.b 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:34.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00948a0 con 0x7f65f0099e40 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 5 monclient: authenticate success, global_id 94198 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _renew_subs 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f0094b80 con 0x7f65f0099e40 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _renew_subs 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f0096f60 con 0x7f65f0099e40 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f008e7a0 con 0x7f65f0099e40 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65e0004450 con 0x7f65f0099e40 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65e0004640 con 0x7f65f0099e40 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 10 monclient: dump: 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:34.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:34.033 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:34.033 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:34.033 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:34.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65ecd99700 4 set_mon_vals no callback set 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.016+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f65e0004820 con 0x7f65f0099e40 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.020+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 3534+0+0 (unknown 1661005669 0 0) 0x7f65e0043060 con 0x7f65f0099e40 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.020+0000 7f65ee59c700 10 monclient: get_auth_request con 0x7f65d00361d0 auth_method 0 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.024+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f65e0005040 con 0x7f65f0099e40 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.024+0000 7f65deffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:34.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.024+0000 7f65deffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.204+0000 7f65f49a9700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.204+0000 7f65f49a9700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.204+0000 7f65f49a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:34.204+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f65f00a2590 con 0x7f65f0099e40 2024-09-16T06:39:35.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.119+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v19) v1 ==== 152+0+0 (unknown 1324558170 0 0) 0x7f65e0012aa0 con 0x7f65f0099e40 2024-09-16T06:39:35.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.119+0000 7f65deffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:35.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.119+0000 7f65deffd700 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-16T06:39:35.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65f49a9700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:35.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65f49a9700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:35.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65f49a9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f65f0093a70 con 0x7f65f0099e40 2024-09-16T06:39:35.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65deffd700 1 -- 172.21.15.189:0/3892575484 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v19) v1 ==== 159+0+0 (unknown 3296235900 0 0) 0x7f65e0014220 con 0x7f65f0099e40 2024-09-16T06:39:35.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65deffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T06:39:35.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.231+0000 7f65deffd700 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-16T06:39:35.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 >> v1:172.21.15.189:6812/3098810674 conn(0x7f65d00361d0 legacy=0x7f65d0038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:35.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 10 monclient: shutdown 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 >> v1:172.21.15.189:6789/0 conn(0x7f65f0099e40 legacy=0x7f65f0096670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 shutdown_connections 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 >> 172.21.15.189:0/3892575484 conn(0x7f65f00a03e0 msgr2=0x7f65f00710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 shutdown_connections 2024-09-16T06:39:35.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.235+0000 7f65f49a9700 1 -- 172.21.15.189:0/3892575484 wait complete. 2024-09-16T06:39:35.237 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'testpool1234' already exists 2024-09-16T06:39:35.246 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: get_monmap_and_config 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: build_initial_monmap 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: monmap: 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.352 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:35.353937+0000 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:35.353937+0000 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:35.353 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 1 Processor -- start 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 1 -- start start 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: init 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-c con 0x7f9750133420 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-a con 0x7f9750132e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-b con 0x7f9750133800 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500719f0 con 0x7f9750132e00 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750133260 con 0x7f9750133420 2024-09-16T06:39:35.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975009f690 con 0x7f9750133800 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f97561ea700 10 monclient(hunting): authenticate will time out at 927.867347s 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9750132e00 legacy=0x7f975009ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9750133420 0x7f975009e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34944/0 (socket says 172.21.15.189:34944) 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974f7fe700 1 -- 172.21.15.189:0/2829996765 learned_addr learned my addr 172.21.15.189:0/2829996765 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133194139 0 0) 0x7f9750133260 con 0x7f9750133420 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 10 monclient(hunting): my global_id is 94201 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.351+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97380035d0 con 0x7f9750133420 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469340036 0 0) 0x7f975009f690 con 0x7f9750133800 2024-09-16T06:39:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient(hunting): my global_id is 4565 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9750133260 con 0x7f9750133800 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1148761331 0 0) 0x7f97380035d0 con 0x7f9750133420 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 >> v1:172.21.15.76:6790/0 conn(0x7f9750133800 legacy=0x7f975013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 >> v1:172.21.15.76:6789/0 conn(0x7f9750132e00 legacy=0x7f975009ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500a1690 con 0x7f9750133420 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 5 monclient: authenticate success, global_id 94201 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9744001610 con 0x7f9750133420 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: dump: 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.360 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:35.361 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9744001880 con 0x7f9750133420 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:35.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 1 -- 172.21.15.189:0/2829996765 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9744001c10 con 0x7f9750133420 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 4 set_mon_vals no callback set 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f974e7fc700 10 monclient: dump: 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:35.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 10 monclient: shutdown 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 1 -- 172.21.15.189:0/2829996765 >> v1:172.21.15.189:6789/0 conn(0x7f9750133420 legacy=0x7f975009e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 1 -- 172.21.15.189:0/2829996765 shutdown_connections 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 1 -- 172.21.15.189:0/2829996765 >> 172.21.15.189:0/2829996765 conn(0x7f97500a44a0 msgr2=0x7f97500a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 1 -- 172.21.15.189:0/2829996765 shutdown_connections 2024-09-16T06:39:35.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 1 -- 172.21.15.189:0/2829996765 wait complete. 2024-09-16T06:39:35.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 10 monclient: build_initial_monmap 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.355+0000 7f97561ea700 10 monclient: monmap: 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 Processor -- start 2024-09-16T06:39:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- start start 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: init 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-b con 0x7f9750132e00 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-c con 0x7f9750133420 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient(hunting): picked mon.noname-a con 0x7f9750133800 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975012f750 con 0x7f9750133800 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975011f2b0 con 0x7f9750132e00 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500a35c0 con 0x7f9750133420 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient(hunting): authenticate will time out at 927.873491s 2024-09-16T06:39:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f974f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9750132e00 0x7f975011cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34956/0 (socket says 172.21.15.189:34956) 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f974f7fe700 1 -- 172.21.15.189:0/4021535750 learned_addr learned my addr 172.21.15.189:0/4021535750 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f974ffff700 1 -- 172.21.15.189:0/4021535750 >> v1:172.21.15.76:6789/0 conn(0x7f9750133800 legacy=0x7f97501cfc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285856245 0 0) 0x7f975011f2b0 con 0x7f9750132e00 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): my global_id is 94204 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c0035d0 con 0x7f9750132e00 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046935254 0 0) 0x7f97500a35c0 con 0x7f9750133420 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): my global_id is 4568 2024-09-16T06:39:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975011f2b0 con 0x7f9750133420 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688138164 0 0) 0x7f973c0035d0 con 0x7f9750132e00 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97500a35c0 con 0x7f9750132e00 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9744001690 con 0x7f9750132e00 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): dump: 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342692615 0 0) 0x7f975011f2b0 con 0x7f9750133420 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f973c0035d0 con 0x7f9750133420 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f97400041f0 con 0x7f9750133420 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient(hunting): dump: 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2647027946 0 0) 0x7f97500a35c0 con 0x7f9750132e00 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 >> v1:172.21.15.76:6790/0 conn(0x7f9750133420 legacy=0x7f975011d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 >> v1:172.21.15.76:6789/0 conn(0x7f9750133800 legacy=0x7f97501cfc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 monclient: found mon.b 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750132160 con 0x7f9750132e00 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 5 monclient: authenticate success, global_id 94204 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _renew_subs 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975009b0d0 con 0x7f9750132e00 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _renew_subs 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750120a40 con 0x7f9750132e00 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97440011b0 con 0x7f9750132e00 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9744004db0 con 0x7f9750132e00 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f9737fff700 10 monclient: dump: 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.359+0000 7f974d7fa700 4 set_mon_vals no callback set 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f97561ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f97561ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f97561ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9730005100 con 0x7f9750132e00 2024-09-16T06:39:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9744005f80 con 0x7f9750132e00 2024-09-16T06:39:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3857+0+0 (unknown 2712981626 0 0) 0x7f9744043580 con 0x7f9750132e00 2024-09-16T06:39:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.363+0000 7f974effd700 10 monclient: get_auth_request con 0x7f973c0362d0 auth_method 0 2024-09-16T06:39:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.367+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f97440065b0 con 0x7f9750132e00 2024-09-16T06:39:35.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.367+0000 7f9737fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.367+0000 7f9737fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:35.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.531+0000 7f97561ea700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:35.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.531+0000 7f97561ea700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:35.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.531+0000 7f97561ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:35.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:35.531+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f9730005930 con 0x7f9750132e00 2024-09-16T06:39:36.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.123+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v20) v1 ==== 143+0+0 (unknown 2928671464 0 0) 0x7f9744012ed0 con 0x7f9750132e00 2024-09-16T06:39:36.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.123+0000 7f9737fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:36.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.123+0000 7f9737fff700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-16T06:39:36.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.231+0000 7f97561ea700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:36.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.231+0000 7f97561ea700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:36.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.231+0000 7f97561ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:36.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:36.231+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f97300053f0 con 0x7f9750132e00 2024-09-16T06:39:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.123+0000 7f9737fff700 1 -- 172.21.15.189:0/4021535750 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v21) v1 ==== 143+0+0 (unknown 2428324039 0 0) 0x7f9744014a60 con 0x7f9750132e00 2024-09-16T06:39:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.123+0000 7f9737fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T06:39:37.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.123+0000 7f9737fff700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-16T06:39:37.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 >> v1:172.21.15.189:6812/3098810674 conn(0x7f973c0362d0 legacy=0x7f973c038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 10 monclient: shutdown 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 >> v1:172.21.15.189:6789/0 conn(0x7f9750132e00 legacy=0x7f975011cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 shutdown_connections 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 >> 172.21.15.189:0/4021535750 conn(0x7f97500a44a0 msgr2=0x7f9750131a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 shutdown_connections 2024-09-16T06:39:37.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.127+0000 7f97561ea700 1 -- 172.21.15.189:0/4021535750 wait complete. 2024-09-16T06:39:37.131 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_min to 2 2024-09-16T06:39:37.144 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 2 2024-09-16T06:39:37.144 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.255+0000 7fd451372700 10 monclient: get_monmap_and_config 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.255+0000 7fd451372700 10 monclient: build_initial_monmap 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.255+0000 7fd451372700 10 monclient: monmap: 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:37.259745+0000 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:37.259745+0000 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:37.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:37.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:37.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 Processor -- start 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- start start 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: init 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:37.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient(hunting): picked mon.noname-b con 0x7fd44c09fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient(hunting): picked mon.noname-a con 0x7fd44c0717e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient(hunting): picked mon.noname-c con 0x7fd44c0a07d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c074a10 con 0x7fd44c0717e0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c098450 con 0x7fd44c0a07d0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c0986f0 con 0x7fd44c09fce0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient(hunting): authenticate will time out at 929.772933s 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd4437fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd44c0717e0 legacy=0x7fd44c0a00c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd450b71700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd44c0a07d0 0x7fd44c13d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34968/0 (socket says 172.21.15.189:34968) 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd450b71700 1 -- 172.21.15.189:0/1662139106 learned_addr learned my addr 172.21.15.189:0/1662139106 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550684456 0 0) 0x7fd44c098450 con 0x7fd44c0a07d0 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): my global_id is 94207 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4380035d0 con 0x7fd44c0a07d0 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281084529 0 0) 0x7fd44c0986f0 con 0x7fd44c09fce0 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient(hunting): my global_id is 4571 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c098450 con 0x7fd44c09fce0 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777606504 0 0) 0x7fd4380035d0 con 0x7fd44c0a07d0 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 >> v1:172.21.15.76:6790/0 conn(0x7fd44c09fce0 legacy=0x7fd44c098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 >> v1:172.21.15.76:6789/0 conn(0x7fd44c0717e0 legacy=0x7fd44c0a00c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0746f0 con 0x7fd44c0a07d0 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 5 monclient: authenticate success, global_id 94207 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd444001990 con 0x7fd44c0a07d0 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: dump: 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd444001c70 con 0x7fd44c0a07d0 2024-09-16T06:39:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:37.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 1 -- 172.21.15.189:0/1662139106 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd444002000 con 0x7fd44c0a07d0 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 4 set_mon_vals no callback set 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd44b7fe700 10 monclient: dump: 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.268 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: shutdown 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- 172.21.15.189:0/1662139106 >> v1:172.21.15.189:6789/0 conn(0x7fd44c0a07d0 legacy=0x7fd44c13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- 172.21.15.189:0/1662139106 shutdown_connections 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- 172.21.15.189:0/1662139106 >> 172.21.15.189:0/1662139106 conn(0x7fd44c0a0fb0 msgr2=0x7fd44c0a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- 172.21.15.189:0/1662139106 shutdown_connections 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 1 -- 172.21.15.189:0/1662139106 wait complete. 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: build_initial_monmap 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.259+0000 7fd451372700 10 monclient: monmap: 2024-09-16T06:39:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:37.271 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 Processor -- start 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- start start 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: init 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient(hunting): picked mon.noname-b con 0x7fd44c0717e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient(hunting): picked mon.noname-c con 0x7fd44c09fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:37.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient(hunting): picked mon.noname-a con 0x7fd44c0a07d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c08b310 con 0x7fd44c0a07d0 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd44bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd44c0717e0 0x7fd44c096db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34974/0 (socket says 172.21.15.189:34974) 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd44bfff700 1 -- 172.21.15.189:0/2600353258 learned_addr learned my addr 172.21.15.189:0/2600353258 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c099fa0 con 0x7fd44c0717e0 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd450b71700 1 -- 172.21.15.189:0/2600353258 >> v1:172.21.15.76:6789/0 conn(0x7fd44c0a07d0 legacy=0x7fd44c1d2280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c09b4c0 con 0x7fd44c09fce0 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient(hunting): authenticate will time out at 929.777695s 2024-09-16T06:39:37.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600272184 0 0) 0x7fd44c099fa0 con 0x7fd44c0717e0 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): my global_id is 94210 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42c0035d0 con 0x7fd44c0717e0 2024-09-16T06:39:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802045616 0 0) 0x7fd42c0035d0 con 0x7fd44c0717e0 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd44c099fa0 con 0x7fd44c0717e0 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd43c001970 con 0x7fd44c0717e0 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): dump: 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187742690 0 0) 0x7fd44c09b4c0 con 0x7fd44c09fce0 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient(hunting): my global_id is 4574 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42c0035d0 con 0x7fd44c09fce0 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 972633257 0 0) 0x7fd44c099fa0 con 0x7fd44c0717e0 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 >> v1:172.21.15.76:6790/0 conn(0x7fd44c09fce0 legacy=0x7fd44c0974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 >> v1:172.21.15.76:6789/0 conn(0x7fd44c0a07d0 legacy=0x7fd44c1d2280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 monclient: found mon.b 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c08d670 con 0x7fd44c0717e0 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 5 monclient: authenticate success, global_id 94210 2024-09-16T06:39:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _renew_subs 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c08ba60 con 0x7fd44c0717e0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _renew_subs 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c08c090 con 0x7fd44c0717e0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd43c001d20 con 0x7fd44c0717e0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd43c003b90 con 0x7fd44c0717e0 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd4497fa700 10 monclient: dump: 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.263+0000 7fd44a7fc700 4 set_mon_vals no callback set 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd43c003d70 con 0x7fd44c0717e0 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd451372700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd451372700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd451372700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 3873+0+0 (unknown 1538181727 0 0) 0x7fd43c0434e0 con 0x7fd44c0717e0 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44c08cdd0 con 0x7fd44c0717e0 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.267+0000 7fd4437fe700 10 monclient: get_auth_request con 0x7fd42c0361b0 auth_method 0 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.275+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd43c014720 con 0x7fd44c0717e0 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.275+0000 7fd4497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.275+0000 7fd4497fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:37.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd451372700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd451372700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd451372700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fd44c067930 con 0x7fd44c0717e0 2024-09-16T06:39:37.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v21) v1 ==== 105+0+14 (unknown 593488084 0 3052767636) 0x7fd43c012dc0 con 0x7fd44c0717e0 2024-09-16T06:39:37.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd4497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.447+0000 7fd4497fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:37.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fd44c13ef20 con 0x7fd44c0717e0 2024-09-16T06:39:37.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd4497fa700 1 -- 172.21.15.189:0/2600353258 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v21) v1 ==== 105+0+14 (unknown 593488084 0 3052767636) 0x7fd43c014540 con 0x7fd44c0717e0 2024-09-16T06:39:37.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd4497fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:37.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd4497fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd42c0361b0 legacy=0x7fd42c038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 10 monclient: shutdown 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 >> v1:172.21.15.189:6789/0 conn(0x7fd44c0717e0 legacy=0x7fd44c096db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 shutdown_connections 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 >> 172.21.15.189:0/2600353258 conn(0x7fd44c0a0fb0 msgr2=0x7fd44c098cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 shutdown_connections 2024-09-16T06:39:37.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.547+0000 7fd451372700 1 -- 172.21.15.189:0/2600353258 wait complete. 2024-09-16T06:39:37.560 INFO:tasks.workunit.client.0.smithi189.stdout:pg_num_min: 2 2024-09-16T06:39:37.560 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-16T06:39:37.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.667+0000 7f00c6c36700 10 monclient: get_monmap_and_config 2024-09-16T06:39:37.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.667+0000 7f00c6c36700 10 monclient: build_initial_monmap 2024-09-16T06:39:37.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.667+0000 7f00c6c36700 10 monclient: monmap: 2024-09-16T06:39:37.669 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:37.671253+0000 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:37.671253+0000 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:37.670 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.667+0000 7f00c6c36700 1 Processor -- start 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- start start 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient: init 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-a con 0x7f00c012ef60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-b con 0x7f00c009e590 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-c con 0x7f00c012e990 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c009ce10 con 0x7f00c012ef60 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c009d560 con 0x7f00c012e990 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c009d840 con 0x7f00c009e590 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient(hunting): authenticate will time out at 930.185354s 2024-09-16T06:39:37.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c49d2700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00c012ef60 legacy=0x7f00c012f340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c51d3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00c012e990 0x7f00c013a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34976/0 (socket says 172.21.15.189:34976) 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c51d3700 1 -- 172.21.15.189:0/2315155062 learned_addr learned my addr 172.21.15.189:0/2315155062 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373114439 0 0) 0x7f00c009d560 con 0x7f00c012e990 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): my global_id is 94213 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a80035d0 con 0x7f00c012e990 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568166154 0 0) 0x7f00c009d840 con 0x7f00c009e590 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient(hunting): my global_id is 4577 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c009d560 con 0x7f00c009e590 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277497161 0 0) 0x7f00a80035d0 con 0x7f00c012e990 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 >> v1:172.21.15.76:6790/0 conn(0x7f00c009e590 legacy=0x7f00c01371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 >> v1:172.21.15.76:6789/0 conn(0x7f00c012ef60 legacy=0x7f00c012f340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c009db20 con 0x7f00c012e990 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:37.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 5 monclient: authenticate success, global_id 94213 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00b8001990 con 0x7f00c012e990 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: dump: 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00b8001c70 con 0x7f00c012e990 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 1 -- 172.21.15.189:0/2315155062 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00b8002000 con 0x7f00c012e990 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 4 set_mon_vals no callback set 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00bf7fe700 10 monclient: dump: 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 10 monclient: shutdown 2024-09-16T06:39:37.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:37.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- 172.21.15.189:0/2315155062 >> v1:172.21.15.189:6789/0 conn(0x7f00c012e990 legacy=0x7f00c013a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- 172.21.15.189:0/2315155062 shutdown_connections 2024-09-16T06:39:37.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.671+0000 7f00c6c36700 1 -- 172.21.15.189:0/2315155062 >> 172.21.15.189:0/2315155062 conn(0x7f00c00a1100 msgr2=0x7f00c00a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:37.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- 172.21.15.189:0/2315155062 shutdown_connections 2024-09-16T06:39:37.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- 172.21.15.189:0/2315155062 wait complete. 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient: build_initial_monmap 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient: monmap: 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:37.685 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 Processor -- start 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- start start 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient: init 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-b con 0x7f00c009e590 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:37.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-c con 0x7f00c012e990 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient(hunting): picked mon.noname-a con 0x7f00c012ef60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c006bae0 con 0x7f00c012ef60 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c00a0430 con 0x7f00c009e590 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c49d2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00c009e590 0x7f00c0097300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34992/0 (socket says 172.21.15.189:34992) 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c51d3700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00c012ef60 legacy=0x7f00c01cf860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c49d2700 1 -- 172.21.15.189:0/91829598 learned_addr learned my addr 172.21.15.189:0/91829598 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c0072220 con 0x7f00c012e990 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00c6c36700 10 monclient(hunting): authenticate will time out at 930.191274s 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916465715 0 0) 0x7f00c00a0430 con 0x7f00c009e590 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): my global_id is 94216 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac0035d0 con 0x7f00c009e590 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869266203 0 0) 0x7f00c0072220 con 0x7f00c012e990 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 10 monclient(hunting): my global_id is 4580 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.675+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c00a0430 con 0x7f00c012e990 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976406484 0 0) 0x7f00ac0035d0 con 0x7f00c009e590 2024-09-16T06:39:37.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00c0072220 con 0x7f00c009e590 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00b0004280 con 0x7f00c009e590 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.689 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839119375 0 0) 0x7f00c00a0430 con 0x7f00c012e990 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00ac0035d0 con 0x7f00c012e990 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00b40018f0 con 0x7f00c012e990 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1634343104 0 0) 0x7f00c0072220 con 0x7f00c009e590 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 >> v1:172.21.15.76:6790/0 conn(0x7f00c012e990 legacy=0x7f00c01cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 >> v1:172.21.15.76:6789/0 conn(0x7f00c012ef60 legacy=0x7f00c01cf860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 monclient: found mon.b 2024-09-16T06:39:37.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c009b610 con 0x7f00c009e590 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 5 monclient: authenticate success, global_id 94216 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _renew_subs 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c00a0840 con 0x7f00c009e590 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _renew_subs 2024-09-16T06:39:37.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c0070b30 con 0x7f00c009e590 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00b0004560 con 0x7f00c009e590 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00b0004870 con 0x7f00c009e590 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00beffd700 4 set_mon_vals no callback set 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 10 monclient: dump: 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:37.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f00b0004a50 con 0x7f00c009e590 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.694 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a0005100 con 0x7f00c009e590 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 3873+0+0 (unknown 1538181727 0 0) 0x7f00b0043440 con 0x7f00c009e590 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.679+0000 7f00bffff700 10 monclient: get_auth_request con 0x7f00ac0362f0 auth_method 0 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.687+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f00b0005270 con 0x7f00c009e590 2024-09-16T06:39:37.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.687+0000 7f00bd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:37.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.687+0000 7f00bd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:37.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.855+0000 7f00c6c36700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:37.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.855+0000 7f00c6c36700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:37.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.855+0000 7f00c6c36700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:37.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:37.855+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f00a0005930 con 0x7f00c009e590 2024-09-16T06:39:38.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.151+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (unknown 1578579183 0 0) 0x7f00b001cea0 con 0x7f00c009e590 2024-09-16T06:39:38.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.151+0000 7f00bd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:38.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.151+0000 7f00bd7fa700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-16T06:39:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.259+0000 7f00c6c36700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.259+0000 7f00c6c36700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:38.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.259+0000 7f00c6c36700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:38.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:38.259+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f00a00053f0 con 0x7f00c009e590 2024-09-16T06:39:39.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.151+0000 7f00bd7fa700 1 -- 172.21.15.189:0/91829598 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v23) v1 ==== 145+0+0 (unknown 2513166620 0 0) 0x7f00b0012d20 con 0x7f00c009e590 2024-09-16T06:39:39.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.151+0000 7f00bd7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T06:39:39.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.151+0000 7f00bd7fa700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-16T06:39:39.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00ac0362f0 legacy=0x7f00ac0387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 10 monclient: shutdown 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 >> v1:172.21.15.189:6789/0 conn(0x7f00c009e590 legacy=0x7f00c0097300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 shutdown_connections 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 >> 172.21.15.189:0/91829598 conn(0x7f00c00a1100 msgr2=0x7f00c012bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 shutdown_connections 2024-09-16T06:39:39.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.159+0000 7f00c6c36700 1 -- 172.21.15.189:0/91829598 wait complete. 2024-09-16T06:39:39.161 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_max to 33 2024-09-16T06:39:39.173 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T06:39:39.173 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 33 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 10 monclient: get_monmap_and_config 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 10 monclient: build_initial_monmap 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 10 monclient: monmap: 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:39.285937+0000 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:39.285937+0000 2024-09-16T06:39:39.284 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 1 Processor -- start 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 1 -- start start 2024-09-16T06:39:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 10 monclient: init 2024-09-16T06:39:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.283+0000 7fa106731700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:39.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient(hunting): picked mon.noname-b con 0x7fa100066480 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient(hunting): picked mon.noname-c con 0x7fa10009b8e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient(hunting): picked mon.noname-a con 0x7fa10009bcf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa100066860 con 0x7fa10009bcf0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10009f7e0 con 0x7fa10009b8e0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000a10e0 con 0x7fa100066480 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient(hunting): authenticate will time out at 931.800271s 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0f77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa10009b8e0 0x7fa1001374d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47718/0 (socket says 172.21.15.189:47718) 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa104cce700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa10009bcf0 legacy=0x7fa10013ac00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0f77fe700 1 -- 172.21.15.189:0/2319115833 learned_addr learned my addr 172.21.15.189:0/2319115833 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086390186 0 0) 0x7fa10009f7e0 con 0x7fa10009b8e0 2024-09-16T06:39:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): my global_id is 94219 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec0035d0 con 0x7fa10009b8e0 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936790749 0 0) 0x7fa1000a10e0 con 0x7fa100066480 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient(hunting): my global_id is 4583 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10009f7e0 con 0x7fa100066480 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230266294 0 0) 0x7fa0ec0035d0 con 0x7fa10009b8e0 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 >> v1:172.21.15.76:6790/0 conn(0x7fa100066480 legacy=0x7fa10009b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 >> v1:172.21.15.76:6789/0 conn(0x7fa10009bcf0 legacy=0x7fa10013ac00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100071970 con 0x7fa10009b8e0 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 5 monclient: authenticate success, global_id 94219 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa0e8001610 con 0x7fa10009b8e0 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: dump: 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8001880 con 0x7fa10009b8e0 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:39.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 1 -- 172.21.15.189:0/2319115833 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa0e8001c10 con 0x7fa10009b8e0 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 4 set_mon_vals no callback set 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa0ff7fe700 10 monclient: dump: 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.294 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient: shutdown 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- 172.21.15.189:0/2319115833 >> v1:172.21.15.189:6789/0 conn(0x7fa10009b8e0 legacy=0x7fa1001374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- 172.21.15.189:0/2319115833 shutdown_connections 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- 172.21.15.189:0/2319115833 >> 172.21.15.189:0/2319115833 conn(0x7fa1000a4410 msgr2=0x7fa1000a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- 172.21.15.189:0/2319115833 shutdown_connections 2024-09-16T06:39:39.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 1 -- 172.21.15.189:0/2319115833 wait complete. 2024-09-16T06:39:39.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient: build_initial_monmap 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.287+0000 7fa106731700 10 monclient: monmap: 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 Processor -- start 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- start start 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: init 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient(hunting): picked mon.noname-a con 0x7fa100066480 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient(hunting): picked mon.noname-c con 0x7fa10009b8e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient(hunting): picked mon.noname-b con 0x7fa10009bcf0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1001a8fd0 con 0x7fa100066480 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10006b210 con 0x7fa10009bcf0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000a3790 con 0x7fa10009b8e0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient(hunting): authenticate will time out at 931.805688s 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa100066480 legacy=0x7fa1001c46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa104cce700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa10009bcf0 0x7fa1001d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47734/0 (socket says 172.21.15.189:47734) 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa104cce700 1 -- 172.21.15.189:0/4154848083 learned_addr learned my addr 172.21.15.189:0/4154848083 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626598225 0 0) 0x7fa10006b210 con 0x7fa10009bcf0 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): my global_id is 94222 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e00035d0 con 0x7fa10009bcf0 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926592640 0 0) 0x7fa1000a3790 con 0x7fa10009b8e0 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): my global_id is 4586 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10006b210 con 0x7fa10009b8e0 2024-09-16T06:39:39.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324970715 0 0) 0x7fa0e00035d0 con 0x7fa10009bcf0 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1000a3790 con 0x7fa10009bcf0 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa0f80044f0 con 0x7fa10009bcf0 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.303 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3639334109 0 0) 0x7fa1000a3790 con 0x7fa10009bcf0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 >> v1:172.21.15.76:6790/0 conn(0x7fa10009b8e0 legacy=0x7fa1001cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 >> v1:172.21.15.76:6789/0 conn(0x7fa100066480 legacy=0x7fa1001c46a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 monclient: found mon.b 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000a2fa0 con 0x7fa10009bcf0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:39.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 5 monclient: authenticate success, global_id 94222 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _renew_subs 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1001a8730 con 0x7fa10009bcf0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _renew_subs 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001c7b60 con 0x7fa10009bcf0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f80047d0 con 0x7fa10009bcf0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa0f8004b60 con 0x7fa10009bcf0 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0feffd700 4 set_mon_vals no callback set 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.291+0000 7fa0fd7fa700 10 monclient: dump: 2024-09-16T06:39:39.305 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa106731700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa106731700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa106731700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4005100 con 0x7fa10009bcf0 2024-09-16T06:39:39.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa0f8004cc0 con 0x7fa10009bcf0 2024-09-16T06:39:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7fa0f80436a0 con 0x7fa10009bcf0 2024-09-16T06:39:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.295+0000 7fa0fffff700 10 monclient: get_auth_request con 0x7fa0e0035fa0 auth_method 0 2024-09-16T06:39:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.299+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa0f8005510 con 0x7fa10009bcf0 2024-09-16T06:39:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.299+0000 7fa0fd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.299+0000 7fa0fd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:39.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa106731700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa106731700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa106731700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7fa0e4005930 con 0x7fa10009bcf0 2024-09-16T06:39:39.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v23) v1 ==== 105+0+15 (unknown 4165109328 0 2995009362) 0x7fa0f8012f70 con 0x7fa10009bcf0 2024-09-16T06:39:39.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa0fd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.471+0000 7fa0fd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:39.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.567+0000 7fa106731700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.567+0000 7fa106731700 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.567+0000 7fa106731700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.567+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7fa0e4005bc0 con 0x7fa10009bcf0 2024-09-16T06:39:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa0fd7fa700 1 -- 172.21.15.189:0/4154848083 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v23) v1 ==== 105+0+15 (unknown 4165109328 0 2995009362) 0x7fa0f80146f0 con 0x7fa10009bcf0 2024-09-16T06:39:39.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa0fd7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:39.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa0fd7fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa0e0035fa0 legacy=0x7fa0e0038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 10 monclient: shutdown 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 >> v1:172.21.15.189:6789/0 conn(0x7fa10009bcf0 legacy=0x7fa1001d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 shutdown_connections 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 >> 172.21.15.189:0/4154848083 conn(0x7fa1000a4410 msgr2=0x7fa10006f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 shutdown_connections 2024-09-16T06:39:39.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.571+0000 7fa106731700 1 -- 172.21.15.189:0/4154848083 wait complete. 2024-09-16T06:39:39.585 INFO:tasks.workunit.client.0.smithi189.stdout:pg_num_max: 33 2024-09-16T06:39:39.585 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T06:39:39.585 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:39.586 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: get_monmap_and_config 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: build_initial_monmap 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: monmap: 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:39.697702+0000 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:39.697702+0000 2024-09-16T06:39:39.696 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:39.697 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.697 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:39.697 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:39.697 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:39.697 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 1 Processor -- start 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 1 -- start start 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: init 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-c con 0x7f59d409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-b con 0x7f59d409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:39.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-a con 0x7f59d4138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d409cca0 con 0x7f59d4138da0 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d409f750 con 0x7f59d409bb00 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d4097620 con 0x7f59d409c5f0 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient(hunting): authenticate will time out at 932.210699s 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59da1b3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f59d409bb00 0x7f59d409bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47744/0 (socket says 172.21.15.189:47744) 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59da1b3700 1 -- 172.21.15.189:0/3006327236 learned_addr learned my addr 172.21.15.189:0/3006327236 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59da9b4700 1 -- 172.21.15.189:0/3006327236 >> v1:172.21.15.76:6789/0 conn(0x7f59d4138da0 legacy=0x7f59d413b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925870800 0 0) 0x7f59d409f750 con 0x7f59d409bb00 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): my global_id is 94225 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c00035d0 con 0x7f59d409bb00 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507982474 0 0) 0x7f59d4097620 con 0x7f59d409c5f0 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient(hunting): my global_id is 4589 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d409f750 con 0x7f59d409c5f0 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888234085 0 0) 0x7f59c00035d0 con 0x7f59d409bb00 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 >> v1:172.21.15.76:6790/0 conn(0x7f59d409c5f0 legacy=0x7f59d4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 >> v1:172.21.15.76:6789/0 conn(0x7f59d4138da0 legacy=0x7f59d413b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: _finish_hunting 0 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 monclient: found mon.noname-c 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4097900 con 0x7f59d409bb00 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: _finish_auth 0 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: _check_auth_tickets 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 5 monclient: authenticate success, global_id 94225 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59c4001610 con 0x7f59d409bb00 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: dump: 2024-09-16T06:39:39.703 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c40018f0 con 0x7f59d409bb00 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:39.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 1 -- 172.21.15.189:0/3006327236 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59c4001c80 con 0x7f59d409bb00 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59dc417700 4 set_mon_vals no callback set 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.695+0000 7f59d91b1700 10 monclient: dump: 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.705 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: shutdown 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- 172.21.15.189:0/3006327236 >> v1:172.21.15.189:6789/0 conn(0x7f59d409bb00 legacy=0x7f59d409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- 172.21.15.189:0/3006327236 shutdown_connections 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- 172.21.15.189:0/3006327236 >> 172.21.15.189:0/3006327236 conn(0x7f59d40a03e0 msgr2=0x7f59d40a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- 172.21.15.189:0/3006327236 shutdown_connections 2024-09-16T06:39:39.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- 172.21.15.189:0/3006327236 wait complete. 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: build_initial_monmap 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: monmap: 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:39.710 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 Processor -- start 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- start start 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: init 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-c con 0x7f59d409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-a con 0x7f59d409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient(hunting): picked mon.noname-b con 0x7f59d4138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:39.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d4090b60 con 0x7f59d409c5f0 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d409a6e0 con 0x7f59d4138da0 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d4071560 con 0x7f59d409bb00 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59dc417700 10 monclient(hunting): authenticate will time out at 932.215913s 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.699+0000 7f59d99b2700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f59d409c5f0 legacy=0x7f59d4090450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59da9b4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f59d4138da0 0x7f59d41d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47754/0 (socket says 172.21.15.189:47754) 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59da9b4700 1 -- 172.21.15.189:0/620996279 learned_addr learned my addr 172.21.15.189:0/620996279 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774877124 0 0) 0x7f59d409a6e0 con 0x7f59d4138da0 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): my global_id is 94228 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b80035d0 con 0x7f59d4138da0 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425328168 0 0) 0x7f59d4071560 con 0x7f59d409bb00 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): my global_id is 4592 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d409a6e0 con 0x7f59d409bb00 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112862391 0 0) 0x7f59b80035d0 con 0x7f59d4138da0 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d4071560 con 0x7f59d4138da0 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59cc0044c0 con 0x7f59d4138da0 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): dump: 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:39.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 455371191 0 0) 0x7f59d4071560 con 0x7f59d4138da0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 >> v1:172.21.15.76:6790/0 conn(0x7f59d409bb00 legacy=0x7f59d408fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 >> v1:172.21.15.76:6789/0 conn(0x7f59d409c5f0 legacy=0x7f59d4090450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 monclient: found mon.b 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40a2490 con 0x7f59d4138da0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 5 monclient: authenticate success, global_id 94228 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _renew_subs 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d409b5c0 con 0x7f59d4138da0 2024-09-16T06:39:39.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _renew_subs 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d4093580 con 0x7f59d4138da0 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59cc004730 con 0x7f59d4138da0 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59cc004ac0 con 0x7f59d4138da0 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 10 monclient: dump: 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:39.716 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59d89b0700 4 set_mon_vals no callback set 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59dc417700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc005100 con 0x7f59d4138da0 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f59cc004c20 con 0x7f59d4138da0 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7f59cc043660 con 0x7f59d4138da0 2024-09-16T06:39:39.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.703+0000 7f59da1b3700 10 monclient: get_auth_request con 0x7f59b8035fa0 auth_method 0 2024-09-16T06:39:39.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.711+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f59cc005470 con 0x7f59d4138da0 2024-09-16T06:39:39.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.711+0000 7f59caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:39.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.711+0000 7f59caffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:39.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59dc417700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T06:39:39.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59dc417700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T06:39:39.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59dc417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:39.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59dc417700 1 -- 172.21.15.189:0/620996279 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7f59bc005930 con 0x7f59d4138da0 2024-09-16T06:39:39.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59caffd700 1 -- 172.21.15.189:0/620996279 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v23) v1 ==== 150+0+0 (unknown 3331375382 0 0) 0x7f59cc012ed0 con 0x7f59d4138da0 2024-09-16T06:39:39.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T06:39:39.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.879+0000 7f59caffd700 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-16T06:39:39.883 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-16T06:39:39.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:39.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T06:39:39.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:39.888 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: get_monmap_and_config 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: build_initial_monmap 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: monmap: 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:39.997881+0000 2024-09-16T06:39:39.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:39.997881+0000 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 1 Processor -- start 2024-09-16T06:39:39.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 1 -- start start 2024-09-16T06:39:39.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: init 2024-09-16T06:39:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.995+0000 7f6c1e9cf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:40.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-a con 0x7f6c180a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-b con 0x7f6c1809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-c con 0x7f6c180a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1809a040 con 0x7f6c180a2350 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180a33d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c18099d60 con 0x7f6c1809aa20 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient(hunting): authenticate will time out at 932.512242s 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c17fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6c180a2350 legacy=0x7f6c180a2730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1cf6c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6c180a3ca0 0x7f6c1813dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47764/0 (socket says 172.21.15.189:47764) 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1cf6c700 1 -- 172.21.15.189:0/1105102230 learned_addr learned my addr 172.21.15.189:0/1105102230 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129017430 0 0) 0x7f6c180a33d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): my global_id is 94231 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c040035d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742583088 0 0) 0x7f6c18099d60 con 0x7f6c1809aa20 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient(hunting): my global_id is 4595 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c180a33d0 con 0x7f6c1809aa20 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1662125269 0 0) 0x7f6c040035d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 >> v1:172.21.15.76:6790/0 conn(0x7f6c1809aa20 legacy=0x7f6c1813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 >> v1:172.21.15.76:6789/0 conn(0x7f6c180a2350 legacy=0x7f6c180a2730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180a3920 con 0x7f6c180a3ca0 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 5 monclient: authenticate success, global_id 94231 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c10001990 con 0x7f6c180a3ca0 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: dump: 2024-09-16T06:39:40.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c10001c70 con 0x7f6c180a3ca0 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:40.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 4 set_mon_vals no callback set 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 1 -- 172.21.15.189:0/1105102230 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c10002000 con 0x7f6c180a3ca0 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c177fe700 10 monclient: dump: 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.006 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient: shutdown 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/1105102230 >> v1:172.21.15.189:6789/0 conn(0x7f6c180a3ca0 legacy=0x7f6c1813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/1105102230 shutdown_connections 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/1105102230 >> 172.21.15.189:0/1105102230 conn(0x7f6c180a7d70 msgr2=0x7f6c180a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/1105102230 shutdown_connections 2024-09-16T06:39:40.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/1105102230 wait complete. 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:39.999+0000 7f6c1e9cf700 10 monclient: monmap: 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 Processor -- start 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- start start 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: init 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-c con 0x7f6c1809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-a con 0x7f6c180a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient(hunting): picked mon.noname-b con 0x7f6c180a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c18120400 con 0x7f6c180a2350 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1806bed0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1809b060 con 0x7f6c1809aa20 2024-09-16T06:39:40.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1cf6c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6c180a3ca0 0x7f6c181d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47776/0 (socket says 172.21.15.189:47776) 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1cf6c700 1 -- 172.21.15.189:0/3159574327 learned_addr learned my addr 172.21.15.189:0/3159574327 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c0ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6c180a2350 legacy=0x7f6c181ceb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient(hunting): authenticate will time out at 932.517496s 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452107595 0 0) 0x7f6c1806bed0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): my global_id is 94234 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf80035d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615428752 0 0) 0x7f6c1809b060 con 0x7f6c1809aa20 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): my global_id is 4598 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1806bed0 con 0x7f6c1809aa20 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990241313 0 0) 0x7f6bf80035d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c1809b060 con 0x7f6c180a3ca0 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c10001950 con 0x7f6c180a3ca0 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): dump: 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.014 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 758833306 0 0) 0x7f6c1806bed0 con 0x7f6c1809aa20 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bf80035d0 con 0x7f6c1809aa20 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c080041f0 con 0x7f6c1809aa20 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient(hunting): dump: 2024-09-16T06:39:40.015 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1821564627 0 0) 0x7f6c1809b060 con 0x7f6c180a3ca0 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 >> v1:172.21.15.76:6790/0 conn(0x7f6c1809aa20 legacy=0x7f6c18134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 >> v1:172.21.15.76:6789/0 conn(0x7f6c180a2350 legacy=0x7f6c181ceb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:40.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 monclient: found mon.b 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1811fa00 con 0x7f6c180a3ca0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 5 monclient: authenticate success, global_id 94234 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _renew_subs 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c18121dd0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _renew_subs 2024-09-16T06:39:40.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1809b590 con 0x7f6c180a3ca0 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c100013e0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c100051c0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c16ffd700 4 set_mon_vals no callback set 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.003+0000 7f6c157fa700 10 monclient: dump: 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.018 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6c100053a0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c1e9cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c1e9cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c1e9cf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bfc005100 con 0x7f6c180a3ca0 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7f6c10043780 con 0x7f6c180a3ca0 2024-09-16T06:39:40.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.007+0000 7f6c17fff700 10 monclient: get_auth_request con 0x7f6bf80362a0 auth_method 0 2024-09-16T06:39:40.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.015+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6c10014c30 con 0x7f6c180a3ca0 2024-09-16T06:39:40.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.015+0000 7f6c157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.015+0000 7f6c157fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:40.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c1e9cf700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T06:39:40.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c1e9cf700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T06:39:40.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c1e9cf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c1e9cf700 1 -- 172.21.15.189:0/3159574327 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7f6bfc005930 con 0x7f6c180a3ca0 2024-09-16T06:39:40.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c157fa700 1 -- 172.21.15.189:0/3159574327 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v23) v1 ==== 150+0+0 (unknown 3319333238 0 0) 0x7f6c100132d0 con 0x7f6c180a3ca0 2024-09-16T06:39:40.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T06:39:40.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.183+0000 7f6c157fa700 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-16T06:39:40.188 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-16T06:39:40.191 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-16T06:39:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:40.192 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-16T06:39:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: get_monmap_and_config 2024-09-16T06:39:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: monmap: 2024-09-16T06:39:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:40.301819+0000 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:40.301819+0000 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:40.301 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 1 Processor -- start 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 1 -- start start 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: init 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-b con 0x7f7060098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-a con 0x7f7060099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-c con 0x7f7060138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706009c460 con 0x7f7060099a60 2024-09-16T06:39:40.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706009c1e0 con 0x7f7060138a80 2024-09-16T06:39:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706009bd60 con 0x7f7060098070 2024-09-16T06:39:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient(hunting): authenticate will time out at 932.814690s 2024-09-16T06:39:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f70651e8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7060138a80 0x7f706013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47792/0 (socket says 172.21.15.189:47792) 2024-09-16T06:39:40.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7060099a60 legacy=0x7f7060099e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f70651e8700 1 -- 172.21.15.189:0/899285706 learned_addr learned my addr 172.21.15.189:0/899285706 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869665985 0 0) 0x7f706009c1e0 con 0x7f7060138a80 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient(hunting): my global_id is 94237 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f704c0035d0 con 0x7f7060138a80 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291805962 0 0) 0x7f704c0035d0 con 0x7f7060138a80 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 >> v1:172.21.15.76:6790/0 conn(0x7f7060098070 legacy=0x7f706009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 >> v1:172.21.15.76:6789/0 conn(0x7f7060099a60 legacy=0x7f7060099e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706009d490 con 0x7f7060138a80 2024-09-16T06:39:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 5 monclient: authenticate success, global_id 94237 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7058001610 con 0x7f7060138a80 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: dump: 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.310 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70580018c0 con 0x7f7060138a80 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f7066c4b700 4 set_mon_vals no callback set 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 1 -- 172.21.15.189:0/899285706 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7058001c50 con 0x7f7060138a80 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.299+0000 7f705f7fe700 10 monclient: dump: 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.312 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: shutdown 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- 172.21.15.189:0/899285706 >> v1:172.21.15.189:6789/0 conn(0x7f7060138a80 legacy=0x7f706013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- 172.21.15.189:0/899285706 shutdown_connections 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- 172.21.15.189:0/899285706 >> 172.21.15.189:0/899285706 conn(0x7f70600a03e0 msgr2=0x7f70600a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- 172.21.15.189:0/899285706 shutdown_connections 2024-09-16T06:39:40.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- 172.21.15.189:0/899285706 wait complete. 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: monmap: 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 Processor -- start 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- start start 2024-09-16T06:39:40.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: init 2024-09-16T06:39:40.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-a con 0x7f7060098070 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-b con 0x7f7060099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient(hunting): picked mon.noname-c con 0x7f7060138a80 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7060094c10 con 0x7f7060098070 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7060093760 con 0x7f7060099a60 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7060098ea0 con 0x7f7060138a80 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f7066c4b700 10 monclient(hunting): authenticate will time out at 932.818902s 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f70649e7700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7060098070 legacy=0x7f7060137ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7060099a60 0x7f70601cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47806/0 (socket says 172.21.15.189:47806) 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705ffff700 1 -- 172.21.15.189:0/249767989 learned_addr learned my addr 172.21.15.189:0/249767989 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696375050 0 0) 0x7f7060093760 con 0x7f7060099a60 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): my global_id is 94240 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70400035d0 con 0x7f7060099a60 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128422650 0 0) 0x7f70400035d0 con 0x7f7060099a60 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7060093760 con 0x7f7060099a60 2024-09-16T06:39:40.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f70480045b0 con 0x7f7060099a60 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): dump: 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.328 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529732373 0 0) 0x7f7060098ea0 con 0x7f7060138a80 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient(hunting): my global_id is 4604 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70400035d0 con 0x7f7060138a80 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2577764533 0 0) 0x7f7060093760 con 0x7f7060099a60 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 >> v1:172.21.15.76:6790/0 conn(0x7f7060138a80 legacy=0x7f70601d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 >> v1:172.21.15.76:6789/0 conn(0x7f7060098070 legacy=0x7f7060137ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.303+0000 7f705d7fa700 1 monclient: found mon.b 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060094930 con 0x7f7060099a60 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 5 monclient: authenticate success, global_id 94240 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _renew_subs 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70600934e0 con 0x7f7060099a60 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _renew_subs 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7060091f50 con 0x7f7060099a60 2024-09-16T06:39:40.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70480047d0 con 0x7f7060099a60 2024-09-16T06:39:40.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7048004b60 con 0x7f7060099a60 2024-09-16T06:39:40.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705effd700 4 set_mon_vals no callback set 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 10 monclient: dump: 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f7066c4b700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7044005100 con 0x7f7060099a60 2024-09-16T06:39:40.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7048004d40 con 0x7f7060099a60 2024-09-16T06:39:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7f7048043540 con 0x7f7060099a60 2024-09-16T06:39:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.307+0000 7f70649e7700 10 monclient: get_auth_request con 0x7f7040036220 auth_method 0 2024-09-16T06:39:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.311+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f70480147d0 con 0x7f7060099a60 2024-09-16T06:39:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.311+0000 7f705d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.311+0000 7f705d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:40.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.479+0000 7f7066c4b700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T06:39:40.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.479+0000 7f7066c4b700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T06:39:40.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.479+0000 7f7066c4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.479+0000 7f7066c4b700 1 -- 172.21.15.189:0/249767989 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7f70440053b0 con 0x7f7060099a60 2024-09-16T06:39:40.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.483+0000 7f705d7fa700 1 -- 172.21.15.189:0/249767989 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v23) v1 ==== 146+0+0 (unknown 3608042942 0 0) 0x7f7048012e70 con 0x7f7060099a60 2024-09-16T06:39:40.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.483+0000 7f705d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T06:39:40.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.483+0000 7f705d7fa700 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-16T06:39:40.485 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-16T06:39:40.487 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:40.487 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-16T06:39:40.487 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:40.487 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-16T06:39:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:39:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 10 monclient: monmap: 2024-09-16T06:39:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.597 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:40.598742+0000 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:40.598742+0000 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:40.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 1 Processor -- start 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 1 -- start start 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.595+0000 7ff17bfff700 10 monclient: init 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff17c1321f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-a con 0x7ff17c06bac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff17c13bc00 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c06a340 con 0x7ff17c06bac0 2024-09-16T06:39:40.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c06bf00 con 0x7ff17c13bc00 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c0a3930 con 0x7ff17c1321f0 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): authenticate will time out at 933.112583s 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff173fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff17c06bac0 legacy=0x7ff17c13a4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff17c13bc00 0x7ff17c13dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47812/0 (socket says 172.21.15.189:47812) 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17b7fe700 1 -- 172.21.15.189:0/336463096 learned_addr learned my addr 172.21.15.189:0/336463096 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274181051 0 0) 0x7ff17c06bf00 con 0x7ff17c13bc00 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient(hunting): my global_id is 94243 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff15c0035d0 con 0x7ff17c13bc00 2024-09-16T06:39:40.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3804258909 0 0) 0x7ff15c0035d0 con 0x7ff17c13bc00 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 >> v1:172.21.15.76:6790/0 conn(0x7ff17c1321f0 legacy=0x7ff17c1325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 >> v1:172.21.15.76:6789/0 conn(0x7ff17c06bac0 legacy=0x7ff17c13a4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 monclient: found mon.noname-c 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c06a820 con 0x7ff17c13bc00 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 5 monclient: authenticate success, global_id 94243 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff174001990 con 0x7ff17c13bc00 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: dump: 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:40.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff174001cb0 con 0x7ff17c13bc00 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 1 -- 172.21.15.189:0/336463096 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff174002040 con 0x7ff17c13bc00 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 4 set_mon_vals no callback set 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17a7fc700 10 monclient: dump: 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.610 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: shutdown 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- 172.21.15.189:0/336463096 >> v1:172.21.15.189:6789/0 conn(0x7ff17c13bc00 legacy=0x7ff17c13dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- 172.21.15.189:0/336463096 shutdown_connections 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- 172.21.15.189:0/336463096 >> 172.21.15.189:0/336463096 conn(0x7ff17c0a4280 msgr2=0x7ff17c0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- 172.21.15.189:0/336463096 shutdown_connections 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- 172.21.15.189:0/336463096 wait complete. 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: monmap: 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 Processor -- start 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- start start 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: init 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-a con 0x7ff17c06bac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff17c1321f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff17c13bc00 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c138340 con 0x7ff17c06bac0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c1358a0 con 0x7ff17c1321f0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17c13b8e0 con 0x7ff17c13bc00 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17bfff700 10 monclient(hunting): authenticate will time out at 933.115593s 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff17affd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff17c06bac0 legacy=0x7ff17c1362b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff173fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff17c1321f0 0x7ff17c1369c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47820/0 (socket says 172.21.15.189:47820) 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.599+0000 7ff173fff700 1 -- 172.21.15.189:0/3325609828 learned_addr learned my addr 172.21.15.189:0/3325609828 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164559746 0 0) 0x7ff17c1358a0 con 0x7ff17c1321f0 2024-09-16T06:39:40.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): my global_id is 94246 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1600035d0 con 0x7ff17c1321f0 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958844538 0 0) 0x7ff17c13b8e0 con 0x7ff17c13bc00 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): my global_id is 4610 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff17c1358a0 con 0x7ff17c13bc00 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3223003742 0 0) 0x7ff1600035d0 con 0x7ff17c1321f0 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff17c13b8e0 con 0x7ff17c1321f0 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff164004230 con 0x7ff17c1321f0 2024-09-16T06:39:40.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): dump: 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:40.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961297606 0 0) 0x7ff17c1358a0 con 0x7ff17c13bc00 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1600035d0 con 0x7ff17c13bc00 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1740017b0 con 0x7ff17c13bc00 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient(hunting): dump: 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.616 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2733720675 0 0) 0x7ff17c13b8e0 con 0x7ff17c1321f0 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 >> v1:172.21.15.76:6790/0 conn(0x7ff17c13bc00 legacy=0x7ff17c1d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 >> v1:172.21.15.76:6789/0 conn(0x7ff17c06bac0 legacy=0x7ff17c1362b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 monclient: found mon.b 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c09e790 con 0x7ff17c1321f0 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 5 monclient: authenticate success, global_id 94246 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _renew_subs 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c138c40 con 0x7ff17c1321f0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _renew_subs 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17c13b390 con 0x7ff17c1321f0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff164004510 con 0x7ff17c1321f0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff179ffb700 4 set_mon_vals no callback set 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff164004820 con 0x7ff17c1321f0 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 10 monclient: dump: 2024-09-16T06:39:40.618 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17bfff700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154005100 con 0x7ff17c1321f0 2024-09-16T06:39:40.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff164004a00 con 0x7ff17c1321f0 2024-09-16T06:39:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7ff164043360 con 0x7ff17c1321f0 2024-09-16T06:39:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.603+0000 7ff17affd700 10 monclient: get_auth_request con 0x7ff1600362a0 auth_method 0 2024-09-16T06:39:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.611+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff164005220 con 0x7ff17c1321f0 2024-09-16T06:39:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.611+0000 7ff1737fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.611+0000 7ff1737fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:40.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.779+0000 7ff17bfff700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T06:39:40.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.779+0000 7ff17bfff700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T06:39:40.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.779+0000 7ff17bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.779+0000 7ff17bfff700 1 -- 172.21.15.189:0/3325609828 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7ff154005930 con 0x7ff17c1321f0 2024-09-16T06:39:40.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.783+0000 7ff1737fe700 1 -- 172.21.15.189:0/3325609828 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v23) v1 ==== 149+0+0 (unknown 978790003 0 0) 0x7ff16401ce10 con 0x7ff17c1321f0 2024-09-16T06:39:40.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.783+0000 7ff1737fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T06:39:40.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.783+0000 7ff1737fe700 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-16T06:39:40.785 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-16T06:39:40.789 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:40.789 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-16T06:39:40.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: get_monmap_and_config 2024-09-16T06:39:40.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: monmap: 2024-09-16T06:39:40.900 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:40.902017+0000 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:40.902017+0000 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:40.901 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 1 Processor -- start 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 1 -- start start 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: init 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-a con 0x7f8a5809bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-b con 0x7f8a5809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-c con 0x7f8a58138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5809cca0 con 0x7f8a5809bb00 2024-09-16T06:39:40.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5809f750 con 0x7f8a58138da0 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a58097620 con 0x7f8a5809c5f0 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 10 monclient(hunting): authenticate will time out at 933.414992s 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a56ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8a5809bb00 legacy=0x7f8a5809bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a577fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8a58138da0 0x7f8a5813b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47832/0 (socket says 172.21.15.189:47832) 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a577fe700 1 -- 172.21.15.189:0/3604338556 learned_addr learned my addr 172.21.15.189:0/3604338556 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100539766 0 0) 0x7f8a5809f750 con 0x7f8a58138da0 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient(hunting): my global_id is 94249 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a380035d0 con 0x7f8a58138da0 2024-09-16T06:39:40.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757437698 0 0) 0x7f8a380035d0 con 0x7f8a58138da0 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 >> v1:172.21.15.76:6790/0 conn(0x7f8a5809c5f0 legacy=0x7f8a58137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 >> v1:172.21.15.76:6789/0 conn(0x7f8a5809bb00 legacy=0x7f8a5809bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 monclient: found mon.noname-c 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a58097900 con 0x7f8a58138da0 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a5d9d2700 5 monclient: authenticate success, global_id 94249 2024-09-16T06:39:40.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8a50001990 con 0x7f8a58138da0 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.899+0000 7f8a567fc700 10 monclient: dump: 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.911 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a50001cb0 con 0x7f8a58138da0 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 1 -- 172.21.15.189:0/3604338556 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8a50002040 con 0x7f8a58138da0 2024-09-16T06:39:40.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a567fc700 10 monclient: dump: 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 4 set_mon_vals no callback set 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: shutdown 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/3604338556 >> v1:172.21.15.189:6789/0 conn(0x7f8a58138da0 legacy=0x7f8a5813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/3604338556 shutdown_connections 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/3604338556 >> 172.21.15.189:0/3604338556 conn(0x7f8a580a03e0 msgr2=0x7f8a580a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/3604338556 shutdown_connections 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/3604338556 wait complete. 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: build_initial_monmap 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: monmap: 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:40.914 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 Processor -- start 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 1 -- start start 2024-09-16T06:39:40.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: init 2024-09-16T06:39:40.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-b con 0x7f8a5809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.903+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-c con 0x7f8a5809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient(hunting): picked mon.noname-a con 0x7f8a58138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a58090b10 con 0x7f8a58138da0 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a56ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8a5809bb00 0x7f8a5808cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47834/0 (socket says 172.21.15.189:47834) 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5808d9b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a56ffd700 1 -- 172.21.15.189:0/1850096323 learned_addr learned my addr 172.21.15.189:0/1850096323 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580994f0 con 0x7f8a5809c5f0 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient(hunting): authenticate will time out at 933.420217s 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a577fe700 1 -- 172.21.15.189:0/1850096323 >> v1:172.21.15.76:6789/0 conn(0x7f8a58138da0 legacy=0x7f8a581cdf60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848004811 0 0) 0x7f8a5808d9b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): my global_id is 94252 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c0035d0 con 0x7f8a5809bb00 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691228278 0 0) 0x7f8a3c0035d0 con 0x7f8a5809bb00 2024-09-16T06:39:40.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a5808d9b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8a48004230 con 0x7f8a5809bb00 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.922 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455216630 0 0) 0x7f8a580994f0 con 0x7f8a5809c5f0 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient(hunting): my global_id is 4616 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c0035d0 con 0x7f8a5809c5f0 2024-09-16T06:39:40.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312203711 0 0) 0x7f8a5808d9b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 >> v1:172.21.15.76:6790/0 conn(0x7f8a5809c5f0 legacy=0x7f8a5808d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 >> v1:172.21.15.76:6789/0 conn(0x7f8a58138da0 legacy=0x7f8a581cdf60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 monclient: found mon.b 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5809a6e0 con 0x7f8a5809bb00 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:40.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 5 monclient: authenticate success, global_id 94252 2024-09-16T06:39:40.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _renew_subs 2024-09-16T06:39:40.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a58090830 con 0x7f8a5809bb00 2024-09-16T06:39:40.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _renew_subs 2024-09-16T06:39:40.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a580a2cb0 con 0x7f8a5809bb00 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a580904b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a48004450 con 0x7f8a5809bb00 2024-09-16T06:39:40.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8a48004640 con 0x7f8a5809bb00 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 10 monclient: dump: 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:40.927 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:40.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a55ffb700 4 set_mon_vals no callback set 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.907+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8a48004820 con 0x7f8a5809bb00 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.911+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 3889+0+0 (unknown 2339080297 0 0) 0x7f8a48043180 con 0x7f8a5809bb00 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.911+0000 7f8a4ffff700 10 monclient: get_auth_request con 0x7f8a3c036200 auth_method 0 2024-09-16T06:39:40.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.915+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8a480143b0 con 0x7f8a5809bb00 2024-09-16T06:39:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.915+0000 7f8a54ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:40.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:40.915+0000 7f8a54ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:41.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.079+0000 7f8a5d9d2700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:41.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.079+0000 7f8a5d9d2700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:41.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.079+0000 7f8a5d9d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:41.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.079+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f8a5808e740 con 0x7f8a5809bb00 2024-09-16T06:39:41.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.179+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (unknown 2342121220 0 0) 0x7f8a480141d0 con 0x7f8a5809bb00 2024-09-16T06:39:41.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.179+0000 7f8a54ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:41.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.179+0000 7f8a54ff9700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-16T06:39:41.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.287+0000 7f8a5d9d2700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:41.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.287+0000 7f8a5d9d2700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:41.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.287+0000 7f8a5d9d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:41.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:41.287+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f8a5808f050 con 0x7f8a5809bb00 2024-09-16T06:39:42.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.191+0000 7f8a54ff9700 1 -- 172.21.15.189:0/1850096323 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v25) v1 ==== 143+0+0 (unknown 3047906603 0 0) 0x7f8a48014f10 con 0x7f8a5809bb00 2024-09-16T06:39:42.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.191+0000 7f8a54ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T06:39:42.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.191+0000 7f8a54ff9700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-16T06:39:42.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8a3c036200 legacy=0x7f8a3c0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 10 monclient: shutdown 2024-09-16T06:39:42.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 >> v1:172.21.15.189:6789/0 conn(0x7f8a5809bb00 legacy=0x7f8a5808cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 shutdown_connections 2024-09-16T06:39:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 >> 172.21.15.189:0/1850096323 conn(0x7f8a580a03e0 msgr2=0x7f8a5809a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 shutdown_connections 2024-09-16T06:39:42.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.195+0000 7f8a5d9d2700 1 -- 172.21.15.189:0/1850096323 wait complete. 2024-09-16T06:39:42.198 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_min to 0 2024-09-16T06:39:42.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-16T06:39:42.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:42.211 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T06:39:42.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: get_monmap_and_config 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: build_initial_monmap 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: monmap: 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:42.314402+0000 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:42.314402+0000 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:42.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:42.314 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 1 Processor -- start 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 1 -- start start 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: init 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-b con 0x7f3ca012b880 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-a con 0x7f3ca012bc60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-c con 0x7f3ca013ba80 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca009f3e0 con 0x7f3ca012bc60 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca009f6c0 con 0x7f3ca013ba80 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca009f9a0 con 0x7f3ca012b880 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.311+0000 7f3ca74d2700 10 monclient(hunting): authenticate will time out at 934.827802s 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca4a6d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3ca012bc60 legacy=0x7f3ca013a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca5a6f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ca013ba80 0x7f3ca013de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47848/0 (socket says 172.21.15.189:47848) 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca5a6f700 1 -- 172.21.15.189:0/684294078 learned_addr learned my addr 172.21.15.189:0/684294078 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668487321 0 0) 0x7f3ca009f6c0 con 0x7f3ca013ba80 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): my global_id is 94255 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c8c0035d0 con 0x7f3ca013ba80 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644504942 0 0) 0x7f3ca009f9a0 con 0x7f3ca012b880 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient(hunting): my global_id is 4619 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca009f6c0 con 0x7f3ca012b880 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350464944 0 0) 0x7f3c8c0035d0 con 0x7f3ca013ba80 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 >> v1:172.21.15.76:6790/0 conn(0x7f3ca012b880 legacy=0x7f3ca009ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 >> v1:172.21.15.76:6789/0 conn(0x7f3ca012bc60 legacy=0x7f3ca013a360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 monclient: found mon.noname-c 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca009e7f0 con 0x7f3ca013ba80 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 5 monclient: authenticate success, global_id 94255 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3c98001990 con 0x7f3ca013ba80 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: dump: 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c98001c70 con 0x7f3ca013ba80 2024-09-16T06:39:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 1 -- 172.21.15.189:0/684294078 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3c98002000 con 0x7f3ca013ba80 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 4 set_mon_vals no callback set 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3c97fff700 10 monclient: dump: 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 10 monclient: shutdown 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 1 -- 172.21.15.189:0/684294078 >> v1:172.21.15.189:6789/0 conn(0x7f3ca013ba80 legacy=0x7f3ca013de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 1 -- 172.21.15.189:0/684294078 shutdown_connections 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 1 -- 172.21.15.189:0/684294078 >> 172.21.15.189:0/684294078 conn(0x7f3ca00a4410 msgr2=0x7f3ca00a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 1 -- 172.21.15.189:0/684294078 shutdown_connections 2024-09-16T06:39:42.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 1 -- 172.21.15.189:0/684294078 wait complete. 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 10 monclient: build_initial_monmap 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.315+0000 7f3ca74d2700 10 monclient: monmap: 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 Processor -- start 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- start start 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: init 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-b con 0x7f3ca012b880 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-c con 0x7f3ca012bc60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient(hunting): picked mon.noname-a con 0x7f3ca013ba80 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca009c8c0 con 0x7f3ca013ba80 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca00a03c0 con 0x7f3ca012b880 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca0136800 con 0x7f3ca012bc60 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca526e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ca012b880 0x7f3ca012a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47858/0 (socket says 172.21.15.189:47858) 2024-09-16T06:39:42.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca526e700 1 -- 172.21.15.189:0/1561843333 learned_addr learned my addr 172.21.15.189:0/1561843333 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient(hunting): authenticate will time out at 934.833007s 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca5a6f700 1 -- 172.21.15.189:0/1561843333 >> v1:172.21.15.76:6789/0 conn(0x7f3ca013ba80 legacy=0x7f3ca01d2570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110543306 0 0) 0x7f3ca00a03c0 con 0x7f3ca012b880 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): my global_id is 94258 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c840035d0 con 0x7f3ca012b880 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748063020 0 0) 0x7f3c840035d0 con 0x7f3ca012b880 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ca00a03c0 con 0x7f3ca012b880 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3c9c001910 con 0x7f3ca012b880 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:42.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567388994 0 0) 0x7f3ca0136800 con 0x7f3ca012bc60 2024-09-16T06:39:42.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient(hunting): my global_id is 4622 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c840035d0 con 0x7f3ca012bc60 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1874745103 0 0) 0x7f3ca00a03c0 con 0x7f3ca012b880 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 >> v1:172.21.15.76:6790/0 conn(0x7f3ca012bc60 legacy=0x7f3ca01cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 >> v1:172.21.15.76:6789/0 conn(0x7f3ca013ba80 legacy=0x7f3ca01d2570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 monclient: found mon.b 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca009d490 con 0x7f3ca012b880 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 5 monclient: authenticate success, global_id 94258 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _renew_subs 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca0136ae0 con 0x7f3ca012b880 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _renew_subs 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c9c003740 con 0x7f3ca012b880 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca009d210 con 0x7f3ca012b880 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3c9c003a50 con 0x7f3ca012b880 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 10 monclient: dump: 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c977fe700 4 set_mon_vals no callback set 2024-09-16T06:39:42.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3c9c003bb0 con 0x7f3ca012b880 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca74d2700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca0134380 con 0x7f3ca012b880 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(25..25 src has 1..25) v4 ==== 3873+0+0 (unknown 3991942991 0 0) 0x7f3c9c043310 con 0x7f3ca012b880 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.319+0000 7f3ca4a6d700 10 monclient: get_auth_request con 0x7f3c840362a0 auth_method 0 2024-09-16T06:39:42.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.327+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3c9c014720 con 0x7f3ca012b880 2024-09-16T06:39:42.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.327+0000 7f3c95ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.327+0000 7f3c95ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:42.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.491+0000 7f3ca74d2700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:42.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.491+0000 7f3ca74d2700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:42.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.491+0000 7f3ca74d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.491+0000 7f3ca74d2700 1 -- 172.21.15.189:0/1561843333 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f3ca0066ed0 con 0x7f3ca012b880 2024-09-16T06:39:42.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.495+0000 7f3c95ffb700 1 -- 172.21.15.189:0/1561843333 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v25) v1 ==== 158+0+0 (unknown 3019490396 0 0) 0x7f3c9c014310 con 0x7f3ca012b880 2024-09-16T06:39:42.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.495+0000 7f3c95ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T06:39:42.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.495+0000 7f3c95ffb700 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T06:39:42.496 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T06:39:42.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:42.500 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.607+0000 7f60893dc700 10 monclient: get_monmap_and_config 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.607+0000 7f60893dc700 10 monclient: build_initial_monmap 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.607+0000 7f60893dc700 10 monclient: monmap: 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:42.612493+0000 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:42.612493+0000 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:42.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:42.612 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:42.612 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 Processor -- start 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- start start 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient: init 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-a con 0x7f608406a340 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-b con 0x7f608409d5b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-c con 0x7f608413bb20 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608406b880 con 0x7f608406a340 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608406be90 con 0x7f608413bb20 2024-09-16T06:39:42.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60840a1a40 con 0x7f608409d5b0 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient(hunting): authenticate will time out at 935.125680s 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6082ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f608406a340 legacy=0x7f608406a720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f608413bb20 0x7f608413df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47862/0 (socket says 172.21.15.189:47862) 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60837fe700 1 -- 172.21.15.189:0/692797633 learned_addr learned my addr 172.21.15.189:0/692797633 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377404782 0 0) 0x7f608406be90 con 0x7f608413bb20 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): my global_id is 94261 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c0035d0 con 0x7f608413bb20 2024-09-16T06:39:42.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266473336 0 0) 0x7f60840a1a40 con 0x7f608409d5b0 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient(hunting): my global_id is 4625 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f608406be90 con 0x7f608409d5b0 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3151799576 0 0) 0x7f606c0035d0 con 0x7f608413bb20 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 >> v1:172.21.15.76:6790/0 conn(0x7f608409d5b0 legacy=0x7f608406ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 >> v1:172.21.15.76:6789/0 conn(0x7f608406a340 legacy=0x7f608406a720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 monclient: found mon.noname-c 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840a16f0 con 0x7f608413bb20 2024-09-16T06:39:42.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 5 monclient: authenticate success, global_id 94261 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f607c001990 con 0x7f608413bb20 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: dump: 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.619 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f607c001c70 con 0x7f608413bb20 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 1 -- 172.21.15.189:0/692797633 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f607c002000 con 0x7f608413bb20 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f6081ffb700 10 monclient: dump: 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.620 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 4 set_mon_vals no callback set 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 10 monclient: shutdown 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:42.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- 172.21.15.189:0/692797633 >> v1:172.21.15.189:6789/0 conn(0x7f608413bb20 legacy=0x7f608413df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- 172.21.15.189:0/692797633 shutdown_connections 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- 172.21.15.189:0/692797633 >> 172.21.15.189:0/692797633 conn(0x7f60840a21f0 msgr2=0x7f60840a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- 172.21.15.189:0/692797633 shutdown_connections 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.611+0000 7f60893dc700 1 -- 172.21.15.189:0/692797633 wait complete. 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: build_initial_monmap 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: monmap: 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:42.622 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:42.623 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:42.623 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 Processor -- start 2024-09-16T06:39:42.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- start start 2024-09-16T06:39:42.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: init 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-b con 0x7f608406a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-c con 0x7f608409d5b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient(hunting): picked mon.noname-a con 0x7f608413bb20 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084134490 con 0x7f608413bb20 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084071470 con 0x7f608406a340 2024-09-16T06:39:42.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608409f870 con 0x7f608409d5b0 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient(hunting): authenticate will time out at 935.130059s 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f6082ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f608406a340 0x7f6084070650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47866/0 (socket says 172.21.15.189:47866) 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f6082ffd700 1 -- 172.21.15.189:0/258657705 learned_addr learned my addr 172.21.15.189:0/258657705 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60837fe700 1 -- 172.21.15.189:0/258657705 >> v1:172.21.15.76:6789/0 conn(0x7f608413bb20 legacy=0x7f60841d20c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847158211 0 0) 0x7f6084071470 con 0x7f608406a340 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): my global_id is 94264 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60700035d0 con 0x7f608406a340 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699607768 0 0) 0x7f60700035d0 con 0x7f608406a340 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6084071470 con 0x7f608406a340 2024-09-16T06:39:42.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6078004280 con 0x7f608406a340 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.628 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227270179 0 0) 0x7f608409f870 con 0x7f608409d5b0 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient(hunting): my global_id is 4628 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60700035d0 con 0x7f608409d5b0 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1751529643 0 0) 0x7f6084071470 con 0x7f608406a340 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 >> v1:172.21.15.76:6790/0 conn(0x7f608409d5b0 legacy=0x7f6084070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 >> v1:172.21.15.76:6789/0 conn(0x7f608413bb20 legacy=0x7f60841d20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 monclient: found mon.b 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608409c700 con 0x7f608406a340 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 5 monclient: authenticate success, global_id 94264 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _renew_subs 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6084072230 con 0x7f608406a340 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _renew_subs 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60841374a0 con 0x7f608406a340 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60780045a0 con 0x7f608406a340 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:42.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6078004930 con 0x7f608406a340 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f60817fa700 4 set_mon_vals no callback set 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.615+0000 7f606b7fe700 10 monclient: dump: 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:42.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f60893dc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f60893dc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f60893dc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6064005100 con 0x7f608406a340 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6078004b10 con 0x7f608406a340 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(25..25 src has 1..25) v4 ==== 3873+0+0 (unknown 3991942991 0 0) 0x7f60780433e0 con 0x7f608406a340 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.619+0000 7f60827fc700 10 monclient: get_auth_request con 0x7f6070035fe0 auth_method 0 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.623+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6078014990 con 0x7f608406a340 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.623+0000 7f606b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:42.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.623+0000 7f606b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:42.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.795+0000 7f60893dc700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.795+0000 7f60893dc700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.795+0000 7f60893dc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:42.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:42.795+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f6064005930 con 0x7f608406a340 2024-09-16T06:39:43.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.191+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (unknown 3024910647 0 0) 0x7f6078012e00 con 0x7f608406a340 2024-09-16T06:39:43.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.191+0000 7f606b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:43.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.191+0000 7f606b7fe700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-16T06:39:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.299+0000 7f60893dc700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:43.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.299+0000 7f60893dc700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.299+0000 7f60893dc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:43.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:43.299+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f60640053f0 con 0x7f608406a340 2024-09-16T06:39:44.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.207+0000 7f606b7fe700 1 -- 172.21.15.189:0/258657705 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v27) v1 ==== 143+0+0 (unknown 2323565336 0 0) 0x7f6078014760 con 0x7f608406a340 2024-09-16T06:39:44.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.207+0000 7f606b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T06:39:44.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.207+0000 7f606b7fe700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6070035fe0 legacy=0x7f60700384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 10 monclient: shutdown 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 >> v1:172.21.15.189:6789/0 conn(0x7f608406a340 legacy=0x7f6084070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 shutdown_connections 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.211+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 >> 172.21.15.189:0/258657705 conn(0x7f60840a21f0 msgr2=0x7f608409e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.215+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 shutdown_connections 2024-09-16T06:39:44.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.215+0000 7f60893dc700 1 -- 172.21.15.189:0/258657705 wait complete. 2024-09-16T06:39:44.216 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 3 pg_num_max to 0 2024-09-16T06:39:44.230 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-16T06:39:44.230 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:44.230 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T06:39:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: get_monmap_and_config 2024-09-16T06:39:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: build_initial_monmap 2024-09-16T06:39:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: monmap: 2024-09-16T06:39:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:44.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:44.342846+0000 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:44.342846+0000 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:44.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 1 Processor -- start 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 1 -- start start 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: init 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-c con 0x7f8ed40a5c70 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-a con 0x7f8ed40a7100 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-b con 0x7f8ed413bab0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:44.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed40a5a90 con 0x7f8ed40a7100 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed40a6a90 con 0x7f8ed40a5c70 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed4099e40 con 0x7f8ed413bab0 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.339+0000 7f8ed9573700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient(hunting): authenticate will time out at 936.856015s 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed27fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ed40a7100 legacy=0x7f8ed413a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ed40a5c70 0x7f8ed40a6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47876/0 (socket says 172.21.15.189:47876) 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed2ffd700 1 -- 172.21.15.189:0/343923354 learned_addr learned my addr 172.21.15.189:0/343923354 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007536440 0 0) 0x7f8ed40a6a90 con 0x7f8ed40a5c70 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): my global_id is 94267 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec00035d0 con 0x7f8ed40a5c70 2024-09-16T06:39:44.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870877118 0 0) 0x7f8ed4099e40 con 0x7f8ed413bab0 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient(hunting): my global_id is 4631 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed40a6a90 con 0x7f8ed413bab0 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899619069 0 0) 0x7f8ec00035d0 con 0x7f8ed40a5c70 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 >> v1:172.21.15.76:6790/0 conn(0x7f8ed413bab0 legacy=0x7f8ed413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 >> v1:172.21.15.76:6789/0 conn(0x7f8ed40a7100 legacy=0x7f8ed413a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 monclient: found mon.noname-c 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed409a120 con 0x7f8ed40a5c70 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 5 monclient: authenticate success, global_id 94267 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec4001610 con 0x7f8ed40a5c70 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: dump: 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec40018f0 con 0x7f8ed40a5c70 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 1 -- 172.21.15.189:0/343923354 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec4001c80 con 0x7f8ed40a5c70 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 4 set_mon_vals no callback set 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed1ffb700 10 monclient: dump: 2024-09-16T06:39:44.349 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient: shutdown 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- 172.21.15.189:0/343923354 >> v1:172.21.15.189:6789/0 conn(0x7f8ed40a5c70 legacy=0x7f8ed40a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- 172.21.15.189:0/343923354 shutdown_connections 2024-09-16T06:39:44.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- 172.21.15.189:0/343923354 >> 172.21.15.189:0/343923354 conn(0x7f8ed40a79e0 msgr2=0x7f8ed40a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- 172.21.15.189:0/343923354 shutdown_connections 2024-09-16T06:39:44.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- 172.21.15.189:0/343923354 wait complete. 2024-09-16T06:39:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient: build_initial_monmap 2024-09-16T06:39:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient: monmap: 2024-09-16T06:39:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:44.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 Processor -- start 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 1 -- start start 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.343+0000 7f8ed9573700 10 monclient: init 2024-09-16T06:39:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-a con 0x7f8ed40a5c70 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-c con 0x7f8ed40a7100 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient(hunting): picked mon.noname-b con 0x7f8ed413bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed40a6f50 con 0x7f8ed40a5c70 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed40a4520 con 0x7f8ed413bab0 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed406bd50 con 0x7f8ed40a7100 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:44.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient(hunting): authenticate will time out at 936.860367s 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed2ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ed40a5c70 legacy=0x7f8ed41c9ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8ed413bab0 0x7f8ed41d25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47886/0 (socket says 172.21.15.189:47886) 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed37fe700 1 -- 172.21.15.189:0/4147950272 learned_addr learned my addr 172.21.15.189:0/4147950272 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675948406 0 0) 0x7f8ed40a4520 con 0x7f8ed413bab0 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): my global_id is 94270 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb40035d0 con 0x7f8ed413bab0 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901752013 0 0) 0x7f8ed406bd50 con 0x7f8ed40a7100 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): my global_id is 4634 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed40a4520 con 0x7f8ed40a7100 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273504100 0 0) 0x7f8eb40035d0 con 0x7f8ed413bab0 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ed406bd50 con 0x7f8ed413bab0 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ecc001c50 con 0x7f8ed413bab0 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.358 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935015766 0 0) 0x7f8ed406bd50 con 0x7f8ed413bab0 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 >> v1:172.21.15.76:6790/0 conn(0x7f8ed40a7100 legacy=0x7f8ed41cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 >> v1:172.21.15.76:6789/0 conn(0x7f8ed40a5c70 legacy=0x7f8ed41c9ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 monclient: found mon.b 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed411e590 con 0x7f8ed413bab0 2024-09-16T06:39:44.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 5 monclient: authenticate success, global_id 94270 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _renew_subs 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed411f500 con 0x7f8ed413bab0 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ecc001fe0 con 0x7f8ed413bab0 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ecc003e90 con 0x7f8ed413bab0 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 10 monclient: dump: 2024-09-16T06:39:44.360 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _renew_subs 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed17fa700 4 set_mon_vals no callback set 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed4134da0 con 0x7f8ed413bab0 2024-09-16T06:39:44.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed9573700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed4134320 con 0x7f8ed413bab0 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8ecc004070 con 0x7f8ed413bab0 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 3857+0+0 (unknown 1224060302 0 0) 0x7f8ecc043780 con 0x7f8ed413bab0 2024-09-16T06:39:44.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.347+0000 7f8ed2ffd700 10 monclient: get_auth_request con 0x7f8eb4036180 auth_method 0 2024-09-16T06:39:44.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.359+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ecc0149d0 con 0x7f8ed413bab0 2024-09-16T06:39:44.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.359+0000 7f8ecb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.359+0000 7f8ecb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:44.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ed9573700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ed9573700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ed9573700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ed9573700 1 -- 172.21.15.189:0/4147950272 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f8ed4067950 con 0x7f8ed413bab0 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ecb7fe700 1 -- 172.21.15.189:0/4147950272 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v27) v1 ==== 158+0+0 (unknown 694716346 0 0) 0x7f8ecc013070 con 0x7f8ed413bab0 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ecb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T06:39:44.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.527+0000 7f8ecb7fe700 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T06:39:44.532 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T06:39:44.535 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:44.535 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: get_monmap_and_config 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: build_initial_monmap 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: monmap: 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:44.649473+0000 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:44.649473+0000 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.648 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:44.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:44.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:44.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 1 Processor -- start 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 1 -- start start 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: init 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-a con 0x7f6dc80707a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-c con 0x7f6dc8070b80 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-b con 0x7f6dc813b510 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80a12a0 con 0x7f6dc80707a0 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80a0670 con 0x7f6dc8070b80 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc809f380 con 0x7f6dc813b510 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:44.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient(hunting): authenticate will time out at 937.162716s 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc6ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6dc80707a0 legacy=0x7f6dc809ff60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6dc8070b80 0x7f6dc8070fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47902/0 (socket says 172.21.15.189:47902) 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc67fc700 1 -- 172.21.15.189:0/4113140907 learned_addr learned my addr 172.21.15.189:0/4113140907 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831285915 0 0) 0x7f6dc80a0670 con 0x7f6dc8070b80 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): my global_id is 94273 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db00035d0 con 0x7f6dc8070b80 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601523339 0 0) 0x7f6dc809f380 con 0x7f6dc813b510 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient(hunting): my global_id is 4637 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc80a0670 con 0x7f6dc813b510 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496035764 0 0) 0x7f6db00035d0 con 0x7f6dc8070b80 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 >> v1:172.21.15.76:6790/0 conn(0x7f6dc813b510 legacy=0x7f6dc813d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 >> v1:172.21.15.76:6789/0 conn(0x7f6dc80707a0 legacy=0x7f6dc809ff60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 monclient: found mon.noname-c 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc809f660 con 0x7f6dc8070b80 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:44.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6db8001610 con 0x7f6dc8070b80 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 5 monclient: authenticate success, global_id 94273 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: dump: 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.661 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6db8001880 con 0x7f6dc8070b80 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dcd439700 4 set_mon_vals no callback set 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 1 -- 172.21.15.189:0/4113140907 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6db8001c10 con 0x7f6dc8070b80 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.647+0000 7f6dc5ffb700 10 monclient: dump: 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.662 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: shutdown 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- 172.21.15.189:0/4113140907 >> v1:172.21.15.189:6789/0 conn(0x7f6dc8070b80 legacy=0x7f6dc8070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- 172.21.15.189:0/4113140907 shutdown_connections 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- 172.21.15.189:0/4113140907 >> 172.21.15.189:0/4113140907 conn(0x7f6dc80a44a0 msgr2=0x7f6dc80a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:44.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- 172.21.15.189:0/4113140907 shutdown_connections 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- 172.21.15.189:0/4113140907 wait complete. 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: build_initial_monmap 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: monmap: 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:44.664 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 Processor -- start 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- start start 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: init 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-b con 0x7f6dc80707a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-a con 0x7f6dc8070b80 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient(hunting): picked mon.noname-c con 0x7f6dc813b510 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:44.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc81374e0 con 0x7f6dc8070b80 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc809d700 con 0x7f6dc80707a0 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc8137200 con 0x7f6dc813b510 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dcd439700 10 monclient(hunting): authenticate will time out at 937.167474s 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dc67fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6dc8070b80 legacy=0x7f6dc809c260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dc6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6dc80707a0 0x7f6dc809bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47918/0 (socket says 172.21.15.189:47918) 2024-09-16T06:39:44.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.651+0000 7f6dc6ffd700 1 -- 172.21.15.189:0/2148536424 learned_addr learned my addr 172.21.15.189:0/2148536424 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588963046 0 0) 0x7f6dc809d700 con 0x7f6dc80707a0 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): my global_id is 94276 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db40035d0 con 0x7f6dc80707a0 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315790658 0 0) 0x7f6dc8137200 con 0x7f6dc813b510 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:44.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): my global_id is 4640 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc809d700 con 0x7f6dc813b510 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102065422 0 0) 0x7f6db40035d0 con 0x7f6dc80707a0 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc8137200 con 0x7f6dc80707a0 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6dbc004230 con 0x7f6dc80707a0 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.668 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197995779 0 0) 0x7f6dc809d700 con 0x7f6dc813b510 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db40035d0 con 0x7f6dc813b510 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6dc0001c30 con 0x7f6dc813b510 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.669 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107637413 0 0) 0x7f6dc8137200 con 0x7f6dc80707a0 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 >> v1:172.21.15.76:6790/0 conn(0x7f6dc813b510 legacy=0x7f6dc809c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 >> v1:172.21.15.76:6789/0 conn(0x7f6dc8070b80 legacy=0x7f6dc809c260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:44.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 monclient: found mon.b 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc81377c0 con 0x7f6dc80707a0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 5 monclient: authenticate success, global_id 94276 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _renew_subs 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc81364e0 con 0x7f6dc80707a0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _renew_subs 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc81d2fe0 con 0x7f6dc80707a0 2024-09-16T06:39:44.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dbc0044a0 con 0x7f6dc80707a0 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dc57fa700 4 set_mon_vals no callback set 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6dbc004830 con 0x7f6dc80707a0 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 10 monclient: dump: 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:44.672 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da8005100 con 0x7f6dc80707a0 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6dbc004a50 con 0x7f6dc80707a0 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(27..27 src has 1..27) v4 ==== 3857+0+0 (unknown 1224060302 0 0) 0x7f6dbc043250 con 0x7f6dc80707a0 2024-09-16T06:39:44.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.655+0000 7f6dc67fc700 10 monclient: get_auth_request con 0x7f6db40362c0 auth_method 0 2024-09-16T06:39:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.663+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6dbc014830 con 0x7f6dc80707a0 2024-09-16T06:39:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.663+0000 7f6daf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:44.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.663+0000 7f6daf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.831+0000 7f6dcd439700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.831+0000 7f6dcd439700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.831+0000 7f6dcd439700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:44.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:44.831+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6da80053b0 con 0x7f6dc80707a0 2024-09-16T06:39:45.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.211+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v28) v1 ==== 176+0+0 (unknown 2591182866 0 0) 0x7f6dbc012ca0 con 0x7f6dc80707a0 2024-09-16T06:39:45.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.211+0000 7f6daf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:45.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.211+0000 7f6daf7fe700 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-16T06:39:45.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.315+0000 7f6dcd439700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:45.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.315+0000 7f6dcd439700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:45.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.315+0000 7f6dcd439700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.315+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6da80010a0 con 0x7f6dc80707a0 2024-09-16T06:39:45.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6daf7fe700 1 -- 172.21.15.189:0/2148536424 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v28) v1 ==== 183+0+0 (unknown 2571058706 0 0) 0x7f6dbc014420 con 0x7f6dc80707a0 2024-09-16T06:39:45.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6daf7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T06:39:45.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6daf7fe700 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-16T06:39:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6db40362c0 legacy=0x7f6db4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 10 monclient: shutdown 2024-09-16T06:39:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 >> v1:172.21.15.189:6789/0 conn(0x7f6dc80707a0 legacy=0x7f6dc809bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 shutdown_connections 2024-09-16T06:39:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 >> 172.21.15.189:0/2148536424 conn(0x7f6dc80a44a0 msgr2=0x7f6dc80a2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 shutdown_connections 2024-09-16T06:39:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.319+0000 7f6dcd439700 1 -- 172.21.15.189:0/2148536424 wait complete. 2024-09-16T06:39:45.322 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'testpool1234' does not exist 2024-09-16T06:39:45.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T06:39:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:39:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T06:39:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: get_monmap_and_config 2024-09-16T06:39:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: build_initial_monmap 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: monmap: 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:45.442052+0000 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:45.442052+0000 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:45.441 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 1 Processor -- start 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 1 -- start start 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: init 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-b con 0x7f95ac12e310 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-a con 0x7f95ac12e6f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-c con 0x7f95ac09d660 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac09c630 con 0x7f95ac12e6f0 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac09c910 con 0x7f95ac09d660 2024-09-16T06:39:45.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac066ed0 con 0x7f95ac12e310 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b139e700 10 monclient(hunting): authenticate will time out at 937.955480s 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95ab7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f95ac12e6f0 legacy=0x7f95ac1362f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b0b9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f95ac09d660 0x7f95ac139a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47928/0 (socket says 172.21.15.189:47928) 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.439+0000 7f95b0b9d700 1 -- 172.21.15.189:0/1716223423 learned_addr learned my addr 172.21.15.189:0/1716223423 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759405584 0 0) 0x7f95ac09c910 con 0x7f95ac09d660 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient(hunting): my global_id is 94279 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95980035d0 con 0x7f95ac09d660 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7159913 0 0) 0x7f95980035d0 con 0x7f95ac09d660 2024-09-16T06:39:45.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 >> v1:172.21.15.76:6790/0 conn(0x7f95ac12e310 legacy=0x7f95ac12dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 >> v1:172.21.15.76:6789/0 conn(0x7f95ac12e6f0 legacy=0x7f95ac1362f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 monclient: found mon.noname-c 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac067060 con 0x7f95ac09d660 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95a4001990 con 0x7f95ac09d660 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 5 monclient: authenticate success, global_id 94279 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:45.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: dump: 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:45.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95a4001c30 con 0x7f95ac09d660 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 4 set_mon_vals no callback set 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 1 -- 172.21.15.189:0/1716223423 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95a4002000 con 0x7f95ac09d660 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95aaffd700 10 monclient: dump: 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 10 monclient: shutdown 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- 172.21.15.189:0/1716223423 >> v1:172.21.15.189:6789/0 conn(0x7f95ac09d660 legacy=0x7f95ac139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- 172.21.15.189:0/1716223423 shutdown_connections 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- 172.21.15.189:0/1716223423 >> 172.21.15.189:0/1716223423 conn(0x7f95ac0a05b0 msgr2=0x7f95ac09de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- 172.21.15.189:0/1716223423 shutdown_connections 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- 172.21.15.189:0/1716223423 wait complete. 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 10 monclient: build_initial_monmap 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 10 monclient: monmap: 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:45.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 Processor -- start 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 1 -- start start 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.443+0000 7f95b139e700 10 monclient: init 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-b con 0x7f95ac09d660 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-c con 0x7f95ac12e310 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient(hunting): picked mon.noname-a con 0x7f95ac12e6f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac0a27e0 con 0x7f95ac12e6f0 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac09a670 con 0x7f95ac09d660 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac06ca20 con 0x7f95ac12e310 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient(hunting): authenticate will time out at 937.960787s 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95abfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f95ac09d660 0x7f95ac0962f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47944/0 (socket says 172.21.15.189:47944) 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95abfff700 1 -- 172.21.15.189:0/848310596 learned_addr learned my addr 172.21.15.189:0/848310596 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b0b9d700 1 -- 172.21.15.189:0/848310596 >> v1:172.21.15.76:6789/0 conn(0x7f95ac12e6f0 legacy=0x7f95ac1cf3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:45.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894204239 0 0) 0x7f95ac09a670 con 0x7f95ac09d660 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): my global_id is 94282 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c0035d0 con 0x7f95ac09d660 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381587067 0 0) 0x7f958c0035d0 con 0x7f95ac09d660 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ac09a670 con 0x7f95ac09d660 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95940019b0 con 0x7f95ac09d660 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2981792662 0 0) 0x7f95ac09a670 con 0x7f95ac09d660 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 >> v1:172.21.15.76:6790/0 conn(0x7f95ac12e310 legacy=0x7f95ac1cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 >> v1:172.21.15.76:6789/0 conn(0x7f95ac12e6f0 legacy=0x7f95ac1cf3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 monclient: found mon.b 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac06d980 con 0x7f95ac09d660 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 5 monclient: authenticate success, global_id 94282 2024-09-16T06:39:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _renew_subs 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac12dfb0 con 0x7f95ac09d660 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _renew_subs 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac13dc70 con 0x7f95ac09d660 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9594003970 con 0x7f95ac09d660 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9594003d00 con 0x7f95ac09d660 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95aa7fc700 4 set_mon_vals no callback set 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 10 monclient: dump: 2024-09-16T06:39:45.459 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9594003ee0 con 0x7f95ac09d660 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.447+0000 7f95b139e700 1 -- 172.21.15.189:0/848310596 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9590005100 con 0x7f95ac09d660 2024-09-16T06:39:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.451+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(28..28 src has 1..28) v4 ==== 3506+0+0 (unknown 1964681941 0 0) 0x7f95940432c0 con 0x7f95ac09d660 2024-09-16T06:39:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.451+0000 7f95ab7fe700 10 monclient: get_auth_request con 0x7f958c035f00 auth_method 0 2024-09-16T06:39:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.455+0000 7f95a8ff9700 1 -- 172.21.15.189:0/848310596 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f95940147d0 con 0x7f95ac09d660 2024-09-16T06:39:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.455+0000 7f95a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.455+0000 7f95a8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:45.624 INFO:tasks.workunit.client.0.smithi189.stderr:foo-rule not valid: foo-rule not one of 'true', 'false' 2024-09-16T06:39:45.624 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-16T06:39:45.624 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] [--yes-i-really-mean-it] : create pool 2024-09-16T06:39:45.624 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T06:39:45.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:39:45.628 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-16T06:39:45.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.735+0000 7f4942183700 10 monclient: get_monmap_and_config 2024-09-16T06:39:45.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.735+0000 7f4942183700 10 monclient: build_initial_monmap 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.735+0000 7f4942183700 10 monclient: monmap: 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:45.739817+0000 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:45.739817+0000 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:45.739 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 Processor -- start 2024-09-16T06:39:45.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- start start 2024-09-16T06:39:45.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: init 2024-09-16T06:39:45.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient(hunting): picked mon.noname-b con 0x7f493c0a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient(hunting): picked mon.noname-c con 0x7f493c09aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient(hunting): picked mon.noname-a con 0x7f493c0a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c09a040 con 0x7f493c0a3ca0 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0a33d0 con 0x7f493c09aa20 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c099d60 con 0x7f493c0a2350 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:45.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient(hunting): authenticate will time out at 938.253034s 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f493c0a3ca0 legacy=0x7f493c13dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f493c09aa20 0x7f493c13a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47952/0 (socket says 172.21.15.189:47952) 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493affd700 1 -- 172.21.15.189:0/2913462206 learned_addr learned my addr 172.21.15.189:0/2913462206 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186036035 0 0) 0x7f493c0a33d0 con 0x7f493c09aa20 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:45.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): my global_id is 94285 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49240035d0 con 0x7f493c09aa20 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074386749 0 0) 0x7f493c099d60 con 0x7f493c0a2350 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient(hunting): my global_id is 4649 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f493c0a33d0 con 0x7f493c0a2350 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226239113 0 0) 0x7f49240035d0 con 0x7f493c09aa20 2024-09-16T06:39:45.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 >> v1:172.21.15.76:6790/0 conn(0x7f493c0a2350 legacy=0x7f493c0a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 >> v1:172.21.15.76:6789/0 conn(0x7f493c0a3ca0 legacy=0x7f493c13dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 monclient: found mon.noname-c 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0a3920 con 0x7f493c09aa20 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 5 monclient: authenticate success, global_id 94285 2024-09-16T06:39:45.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f492c001610 con 0x7f493c09aa20 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: dump: 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.749 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f492c0018f0 con 0x7f493c09aa20 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 1 -- 172.21.15.189:0/2913462206 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f492c001c80 con 0x7f493c09aa20 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 4 set_mon_vals no callback set 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f493a7fc700 10 monclient: dump: 2024-09-16T06:39:45.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: shutdown 2024-09-16T06:39:45.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- 172.21.15.189:0/2913462206 >> v1:172.21.15.189:6789/0 conn(0x7f493c09aa20 legacy=0x7f493c13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- 172.21.15.189:0/2913462206 shutdown_connections 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- 172.21.15.189:0/2913462206 >> 172.21.15.189:0/2913462206 conn(0x7f493c0a7d70 msgr2=0x7f493c0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- 172.21.15.189:0/2913462206 shutdown_connections 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 1 -- 172.21.15.189:0/2913462206 wait complete. 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: build_initial_monmap 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.739+0000 7f4942183700 10 monclient: monmap: 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 Processor -- start 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- start start 2024-09-16T06:39:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: init 2024-09-16T06:39:45.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient(hunting): picked mon.noname-b con 0x7f493c09aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient(hunting): picked mon.noname-a con 0x7f493c0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient(hunting): picked mon.noname-c con 0x7f493c0a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c122dc0 con 0x7f493c0a2350 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c121910 con 0x7f493c09aa20 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c1361b0 con 0x7f493c0a3ca0 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient(hunting): authenticate will time out at 938.257493s 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f493affd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f493c0a2350 legacy=0x7f493c1cee90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:45.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f493b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f493c09aa20 0x7f493c1356e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47962/0 (socket says 172.21.15.189:47962) 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f493b7fe700 1 -- 172.21.15.189:0/1242201221 learned_addr learned my addr 172.21.15.189:0/1242201221 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730449789 0 0) 0x7f493c121910 con 0x7f493c09aa20 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): my global_id is 94288 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49280035d0 con 0x7f493c09aa20 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921732014 0 0) 0x7f49280035d0 con 0x7f493c09aa20 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f493c121910 con 0x7f493c09aa20 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4930001910 con 0x7f493c09aa20 2024-09-16T06:39:45.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): dump: 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.760 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2348443116 0 0) 0x7f493c121910 con 0x7f493c09aa20 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 >> v1:172.21.15.76:6790/0 conn(0x7f493c0a3ca0 legacy=0x7f493c1d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 >> v1:172.21.15.76:6789/0 conn(0x7f493c0a2350 legacy=0x7f493c1cee90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 monclient: found mon.b 2024-09-16T06:39:45.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c122ae0 con 0x7f493c09aa20 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 5 monclient: authenticate success, global_id 94288 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _renew_subs 2024-09-16T06:39:45.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c121690 con 0x7f493c09aa20 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _renew_subs 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c1c9b10 con 0x7f493c09aa20 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4930001d20 con 0x7f493c09aa20 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4930003b10 con 0x7f493c09aa20 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4939ffb700 4 set_mon_vals no callback set 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:45.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.743+0000 7f4923fff700 10 monclient: dump: 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4942183700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4942183700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4942183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c005100 con 0x7f493c09aa20 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4930003d30 con 0x7f493c09aa20 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(28..28 src has 1..28) v4 ==== 3506+0+0 (unknown 1964681941 0 0) 0x7f4930043300 con 0x7f493c09aa20 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.747+0000 7f493affd700 10 monclient: get_auth_request con 0x7f4928036080 auth_method 0 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.755+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f49300148e0 con 0x7f493c09aa20 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.755+0000 7f4923fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:45.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.755+0000 7f4923fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.915+0000 7f4942183700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.915+0000 7f4942183700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.915+0000 7f4942183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:45.915+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f491c005930 con 0x7f493c09aa20 2024-09-16T06:39:46.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.235+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v29) v1 ==== 154+0+0 (unknown 2343297418 0 0) 0x7f4930012d50 con 0x7f493c09aa20 2024-09-16T06:39:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.235+0000 7f4923fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:46.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.235+0000 7f4923fff700 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-16T06:39:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4942183700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4942183700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4942183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f491c005420 con 0x7f493c09aa20 2024-09-16T06:39:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4923fff700 1 -- 172.21.15.189:0/1242201221 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v29) v1 ==== 161+0+0 (unknown 2157755710 0 0) 0x7f49300144d0 con 0x7f493c09aa20 2024-09-16T06:39:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4923fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T06:39:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.347+0000 7f4923fff700 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4928036080 legacy=0x7f4928038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 10 monclient: shutdown 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 >> v1:172.21.15.189:6789/0 conn(0x7f493c09aa20 legacy=0x7f493c1356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 shutdown_connections 2024-09-16T06:39:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 >> 172.21.15.189:0/1242201221 conn(0x7f493c0a7d70 msgr2=0x7f493c1bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 shutdown_connections 2024-09-16T06:39:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.351+0000 7f4942183700 1 -- 172.21.15.189:0/1242201221 wait complete. 2024-09-16T06:39:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2024-09-16T06:39:46.363 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 10 monclient: get_monmap_and_config 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 10 monclient: build_initial_monmap 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 10 monclient: monmap: 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:46.479386+0000 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:46.479386+0000 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:46.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:46.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 1 Processor -- start 2024-09-16T06:39:46.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 1 -- start start 2024-09-16T06:39:46.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.475+0000 7f4a6a8af700 10 monclient: init 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-b con 0x7f4a64099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-c con 0x7f4a64099e40 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-a con 0x7f4a6409a250 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6409c460 con 0x7f4a6409a250 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6409c1e0 con 0x7f4a64099e40 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6409bd60 con 0x7f4a64099a60 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:46.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): authenticate will time out at 938.992721s 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a637fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a64099e40 0x7f4a64136960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47970/0 (socket says 172.21.15.189:47970) 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a637fe700 1 -- 172.21.15.189:0/1193959968 learned_addr learned my addr 172.21.15.189:0/1193959968 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a68e4c700 1 -- 172.21.15.189:0/1193959968 >> v1:172.21.15.76:6789/0 conn(0x7f4a6409a250 legacy=0x7f4a6413a090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328153198 0 0) 0x7f4a6409c1e0 con 0x7f4a64099e40 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): my global_id is 94291 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c0034f0 con 0x7f4a64099e40 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600843462 0 0) 0x7f4a6409bd60 con 0x7f4a64099a60 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:46.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient(hunting): my global_id is 4652 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a6409c1e0 con 0x7f4a64099a60 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410132608 0 0) 0x7f4a4c0034f0 con 0x7f4a64099e40 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 >> v1:172.21.15.76:6790/0 conn(0x7f4a64099a60 legacy=0x7f4a64098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 >> v1:172.21.15.76:6789/0 conn(0x7f4a6409a250 legacy=0x7f4a6413a090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 monclient: found mon.noname-c 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6409d490 con 0x7f4a64099e40 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4a54001610 con 0x7f4a64099e40 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: dump: 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 5 monclient: authenticate success, global_id 94291 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a540018f0 con 0x7f4a64099e40 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 4 set_mon_vals no callback set 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 1 -- 172.21.15.189:0/1193959968 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4a54001c80 con 0x7f4a64099e40 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a62ffd700 10 monclient: dump: 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: shutdown 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1193959968 >> v1:172.21.15.189:6789/0 conn(0x7f4a64099e40 legacy=0x7f4a64136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1193959968 shutdown_connections 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1193959968 >> 172.21.15.189:0/1193959968 conn(0x7f4a640a03e0 msgr2=0x7f4a640a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1193959968 shutdown_connections 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1193959968 wait complete. 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: build_initial_monmap 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: monmap: 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 Processor -- start 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 1 -- start start 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: init 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.479+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-b con 0x7f4a64099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-c con 0x7f4a64099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient(hunting): picked mon.noname-a con 0x7f4a6409a250 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6408f750 con 0x7f4a6409a250 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64090810 con 0x7f4a64099a60 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a641c63e0 con 0x7f4a64099e40 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient(hunting): authenticate will time out at 938.996072s 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a63fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4a64099a60 0x7f4a64096620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47976/0 (socket says 172.21.15.189:47976) 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a63fff700 1 -- 172.21.15.189:0/1091273164 learned_addr learned my addr 172.21.15.189:0/1091273164 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a68e4c700 1 -- 172.21.15.189:0/1091273164 >> v1:172.21.15.76:6789/0 conn(0x7f4a6409a250 legacy=0x7f4a641ce3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524358294 0 0) 0x7f4a64090810 con 0x7f4a64099a60 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): my global_id is 94294 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a500035d0 con 0x7f4a64099a60 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532756513 0 0) 0x7f4a641c63e0 con 0x7f4a64099e40 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): my global_id is 4655 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a64090810 con 0x7f4a64099e40 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056320498 0 0) 0x7f4a500035d0 con 0x7f4a64099a60 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a641c63e0 con 0x7f4a64099a60 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4a58001850 con 0x7f4a64099a60 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 626395122 0 0) 0x7f4a641c63e0 con 0x7f4a64099a60 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 >> v1:172.21.15.76:6790/0 conn(0x7f4a64099e40 legacy=0x7f4a641caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 >> v1:172.21.15.76:6789/0 conn(0x7f4a6409a250 legacy=0x7f4a641ce3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 monclient: found mon.b 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6409a6e0 con 0x7f4a64099a60 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 5 monclient: authenticate success, global_id 94294 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _renew_subs 2024-09-16T06:39:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a640a30d0 con 0x7f4a64099a60 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _renew_subs 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a64090dd0 con 0x7f4a64099a60 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a64070970 con 0x7f4a64099a60 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a580036c0 con 0x7f4a64099a60 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4a58003a50 con 0x7f4a64099a60 2024-09-16T06:39:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 10 monclient: dump: 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a627fc700 4 set_mon_vals no callback set 2024-09-16T06:39:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.483+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4a58003c30 con 0x7f4a64099a60 2024-09-16T06:39:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.487+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(29..29 src has 1..29) v4 ==== 3848+0+0 (unknown 2301446092 0 0) 0x7f4a58043310 con 0x7f4a64099a60 2024-09-16T06:39:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.487+0000 7f4a637fe700 10 monclient: get_auth_request con 0x7f4a500361d0 auth_method 0 2024-09-16T06:39:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.491+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4a580147a0 con 0x7f4a64099a60 2024-09-16T06:39:46.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.491+0000 7f4a60ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:46.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.491+0000 7f4a60ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.659+0000 7f4a6a8af700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.659+0000 7f4a6a8af700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.659+0000 7f4a6a8af700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:46.659+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f4a6408fb70 con 0x7f4a64099a60 2024-09-16T06:39:48.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.251+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v31) v1 ==== 188+0+0 (unknown 3355072401 0 0) 0x7f4a6408fb70 con 0x7f4a64099a60 2024-09-16T06:39:48.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.251+0000 7f4a60ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:48.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.251+0000 7f4a60ff9700 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-16T06:39:48.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.367+0000 7f4a6a8af700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:48.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.367+0000 7f4a6a8af700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:48.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.367+0000 7f4a6a8af700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.367+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f4a6408fb70 con 0x7f4a64099a60 2024-09-16T06:39:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a60ff9700 1 -- 172.21.15.189:0/1091273164 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v31) v1 ==== 195+0+0 (unknown 4178347572 0 0) 0x7f4a6408fb70 con 0x7f4a64099a60 2024-09-16T06:39:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a60ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T06:39:48.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a60ff9700 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4a500361d0 legacy=0x7f4a50038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 10 monclient: shutdown 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 >> v1:172.21.15.189:6789/0 conn(0x7f4a64099a60 legacy=0x7f4a64096620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 shutdown_connections 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 >> 172.21.15.189:0/1091273164 conn(0x7f4a640a03e0 msgr2=0x7f4a6406da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 shutdown_connections 2024-09-16T06:39:48.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.371+0000 7f4a6a8af700 1 -- 172.21.15.189:0/1091273164 wait complete. 2024-09-16T06:39:48.375 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' already exists 2024-09-16T06:39:48.384 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foooo 123 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.487+0000 7f444d8eb700 10 monclient: get_monmap_and_config 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.487+0000 7f444d8eb700 10 monclient: build_initial_monmap 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: monmap: 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:48.492540+0000 2024-09-16T06:39:48.491 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:48.492540+0000 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:48.492 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 Processor -- start 2024-09-16T06:39:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- start start 2024-09-16T06:39:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: init 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-b con 0x7f44481321f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-a con 0x7f444806bac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-c con 0x7f444813bc00 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444806a340 con 0x7f444806bac0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444806bf00 con 0x7f444813bc00 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44480a3930 con 0x7f44481321f0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient(hunting): authenticate will time out at 941.005852s 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f4447fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f444806bac0 legacy=0x7f444813a4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d0ea700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f444813bc00 0x7f444813dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45078/0 (socket says 172.21.15.189:45078) 2024-09-16T06:39:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d0ea700 1 -- 172.21.15.189:0/4153094944 learned_addr learned my addr 172.21.15.189:0/4153094944 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353118248 0 0) 0x7f444806bf00 con 0x7f444813bc00 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): my global_id is 94297 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44300035d0 con 0x7f444813bc00 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248725297 0 0) 0x7f44480a3930 con 0x7f44481321f0 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient(hunting): my global_id is 4658 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f444806bf00 con 0x7f44481321f0 2024-09-16T06:39:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226466714 0 0) 0x7f44300035d0 con 0x7f444813bc00 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 >> v1:172.21.15.76:6790/0 conn(0x7f44481321f0 legacy=0x7f44481325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 >> v1:172.21.15.76:6789/0 conn(0x7f444806bac0 legacy=0x7f444813a4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 monclient: found mon.noname-c 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444806a820 con 0x7f444813bc00 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4440001990 con 0x7f444813bc00 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 5 monclient: authenticate success, global_id 94297 2024-09-16T06:39:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: dump: 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4440001bb0 con 0x7f444813bc00 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 1 -- 172.21.15.189:0/4153094944 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4440001f40 con 0x7f444813bc00 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f44477fe700 10 monclient: dump: 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 4 set_mon_vals no callback set 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 10 monclient: shutdown 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- 172.21.15.189:0/4153094944 >> v1:172.21.15.189:6789/0 conn(0x7f444813bc00 legacy=0x7f444813dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- 172.21.15.189:0/4153094944 shutdown_connections 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.491+0000 7f444d8eb700 1 -- 172.21.15.189:0/4153094944 >> 172.21.15.189:0/4153094944 conn(0x7f44480a4280 msgr2=0x7f44480a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- 172.21.15.189:0/4153094944 shutdown_connections 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- 172.21.15.189:0/4153094944 wait complete. 2024-09-16T06:39:48.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: build_initial_monmap 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: monmap: 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 Processor -- start 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- start start 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: init 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-b con 0x7f444806bac0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-a con 0x7f44481321f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient(hunting): picked mon.noname-c con 0x7f444813bc00 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44481370d0 con 0x7f44481321f0 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444809ddb0 con 0x7f444806bac0 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444809e790 con 0x7f444813bc00 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient(hunting): authenticate will time out at 941.010129s 2024-09-16T06:39:48.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f4447fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f44481321f0 legacy=0x7f44481369c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444c8e9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f444806bac0 0x7f44481362b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45080/0 (socket says 172.21.15.189:45080) 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444c8e9700 1 -- 172.21.15.189:0/526696549 learned_addr learned my addr 172.21.15.189:0/526696549 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875076064 0 0) 0x7f444809ddb0 con 0x7f444806bac0 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): my global_id is 94300 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44340035d0 con 0x7f444806bac0 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145572879 0 0) 0x7f444809e790 con 0x7f444813bc00 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): my global_id is 4661 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f444809ddb0 con 0x7f444813bc00 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1785244003 0 0) 0x7f44340035d0 con 0x7f444806bac0 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f444809e790 con 0x7f444806bac0 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4438001970 con 0x7f444806bac0 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): dump: 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266770037 0 0) 0x7f444809e790 con 0x7f444806bac0 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 >> v1:172.21.15.76:6790/0 conn(0x7f444813bc00 legacy=0x7f44481d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 >> v1:172.21.15.76:6789/0 conn(0x7f44481321f0 legacy=0x7f44481369c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 monclient: found mon.b 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448138510 con 0x7f444806bac0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 5 monclient: authenticate success, global_id 94300 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _renew_subs 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44481387f0 con 0x7f444806bac0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _renew_subs 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4448138150 con 0x7f444806bac0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4438001ce0 con 0x7f444806bac0 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4438003b10 con 0x7f444806bac0 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44467fc700 4 set_mon_vals no callback set 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.495+0000 7f44457fa700 10 monclient: dump: 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.515 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4438003cf0 con 0x7f444806bac0 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(31..31 src has 1..31) v4 ==== 4290+0+0 (unknown 1179479754 0 0) 0x7f4438043680 con 0x7f444806bac0 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f4447fff700 10 monclient: get_auth_request con 0x7f4434036180 auth_method 0 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f444d8eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f444d8eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f444d8eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.499+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4428005100 con 0x7f444806bac0 2024-09-16T06:39:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.507+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4438014720 con 0x7f444806bac0 2024-09-16T06:39:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.507+0000 7f44457fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.507+0000 7f44457fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:48.630 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:39:48.630 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:39:48.630 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:39:48.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:39:48.633 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:39:48.633 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:39:48.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:48.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.679+0000 7f444d8eb700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:48.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.679+0000 7f444d8eb700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:48.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.679+0000 7f444d8eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:48.679+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f4428005930 con 0x7f444806bac0 2024-09-16T06:39:48.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: get_monmap_and_config 2024-09-16T06:39:48.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: build_initial_monmap 2024-09-16T06:39:48.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: monmap: 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:48.753140+0000 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:48.753140+0000 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:48.752 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 1 Processor -- start 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 1 -- start start 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: init 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-c con 0x7f35ec070d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-a con 0x7f35ec09e920 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-b con 0x7f35ec0a0240 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec073f10 con 0x7f35ec09e920 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec09efc0 con 0x7f35ec070d10 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec09f2a0 con 0x7f35ec0a0240 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:48.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f255e700 10 monclient(hunting): authenticate will time out at 935.887266s 2024-09-16T06:39:48.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f1d5d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f35ec0a0240 0x7f35ec13db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49622/0 (socket says 172.21.15.76:49622) 2024-09-16T06:39:48.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.751+0000 7f35f1d5d700 1 -- 172.21.15.76:0/1680902172 learned_addr learned my addr 172.21.15.76:0/1680902172 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563617931 0 0) 0x7f35ec09efc0 con 0x7f35ec070d10 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 10 monclient(hunting): my global_id is 94303 2024-09-16T06:39:48.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.775+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d40035d0 con 0x7f35ec070d10 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 923600013 0 0) 0x7f35d40035d0 con 0x7f35ec070d10 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 >> v1:172.21.15.76:6790/0 conn(0x7f35ec0a0240 legacy=0x7f35ec13db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 >> v1:172.21.15.76:6789/0 conn(0x7f35ec09e920 legacy=0x7f35ec13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 monclient: found mon.noname-c 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec09f580 con 0x7f35ec070d10 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: _finish_auth 0 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:48.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35e4001990 con 0x7f35ec070d10 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 5 monclient: authenticate success, global_id 94303 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: dump: 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e4001c40 con 0x7f35ec070d10 2024-09-16T06:39:48.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 1 -- 172.21.15.76:0/1680902172 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35e4002010 con 0x7f35ec070d10 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 4 set_mon_vals no callback set 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35ebfff700 10 monclient: dump: 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:48.783 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.784 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.784 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.784 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 10 monclient: shutdown 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 1 -- 172.21.15.76:0/1680902172 >> v1:172.21.15.189:6789/0 conn(0x7f35ec070d10 legacy=0x7f35ec0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 1 -- 172.21.15.76:0/1680902172 shutdown_connections 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 1 -- 172.21.15.76:0/1680902172 >> 172.21.15.76:0/1680902172 conn(0x7f35ec0a4430 msgr2=0x7f35ec0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 1 -- 172.21.15.76:0/1680902172 shutdown_connections 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 1 -- 172.21.15.76:0/1680902172 wait complete. 2024-09-16T06:39:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 10 monclient: build_initial_monmap 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.779+0000 7f35f255e700 10 monclient: monmap: 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 Processor -- start 2024-09-16T06:39:48.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- start start 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: init 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-c con 0x7f35ec070d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-a con 0x7f35ec0a0240 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient(hunting): picked mon.noname-b con 0x7f35ec137ee0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec13acd0 con 0x7f35ec0a0240 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec09b7d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec09bd20 con 0x7f35ec070d10 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient(hunting): authenticate will time out at 935.917483s 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f0d5b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f35ec0a0240 0x7f35ec1377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39340/0 (socket says 172.21.15.76:39340) 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f0d5b700 1 -- 172.21.15.76:0/2020207914 learned_addr learned my addr 172.21.15.76:0/2020207914 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:48.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538301491 0 0) 0x7f35ec09b7d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): my global_id is 94306 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d80035d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667887019 0 0) 0x7f35d80035d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35ec09b7d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35e0001720 con 0x7f35ec137ee0 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3064586415 0 0) 0x7f35ec09b7d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 >> v1:172.21.15.76:6790/0 conn(0x7f35ec070d10 legacy=0x7f35ec1370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 >> v1:172.21.15.76:6789/0 conn(0x7f35ec0a0240 legacy=0x7f35ec1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 monclient: found mon.b 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec09b0f0 con 0x7f35ec137ee0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 5 monclient: authenticate success, global_id 94306 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _renew_subs 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec138900 con 0x7f35ec137ee0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _renew_subs 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec1d38f0 con 0x7f35ec137ee0 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e0001d20 con 0x7f35ec137ee0 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35e0004a30 con 0x7f35ec137ee0 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35e9ffb700 10 monclient: dump: 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:48.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.783+0000 7f35eb7fe700 4 set_mon_vals no callback set 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f35e0004c10 con 0x7f35ec137ee0 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(31..31 src has 1..31) v4 ==== 4290+0+0 (unknown 1179479754 0 0) 0x7f35e0043490 con 0x7f35ec137ee0 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35f155c700 10 monclient: get_auth_request con 0x7f35d8036150 auth_method 0 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35f255e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35f255e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35f255e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.787+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec0678b0 con 0x7f35ec137ee0 2024-09-16T06:39:48.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.799+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f35e00144d0 con 0x7f35ec137ee0 2024-09-16T06:39:48.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.799+0000 7f35e9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:48.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:48.799+0000 7f35e9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:49.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:49.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f35ec13a550 con 0x7f35ec137ee0 2024-09-16T06:39:49.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35e9ffb700 1 -- 172.21.15.76:0/2020207914 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1418067397) 0x7f35e001cd50 con 0x7f35ec137ee0 2024-09-16T06:39:49.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35e9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:49.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35e9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:49.036 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 >> v1:172.21.15.189:6812/3098810674 conn(0x7f35d8036150 legacy=0x7f35d8038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 10 monclient: shutdown 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 >> v1:172.21.15.189:6789/0 conn(0x7f35ec137ee0 legacy=0x7f35ec1d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 shutdown_connections 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 >> 172.21.15.76:0/2020207914 conn(0x7f35ec0a4430 msgr2=0x7f35ec09c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 shutdown_connections 2024-09-16T06:39:49.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.035+0000 7f35f255e700 1 -- 172.21.15.76:0/2020207914 wait complete. 2024-09-16T06:39:49.050 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:39:49.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:39:49.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:39:49.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: monmap: 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:49.160353+0000 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:49.160353+0000 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 Processor -- start 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- start start 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: init 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:49.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c070d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c09e920 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c071800 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c073f50 con 0x7f7e1c09e920 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c09efc0 con 0x7f7e1c070d10 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c09f2a0 con 0x7f7e1c071800 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient(hunting): authenticate will time out at 936.294006s 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7e1c071800 0x7f7e1c13db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49628/0 (socket says 172.21.15.76:49628) 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1bfff700 1 -- 172.21.15.76:0/543198420 learned_addr learned my addr 172.21.15.76:0/543198420 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535573862 0 0) 0x7f7e1c09f2a0 con 0x7f7e1c071800 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): my global_id is 4664 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e040035d0 con 0x7f7e1c071800 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400681642 0 0) 0x7f7e1c073f50 con 0x7f7e1c09e920 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): my global_id is 104097 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c09f2a0 con 0x7f7e1c09e920 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461437763 0 0) 0x7f7e1c09efc0 con 0x7f7e1c070d10 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient(hunting): my global_id is 94309 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c073f50 con 0x7f7e1c070d10 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471978735 0 0) 0x7f7e1c09f2a0 con 0x7f7e1c09e920 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 >> v1:172.21.15.76:6790/0 conn(0x7f7e1c071800 legacy=0x7f7e1c13db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 >> v1:172.21.15.189:6789/0 conn(0x7f7e1c070d10 legacy=0x7f7e1c0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0a4ab0 con 0x7f7e1c09e920 2024-09-16T06:39:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e0c0015f0 con 0x7f7e1c09e920 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: dump: 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 5 monclient: authenticate success, global_id 104097 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e0c001910 con 0x7f7e1c09e920 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 4 set_mon_vals no callback set 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 1 -- 172.21.15.76:0/543198420 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e0c001ce0 con 0x7f7e1c09e920 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e1a7fc700 10 monclient: dump: 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 10 monclient: shutdown 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- 172.21.15.76:0/543198420 >> v1:172.21.15.76:6789/0 conn(0x7f7e1c09e920 legacy=0x7f7e1c13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- 172.21.15.76:0/543198420 shutdown_connections 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- 172.21.15.76:0/543198420 >> 172.21.15.76:0/543198420 conn(0x7f7e1c0a4430 msgr2=0x7f7e1c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- 172.21.15.76:0/543198420 shutdown_connections 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.159+0000 7f7e20dc3700 1 -- 172.21.15.76:0/543198420 wait complete. 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: monmap: 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:49.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 Processor -- start 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- start start 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: init 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c070d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c071800 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c1cf8e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c13d4e0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c130100 con 0x7f7e1c071800 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c13a7e0 con 0x7f7e1c070d10 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient(hunting): authenticate will time out at 936.297766s 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e1b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7e1c070d10 0x7f7e1c13ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49632/0 (socket says 172.21.15.76:49632) 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e1b7fe700 1 -- 172.21.15.76:0/3540604271 learned_addr learned my addr 172.21.15.76:0/3540604271 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282451774 0 0) 0x7f7e1c13d4e0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): my global_id is 104100 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e080035d0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135052970 0 0) 0x7f7e1c13a7e0 con 0x7f7e1c070d10 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): my global_id is 4667 2024-09-16T06:39:49.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c13d4e0 con 0x7f7e1c070d10 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744093358 0 0) 0x7f7e1c130100 con 0x7f7e1c071800 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): my global_id is 94312 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c13a7e0 con 0x7f7e1c071800 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906506069 0 0) 0x7f7e080035d0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e1c130100 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e14001490 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): dump: 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.179 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993749129 0 0) 0x7f7e1c13d4e0 con 0x7f7e1c070d10 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e080035d0 con 0x7f7e1c070d10 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e10001990 con 0x7f7e1c070d10 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient(hunting): dump: 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013461646 0 0) 0x7f7e1c130100 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6790/0 conn(0x7f7e1c070d10 legacy=0x7f7e1c13ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.189:6789/0 conn(0x7f7e1c071800 legacy=0x7f7e1c1ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 monclient: found mon.a 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c12fbb0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 5 monclient: authenticate success, global_id 104100 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _renew_subs 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c09b0f0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _renew_subs 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c138480 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e140019c0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e14004db0 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e19ffb700 4 set_mon_vals no callback set 2024-09-16T06:39:49.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.163+0000 7f7e03fff700 10 monclient: dump: 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7e1400f820 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e20dc3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7dfc006590 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(31..31 src has 1..31) v4 ==== 4290+0+0 (unknown 1179479754 0 0) 0x7f7e14043660 con 0x7f7e1c1cf8e0 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e1affd700 10 monclient: get_auth_request con 0x7f7e080364a0 auth_method 0 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287922451 0 0) 0x7f7dfc006590 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient(hunting): my global_id is 104103 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c130100 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3337943043 0 0) 0x7f7e1c130100 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dfc006590 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e10003b90 con 0x7f7dfc002830 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient: dump: 2024-09-16T06:39:49.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.186 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4069864125 0 0) 0x7f7dfc006590 con 0x7f7dfc002830 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dfc00a190 con 0x7f7dfc002830 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7e10005870 con 0x7f7dfc002830 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6789/0 conn(0x7f7dfc002830 legacy=0x7f7dfc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.167+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6789/0 conn(0x7f7dfc002830 legacy=0x7f7dfc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7dfc009e30 con 0x7f7dfc009150 2024-09-16T06:39:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279140723 0 0) 0x7f7dfc009e30 con 0x7f7dfc009150 2024-09-16T06:39:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient(hunting): my global_id is 104106 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc006590 con 0x7f7dfc009150 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1180387258 0 0) 0x7f7dfc006590 con 0x7f7dfc009150 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dfc009e30 con 0x7f7dfc009150 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7e100042a0 con 0x7f7dfc009150 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient: dump: 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.190 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2633074268 0 0) 0x7f7dfc009e30 con 0x7f7dfc009150 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7dfc00c340 con 0x7f7dfc009150 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3174396740) 0x7f7e100059e0 con 0x7f7dfc009150 2024-09-16T06:39:49.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6789/0 conn(0x7f7dfc009150 legacy=0x7f7dfc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e03fff700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6789/0 conn(0x7f7dfc009150 legacy=0x7f7dfc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:49.192 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.193 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:49.194 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:49.195 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:49.196 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.197 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.198 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.200 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:49.201 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.202 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.205 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:49.205 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:49.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7e080364a0 legacy=0x7f7e08038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 10 monclient: shutdown 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 >> v1:172.21.15.76:6789/0 conn(0x7f7e1c1cf8e0 legacy=0x7f7e1c1d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e1bfff700 1 -- 172.21.15.76:0/3540604271 reap_dead start 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e1bfff700 1 -- 172.21.15.76:0/3540604271 reap_dead start 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 shutdown_connections 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 >> 172.21.15.76:0/3540604271 conn(0x7f7e1c0a4430 msgr2=0x7f7e1c1313b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 shutdown_connections 2024-09-16T06:39:49.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.179+0000 7f7e20dc3700 1 -- 172.21.15.76:0/3540604271 wait complete. 2024-09-16T06:39:49.209 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:49.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:39:49.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:39:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.259+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v32) v1 ==== 115+0+0 (unknown 4150984509 0 0) 0x7f4438012dc0 con 0x7f444806bac0 2024-09-16T06:39:49.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.259+0000 7f44457fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:49.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.259+0000 7f44457fa700 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: monmap: 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:49.347979+0000 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:49.347979+0000 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.348 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 Processor -- start 2024-09-16T06:39:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- start start 2024-09-16T06:39:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: init 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient(hunting): picked mon.noname-b con 0x7f3714070ac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient(hunting): picked mon.noname-c con 0x7f3714070ea0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient(hunting): picked mon.noname-a con 0x7f371413b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714072610 con 0x7f371413b980 2024-09-16T06:39:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371409f560 con 0x7f3714070ea0 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714072330 con 0x7f3714070ac0 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient(hunting): authenticate will time out at 936.481270s 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3712d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3714070ac0 0x7f3714097bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49634/0 (socket says 172.21.15.76:49634) 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3712d9d700 1 -- 172.21.15.76:0/3505812867 learned_addr learned my addr 172.21.15.76:0/3505812867 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204281792 0 0) 0x7f3714072330 con 0x7f3714070ac0 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): my global_id is 4670 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37000035d0 con 0x7f3714070ac0 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594909995 0 0) 0x7f3714072610 con 0x7f371413b980 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): my global_id is 104109 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714072330 con 0x7f371413b980 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940043719 0 0) 0x7f371409f560 con 0x7f3714070ea0 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient(hunting): my global_id is 94315 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714072610 con 0x7f3714070ea0 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687878608 0 0) 0x7f37000035d0 con 0x7f3714070ac0 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 >> v1:172.21.15.189:6789/0 conn(0x7f3714070ea0 legacy=0x7f37140712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 >> v1:172.21.15.76:6789/0 conn(0x7f371413b980 legacy=0x7f371413dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 monclient: found mon.noname-b 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140974f0 con 0x7f3714070ac0 2024-09-16T06:39:49.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001610 con 0x7f3714070ac0 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: dump: 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.361 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 5 monclient: authenticate success, global_id 4670 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37040018f0 con 0x7f3714070ac0 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 1 -- 172.21.15.76:0/3505812867 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001cc0 con 0x7f3714070ac0 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 4 set_mon_vals no callback set 2024-09-16T06:39:49.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3711d9b700 10 monclient: dump: 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: shutdown 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- 172.21.15.76:0/3505812867 >> v1:172.21.15.76:6790/0 conn(0x7f3714070ac0 legacy=0x7f3714097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- 172.21.15.76:0/3505812867 shutdown_connections 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- 172.21.15.76:0/3505812867 >> 172.21.15.76:0/3505812867 conn(0x7f37140a0030 msgr2=0x7f37140a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- 172.21.15.76:0/3505812867 shutdown_connections 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- 172.21.15.76:0/3505812867 wait complete. 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: monmap: 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.364 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 Processor -- start 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 1 -- start start 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.347+0000 7f3718953700 10 monclient: init 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient(hunting): picked mon.noname-a con 0x7f3714070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient(hunting): picked mon.noname-c con 0x7f3714070ea0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient(hunting): picked mon.noname-b con 0x7f371413b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371409a5b0 con 0x7f3714070ac0 2024-09-16T06:39:49.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714137770 con 0x7f371413b980 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371413a910 con 0x7f3714070ea0 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient(hunting): authenticate will time out at 936.484473s 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3712d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3714070ac0 0x7f3714136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39392/0 (socket says 172.21.15.76:39392) 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3712d9d700 1 -- 172.21.15.76:0/4090006501 learned_addr learned my addr 172.21.15.76:0/4090006501 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856608552 0 0) 0x7f37140974f0 con 0x7f3714070ac0 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): my global_id is 104112 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f40035d0 con 0x7f3714070ac0 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007546887 0 0) 0x7f371413a910 con 0x7f3714070ea0 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): my global_id is 4673 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37140974f0 con 0x7f3714070ea0 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391381933 0 0) 0x7f3714137770 con 0x7f371413b980 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): my global_id is 94318 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371413a910 con 0x7f371413b980 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781383547 0 0) 0x7f36f40035d0 con 0x7f3714070ac0 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3714137770 con 0x7f3714070ac0 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001b20 con 0x7f3714070ac0 2024-09-16T06:39:49.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476526282 0 0) 0x7f37140974f0 con 0x7f3714070ea0 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f40035d0 con 0x7f3714070ea0 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4064603397 0 0) 0x7f371413a910 con 0x7f371413b980 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37140974f0 con 0x7f371413b980 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36fc0015f0 con 0x7f3714070ea0 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f370c001b30 con 0x7f371413b980 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): dump: 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460481926 0 0) 0x7f3714137770 con 0x7f3714070ac0 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6790/0 conn(0x7f3714070ea0 legacy=0x7f3714137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.189:6789/0 conn(0x7f371413b980 legacy=0x7f37141d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 monclient: found mon.a 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3714139390 con 0x7f3714070ac0 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 5 monclient: authenticate success, global_id 104112 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _renew_subs 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3714139670 con 0x7f3714070ac0 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _renew_subs 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3714138060 con 0x7f3714070ac0 2024-09-16T06:39:49.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3704004ca0 con 0x7f3714070ac0 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704004fb0 con 0x7f3714070ac0 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: dump: 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.373 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f371159a700 4 set_mon_vals no callback set 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36f8006590 con 0x7f36f8002830 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f370400fa20 con 0x7f3714070ac0 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(32..32 src has 1..32) v4 ==== 4634+0+0 (unknown 3993429036 0 0) 0x7f3704043610 con 0x7f3714070ac0 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484328871 0 0) 0x7f36f8006590 con 0x7f36f8002830 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient(hunting): my global_id is 4676 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714137770 con 0x7f36f8002830 2024-09-16T06:39:49.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f371359e700 10 monclient: get_auth_request con 0x7f36f4036550 auth_method 0 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 381180506 0 0) 0x7f3714137770 con 0x7f36f8002830 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f8006590 con 0x7f36f8002830 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36fc003a30 con 0x7f36f8002830 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: dump: 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3419942969 0 0) 0x7f36f8006590 con 0x7f36f8002830 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36f800a190 con 0x7f36f8002830 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f36fc003d60 con 0x7f36f8002830 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6790/0 conn(0x7f36f8002830 legacy=0x7f36f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.351+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6790/0 conn(0x7f36f8002830 legacy=0x7f36f8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f3718953700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f36f8009e50 con 0x7f36f8009170 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821638460 0 0) 0x7f36f8009e50 con 0x7f36f8009170 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 10 monclient(hunting): my global_id is 4679 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.359+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f8006590 con 0x7f36f8009170 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3575573433 0 0) 0x7f36f8006590 con 0x7f36f8009170 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f8009e50 con 0x7f36f8009170 2024-09-16T06:39:49.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36fc0043b0 con 0x7f36f8009170 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 10 monclient: dump: 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.379 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3453988013 0 0) 0x7f36f8009e50 con 0x7f36f8009170 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f36f800c340 con 0x7f36f8009170 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1420107130) 0x7f36fc005730 con 0x7f36f8009170 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6790/0 conn(0x7f36f8009170 legacy=0x7f36f8009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f370b7fe700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6790/0 conn(0x7f36f8009170 legacy=0x7f36f8009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:39:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:49.392 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:49.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.189:6812/3098810674 conn(0x7f36f4036550 legacy=0x7f36f4038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 10 monclient: shutdown 2024-09-16T06:39:49.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 >> v1:172.21.15.76:6789/0 conn(0x7f3714070ac0 legacy=0x7f3714136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f371359e700 1 -- 172.21.15.76:0/4090006501 reap_dead start 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f371359e700 1 -- 172.21.15.76:0/4090006501 reap_dead start 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 shutdown_connections 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 >> 172.21.15.76:0/4090006501 conn(0x7f37140a0030 msgr2=0x7f371413dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 shutdown_connections 2024-09-16T06:39:49.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.363+0000 7f3718953700 1 -- 172.21.15.76:0/4090006501 wait complete. 2024-09-16T06:39:49.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:49.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:39:49.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:39:49.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.415+0000 7f444d8eb700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:49.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.415+0000 7f444d8eb700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:49.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.415+0000 7f444d8eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.415+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f44280053b0 con 0x7f444806bac0 2024-09-16T06:39:49.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f44457fa700 1 -- 172.21.15.189:0/526696549 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v32) v1 ==== 122+0+0 (unknown 1996310298 0 0) 0x7f4438014540 con 0x7f444806bac0 2024-09-16T06:39:49.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f44457fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T06:39:49.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f44457fa700 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4434036180 legacy=0x7f4434038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 10 monclient: shutdown 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 >> v1:172.21.15.189:6789/0 conn(0x7f444806bac0 legacy=0x7f44481362b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 shutdown_connections 2024-09-16T06:39:49.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 >> 172.21.15.189:0/526696549 conn(0x7f44480a4280 msgr2=0x7f44480a0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 shutdown_connections 2024-09-16T06:39:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.419+0000 7f444d8eb700 1 -- 172.21.15.189:0/526696549 wait complete. 2024-09-16T06:39:49.424 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foooo' already exists 2024-09-16T06:39:49.433 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create foo 123 2024-09-16T06:39:49.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: monmap: 2024-09-16T06:39:49.511 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:49.512921+0000 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:49.512921+0000 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.512 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 1 Processor -- start 2024-09-16T06:39:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 1 -- start start 2024-09-16T06:39:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: init 2024-09-16T06:39:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fc7600964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fc76009b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fc760137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc760098560 con 0x7fc760137e70 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc760099ec0 con 0x7fc76009b1c0 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76009b5f0 con 0x7fc7600964f0 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient(hunting): authenticate will time out at 936.646427s 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc7600964f0 0x7fc7600968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49666/0 (socket says 172.21.15.76:49666) 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75effd700 1 -- 172.21.15.76:0/1173906293 learned_addr learned my addr 172.21.15.76:0/1173906293 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089925303 0 0) 0x7fc760098560 con 0x7fc760137e70 2024-09-16T06:39:49.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): my global_id is 104115 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc74c0034f0 con 0x7fc760137e70 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723439168 0 0) 0x7fc76009b5f0 con 0x7fc7600964f0 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): my global_id is 4682 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc760098560 con 0x7fc7600964f0 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052393394 0 0) 0x7fc760099ec0 con 0x7fc76009b1c0 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient(hunting): my global_id is 94321 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76009b5f0 con 0x7fc76009b1c0 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785046428 0 0) 0x7fc74c0034f0 con 0x7fc760137e70 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 >> v1:172.21.15.76:6790/0 conn(0x7fc7600964f0 legacy=0x7fc7600968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 >> v1:172.21.15.189:6789/0 conn(0x7fc76009b1c0 legacy=0x7fc760136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760099be0 con 0x7fc760137e70 2024-09-16T06:39:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 5 monclient: authenticate success, global_id 104115 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001990 con 0x7fc760137e70 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: dump: 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc758001bf0 con 0x7fc760137e70 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 1 -- 172.21.15.76:0/1173906293 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001fc0 con 0x7fc760137e70 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75ffff700 4 set_mon_vals no callback set 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.511+0000 7fc75e7fc700 10 monclient: dump: 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.527 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: shutdown 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/1173906293 >> v1:172.21.15.76:6789/0 conn(0x7fc760137e70 legacy=0x7fc76013a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/1173906293 shutdown_connections 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/1173906293 >> 172.21.15.76:0/1173906293 conn(0x7fc7600a0030 msgr2=0x7fc7600a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/1173906293 shutdown_connections 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/1173906293 wait complete. 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: monmap: 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.528 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 Processor -- start 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- start start 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: init 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:49.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fc7600964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fc76009b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fc760137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc760093530 con 0x7fc760137e70 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7600919b0 con 0x7fc7600964f0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc760091eb0 con 0x7fc76009b1c0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient(hunting): authenticate will time out at 936.650294s 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc760137e70 0x7fc760090410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39396/0 (socket says 172.21.15.76:39396) 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75f7fe700 1 -- 172.21.15.76:0/2513774721 learned_addr learned my addr 172.21.15.76:0/2513774721 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc7567fc700 1 --1- 172.21.15.76:0/2513774721 >> v1:172.21.15.76:6790/0 conn(0x7fc76009b1c0 0x7fc76008fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49678/0 (socket says 172.21.15.76:49678) 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776378677 0 0) 0x7fc760091eb0 con 0x7fc76009b1c0 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): my global_id is 4685 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7400035d0 con 0x7fc76009b1c0 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551853314 0 0) 0x7fc760093530 con 0x7fc760137e70 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): my global_id is 104118 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc760091eb0 con 0x7fc760137e70 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876903361 0 0) 0x7fc7600919b0 con 0x7fc7600964f0 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): my global_id is 94324 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc760093530 con 0x7fc7600964f0 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3976344235 0 0) 0x7fc760093530 con 0x7fc7600964f0 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7600919b0 con 0x7fc7600964f0 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc750001250 con 0x7fc7600964f0 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): dump: 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159206790 0 0) 0x7fc7400035d0 con 0x7fc76009b1c0 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc760093530 con 0x7fc76009b1c0 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097421229 0 0) 0x7fc760091eb0 con 0x7fc760137e70 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7400035d0 con 0x7fc760137e70 2024-09-16T06:39:49.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc748001830 con 0x7fc76009b1c0 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): dump: 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001f00 con 0x7fc760137e70 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient(hunting): dump: 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.535 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2815063450 0 0) 0x7fc7600919b0 con 0x7fc7600964f0 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.76:6790/0 conn(0x7fc76009b1c0 legacy=0x7fc76008fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.76:6789/0 conn(0x7fc760137e70 legacy=0x7fc760090410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 monclient: found mon.b 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76009a540 con 0x7fc7600964f0 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 5 monclient: authenticate success, global_id 94324 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _renew_subs 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76009bee0 con 0x7fc7600964f0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _renew_subs 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc760092c90 con 0x7fc7600964f0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc750004890 con 0x7fc7600964f0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc750004ba0 con 0x7fc7600964f0 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc75dffb700 4 set_mon_vals no callback set 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.515+0000 7fc757fff700 10 monclient: dump: 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc75ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc75000f610 con 0x7fc7600964f0 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc744006590 con 0x7fc744002830 2024-09-16T06:39:49.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(32..32 src has 1..32) v4 ==== 4634+0+0 (unknown 3993429036 0 0) 0x7fc750043570 con 0x7fc7600964f0 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc75f7fe700 10 monclient: get_auth_request con 0x7fc7400365e0 auth_method 0 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528209865 0 0) 0x7fc744006590 con 0x7fc744002830 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient(hunting): my global_id is 94327 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7600919b0 con 0x7fc744002830 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4124155018 0 0) 0x7fc7600919b0 con 0x7fc744002830 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc744006590 con 0x7fc744002830 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc748003860 con 0x7fc744002830 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient: dump: 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1265003193 0 0) 0x7fc744006590 con 0x7fc744002830 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc74400a190 con 0x7fc744002830 2024-09-16T06:39:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc748001d70 con 0x7fc744002830 2024-09-16T06:39:49.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6789/0 conn(0x7fc744002830 legacy=0x7fc744004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.519+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6789/0 conn(0x7fc744002830 legacy=0x7fc744004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 10 monclient: monmap: 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:49.541804+0000 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:49.541804+0000 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.542 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 1 Processor -- start 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 1 -- start start 2024-09-16T06:39:49.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.539+0000 7ff966527700 10 monclient: init 2024-09-16T06:39:49.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc75ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:49.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc744009e30 con 0x7fc744009150 2024-09-16T06:39:49.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201811360 0 0) 0x7fc744009e30 con 0x7fc744009150 2024-09-16T06:39:49.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient(hunting): my global_id is 94330 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744006590 con 0x7fc744009150 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 451739073 0 0) 0x7fc744006590 con 0x7fc744009150 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc744009e30 con 0x7fc744009150 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc748003ee0 con 0x7fc744009150 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient: dump: 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2897614909 0 0) 0x7fc744009e30 con 0x7fc744009150 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc74400c340 con 0x7fc744009150 2024-09-16T06:39:49.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3016556631) 0x7fc748004240 con 0x7fc744009150 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6789/0 conn(0x7fc744009150 legacy=0x7fc7440095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.531+0000 7fc757fff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6789/0 conn(0x7fc744009150 legacy=0x7fc7440095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:49.559 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7400365e0 legacy=0x7fc740038aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 10 monclient: shutdown 2024-09-16T06:39:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 >> v1:172.21.15.189:6789/0 conn(0x7fc7600964f0 legacy=0x7fc76008f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75f7fe700 1 -- 172.21.15.76:0/2513774721 reap_dead start 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75f7fe700 1 -- 172.21.15.76:0/2513774721 reap_dead start 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 shutdown_connections 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 >> 172.21.15.76:0/2513774721 conn(0x7fc7600a0030 msgr2=0x7fc7600a1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 shutdown_connections 2024-09-16T06:39:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.535+0000 7fc75ffff700 1 -- 172.21.15.76:0/2513774721 wait complete. 2024-09-16T06:39:49.562 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:49.562 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:39:49.563 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:39:49.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-b con 0x7ff96009d830 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-a con 0x7ff96009dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-c con 0x7ff960137dd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96009a790 con 0x7ff96009dc10 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff965525700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff96009d830 0x7ff960131930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:34020/0 (socket says 172.21.15.189:34020) 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff965525700 1 -- 172.21.15.189:0/1373910303 learned_addr learned my addr 172.21.15.189:0/1373910303 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960071510 con 0x7ff960137dd0 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960132040 con 0x7ff96009d830 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): authenticate will time out at 942.056734s 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133183658 0 0) 0x7ff960132040 con 0x7ff96009d830 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): my global_id is 4688 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff94c0035d0 con 0x7ff96009d830 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706096500 0 0) 0x7ff9580013c0 con 0x7ff960137dd0 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): my global_id is 94333 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff960132040 con 0x7ff960137dd0 2024-09-16T06:39:49.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286954612 0 0) 0x7ff96009a790 con 0x7ff96009dc10 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient(hunting): my global_id is 104121 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9580013c0 con 0x7ff96009dc10 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278866948 0 0) 0x7ff94c0035d0 con 0x7ff96009d830 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 >> v1:172.21.15.189:6789/0 conn(0x7ff960137dd0 legacy=0x7ff96013a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 >> v1:172.21.15.76:6789/0 conn(0x7ff96009dc10 legacy=0x7ff9601366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 monclient: found mon.noname-b 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96009e060 con 0x7ff96009d830 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 5 monclient: authenticate success, global_id 4688 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff95c001610 con 0x7ff96009d830 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: dump: 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff95c0018f0 con 0x7ff96009d830 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 4 set_mon_vals no callback set 2024-09-16T06:39:49.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 1 -- 172.21.15.189:0/1373910303 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff95c001c80 con 0x7ff96009d830 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff957fff700 10 monclient: dump: 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: shutdown 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 >> v1:172.21.15.76:6790/0 conn(0x7ff96009d830 legacy=0x7ff960131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 shutdown_connections 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 >> 172.21.15.189:0/1373910303 conn(0x7ff9600a4320 msgr2=0x7ff9600a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 shutdown_connections 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- 172.21.15.189:0/1373910303 wait complete. 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: monmap: 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:49.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 Processor -- start 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- start start 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: init 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-c con 0x7ff96009d830 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-a con 0x7ff96009dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): picked mon.noname-b con 0x7ff960137dd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960136310 con 0x7ff96009dc10 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96009f700 con 0x7ff960137dd0 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960072210 con 0x7ff96009d830 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.543+0000 7ff966527700 10 monclient(hunting): authenticate will time out at 942.059875s 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff965d26700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff960137dd0 0x7ff96011b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45108/0 (socket says 172.21.15.189:45108) 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff965d26700 1 -- 172.21.15.189:0/2973884589 learned_addr learned my addr 172.21.15.189:0/2973884589 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962083461 0 0) 0x7ff960072210 con 0x7ff96009d830 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): my global_id is 4691 2024-09-16T06:39:49.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9440035d0 con 0x7ff96009d830 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091836883 0 0) 0x7ff96009f700 con 0x7ff960137dd0 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): my global_id is 94336 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff960072210 con 0x7ff960137dd0 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670345433 0 0) 0x7ff960136310 con 0x7ff96009dc10 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): my global_id is 104124 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96009f700 con 0x7ff96009dc10 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544917014 0 0) 0x7ff9440035d0 con 0x7ff96009d830 2024-09-16T06:39:49.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff960136310 con 0x7ff96009d830 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff95c001c00 con 0x7ff96009d830 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652639081 0 0) 0x7ff960072210 con 0x7ff960137dd0 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9440035d0 con 0x7ff960137dd0 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff958002050 con 0x7ff960137dd0 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814655078 0 0) 0x7ff96009f700 con 0x7ff96009dc10 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff960072210 con 0x7ff96009dc10 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff950001830 con 0x7ff96009dc10 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2263283228 0 0) 0x7ff960136310 con 0x7ff96009d830 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 >> v1:172.21.15.189:6789/0 conn(0x7ff960137dd0 legacy=0x7ff96011b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 >> v1:172.21.15.76:6789/0 conn(0x7ff96009dc10 legacy=0x7ff96011ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 monclient: found mon.c 2024-09-16T06:39:49.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96011e350 con 0x7ff96009d830 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 5 monclient: authenticate success, global_id 4691 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _renew_subs 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96011e630 con 0x7ff96009d830 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _renew_subs 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601d2ff0 con 0x7ff96009d830 2024-09-16T06:39:49.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff95c004ca0 con 0x7ff96009d830 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff95c001210 con 0x7ff96009d830 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff9577fe700 4 set_mon_vals no callback set 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 10 monclient: dump: 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.585 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff95c00f8c0 con 0x7ff96009d830 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(32..32 src has 1..32) v4 ==== 4634+0+0 (unknown 3993429036 0 0) 0x7ff95c0437e0 con 0x7ff96009d830 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff964d24700 10 monclient: get_auth_request con 0x7ff944036440 auth_method 0 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.547+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff940005100 con 0x7ff96009d830 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.559+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff95c0147f0 con 0x7ff96009d830 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.559+0000 7ff955ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.559+0000 7ff955ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.675+0000 7f161575f700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.675+0000 7f161575f700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.675+0000 7f161575f700 10 monclient: monmap: 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:49.679279+0000 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:49.679279+0000 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.678 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.675+0000 7f161575f700 1 Processor -- start 2024-09-16T06:39:49.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- start start 2024-09-16T06:39:49.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: init 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient(hunting): picked mon.noname-a con 0x7f161006e4d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient(hunting): picked mon.noname-c con 0x7f1610070cb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient(hunting): picked mon.noname-b con 0x7f16100a1b40 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161006fff0 con 0x7f161006e4d0 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16100a21a0 con 0x7f1610070cb0 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16100a2330 con 0x7f16100a1b40 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient(hunting): authenticate will time out at 936.812522s 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f161006e4d0 0x7f161006e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39398/0 (socket says 172.21.15.76:39398) 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f16100a1b40 0x7f161013bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49682/0 (socket says 172.21.15.76:49682) 2024-09-16T06:39:49.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160effd700 1 -- 172.21.15.76:0/1123944270 learned_addr learned my addr 172.21.15.76:0/1123944270 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964327781 0 0) 0x7f16100a2330 con 0x7f16100a1b40 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): my global_id is 4694 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f80035d0 con 0x7f16100a1b40 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901071870 0 0) 0x7f161006fff0 con 0x7f161006e4d0 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): my global_id is 104127 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16100a2330 con 0x7f161006e4d0 2024-09-16T06:39:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270459928 0 0) 0x7f16100a21a0 con 0x7f1610070cb0 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient(hunting): my global_id is 94339 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161006fff0 con 0x7f1610070cb0 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752036894 0 0) 0x7f15f80035d0 con 0x7f16100a1b40 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 >> v1:172.21.15.189:6789/0 conn(0x7f1610070cb0 legacy=0x7f16101384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 >> v1:172.21.15.76:6789/0 conn(0x7f161006e4d0 legacy=0x7f161006e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 monclient: found mon.noname-b 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161009fce0 con 0x7f16100a1b40 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 5 monclient: authenticate success, global_id 4694 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f16080019d0 con 0x7f16100a1b40 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: dump: 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1608001bf0 con 0x7f16100a1b40 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 4 set_mon_vals no callback set 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 1 -- 172.21.15.76:0/1123944270 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1608001f80 con 0x7f16100a1b40 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f160dffb700 10 monclient: dump: 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.689 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: shutdown 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- 172.21.15.76:0/1123944270 >> v1:172.21.15.76:6790/0 conn(0x7f16100a1b40 legacy=0x7f161013bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- 172.21.15.76:0/1123944270 shutdown_connections 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- 172.21.15.76:0/1123944270 >> 172.21.15.76:0/1123944270 conn(0x7f16100a25c0 msgr2=0x7f16100a29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- 172.21.15.76:0/1123944270 shutdown_connections 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 1 -- 172.21.15.76:0/1123944270 wait complete. 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.679+0000 7f161575f700 10 monclient: monmap: 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 Processor -- start 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- start start 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: init 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient(hunting): picked mon.noname-b con 0x7f161006e4d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient(hunting): picked mon.noname-a con 0x7f1610070cb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient(hunting): picked mon.noname-c con 0x7f16100a1b40 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1610072760 con 0x7f1610070cb0 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1610133290 con 0x7f161006e4d0 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1610131ca0 con 0x7f16100a1b40 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient(hunting): authenticate will time out at 936.817379s 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f160f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f16100a1b40 0x7f161012f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49698/0 (socket says 172.21.15.76:49698) 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f160f7fe700 1 -- 172.21.15.76:0/4189639400 learned_addr learned my addr 172.21.15.76:0/4189639400 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:49.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f160e7fc700 1 --1- 172.21.15.76:0/4189639400 >> v1:172.21.15.76:6789/0 conn(0x7f1610070cb0 0x7f161012f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39402/0 (socket says 172.21.15.76:39402) 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464368878 0 0) 0x7f1610131ca0 con 0x7f16100a1b40 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): my global_id is 4697 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc0035d0 con 0x7f16100a1b40 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408367620 0 0) 0x7f161006fff0 con 0x7f1610070cb0 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): my global_id is 104130 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1610131ca0 con 0x7f1610070cb0 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022701744 0 0) 0x7f1610133290 con 0x7f161006e4d0 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): my global_id is 94342 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161006fff0 con 0x7f161006e4d0 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217164369 0 0) 0x7f15fc0035d0 con 0x7f16100a1b40 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1610133290 con 0x7f16100a1b40 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1608001f00 con 0x7f16100a1b40 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:49.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): dump: 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014974457 0 0) 0x7f1610131ca0 con 0x7f1610070cb0 2024-09-16T06:39:49.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15fc0035d0 con 0x7f1610070cb0 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1600001690 con 0x7f1610070cb0 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient(hunting): dump: 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.700 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 674816843 0 0) 0x7f1610133290 con 0x7f16100a1b40 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 >> v1:172.21.15.189:6789/0 conn(0x7f161006e4d0 legacy=0x7f161012eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 >> v1:172.21.15.76:6789/0 conn(0x7f1610070cb0 legacy=0x7f161012f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 monclient: found mon.c 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610131f80 con 0x7f16100a1b40 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 5 monclient: authenticate success, global_id 4697 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _renew_subs 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1610132260 con 0x7f16100a1b40 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _renew_subs 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1610133bc0 con 0x7f16100a1b40 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1608004ec0 con 0x7f16100a1b40 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f160d7fa700 4 set_mon_vals no callback set 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1608005250 con 0x7f16100a1b40 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.683+0000 7f15f77fe700 10 monclient: dump: 2024-09-16T06:39:49.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f161575f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f161575f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f161575f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f0005100 con 0x7f16100a1b40 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f16080053b0 con 0x7f16100a1b40 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(32..32 src has 1..32) v4 ==== 4634+0+0 (unknown 3993429036 0 0) 0x7f16080438b0 con 0x7f16100a1b40 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.687+0000 7f160effd700 10 monclient: get_auth_request con 0x7f15fc036170 auth_method 0 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.691+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1608014c50 con 0x7f16100a1b40 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.691+0000 7f15f77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.691+0000 7f15f77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:49.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff966527700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff966527700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff966527700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7ff940005930 con 0x7ff96009d830 2024-09-16T06:39:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v32) v1 ==== 118+0+0 (unknown 1439792772 0 0) 0x7ff95c012e90 con 0x7ff96009d830 2024-09-16T06:39:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff955ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.731+0000 7ff955ffb700 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-16T06:39:49.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff966527700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff966527700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff966527700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7ff940005bc0 con 0x7ff96009d830 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff955ffb700 1 -- 172.21.15.189:0/2973884589 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v32) v1 ==== 118+0+0 (unknown 1439792772 0 0) 0x7ff95c012e90 con 0x7ff96009d830 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff955ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T06:39:49.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.831+0000 7ff955ffb700 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T06:39:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff944036440 legacy=0x7ff944038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 10 monclient: shutdown 2024-09-16T06:39:49.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 >> v1:172.21.15.76:6790/0 conn(0x7ff96009d830 legacy=0x7ff96011a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 shutdown_connections 2024-09-16T06:39:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 >> 172.21.15.189:0/2973884589 conn(0x7ff9600a4320 msgr2=0x7ff9601366d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 shutdown_connections 2024-09-16T06:39:49.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.835+0000 7ff966527700 1 -- 172.21.15.189:0/2973884589 wait complete. 2024-09-16T06:39:49.837 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' already exists 2024-09-16T06:39:49.846 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-16T06:39:49.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f161575f700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:39:49.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f161575f700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:39:49.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f161575f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:49.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f15f0005bc0 con 0x7f16100a1b40 2024-09-16T06:39:49.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f15f77fe700 1 -- 172.21.15.76:0/4189639400 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f16080130c0 con 0x7f16100a1b40 2024-09-16T06:39:49.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f15f77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:39:49.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f15f77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=10,kv=13,logm=57} crc {auth=1513501807,config=1097430493,health=4205800003,kv=1001081567,logm=1351220306}) 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=12,logm=57} crc {auth=4087508988,config=1097430493,health=305525474,kv=1414813804,logm=1910575008}) 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=10,kv=13,logm=57} crc {auth=1513501807,config=1097430493,health=4205800003,kv=1001081567,logm=1351220306}) 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.931+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=13,logm=58} crc {auth=1955667829,config=1097430493,health=286398009,kv=1001081567,logm=3672884443}) 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2607319278}) 2024-09-16T06:39:49.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649923165}) 2024-09-16T06:39:49.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2607319278}) 2024-09-16T06:39:49.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147224617}) 2024-09-16T06:39:49.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 >> v1:172.21.15.189:6812/3098810674 conn(0x7f15fc036170 legacy=0x7f15fc038630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 10 monclient: shutdown 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 >> v1:172.21.15.76:6790/0 conn(0x7f16100a1b40 legacy=0x7f161012f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 shutdown_connections 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 >> 172.21.15.76:0/4189639400 conn(0x7f16100a25c0 msgr2=0x7f16100a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 shutdown_connections 2024-09-16T06:39:49.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f161575f700 1 -- 172.21.15.76:0/4189639400 wait complete. 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140746556}) 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327094750}) 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140746556}) 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734482372}) 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.935+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433438746}) 2024-09-16T06:39:49.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=966933404}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433438746}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318315990}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1277876388,mdsmap=1531151339,mgr=663292968}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=3016800403,mdsmap=1531151339,mgr=3505470075}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1277876388,mdsmap=1531151339,mgr=663292968}) 2024-09-16T06:39:49.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.939+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=89,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=1} crc {logm=1514302039,mdsmap=1531151339,mgr=1877095224,mgr_metadata=1347778423,mgrstat=759586200}) 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_metadata=1,mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=2887348009,mgr_metadata=1347778423,mgrstat=1253859867,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=4256882498,osd_snap=2033388227,osdmap=3320760809}) 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mgr=2887348009,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2440537806,mon_config_key=961403829,monmap=1887442722,osd_metadata=2882727975,osd_pg_creating=4256882498,osd_snap=2033388227,osdmap=3538387744}) 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_metadata=1,mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=2887348009,mgr_metadata=1347778423,mgrstat=1253859867,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=4256882498,osd_snap=2033388227,osdmap=3320760809}) 2024-09-16T06:39:49.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=2887348009,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1253859867,mon_config_key=4233526517,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=4256882498,osd_snap=2033388227,osdmap=2266049259}) 2024-09-16T06:39:49.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=7} crc {osdmap=1093630606}) 2024-09-16T06:39:49.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=531756987}) 2024-09-16T06:39:49.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:39:49.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=7} crc {osdmap=1093630606}) 2024-09-16T06:39:49.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:39:49.943+0000 7f923232e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=8} crc {osdmap=531756987}) 2024-09-16T06:39:49.951 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:39:49.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.951+0000 7f25e2f6a700 10 monclient: get_monmap_and_config 2024-09-16T06:39:49.954 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.951+0000 7f25e2f6a700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.951+0000 7f25e2f6a700 10 monclient: monmap: 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:49.956296+0000 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:49.956296+0000 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:49.955 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 Processor -- start 2024-09-16T06:39:49.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- start start 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: init 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-a con 0x7f25dc09eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-b con 0x7f25dc09beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-c con 0x7f25dc09c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc099ed0 con 0x7f25dc09eba0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc09d2e0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc09ef80 con 0x7f25dc09beb0 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient(hunting): authenticate will time out at 942.469011s 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2769700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f25dc09c2e0 0x7f25dc13ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45124/0 (socket says 172.21.15.189:45124) 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2769700 1 -- 172.21.15.189:0/1696961017 learned_addr learned my addr 172.21.15.189:0/1696961017 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948517505 0 0) 0x7f25dc09d2e0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): my global_id is 94345 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c80035d0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798628031 0 0) 0x7f25dc09ef80 con 0x7f25dc09beb0 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): my global_id is 4700 2024-09-16T06:39:49.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc09d2e0 con 0x7f25dc09beb0 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183907884 0 0) 0x7f25dc099ed0 con 0x7f25dc09eba0 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient(hunting): my global_id is 104133 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc09ef80 con 0x7f25dc09eba0 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384631473 0 0) 0x7f25c80035d0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 >> v1:172.21.15.76:6790/0 conn(0x7f25dc09beb0 legacy=0x7f25dc137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 >> v1:172.21.15.76:6789/0 conn(0x7f25dc09eba0 legacy=0x7f25dc09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 monclient: found mon.noname-c 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc09cc70 con 0x7f25dc09c2e0 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 5 monclient: authenticate success, global_id 94345 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25d4001990 con 0x7f25dc09c2e0 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: dump: 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d4001c00 con 0x7f25dc09c2e0 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 4 set_mon_vals no callback set 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 1 -- 172.21.15.189:0/1696961017 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25d4001f90 con 0x7f25dc09c2e0 2024-09-16T06:39:49.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e0f66700 10 monclient: dump: 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: shutdown 2024-09-16T06:39:49.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1696961017 >> v1:172.21.15.189:6789/0 conn(0x7f25dc09c2e0 legacy=0x7f25dc13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1696961017 shutdown_connections 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1696961017 >> 172.21.15.189:0/1696961017 conn(0x7f25dc09fe70 msgr2=0x7f25dc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1696961017 shutdown_connections 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1696961017 wait complete. 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: build_initial_monmap 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.955+0000 7f25e2f6a700 10 monclient: monmap: 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:49.972 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 Processor -- start 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- start start 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: init 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-a con 0x7f25dc09beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:49.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-c con 0x7f25dc09c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient(hunting): picked mon.noname-b con 0x7f25dc09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0a12b0 con 0x7f25dc09beb0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc12b3e0 con 0x7f25dc09eba0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc12cf50 con 0x7f25dc09c2e0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient(hunting): authenticate will time out at 942.472932s 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2769700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f25dc09eba0 0x7f25dc1cd950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45130/0 (socket says 172.21.15.189:45130) 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2769700 1 -- 172.21.15.189:0/1483992203 learned_addr learned my addr 172.21.15.189:0/1483992203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790845239 0 0) 0x7f25dc12b3e0 con 0x7f25dc09eba0 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): my global_id is 94348 2024-09-16T06:39:49.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c00035d0 con 0x7f25dc09eba0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454126233 0 0) 0x7f25dc0a12b0 con 0x7f25dc09beb0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): my global_id is 104136 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc12b3e0 con 0x7f25dc09beb0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582832375 0 0) 0x7f25dc12cf50 con 0x7f25dc09c2e0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): my global_id is 4703 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc0a12b0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040587971 0 0) 0x7f25dc12b3e0 con 0x7f25dc09beb0 2024-09-16T06:39:49.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc12cf50 con 0x7f25dc09beb0 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25cc0018b0 con 0x7f25dc09beb0 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): dump: 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.976 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990739945 0 0) 0x7f25c00035d0 con 0x7f25dc09eba0 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc12b3e0 con 0x7f25dc09eba0 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25d4001950 con 0x7f25dc09eba0 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): dump: 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.977 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529521426 0 0) 0x7f25dc0a12b0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c00035d0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25d80016d0 con 0x7f25dc09c2e0 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient(hunting): dump: 2024-09-16T06:39:49.978 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 629900990 0 0) 0x7f25dc12cf50 con 0x7f25dc09beb0 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 >> v1:172.21.15.76:6790/0 conn(0x7f25dc09c2e0 legacy=0x7f25dc129940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 >> v1:172.21.15.189:6789/0 conn(0x7f25dc09eba0 legacy=0x7f25dc1cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 monclient: found mon.a 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc12a050 con 0x7f25dc09beb0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: _finish_auth 0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 5 monclient: authenticate success, global_id 104136 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _renew_subs 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc12c3d0 con 0x7f25dc09beb0 2024-09-16T06:39:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _renew_subs 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc12aed0 con 0x7f25dc09beb0 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25cc001c40 con 0x7f25dc09beb0 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25cc003b10 con 0x7f25dc09beb0 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 10 monclient: dump: 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d3fff700 4 set_mon_vals no callback set 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f25cc00e580 con 0x7f25dc09beb0 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(32..32 src has 1..32) v4 ==== 4634+0+0 (unknown 3993429036 0 0) 0x7f25cc042410 con 0x7f25dc09beb0 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.959+0000 7f25e1767700 10 monclient: get_auth_request con 0x7f25c0036540 auth_method 0 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.963+0000 7f25e2f6a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.963+0000 7f25e2f6a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.963+0000 7f25e2f6a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.963+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c4005100 con 0x7f25dc09beb0 2024-09-16T06:39:49.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.967+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f25cc013670 con 0x7f25dc09beb0 2024-09-16T06:39:49.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.967+0000 7f25d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:49.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:49.967+0000 7f25d27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:50.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.139+0000 7f25e2f6a700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:50.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.139+0000 7f25e2f6a700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:50.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.139+0000 7f25e2f6a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:50.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.139+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25c4005930 con 0x7f25dc09beb0 2024-09-16T06:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.263+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v33) v1 ==== 152+0+0 (unknown 2112903970 0 0) 0x7f25cc011ae0 con 0x7f25dc09beb0 2024-09-16T06:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.263+0000 7f25d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:50.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.263+0000 7f25d27fc700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-16T06:39:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.403+0000 7f25e2f6a700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.403+0000 7f25e2f6a700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.403+0000 7f25e2f6a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:50.403+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25c4005420 con 0x7f25dc09beb0 2024-09-16T06:39:50.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:39:50.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:51.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 10 monclient: get_monmap_and_config 2024-09-16T06:39:51.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 10 monclient: monmap: 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:51.071014+0000 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:51.071014+0000 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:51.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 1 Processor -- start 2024-09-16T06:39:51.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 1 -- start start 2024-09-16T06:39:51.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.066+0000 7faa564aa700 10 monclient: init 2024-09-16T06:39:51.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-a con 0x7faa50132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-b con 0x7faa5006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-c con 0x7faa5006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5006c280 con 0x7faa50132330 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5006bfa0 con 0x7faa5006cf50 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5009ace0 con 0x7faa5006d480 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient(hunting): authenticate will time out at 938.204428s 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa554a8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faa50132330 0x7faa50132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39418/0 (socket says 172.21.15.76:39418) 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa554a8700 1 -- 172.21.15.76:0/3573154552 learned_addr learned my addr 172.21.15.76:0/3573154552 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248047154 0 0) 0x7faa5006c280 con 0x7faa50132330 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): my global_id is 104139 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c0035d0 con 0x7faa50132330 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623918598 0 0) 0x7faa5009ace0 con 0x7faa5006d480 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): my global_id is 4706 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5006c280 con 0x7faa5006d480 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976693973 0 0) 0x7faa5006bfa0 con 0x7faa5006cf50 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient(hunting): my global_id is 94351 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5009ace0 con 0x7faa5006cf50 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534283257 0 0) 0x7faa3c0035d0 con 0x7faa50132330 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 >> v1:172.21.15.76:6790/0 conn(0x7faa5006d480 legacy=0x7faa5013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 >> v1:172.21.15.189:6789/0 conn(0x7faa5006cf50 legacy=0x7faa5013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 monclient: found mon.noname-a 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5009ae70 con 0x7faa50132330 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c001990 con 0x7faa50132330 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: dump: 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 5 monclient: authenticate success, global_id 104139 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa4c001c70 con 0x7faa50132330 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 4 set_mon_vals no callback set 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 1 -- 172.21.15.76:0/3573154552 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c001e60 con 0x7faa50132330 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa43fff700 10 monclient: dump: 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.082 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: shutdown 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- 172.21.15.76:0/3573154552 >> v1:172.21.15.76:6789/0 conn(0x7faa50132330 legacy=0x7faa50132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- 172.21.15.76:0/3573154552 shutdown_connections 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- 172.21.15.76:0/3573154552 >> 172.21.15.76:0/3573154552 conn(0x7faa500a41c0 msgr2=0x7faa500a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- 172.21.15.76:0/3573154552 shutdown_connections 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- 172.21.15.76:0/3573154552 wait complete. 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: monmap: 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 Processor -- start 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 1 -- start start 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.070+0000 7faa564aa700 10 monclient: init 2024-09-16T06:39:51.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-c con 0x7faa5006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-b con 0x7faa5006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient(hunting): picked mon.noname-a con 0x7faa50132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5012fb60 con 0x7faa50132330 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5012fdb0 con 0x7faa5006d480 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5012ead0 con 0x7faa5006cf50 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient(hunting): authenticate will time out at 938.208205s 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa554a8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faa5006cf50 0x7faa5013e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49718/0 (socket says 172.21.15.76:49718) 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa554a8700 1 -- 172.21.15.76:0/4162113430 learned_addr learned my addr 172.21.15.76:0/4162113430 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160613152 0 0) 0x7faa5012ead0 con 0x7faa5006cf50 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): my global_id is 4709 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa340035d0 con 0x7faa5006cf50 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774876091 0 0) 0x7faa5012fb60 con 0x7faa50132330 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): my global_id is 104142 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5012ead0 con 0x7faa50132330 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982673599 0 0) 0x7faa5012fdb0 con 0x7faa5006d480 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): my global_id is 94354 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5012fb60 con 0x7faa5006d480 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3792020181 0 0) 0x7faa340035d0 con 0x7faa5006cf50 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa5012fdb0 con 0x7faa5006cf50 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c004c10 con 0x7faa5006cf50 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1266367392 0 0) 0x7faa5012ead0 con 0x7faa50132330 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa340035d0 con 0x7faa50132330 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa480018b0 con 0x7faa50132330 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient(hunting): dump: 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.088 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910835751 0 0) 0x7faa5012fdb0 con 0x7faa5006cf50 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 >> v1:172.21.15.189:6789/0 conn(0x7faa5006d480 legacy=0x7faa501cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 >> v1:172.21.15.76:6789/0 conn(0x7faa50132330 legacy=0x7faa501d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 monclient: found mon.c 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5012c5f0 con 0x7faa5006cf50 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 5 monclient: authenticate success, global_id 4709 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _renew_subs 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5013ea70 con 0x7faa5006cf50 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _renew_subs 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa50130a70 con 0x7faa5006cf50 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa4c005020 con 0x7faa5006cf50 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c0053b0 con 0x7faa5006cf50 2024-09-16T06:39:51.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 10 monclient: dump: 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa437fe700 4 set_mon_vals no callback set 2024-09-16T06:39:51.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faa4c0055d0 con 0x7faa5006cf50 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(33..33 src has 1..33) v4 ==== 4634+0+0 (unknown 3589232567 0 0) 0x7faa4c043af0 con 0x7faa5006cf50 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa54ca7700 10 monclient: get_auth_request con 0x7faa340363b0 auth_method 0 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.074+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa30005100 con 0x7faa5006cf50 2024-09-16T06:39:51.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.086+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faa4c014c60 con 0x7faa5006cf50 2024-09-16T06:39:51.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.086+0000 7faa41ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.086+0000 7faa41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.267+0000 7f25d27fc700 1 -- 172.21.15.189:0/1483992203 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v34) v1 ==== 152+0+0 (unknown 4035156538 0 0) 0x7f25cc013440 con 0x7f25dc09beb0 2024-09-16T06:39:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.267+0000 7f25d27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T06:39:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.267+0000 7f25d27fc700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.279+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 >> v1:172.21.15.189:6812/3098810674 conn(0x7f25c0036540 legacy=0x7f25c0038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.279+0000 7f25e2f6a700 10 monclient: shutdown 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.279+0000 7f25e2f6a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.279+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 >> v1:172.21.15.76:6789/0 conn(0x7f25dc09beb0 legacy=0x7f25dc129230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.283+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 shutdown_connections 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.283+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 >> 172.21.15.189:0/1483992203 conn(0x7f25dc09fe70 msgr2=0x7f25dc1b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.283+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 shutdown_connections 2024-09-16T06:39:51.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.283+0000 7f25e2f6a700 1 -- 172.21.15.189:0/1483992203 wait complete. 2024-09-16T06:39:51.285 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 1 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 1 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=1 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T06:39:51.305 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T06:39:51.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:51.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:51.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faa30005930 con 0x7faa5006cf50 2024-09-16T06:39:51.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa41ffb700 1 -- 172.21.15.76:0/4162113430 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 545994058) 0x7faa4c014a80 con 0x7faa5006cf50 2024-09-16T06:39:51.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa41ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:51.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:51.339 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 >> v1:172.21.15.189:6812/3098810674 conn(0x7faa340363b0 legacy=0x7faa34038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 10 monclient: shutdown 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 >> v1:172.21.15.76:6790/0 conn(0x7faa5006cf50 legacy=0x7faa5013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 shutdown_connections 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 >> 172.21.15.76:0/4162113430 conn(0x7faa500a41c0 msgr2=0x7faa5009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 shutdown_connections 2024-09-16T06:39:51.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.338+0000 7faa564aa700 1 -- 172.21.15.76:0/4162113430 wait complete. 2024-09-16T06:39:51.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:39:51.352 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:39:51.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:39:51.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 10 monclient: get_monmap_and_config 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 10 monclient: monmap: 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.461 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:51.462903+0000 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:51.462903+0000 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:51.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 1 Processor -- start 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 1 -- start start 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.458+0000 7fe7bb045700 10 monclient: init 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:51.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-c con 0x7fe7ac01f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-a con 0x7fe7ac025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-b con 0x7fe7ac025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac01f8a0 con 0x7fe7ac025410 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac0233f0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac020050 con 0x7fe7ac025860 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient(hunting): authenticate will time out at 938.596152s 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7ba844700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe7ac025860 0x7fe7ac0c2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49724/0 (socket says 172.21.15.76:49724) 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9842700 1 --1- 172.21.15.76:0/2294078063 >> v1:172.21.15.76:6789/0 conn(0x7fe7ac025410 0x7fe7ac0bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39430/0 (socket says 172.21.15.76:39430) 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7ba844700 1 -- 172.21.15.76:0/2294078063 learned_addr learned my addr 172.21.15.76:0/2294078063 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726479517 0 0) 0x7fe7ac01f8a0 con 0x7fe7ac025410 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): my global_id is 104151 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a00035d0 con 0x7fe7ac025410 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188235675 0 0) 0x7fe7ac020050 con 0x7fe7ac025860 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): my global_id is 4718 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac01f8a0 con 0x7fe7ac025860 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708332651 0 0) 0x7fe7ac0233f0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient(hunting): my global_id is 94363 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac020050 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910363867 0 0) 0x7fe7a00035d0 con 0x7fe7ac025410 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 >> v1:172.21.15.76:6790/0 conn(0x7fe7ac025860 legacy=0x7fe7ac0c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 >> v1:172.21.15.189:6789/0 conn(0x7fe7ac01f4c0 legacy=0x7fe7ac024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 monclient: found mon.noname-a 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac021a90 con 0x7fe7ac025410 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 5 monclient: authenticate success, global_id 104151 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7a4001610 con 0x7fe7ac025410 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: dump: 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.469 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7a4001880 con 0x7fe7ac025410 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 4 set_mon_vals no callback set 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 1 -- 172.21.15.76:0/2294078063 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7a4001c10 con 0x7fe7ac025410 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7b9041700 10 monclient: dump: 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: shutdown 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- 172.21.15.76:0/2294078063 >> v1:172.21.15.76:6789/0 conn(0x7fe7ac025410 legacy=0x7fe7ac0bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- 172.21.15.76:0/2294078063 shutdown_connections 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- 172.21.15.76:0/2294078063 >> 172.21.15.76:0/2294078063 conn(0x7fe7ac005b40 msgr2=0x7fe7ac005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- 172.21.15.76:0/2294078063 shutdown_connections 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- 172.21.15.76:0/2294078063 wait complete. 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: monmap: 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 Processor -- start 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 1 -- start start 2024-09-16T06:39:51.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.462+0000 7fe7bb045700 10 monclient: init 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-c con 0x7fe7ac01f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-b con 0x7fe7ac025410 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient(hunting): picked mon.noname-a con 0x7fe7ac025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac0204a0 con 0x7fe7ac025860 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac14a6d0 con 0x7fe7ac025410 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac01efd0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient(hunting): authenticate will time out at 938.600230s 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7ba043700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe7ac01f4c0 0x7fe7ac149fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49738/0 (socket says 172.21.15.76:49738) 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7ba043700 1 -- 172.21.15.76:0/369218339 learned_addr learned my addr 172.21.15.76:0/369218339 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164484466 0 0) 0x7fe7ac01efd0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): my global_id is 4721 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7980035d0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371302949 0 0) 0x7fe7ac0204a0 con 0x7fe7ac025860 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): my global_id is 104154 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac01efd0 con 0x7fe7ac025860 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664447139 0 0) 0x7fe7ac14a6d0 con 0x7fe7ac025410 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): my global_id is 94366 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac0204a0 con 0x7fe7ac025410 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4023927309 0 0) 0x7fe7980035d0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7ac14a6d0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7b4069400 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3785761058 0 0) 0x7fe7ac01efd0 con 0x7fe7ac025860 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7980035d0 con 0x7fe7ac025860 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7b00018d0 con 0x7fe7ac025860 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.480 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2385315083 0 0) 0x7fe7ac14a6d0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.189:6789/0 conn(0x7fe7ac025410 legacy=0x7fe7ac155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6789/0 conn(0x7fe7ac025860 legacy=0x7fe7ac158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 monclient: found mon.c 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac01d320 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 5 monclient: authenticate success, global_id 4721 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _renew_subs 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac131940 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _renew_subs 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac12f600 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7b404ec20 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7b4066de0 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7abfff700 4 set_mon_vals no callback set 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: dump: 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.483 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe7b4066f40 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe79c006590 con 0x7fe79c002830 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7fe7b40bd160 con 0x7fe7ac01f4c0 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7b9842700 10 monclient: get_auth_request con 0x7fe798036240 auth_method 0 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875905882 0 0) 0x7fe79c006590 con 0x7fe79c002830 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient(hunting): my global_id is 104157 2024-09-16T06:39:51.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac14a6d0 con 0x7fe79c002830 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2494481678 0 0) 0x7fe7ac14a6d0 con 0x7fe79c002830 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe79c006590 con 0x7fe79c002830 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7b0004890 con 0x7fe79c002830 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 10 monclient: dump: 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 998330431 0 0) 0x7fe79c006590 con 0x7fe79c002830 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.466+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe79c00a190 con 0x7fe79c002830 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.470+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe7b0005980 con 0x7fe79c002830 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.470+0000 7fe7aaffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.470+0000 7fe7aaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.470+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6789/0 conn(0x7fe79c002830 legacy=0x7fe79c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.470+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6789/0 conn(0x7fe79c002830 legacy=0x7fe79c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7bb045700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe79c009e50 con 0x7fe79c009170 2024-09-16T06:39:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229840944 0 0) 0x7fe79c009e50 con 0x7fe79c009170 2024-09-16T06:39:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient(hunting): my global_id is 104160 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c006590 con 0x7fe79c009170 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3412854213 0 0) 0x7fe79c006590 con 0x7fe79c009170 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe79c009e50 con 0x7fe79c009170 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7b0004fc0 con 0x7fe79c009170 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 10 monclient: dump: 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1480728235 0 0) 0x7fe79c009e50 con 0x7fe79c009170 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.474+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe79c00c340 con 0x7fe79c009170 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2057801944) 0x7fe7b0005af0 con 0x7fe79c009170 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7aaffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7aaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6789/0 conn(0x7fe79c009170 legacy=0x7fe79c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7aaffd700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6789/0 conn(0x7fe79c009170 legacy=0x7fe79c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:39:51.490 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.491 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:51.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.494 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.495 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.498 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:51.500 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe798036240 legacy=0x7fe798038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 10 monclient: shutdown 2024-09-16T06:39:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 >> v1:172.21.15.76:6790/0 conn(0x7fe7ac01f4c0 legacy=0x7fe7ac149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7ba844700 1 -- 172.21.15.76:0/369218339 reap_dead start 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 shutdown_connections 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 >> 172.21.15.76:0/369218339 conn(0x7fe7ac005b40 msgr2=0x7fe7ac01b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 shutdown_connections 2024-09-16T06:39:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.478+0000 7fe7bb045700 1 -- 172.21.15.76:0/369218339 wait complete. 2024-09-16T06:39:51.502 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:51.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:39:51.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:39:51.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: get_monmap_and_config 2024-09-16T06:39:51.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: monmap: 2024-09-16T06:39:51.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:51.620807+0000 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:51.620807+0000 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 1 Processor -- start 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 1 -- start start 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: init 2024-09-16T06:39:51.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-a con 0x7f04b409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-c con 0x7f04b4098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-b con 0x7f04b4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b409d560 con 0x7f04b409ed60 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b409f140 con 0x7f04b4098be0 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b409d790 con 0x7f04b4099030 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient(hunting): authenticate will time out at 938.754096s 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f04b409ed60 0x7f04b40984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39474/0 (socket says 172.21.15.76:39474) 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b3fff700 1 -- 172.21.15.76:0/2827990856 learned_addr learned my addr 172.21.15.76:0/2827990856 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501472864 0 0) 0x7f04b409d790 con 0x7f04b4099030 2024-09-16T06:39:51.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): my global_id is 4724 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a00035d0 con 0x7f04b4099030 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249224790 0 0) 0x7f04b409f140 con 0x7f04b4098be0 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): my global_id is 94369 2024-09-16T06:39:51.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b409d790 con 0x7f04b4098be0 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893557820 0 0) 0x7f04b409d560 con 0x7f04b409ed60 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient(hunting): my global_id is 104163 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b409f140 con 0x7f04b409ed60 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566800874 0 0) 0x7f04a00035d0 con 0x7f04b4099030 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 >> v1:172.21.15.189:6789/0 conn(0x7f04b4098be0 legacy=0x7f04b41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 >> v1:172.21.15.76:6789/0 conn(0x7f04b409ed60 legacy=0x7f04b40984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 monclient: found mon.noname-b 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b409f620 con 0x7f04b4099030 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 5 monclient: authenticate success, global_id 4724 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04ac001990 con 0x7f04b4099030 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: dump: 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.640 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04ac001c70 con 0x7f04b4099030 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b9693700 4 set_mon_vals no callback set 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 1 -- 172.21.15.76:0/2827990856 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04ac002040 con 0x7f04b4099030 2024-09-16T06:39:51.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.618+0000 7f04b37fe700 10 monclient: dump: 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.642 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: shutdown 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- 172.21.15.76:0/2827990856 >> v1:172.21.15.76:6790/0 conn(0x7f04b4099030 legacy=0x7f04b413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- 172.21.15.76:0/2827990856 shutdown_connections 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- 172.21.15.76:0/2827990856 >> 172.21.15.76:0/2827990856 conn(0x7f04b40a0030 msgr2=0x7f04b40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- 172.21.15.76:0/2827990856 shutdown_connections 2024-09-16T06:39:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- 172.21.15.76:0/2827990856 wait complete. 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: monmap: 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:51.644 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 Processor -- start 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- start start 2024-09-16T06:39:51.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: init 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-b con 0x7f04b4098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-c con 0x7f04b4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient(hunting): picked mon.noname-a con 0x7f04b409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b408eba0 con 0x7f04b409ed60 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40986f0 con 0x7f04b4098be0 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b408de10 con 0x7f04b4099030 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b9693700 10 monclient(hunting): authenticate will time out at 938.759250s 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b8e92700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f04b409ed60 0x7f04b41d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39480/0 (socket says 172.21.15.76:39480) 2024-09-16T06:39:51.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b8e92700 1 -- 172.21.15.76:0/3009219824 learned_addr learned my addr 172.21.15.76:0/3009219824 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912524816 0 0) 0x7f04b408de10 con 0x7f04b4099030 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): my global_id is 4727 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04940035d0 con 0x7f04b4099030 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329890951 0 0) 0x7f04b408eba0 con 0x7f04b409ed60 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): my global_id is 104166 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b408de10 con 0x7f04b409ed60 2024-09-16T06:39:51.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967317876 0 0) 0x7f04b40986f0 con 0x7f04b4098be0 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 10 monclient(hunting): my global_id is 94372 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.622+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b408eba0 con 0x7f04b4098be0 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147803628 0 0) 0x7f04940035d0 con 0x7f04b4099030 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b40986f0 con 0x7f04b4099030 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f049c0017d0 con 0x7f04b4099030 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): dump: 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.648 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106245760 0 0) 0x7f04b408eba0 con 0x7f04b4098be0 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04940035d0 con 0x7f04b4098be0 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04a4001840 con 0x7f04b4098be0 2024-09-16T06:39:51.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): dump: 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1391834474 0 0) 0x7f04b408de10 con 0x7f04b409ed60 2024-09-16T06:39:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b408eba0 con 0x7f04b409ed60 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2582110484 0 0) 0x7f04b40986f0 con 0x7f04b4099030 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.189:6789/0 conn(0x7f04b4098be0 legacy=0x7f04b408cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6789/0 conn(0x7f04b409ed60 legacy=0x7f04b41d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 monclient: found mon.c 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4090f70 con 0x7f04b4099030 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 5 monclient: authenticate success, global_id 4727 2024-09-16T06:39:51.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 10 monclient: _renew_subs 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b409a540 con 0x7f04b4099030 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c0036c0 con 0x7f04b4099030 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 10 monclient: _renew_subs 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b2ffd700 4 set_mon_vals no callback set 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b40909b0 con 0x7f04b4099030 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f049c0039d0 con 0x7f04b4099030 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: dump: 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.652 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f049c00e440 con 0x7f04b4099030 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04b413bf80 con 0x7f04b413bba0 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7f049c042290 con 0x7f04b4099030 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875487232 0 0) 0x7f04b413bf80 con 0x7f04b413bba0 2024-09-16T06:39:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient(hunting): my global_id is 4730 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b40986f0 con 0x7f04b413bba0 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b8e92700 10 monclient: get_auth_request con 0x7f0494036520 auth_method 0 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2106577756 0 0) 0x7f04b40986f0 con 0x7f04b413bba0 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b413bf80 con 0x7f04b413bba0 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04a4004890 con 0x7f04b413bba0 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 10 monclient: dump: 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1057447667 0 0) 0x7f04b413bf80 con 0x7f04b413bba0 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.626+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b40a16a0 con 0x7f04b413bba0 2024-09-16T06:39:51.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.630+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f04a4006b10 con 0x7f04b413bba0 2024-09-16T06:39:51.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.630+0000 7f04b17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.630+0000 7f04b17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:51.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.630+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6790/0 conn(0x7f04b413bba0 legacy=0x7f04b4061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.630+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6790/0 conn(0x7f04b413bba0 legacy=0x7f04b4061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04b4090d50 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863879344 0 0) 0x7f04b4090d50 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient(hunting): my global_id is 4733 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b413bf80 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1200184251 0 0) 0x7f04b413bf80 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b4090d50 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04a4004fd0 con 0x7f04b402c820 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient: dump: 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 466029448 0 0) 0x7f04b4090d50 con 0x7f04b402c820 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04b409aaa0 con 0x7f04b402c820 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1915028065) 0x7f04a40063a0 con 0x7f04b402c820 2024-09-16T06:39:51.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6790/0 conn(0x7f04b402c820 legacy=0x7f04b40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b17fa700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6790/0 conn(0x7f04b402c820 legacy=0x7f04b40705d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:51.660 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.661 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.662 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:51.663 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.664 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.666 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.667 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.669 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.670 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:51.671 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.671 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.671 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:51.671 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0494036520 legacy=0x7f04940389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 10 monclient: shutdown 2024-09-16T06:39:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 >> v1:172.21.15.76:6790/0 conn(0x7f04b4099030 legacy=0x7f04b408d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b8e92700 1 -- 172.21.15.76:0/3009219824 reap_dead start 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b8e92700 1 -- 172.21.15.76:0/3009219824 reap_dead start 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 shutdown_connections 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 >> 172.21.15.76:0/3009219824 conn(0x7f04b40a0030 msgr2=0x7f04b406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.638+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 shutdown_connections 2024-09-16T06:39:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.642+0000 7f04b9693700 1 -- 172.21.15.76:0/3009219824 wait complete. 2024-09-16T06:39:51.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:51.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:39:51.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:39:51.678 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 1' 2024-09-16T06:39:51.678 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T06:39:51.679 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 1 2024-09-16T06:39:51.679 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T06:39:51.682 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:1 2024-09-16T06:39:51.683 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:1 2024-09-16T06:39:51.683 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T06:39:51.684 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=1 2024-09-16T06:39:51.684 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 1 2024-09-16T06:39:51.685 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T06:39:51.685 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=1 2024-09-16T06:39:51.685 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 1 != \1 ]] 2024-09-16T06:39:51.685 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 4 2024-09-16T06:39:51.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: get_monmap_and_config 2024-09-16T06:39:51.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: monmap: 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:39:51.786750+0000 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:39:51.786750+0000 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:51.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:51.787 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 1 Processor -- start 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 1 -- start start 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: init 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0a0f40 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc09d7a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc09dc20 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc135500 con 0x7fc4dc09dc20 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0a1320 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc099c90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.783+0000 7fc4e3360700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 10 monclient(hunting): authenticate will time out at 944.300010s 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e235e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4dc0a0f40 0x7fc4dc134df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45164/0 (socket says 172.21.15.189:45164) 2024-09-16T06:39:51.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e235e700 1 -- 172.21.15.189:0/803579728 learned_addr learned my addr 172.21.15.189:0/803579728 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190111295 0 0) 0x7fc4dc0a1320 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): my global_id is 94375 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c40035d0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739587779 0 0) 0x7fc4dc099c90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): my global_id is 4736 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc0a1320 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469699243 0 0) 0x7fc4dc135500 con 0x7fc4dc09dc20 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient(hunting): my global_id is 104169 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc099c90 con 0x7fc4dc09dc20 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119104805 0 0) 0x7fc4c40035d0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 >> v1:172.21.15.76:6790/0 conn(0x7fc4dc09d7a0 legacy=0x7fc4dc13a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 >> v1:172.21.15.76:6789/0 conn(0x7fc4dc09dc20 legacy=0x7fc4dc13d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 monclient: found mon.noname-c 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc099f70 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4cc001610 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: dump: 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.796 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 5 monclient: authenticate success, global_id 94375 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4cc0018f0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 1 -- 172.21.15.189:0/803579728 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4cc001c80 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e135c700 10 monclient: dump: 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:51.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 4 set_mon_vals no callback set 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 10 monclient: shutdown 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 1 -- 172.21.15.189:0/803579728 >> v1:172.21.15.189:6789/0 conn(0x7fc4dc0a0f40 legacy=0x7fc4dc134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 1 -- 172.21.15.189:0/803579728 shutdown_connections 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 1 -- 172.21.15.189:0/803579728 >> 172.21.15.189:0/803579728 conn(0x7fc4dc0a43d0 msgr2=0x7fc4dc0a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 1 -- 172.21.15.189:0/803579728 shutdown_connections 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 1 -- 172.21.15.189:0/803579728 wait complete. 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.787+0000 7fc4e3360700 10 monclient: monmap: 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:51.800 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:51.801 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 Processor -- start 2024-09-16T06:39:51.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- start start 2024-09-16T06:39:51.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: init 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc09d7a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc09dc20 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0a0f40 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc13b500 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc134a90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0721d0 con 0x7fc4dc09dc20 2024-09-16T06:39:51.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e235e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc4dc09d7a0 0x7fc4dc1377c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45172/0 (socket says 172.21.15.189:45172) 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e235e700 1 -- 172.21.15.189:0/417073816 learned_addr learned my addr 172.21.15.189:0/417073816 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient(hunting): authenticate will time out at 944.305783s 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896096172 0 0) 0x7fc4dc134a90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): my global_id is 94378 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c80035d0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174681343 0 0) 0x7fc4dc0721d0 con 0x7fc4dc09dc20 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): my global_id is 4739 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc134a90 con 0x7fc4dc09dc20 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477349703 0 0) 0x7fc4dc13b500 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): my global_id is 104172 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc0721d0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545054836 0 0) 0x7fc4dc134a90 con 0x7fc4dc09dc20 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4dc13b500 con 0x7fc4dc09dc20 2024-09-16T06:39:51.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4d00018b0 con 0x7fc4dc09dc20 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.814 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474933518 0 0) 0x7fc4c80035d0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4dc134a90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4cc0015d0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.815 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287889542 0 0) 0x7fc4dc0721d0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c80035d0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4d4001ae0 con 0x7fc4dc0a0f40 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.816 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849400717 0 0) 0x7fc4dc134a90 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 >> v1:172.21.15.76:6790/0 conn(0x7fc4dc09dc20 legacy=0x7fc4dc137ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 >> v1:172.21.15.76:6789/0 conn(0x7fc4dc0a0f40 legacy=0x7fc4dc1d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 monclient: found mon.b 2024-09-16T06:39:51.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc132d60 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 5 monclient: authenticate success, global_id 94378 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _renew_subs 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc13acc0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _renew_subs 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4cc004ca0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc1391a0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4cc004fb0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4dbfff700 4 set_mon_vals no callback set 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.791+0000 7fc4daffd700 10 monclient: dump: 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.819 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc4cc00fa20 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4e3360700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4e3360700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4e3360700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc0008d0 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7fc4cc043780 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.795+0000 7fc4e1b5d700 10 monclient: get_auth_request con 0x7fc4c8036540 auth_method 0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.803+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc4cc014790 con 0x7fc4dc09d7a0 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.803+0000 7fc4daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.803+0000 7fc4daffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.790+0000 7f148759e700 10 monclient: get_monmap_and_config 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.790+0000 7f148759e700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.790+0000 7f148759e700 10 monclient: monmap: 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:51.795522+0000 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:51.795522+0000 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 Processor -- start 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- start start 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient: init 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient(hunting): picked mon.noname-c con 0x7f148809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient(hunting): picked mon.noname-a con 0x7f1488098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient(hunting): picked mon.noname-b con 0x7f1488099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148809d560 con 0x7f1488098be0 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148809f140 con 0x7f148809ed60 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148809d790 con 0x7f1488099030 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient(hunting): authenticate will time out at 938.928892s 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1486d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1488099030 0x7f148813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:49766/0 (socket says 172.21.15.76:49766) 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1486d9d700 1 -- 172.21.15.76:0/3424793928 learned_addr learned my addr 172.21.15.76:0/3424793928 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110389304 0 0) 0x7f148809d790 con 0x7f1488099030 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): my global_id is 4742 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14680035d0 con 0x7f1488099030 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097119073 0 0) 0x7f148809f140 con 0x7f148809ed60 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): my global_id is 94381 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148809d790 con 0x7f148809ed60 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568412400 0 0) 0x7f148809d560 con 0x7f1488098be0 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient(hunting): my global_id is 104175 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148809f140 con 0x7f1488098be0 2024-09-16T06:39:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049824360 0 0) 0x7f148809d790 con 0x7f148809ed60 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 >> v1:172.21.15.76:6790/0 conn(0x7f1488099030 legacy=0x7f148813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 >> v1:172.21.15.76:6789/0 conn(0x7f1488098be0 legacy=0x7f14881374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 monclient: found mon.noname-c 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148809f620 con 0x7f148809ed60 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14700015f0 con 0x7f148809ed60 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: dump: 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 5 monclient: authenticate success, global_id 94381 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1470001910 con 0x7f148809ed60 2024-09-16T06:39:51.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 1 -- 172.21.15.76:0/3424793928 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1470001ce0 con 0x7f148809ed60 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f1485d9b700 10 monclient: dump: 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.831 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 4 set_mon_vals no callback set 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 10 monclient: shutdown 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- 172.21.15.76:0/3424793928 >> v1:172.21.15.189:6789/0 conn(0x7f148809ed60 legacy=0x7f14880984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- 172.21.15.76:0/3424793928 shutdown_connections 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- 172.21.15.76:0/3424793928 >> 172.21.15.76:0/3424793928 conn(0x7f14880a0030 msgr2=0x7f14880a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- 172.21.15.76:0/3424793928 shutdown_connections 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.794+0000 7f148759e700 1 -- 172.21.15.76:0/3424793928 wait complete. 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient: build_initial_monmap 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient: monmap: 2024-09-16T06:39:51.832 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 1 Processor -- start 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 1 -- start start 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient: init 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:51.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient(hunting): picked mon.noname-b con 0x7f1488098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient(hunting): picked mon.noname-a con 0x7f1488099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient(hunting): picked mon.noname-c con 0x7f148809ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14881b1cf0 con 0x7f1488099030 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14881b01c0 con 0x7f1488098be0 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148809a540 con 0x7f148809ed60 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f148759e700 10 monclient(hunting): authenticate will time out at 938.935242s 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1488099030 0x7f14881ca740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39506/0 (socket says 172.21.15.76:39506) 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147ffff700 1 -- 172.21.15.76:0/275371554 learned_addr learned my addr 172.21.15.76:0/275371554 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184418515 0 0) 0x7f14881b1cf0 con 0x7f1488099030 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): my global_id is 104178 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146c0035d0 con 0x7f1488099030 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022780916 0 0) 0x7f148809a540 con 0x7f148809ed60 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 10 monclient(hunting): my global_id is 4745 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.798+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14881b1cf0 con 0x7f148809ed60 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361546303 0 0) 0x7f146c0035d0 con 0x7f1488099030 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f148809a540 con 0x7f1488099030 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1478001960 con 0x7f1488099030 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:51.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104015421 0 0) 0x7f14881b1cf0 con 0x7f148809ed60 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f146c0035d0 con 0x7f148809ed60 2024-09-16T06:39:51.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1480001420 con 0x7f148809ed60 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): dump: 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3966683996 0 0) 0x7f148809a540 con 0x7f1488099030 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.76:6790/0 conn(0x7f148809ed60 legacy=0x7f14881cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6789/0 conn(0x7f1488098be0 legacy=0x7f14880963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 monclient: found mon.a 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14881b2590 con 0x7f1488099030 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: _finish_auth 0 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:51.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 5 monclient: authenticate success, global_id 104178 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 10 monclient: _renew_subs 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14880983f0 con 0x7f1488099030 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 10 monclient: _renew_subs 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:51.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14881b0fb0 con 0x7f1488099030 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1478001cd0 con 0x7f1488099030 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1478003af0 con 0x7f1488099030 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:51.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: dump: 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.841 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148559a700 4 set_mon_vals no callback set 2024-09-16T06:39:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1478003cd0 con 0x7f1488099030 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1460006590 con 0x7f1460002830 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7f14780436a0 con 0x7f1488099030 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f148659c700 10 monclient: get_auth_request con 0x7f146c036310 auth_method 0 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166201878 0 0) 0x7f1460006590 con 0x7f1460002830 2024-09-16T06:39:51.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient(hunting): my global_id is 94384 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148809a540 con 0x7f1460002830 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 720607825 0 0) 0x7f148809a540 con 0x7f1460002830 2024-09-16T06:39:51.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1460006590 con 0x7f1460002830 2024-09-16T06:39:51.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1480004c10 con 0x7f1460002830 2024-09-16T06:39:51.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: dump: 2024-09-16T06:39:51.845 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2058689932 0 0) 0x7f1460006590 con 0x7f1460002830 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f146000a190 con 0x7f1460002830 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1480004f80 con 0x7f1460002830 2024-09-16T06:39:51.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6789/0 conn(0x7f1460002830 legacy=0x7f1460004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.802+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6789/0 conn(0x7f1460002830 legacy=0x7f1460004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f148759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1460009e50 con 0x7f1460009170 2024-09-16T06:39:51.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107870310 0 0) 0x7f1460009e50 con 0x7f1460009170 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient(hunting): my global_id is 94387 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1460006590 con 0x7f1460009170 2024-09-16T06:39:51.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2083088047 0 0) 0x7f1460006590 con 0x7f1460009170 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1460009e50 con 0x7f1460009170 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14800054c0 con 0x7f1460009170 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient: dump: 2024-09-16T06:39:51.850 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:51.851 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2047225766 0 0) 0x7f1460009e50 con 0x7f1460009170 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f146000c340 con 0x7f1460009170 2024-09-16T06:39:51.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2485166424) 0x7f14800068b0 con 0x7f1460009170 2024-09-16T06:39:51.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:39:51.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:51.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6789/0 conn(0x7f1460009170 legacy=0x7f1460009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.814+0000 7f147effd700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6789/0 conn(0x7f1460009170 legacy=0x7f1460009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:39:51.854 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:39:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:39:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 24, 2024-09-16T06:39:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:39:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:39:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:39:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:39:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:39:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:39:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:39:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:39:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:39:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:39:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:39:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:39:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:39:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:39:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:39:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:39:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:39:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:39:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:39:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:39:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:39:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:39:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:39:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:39:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:39:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:39:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:39:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:39:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:39:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:39:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:39:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:39:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:39:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:39:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:39:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:39:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:39:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:39:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:39:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:39:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:39:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:39:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:39:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:39:51.879 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:39:51.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.189:6812/3098810674 conn(0x7f146c036310 legacy=0x7f146c0387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 10 monclient: shutdown 2024-09-16T06:39:51.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 >> v1:172.21.15.76:6789/0 conn(0x7f1488099030 legacy=0x7f14881ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f1486d9d700 1 -- 172.21.15.76:0/275371554 reap_dead start 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f1486d9d700 1 -- 172.21.15.76:0/275371554 reap_dead start 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 shutdown_connections 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 >> 172.21.15.76:0/275371554 conn(0x7f14880a0030 msgr2=0x7f14881b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 shutdown_connections 2024-09-16T06:39:51.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:51.818+0000 7f148759e700 1 -- 172.21.15.76:0/275371554 wait complete. 2024-09-16T06:39:51.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:39:51.883 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:39:51.883 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:39:51.883 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:39:51.883 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:39:51.884 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:39:51.884 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:39:51.927 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:39:51.927 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:39:51.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:51.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.971+0000 7fc4e3360700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:39:51.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.971+0000 7fc4e3360700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:39:51.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.971+0000 7fc4e3360700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:39:51.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:51.971+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7fc4dc13afa0 con 0x7fc4dc09d7a0 2024-09-16T06:39:52.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: get_monmap_and_config 2024-09-16T06:39:52.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: build_initial_monmap 2024-09-16T06:39:52.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: monmap: 2024-09-16T06:39:52.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:52.050041+0000 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:52.050041+0000 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:52.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 1 Processor -- start 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 1 -- start start 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: init 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient(hunting): picked mon.noname-b con 0x7f2098097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient(hunting): picked mon.noname-c con 0x7f2098099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient(hunting): picked mon.noname-a con 0x7f2098139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:52.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2098098990 con 0x7f2098139500 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2098097f40 con 0x7f2098099090 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209809d610 con 0x7f2098097b60 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209da25700 10 monclient(hunting): authenticate will time out at 939.183302s 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209ca23700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2098097b60 legacy=0x7f209809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209d224700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2098139500 0x7f209813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39544/0 (socket says 172.21.15.76:39544) 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f209d224700 1 -- 172.21.15.76:0/557254274 learned_addr learned my addr 172.21.15.76:0/557254274 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.046+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934143652 0 0) 0x7f2098098990 con 0x7f2098139500 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): my global_id is 104181 2024-09-16T06:39:52.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20800035d0 con 0x7f2098139500 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188440502 0 0) 0x7f2098097f40 con 0x7f2098099090 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient(hunting): my global_id is 94390 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2098098990 con 0x7f2098099090 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097730852 0 0) 0x7f20800035d0 con 0x7f2098139500 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 >> v1:172.21.15.76:6790/0 conn(0x7f2098097b60 legacy=0x7f209809ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 >> v1:172.21.15.189:6789/0 conn(0x7f2098099090 legacy=0x7f209809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: _finish_hunting 0 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 monclient: found mon.noname-a 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980986b0 con 0x7f2098139500 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: _finish_auth 0 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: _check_auth_tickets 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 5 monclient: authenticate success, global_id 104181 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2090001610 con 0x7f2098139500 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: dump: 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.059 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20900018f0 con 0x7f2098139500 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 4 set_mon_vals no callback set 2024-09-16T06:39:52.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 1 -- 172.21.15.76:0/557254274 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2090001c80 con 0x7f2098139500 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20977fe700 10 monclient: dump: 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:52.061 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: shutdown 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- 172.21.15.76:0/557254274 >> v1:172.21.15.76:6789/0 conn(0x7f2098139500 legacy=0x7f209813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- 172.21.15.76:0/557254274 shutdown_connections 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- 172.21.15.76:0/557254274 >> 172.21.15.76:0/557254274 conn(0x7f20980a0030 msgr2=0x7f20980a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- 172.21.15.76:0/557254274 shutdown_connections 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- 172.21.15.76:0/557254274 wait complete. 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: build_initial_monmap 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: monmap: 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:52.062 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 Processor -- start 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- start start 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: init 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:39:52.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient(hunting): picked mon.noname-b con 0x7f2098097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient(hunting): picked mon.noname-c con 0x7f2098099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient(hunting): picked mon.noname-a con 0x7f2098139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209813c120 con 0x7f2098139500 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980701d0 con 0x7f2098097b60 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f2097fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2098099090 legacy=0x7f20981d2f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980a2d30 con 0x7f2098099090 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209da25700 10 monclient(hunting): authenticate will time out at 939.187008s 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209d224700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2098139500 0x7f20981d6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39548/0 (socket says 172.21.15.76:39548) 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f209d224700 1 -- 172.21.15.76:0/1929515291 learned_addr learned my addr 172.21.15.76:0/1929515291 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:52.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697034100 0 0) 0x7f209813c120 con 0x7f2098139500 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): my global_id is 104184 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20840035d0 con 0x7f2098139500 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014642673 0 0) 0x7f20980701d0 con 0x7f2098097b60 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): my global_id is 94393 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209813c120 con 0x7f2098097b60 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3579364876 0 0) 0x7f20840035d0 con 0x7f2098139500 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20980701d0 con 0x7f2098139500 2024-09-16T06:39:52.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2090001c00 con 0x7f2098139500 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): dump: 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.050+0000 7f20957fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3812914352 0 0) 0x7f20980701d0 con 0x7f2098139500 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 >> v1:172.21.15.76:6790/0 conn(0x7f2098099090 legacy=0x7f20981d2f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 >> v1:172.21.15.189:6789/0 conn(0x7f2098097b60 legacy=0x7f2098138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: _finish_hunting 0 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 monclient: found mon.a 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209809a540 con 0x7f2098139500 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: _finish_auth 0 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: _check_auth_tickets 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 5 monclient: authenticate success, global_id 104184 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _renew_subs 2024-09-16T06:39:52.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209806fc30 con 0x7f2098139500 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _renew_subs 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2098070790 con 0x7f2098139500 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2090001100 con 0x7f2098139500 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2090004e40 con 0x7f2098139500 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f2096ffd700 4 set_mon_vals no callback set 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 10 monclient: dump: 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:52.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2078005100 con 0x7f2098139500 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f209000f8b0 con 0x7f2098139500 2024-09-16T06:39:52.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7f2090043760 con 0x7f2098139500 2024-09-16T06:39:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.054+0000 7f209ca23700 10 monclient: get_auth_request con 0x7f20840361d0 auth_method 0 2024-09-16T06:39:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.058+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f20900147d0 con 0x7f2098139500 2024-09-16T06:39:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.058+0000 7f20957fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:52.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.058+0000 7f20957fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:52.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f209da25700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f209da25700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f209da25700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2078005930 con 0x7f2098139500 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f20957fa700 1 -- 172.21.15.76:0/1929515291 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2558159477) 0x7f2090012e70 con 0x7f2098139500 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f20957fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:52.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.294+0000 7f20957fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:52.298 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:52.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 >> v1:172.21.15.189:6812/3098810674 conn(0x7f20840361d0 legacy=0x7f2084038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:52.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 10 monclient: shutdown 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 >> v1:172.21.15.76:6789/0 conn(0x7f2098139500 legacy=0x7f20981d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 shutdown_connections 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 >> 172.21.15.76:0/1929515291 conn(0x7f20980a0030 msgr2=0x7f209813c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 shutdown_connections 2024-09-16T06:39:52.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:52.298+0000 7f209da25700 1 -- 172.21.15.76:0/1929515291 wait complete. 2024-09-16T06:39:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:54.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:54.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:54.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:54.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:54.791+0000 7fc4d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:55.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:55.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: get_monmap_and_config 2024-09-16T06:39:55.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: build_initial_monmap 2024-09-16T06:39:55.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: monmap: 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:55.430121+0000 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:55.430121+0000 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:55.429 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 1 Processor -- start 2024-09-16T06:39:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 1 -- start start 2024-09-16T06:39:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: init 2024-09-16T06:39:55.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-c con 0x7f22880973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-a con 0x7f2288097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-b con 0x7f22881400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2288099350 con 0x7f2288097eb0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228809f690 con 0x7f22880973c0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2288099bf0 con 0x7f22881400d0 2024-09-16T06:39:55.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228d4ec700 10 monclient(hunting): authenticate will time out at 942.563450s 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f228cceb700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f22881400d0 legacy=0x7f22881424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f22877fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2288097eb0 0x7f228813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60506/0 (socket says 172.21.15.76:60506) 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.426+0000 7f22877fe700 1 -- 172.21.15.76:0/1512417231 learned_addr learned my addr 172.21.15.76:0/1512417231 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199773593 0 0) 0x7f2288099350 con 0x7f2288097eb0 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): my global_id is 104187 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22700035d0 con 0x7f2288097eb0 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986033926 0 0) 0x7f228809f690 con 0x7f22880973c0 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:39:55.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient(hunting): my global_id is 94396 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288099350 con 0x7f22880973c0 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289474396 0 0) 0x7f22700035d0 con 0x7f2288097eb0 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 >> v1:172.21.15.76:6790/0 conn(0x7f22881400d0 legacy=0x7f22881424c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 >> v1:172.21.15.189:6789/0 conn(0x7f22880973c0 legacy=0x7f22880977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 monclient: found mon.noname-a 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288098360 con 0x7f2288097eb0 2024-09-16T06:39:55.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: _finish_auth 0 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2278001610 con 0x7f2288097eb0 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: dump: 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.437 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 5 monclient: authenticate success, global_id 104187 2024-09-16T06:39:55.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22780018f0 con 0x7f2288097eb0 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 10 monclient: get_monmap_and_config success 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 4 set_mon_vals no callback set 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 1 -- 172.21.15.76:0/1512417231 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2278001c80 con 0x7f2288097eb0 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f2286ffd700 10 monclient: dump: 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:55.439 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 10 monclient: shutdown 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- 172.21.15.76:0/1512417231 >> v1:172.21.15.76:6789/0 conn(0x7f2288097eb0 legacy=0x7f228813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- 172.21.15.76:0/1512417231 shutdown_connections 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- 172.21.15.76:0/1512417231 >> 172.21.15.76:0/1512417231 conn(0x7f22880a0030 msgr2=0x7f22880a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:55.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- 172.21.15.76:0/1512417231 shutdown_connections 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- 172.21.15.76:0/1512417231 wait complete. 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 10 monclient: build_initial_monmap 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 10 monclient: monmap: 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:39:55.441 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:55.442 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:39:55.442 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:55.442 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:39:55.442 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:39:55.442 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 Processor -- start 2024-09-16T06:39:55.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 1 -- start start 2024-09-16T06:39:55.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.430+0000 7f228d4ec700 10 monclient: init 2024-09-16T06:39:55.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:55.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-a con 0x7f22880973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-c con 0x7f2288097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient(hunting): picked mon.noname-b con 0x7f22881400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2288072660 con 0x7f22880973c0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228813fdc0 con 0x7f22881400d0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228809a540 con 0x7f2288097eb0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f22877fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2288097eb0 legacy=0x7f22881d3570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient(hunting): authenticate will time out at 942.568448s 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2287fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f22880973c0 0x7f22881ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60520/0 (socket says 172.21.15.76:60520) 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2287fff700 1 -- 172.21.15.76:0/1217670000 learned_addr learned my addr 172.21.15.76:0/1217670000 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720760127 0 0) 0x7f2288072660 con 0x7f22880973c0 2024-09-16T06:39:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): my global_id is 104190 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22740035d0 con 0x7f22880973c0 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754323491 0 0) 0x7f228813fdc0 con 0x7f22881400d0 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): my global_id is 94399 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288072660 con 0x7f22881400d0 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1156037672 0 0) 0x7f22740035d0 con 0x7f22880973c0 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228813fdc0 con 0x7f22880973c0 2024-09-16T06:39:55.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f227c001970 con 0x7f22880973c0 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): dump: 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:55.449 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3999369252 0 0) 0x7f228813fdc0 con 0x7f22880973c0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 >> v1:172.21.15.76:6790/0 conn(0x7f2288097eb0 legacy=0x7f22881d3570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 >> v1:172.21.15.189:6789/0 conn(0x7f22881400d0 legacy=0x7f22881d6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 monclient: found mon.a 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228806ca30 con 0x7f22880973c0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: _finish_auth 0 2024-09-16T06:39:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 5 monclient: authenticate success, global_id 104190 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _renew_subs 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228813ec50 con 0x7f22880973c0 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _renew_subs 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f228813f8d0 con 0x7f22880973c0 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f227c001c80 con 0x7f22880973c0 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:39:55.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f22867fc700 4 set_mon_vals no callback set 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f227c003b10 con 0x7f22880973c0 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 10 monclient: dump: 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:55.452 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f227c003cf0 con 0x7f22880973c0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2268005100 con 0x7f22880973c0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7f227c0435b0 con 0x7f22880973c0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.434+0000 7f22877fe700 10 monclient: get_auth_request con 0x7f22740361b0 auth_method 0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.442+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f227c004510 con 0x7f22880973c0 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.442+0000 7f2284ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:39:55.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.442+0000 7f2284ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:39:55.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f228d4ec700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:39:55.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f228d4ec700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f228d4ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2268005930 con 0x7f22880973c0 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f2284ff9700 1 -- 172.21.15.76:0/1217670000 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3365315668) 0x7f227c012d20 con 0x7f22880973c0 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f2284ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.674+0000 7f2284ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:39:55.677 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:39:55.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 >> v1:172.21.15.189:6812/3098810674 conn(0x7f22740361b0 legacy=0x7f2274038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 10 monclient: shutdown 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 >> v1:172.21.15.76:6789/0 conn(0x7f22880973c0 legacy=0x7f22881ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 shutdown_connections 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 >> 172.21.15.76:0/1217670000 conn(0x7f22880a0030 msgr2=0x7f228809ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 shutdown_connections 2024-09-16T06:39:55.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:55.678+0000 7f228d4ec700 1 -- 172.21.15.76:0/1217670000 wait complete. 2024-09-16T06:39:55.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:55.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:55.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:55.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:55.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:55.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:56.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:56.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:56.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:56.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:56.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:56.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:57.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:57.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:57.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:57.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:58.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:39:58.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:58.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:58.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:58.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:58.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:58.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: get_monmap_and_config 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: build_initial_monmap 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: monmap: 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:39:58.812617+0000 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:39:58.812617+0000 2024-09-16T06:39:58.811 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:39:58.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:39:58.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:39:58.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:39:58.812 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:39:58.812 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:39:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 1 Processor -- start 2024-09-16T06:39:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 1 -- start start 2024-09-16T06:39:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: init 2024-09-16T06:39:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-b con 0x7fde4c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-a con 0x7fde4c098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-c con 0x7fde4c099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09d560 con 0x7fde4c098be0 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09f140 con 0x7fde4c099030 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09d790 con 0x7fde4c09ed60 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde513e9700 10 monclient(hunting): authenticate will time out at 945.945861s 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde4bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c0984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde4b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fde4c098be0 0x7fde4c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60528/0 (socket says 172.21.15.76:60528) 2024-09-16T06:39:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:58.810+0000 7fde4b7fe700 1 -- 172.21.15.76:0/1785730835 learned_addr learned my addr 172.21.15.76:0/1785730835 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:39:59.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:59.014+0000 7fde4bfff700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c0984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:59.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:39:59.418+0000 7fde4bfff700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c0984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:39:59.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:59.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:39:59.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:59.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:39:59.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:39:59.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:00.222+0000 7fde4bfff700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c0984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:00.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:00.791+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:00.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:00.791+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:00.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:00.791+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:01.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:01.790+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:01.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:01.790+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:01.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:01.790+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): tick 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c0984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:01.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.189:6789/0 conn(0x7fde4c099030 legacy=0x7fde4c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6789/0 conn(0x7fde4c098be0 legacy=0x7fde4c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-c con 0x7fde2c005950 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-b con 0x7fde4c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-a con 0x7fde4c098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09f620 con 0x7fde4c098be0 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09d790 con 0x7fde2c005950 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c012600 con 0x7fde4c09ed60 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde49ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:01.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:01.810+0000 7fde4b7fe700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:02.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:02.014+0000 7fde4b7fe700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:02.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:02.418+0000 7fde4b7fe700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:02.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:02.790+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:02.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:02.790+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:02.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:02.790+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:03.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:03.222+0000 7fde4b7fe700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:03.790+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:03.790+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:03.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:03.790+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:04.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:04.794+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:04.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:04.794+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:04.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:04.794+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:04.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:04.794+0000 7fc4d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:04.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): tick 2024-09-16T06:40:04.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:40:04.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:40:04.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:40:04.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.189:6789/0 conn(0x7fde2c005950 legacy=0x7fde2c007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6789/0 conn(0x7fde4c098be0 legacy=0x7fde2c00ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-a con 0x7fde2c0128c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-c con 0x7fde2c005950 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): picked mon.noname-b con 0x7fde4c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c013810 con 0x7fde2c0128c0 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c012600 con 0x7fde2c005950 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c013550 con 0x7fde4c09ed60 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde49ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:04.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:04.810+0000 7fde50be8700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:05.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:05.014+0000 7fde50be8700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:05.418+0000 7fde50be8700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:05.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:05.794+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:05.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:05.794+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:05.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:05.794+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.222+0000 7fde50be8700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051931261 0 0) 0x7fde2c013810 con 0x7fde2c0128c0 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient(hunting): my global_id is 104193 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde380035d0 con 0x7fde2c0128c0 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107929272 0 0) 0x7fde380035d0 con 0x7fde2c0128c0 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde2c00b670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.189:6789/0 conn(0x7fde2c005950 legacy=0x7fde2c007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 monclient: found mon.noname-a 2024-09-16T06:40:06.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c00ee40 con 0x7fde2c0128c0 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fde3c004200 con 0x7fde2c0128c0 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 5 monclient: authenticate success, global_id 104193 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: dump: 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde3c0044a0 con 0x7fde2c0128c0 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 1 -- 172.21.15.76:0/1785730835 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fde3c004870 con 0x7fde2c0128c0 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 4 set_mon_vals no callback set 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde4affd700 10 monclient: dump: 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 10 monclient: shutdown 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 1 -- 172.21.15.76:0/1785730835 >> v1:172.21.15.76:6789/0 conn(0x7fde2c0128c0 legacy=0x7fde2c012ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 1 -- 172.21.15.76:0/1785730835 shutdown_connections 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.318+0000 7fde513e9700 1 -- 172.21.15.76:0/1785730835 >> 172.21.15.76:0/1785730835 conn(0x7fde4c0a0030 msgr2=0x7fde4c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- 172.21.15.76:0/1785730835 shutdown_connections 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- 172.21.15.76:0/1785730835 wait complete. 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient: build_initial_monmap 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient: monmap: 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 Processor -- start 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- start start 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient: init 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-a con 0x7fde2c005950 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-b con 0x7fde2c0128c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient(hunting): picked mon.noname-c con 0x7fde4c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c09a540 con 0x7fde2c005950 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c090d90 con 0x7fde2c0128c0 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde50be8700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c1ce300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c091350 con 0x7fde4c09ed60 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde4bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fde2c005950 0x7fde4c08d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60788/0 (socket says 172.21.15.76:60788) 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde4bfff700 1 -- 172.21.15.76:0/1287847858 learned_addr learned my addr 172.21.15.76:0/1287847858 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde513e9700 10 monclient(hunting): authenticate will time out at 953.459206s 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695441053 0 0) 0x7fde4c09a540 con 0x7fde2c005950 2024-09-16T06:40:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 10 monclient(hunting): my global_id is 104196 2024-09-16T06:40:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.322+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2c013810 con 0x7fde2c005950 2024-09-16T06:40:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730509603 0 0) 0x7fde4c090d90 con 0x7fde2c0128c0 2024-09-16T06:40:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): my global_id is 94402 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c09a540 con 0x7fde2c0128c0 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741303649 0 0) 0x7fde2c013810 con 0x7fde2c005950 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde4c090d90 con 0x7fde2c005950 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fde3c003f60 con 0x7fde2c005950 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1468302915 0 0) 0x7fde4c09a540 con 0x7fde2c0128c0 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde2c013810 con 0x7fde2c0128c0 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fde34001b20 con 0x7fde2c0128c0 2024-09-16T06:40:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1138068444 0 0) 0x7fde4c090d90 con 0x7fde2c005950 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 >> v1:172.21.15.76:6790/0 conn(0x7fde4c09ed60 legacy=0x7fde4c1ce300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 >> v1:172.21.15.189:6789/0 conn(0x7fde2c0128c0 legacy=0x7fde4c08dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 monclient: found mon.a 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0a1250 con 0x7fde2c005950 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 5 monclient: authenticate success, global_id 104196 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _renew_subs 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c090ab0 con 0x7fde2c005950 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _renew_subs 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c08e2f0 con 0x7fde2c005950 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde3c004cd0 con 0x7fde2c005950 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fde3c004fe0 con 0x7fde2c005950 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde4a7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 10 monclient: dump: 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fde3c00fa50 con 0x7fde2c005950 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde30005100 con 0x7fde2c005950 2024-09-16T06:40:06.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 4634+0+0 (unknown 2580906778 0 0) 0x7fde3c0436a0 con 0x7fde2c005950 2024-09-16T06:40:06.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.338+0000 7fde4b7fe700 10 monclient: get_auth_request con 0x7fde2c039440 auth_method 0 2024-09-16T06:40:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.346+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fde3c014730 con 0x7fde2c005950 2024-09-16T06:40:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.346+0000 7fde48ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.346+0000 7fde48ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:06.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.398+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v35) v1 ==== 122+0+0 (unknown 1724667936 0 0) 0x7fc4cc012e30 con 0x7fc4dc09d7a0 2024-09-16T06:40:06.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.398+0000 7fc4daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:40:06.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.398+0000 7fc4daffd700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-16T06:40:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.522+0000 7fc4e3360700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:40:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.522+0000 7fc4e3360700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:40:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.522+0000 7fc4e3360700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:06.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.522+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7fc4dc1385e0 con 0x7fc4dc09d7a0 2024-09-16T06:40:06.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde513e9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:06.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde513e9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:06.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde513e9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde30005930 con 0x7fde2c005950 2024-09-16T06:40:06.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde48ff9700 1 -- 172.21.15.76:0/1287847858 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3189569759) 0x7fde3c01cef0 con 0x7fde2c005950 2024-09-16T06:40:06.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde48ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:06.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.602+0000 7fde48ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:06.605 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:06.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 >> v1:172.21.15.189:6812/3098810674 conn(0x7fde2c039440 legacy=0x7fde2c03b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 10 monclient: shutdown 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 >> v1:172.21.15.76:6789/0 conn(0x7fde2c005950 legacy=0x7fde4c08d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 shutdown_connections 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 >> 172.21.15.76:0/1287847858 conn(0x7fde4c0a0030 msgr2=0x7fde4c06ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 shutdown_connections 2024-09-16T06:40:06.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.606+0000 7fde513e9700 1 -- 172.21.15.76:0/1287847858 wait complete. 2024-09-16T06:40:06.618 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:40:06.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:40:06.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:40:06.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.730+0000 7ffb497ed700 10 monclient: get_monmap_and_config 2024-09-16T06:40:06.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.730+0000 7ffb497ed700 10 monclient: build_initial_monmap 2024-09-16T06:40:06.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.730+0000 7ffb497ed700 10 monclient: monmap: 2024-09-16T06:40:06.733 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:06.733 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:06.735034+0000 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:06.735034+0000 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:06.734 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 1 Processor -- start 2024-09-16T06:40:06.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 1 -- start start 2024-09-16T06:40:06.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient: init 2024-09-16T06:40:06.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:06.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-c con 0x7ffb4409ea50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-a con 0x7ffb4409f540 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-b con 0x7ffb4409c1f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409e560 con 0x7ffb4409f540 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb440a04f0 con 0x7ffb4409ea50 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409b550 con 0x7ffb4409c1f0 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient(hunting): authenticate will time out at 953.869967s 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb437fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ffb4409c1f0 legacy=0x7ffb4413a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb427fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffb4409f540 0x7ffb44136c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60796/0 (socket says 172.21.15.76:60796) 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb427fc700 1 -- 172.21.15.76:0/2099562843 learned_addr learned my addr 172.21.15.76:0/2099562843 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:06.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb42ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb4409ea50 0x7ffb4409ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:44450/0 (socket says 172.21.15.76:44450) 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469097573 0 0) 0x7ffb440a04f0 con 0x7ffb4409ea50 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): my global_id is 94405 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2c0035d0 con 0x7ffb4409ea50 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002766841 0 0) 0x7ffb4409e560 con 0x7ffb4409f540 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient(hunting): my global_id is 104199 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb440a04f0 con 0x7ffb4409f540 2024-09-16T06:40:06.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043720344 0 0) 0x7ffb2c0035d0 con 0x7ffb4409ea50 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 >> v1:172.21.15.76:6790/0 conn(0x7ffb4409c1f0 legacy=0x7ffb4413a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 >> v1:172.21.15.76:6789/0 conn(0x7ffb4409f540 legacy=0x7ffb44136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 monclient: found mon.noname-c 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4409bdd0 con 0x7ffb4409ea50 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb34001610 con 0x7ffb4409ea50 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: dump: 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 5 monclient: authenticate success, global_id 94405 2024-09-16T06:40:06.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb340018f0 con 0x7ffb4409ea50 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 1 -- 172.21.15.76:0/2099562843 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb34001c80 con 0x7ffb4409ea50 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb41ffb700 10 monclient: dump: 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:06.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.734+0000 7ffb497ed700 4 set_mon_vals no callback set 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: shutdown 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- 172.21.15.76:0/2099562843 >> v1:172.21.15.189:6789/0 conn(0x7ffb4409ea50 legacy=0x7ffb4409ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- 172.21.15.76:0/2099562843 shutdown_connections 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- 172.21.15.76:0/2099562843 >> 172.21.15.76:0/2099562843 conn(0x7ffb440a0d30 msgr2=0x7ffb440a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- 172.21.15.76:0/2099562843 shutdown_connections 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- 172.21.15.76:0/2099562843 wait complete. 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: build_initial_monmap 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: monmap: 2024-09-16T06:40:06.748 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 Processor -- start 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- start start 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: init 2024-09-16T06:40:06.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-a con 0x7ffb4409c1f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-c con 0x7ffb4409ea50 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient(hunting): picked mon.noname-b con 0x7ffb4409f540 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb440a2a00 con 0x7ffb4409c1f0 2024-09-16T06:40:06.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4409b240 con 0x7ffb4409f540 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb440719f0 con 0x7ffb4409ea50 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb497ed700 10 monclient(hunting): authenticate will time out at 953.875769s 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb427fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ffb4409ea50 legacy=0x7ffb441cae30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb42ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffb4409c1f0 0x7ffb441319a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60802/0 (socket says 172.21.15.76:60802) 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.738+0000 7ffb42ffd700 1 -- 172.21.15.76:0/3818033439 learned_addr learned my addr 172.21.15.76:0/3818033439 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085343934 0 0) 0x7ffb440a2a00 con 0x7ffb4409c1f0 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): my global_id is 104202 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300035d0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701762415 0 0) 0x7ffb300035d0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb440a2a00 con 0x7ffb4409c1f0 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb340015d0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.754 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272121851 0 0) 0x7ffb4409b240 con 0x7ffb4409f540 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): my global_id is 94408 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300035d0 con 0x7ffb4409f540 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3455972291 0 0) 0x7ffb440a2a00 con 0x7ffb4409c1f0 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6790/0 conn(0x7ffb4409ea50 legacy=0x7ffb441cae30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.189:6789/0 conn(0x7ffb4409f540 legacy=0x7ffb441ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 monclient: found mon.a 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44121af0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 5 monclient: authenticate success, global_id 104202 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 10 monclient: _renew_subs 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb44122390 con 0x7ffb4409c1f0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 10 monclient: _renew_subs 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb44120030 con 0x7ffb4409c1f0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb34004d20 con 0x7ffb4409c1f0 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb340050b0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: dump: 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb40ff9700 4 set_mon_vals no callback set 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ffb34005290 con 0x7ffb4409c1f0 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffb24006590 con 0x7ffb24002830 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(35..35 src has 1..35) v4 ==== 4634+0+0 (unknown 2721853569 0 0) 0x7ffb340439e0 con 0x7ffb4409c1f0 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077071003 0 0) 0x7ffb24006590 con 0x7ffb24002830 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient(hunting): my global_id is 104205 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb440a2a00 con 0x7ffb24002830 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb437fe700 10 monclient: get_auth_request con 0x7ffb30036190 auth_method 0 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 198954998 0 0) 0x7ffb440a2a00 con 0x7ffb24002830 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb24006590 con 0x7ffb24002830 2024-09-16T06:40:06.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb38001740 con 0x7ffb24002830 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: dump: 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 361597242 0 0) 0x7ffb24006590 con 0x7ffb24002830 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb2400a190 con 0x7ffb24002830 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ffb38001a70 con 0x7ffb24002830 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6789/0 conn(0x7ffb24002830 legacy=0x7ffb24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.742+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6789/0 conn(0x7ffb24002830 legacy=0x7ffb24004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:06.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb497ed700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:06.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ffb24009e30 con 0x7ffb24009150 2024-09-16T06:40:06.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626928954 0 0) 0x7ffb24009e30 con 0x7ffb24009150 2024-09-16T06:40:06.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 10 monclient(hunting): my global_id is 104208 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb24006590 con 0x7ffb24009150 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.750+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 82887070 0 0) 0x7ffb24006590 con 0x7ffb24009150 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb24009e30 con 0x7ffb24009150 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb38004af0 con 0x7ffb24009150 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 10 monclient: dump: 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3819929004 0 0) 0x7ffb24009e30 con 0x7ffb24009150 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffb2400c340 con 0x7ffb24009150 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 2983166345) 0x7ffb38005fd0 con 0x7ffb24009150 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:06.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6789/0 conn(0x7ffb24009150 legacy=0x7ffb240095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb2b7fe700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6789/0 conn(0x7ffb24009150 legacy=0x7ffb240095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 28, 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:06.764 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.765 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.766 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:06.767 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.768 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:06.769 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.770 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.774 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:06.775 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:06.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.189:6812/3098810674 conn(0x7ffb30036190 legacy=0x7ffb30038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 10 monclient: shutdown 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 >> v1:172.21.15.76:6789/0 conn(0x7ffb4409c1f0 legacy=0x7ffb441319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb437fe700 1 -- 172.21.15.76:0/3818033439 reap_dead start 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb437fe700 1 -- 172.21.15.76:0/3818033439 reap_dead start 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 shutdown_connections 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 >> 172.21.15.76:0/3818033439 conn(0x7ffb440a0d30 msgr2=0x7ffb4406f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 shutdown_connections 2024-09-16T06:40:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.754+0000 7ffb497ed700 1 -- 172.21.15.76:0/3818033439 wait complete. 2024-09-16T06:40:06.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:40:06.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:40:06.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:40:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.790+0000 7fc4d9ffb700 10 monclient: tick 2024-09-16T06:40:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.790+0000 7fc4d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:06.790+0000 7fc4d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.894+0000 7fcec51f3700 10 monclient: get_monmap_and_config 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.894+0000 7fcec51f3700 10 monclient: build_initial_monmap 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.894+0000 7fcec51f3700 10 monclient: monmap: 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:06.899620+0000 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:06.899620+0000 2024-09-16T06:40:06.898 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:06.899 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.899 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:06.899 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:06.899 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:06.899 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 Processor -- start 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- start start 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: init 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:06.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-a con 0x7fcec0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-b con 0x7fcec006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-c con 0x7fcec006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec006c280 con 0x7fcec0132330 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec006bfa0 con 0x7fcec006cf50 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec009ace0 con 0x7fcec006d480 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient(hunting): authenticate will time out at 954.033029s 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebf7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fcec006d480 legacy=0x7fcec013a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcec0132330 0x7fcec0132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60822/0 (socket says 172.21.15.76:60822) 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebffff700 1 -- 172.21.15.76:0/4252503908 learned_addr learned my addr 172.21.15.76:0/4252503908 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870360613 0 0) 0x7fcec006c280 con 0x7fcec0132330 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient(hunting): my global_id is 104211 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea80035d0 con 0x7fcec0132330 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3758766483 0 0) 0x7fcea80035d0 con 0x7fcec0132330 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 >> v1:172.21.15.76:6790/0 conn(0x7fcec006d480 legacy=0x7fcec013a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 >> v1:172.21.15.189:6789/0 conn(0x7fcec006cf50 legacy=0x7fcec013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 monclient: found mon.noname-a 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec009ae70 con 0x7fcec0132330 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 5 monclient: authenticate success, global_id 104211 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb0001990 con 0x7fcec0132330 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: dump: 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.910 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceb0001c30 con 0x7fcec0132330 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 1 -- 172.21.15.76:0/4252503908 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb0001e60 con 0x7fcec0132330 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 4 set_mon_vals no callback set 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcebeffd700 10 monclient: dump: 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.912 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: shutdown 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- 172.21.15.76:0/4252503908 >> v1:172.21.15.76:6789/0 conn(0x7fcec0132330 legacy=0x7fcec0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- 172.21.15.76:0/4252503908 shutdown_connections 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- 172.21.15.76:0/4252503908 >> 172.21.15.76:0/4252503908 conn(0x7fcec00a41c0 msgr2=0x7fcec00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- 172.21.15.76:0/4252503908 shutdown_connections 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 1 -- 172.21.15.76:0/4252503908 wait complete. 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: build_initial_monmap 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.898+0000 7fcec51f3700 10 monclient: monmap: 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:06.913 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 Processor -- start 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- start start 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: init 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:06.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-a con 0x7fcec006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-c con 0x7fcec006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient(hunting): picked mon.noname-b con 0x7fcec0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec009e8b0 con 0x7fcec006cf50 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec01c9e80 con 0x7fcec0132330 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec0137820 con 0x7fcec006d480 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebf7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fcec006d480 legacy=0x7fcec013e520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient(hunting): authenticate will time out at 954.036890s 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcec006cf50 0x7fcec01c9770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60834/0 (socket says 172.21.15.76:60834) 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebffff700 1 -- 172.21.15.76:0/1063295098 learned_addr learned my addr 172.21.15.76:0/1063295098 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624809469 0 0) 0x7fcec009e8b0 con 0x7fcec006cf50 2024-09-16T06:40:06.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): my global_id is 104214 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceac0035d0 con 0x7fcec006cf50 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168767380 0 0) 0x7fcec01c9e80 con 0x7fcec0132330 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): my global_id is 94414 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec009e8b0 con 0x7fcec0132330 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 752619526 0 0) 0x7fceac0035d0 con 0x7fcec006cf50 2024-09-16T06:40:06.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcec01c9e80 con 0x7fcec006cf50 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb0004c10 con 0x7fcec006cf50 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): dump: 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.917 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 894536505 0 0) 0x7fcec01c9e80 con 0x7fcec006cf50 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.76:6790/0 conn(0x7fcec006d480 legacy=0x7fcec013e520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6789/0 conn(0x7fcec0132330 legacy=0x7fcec01d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 monclient: found mon.a 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec0131400 con 0x7fcec006cf50 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:06.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 5 monclient: authenticate success, global_id 104214 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _renew_subs 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec0138fb0 con 0x7fcec006cf50 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _renew_subs 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec0136c50 con 0x7fcec006cf50 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceb0005020 con 0x7fcec006cf50 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb00053b0 con 0x7fcec006cf50 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 10 monclient: dump: 2024-09-16T06:40:06.919 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebe7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fceb00055d0 con 0x7fcec006cf50 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(35..35 src has 1..35) v4 ==== 4634+0+0 (unknown 2721853569 0 0) 0x7fceb0043bf0 con 0x7fcec006cf50 2024-09-16T06:40:06.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.902+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcec0131da0 con 0x7fcec013ec30 2024-09-16T06:40:06.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebf7fe700 10 monclient: get_auth_request con 0x7fceac036040 auth_method 0 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337884323 0 0) 0x7fcec0131da0 con 0x7fcec013ec30 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient(hunting): my global_id is 94417 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec01c9e80 con 0x7fcec013ec30 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2506644757 0 0) 0x7fcec01c9e80 con 0x7fcec013ec30 2024-09-16T06:40:06.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcec0131da0 con 0x7fcec013ec30 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb8001d50 con 0x7fcec013ec30 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient: dump: 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1697044866 0 0) 0x7fcec0131da0 con 0x7fcec013ec30 2024-09-16T06:40:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcec00a3450 con 0x7fcec013ec30 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fceb80059d0 con 0x7fcec013ec30 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6789/0 conn(0x7fcec013ec30 legacy=0x7fcec0061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.906+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6789/0 conn(0x7fcec013ec30 legacy=0x7fcec0061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcec51f3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcec002cc30 con 0x7fcec002c820 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825379182 0 0) 0x7fcec002cc30 con 0x7fcec002c820 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient(hunting): my global_id is 94420 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec0131da0 con 0x7fcec002c820 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2642591341 0 0) 0x7fcec0131da0 con 0x7fcec002c820 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcec002cc30 con 0x7fcec002c820 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fceb8003f00 con 0x7fcec002c820 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient: dump: 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 687417740 0 0) 0x7fcec002cc30 con 0x7fcec002c820 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcec00a01e0 con 0x7fcec002c820 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1628914654) 0x7fceb8005b40 con 0x7fcec002c820 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6789/0 conn(0x7fcec002c820 legacy=0x7fcec009fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.914+0000 7fcebcff9700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6789/0 conn(0x7fcec002c820 legacy=0x7fcec009fad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:06.930 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 28, 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:40:06.931 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.932 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.933 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:06.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:06.935 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:06.940 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.941 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.942 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:06.943 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:06.944 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.189:6812/3098810674 conn(0x7fceac036040 legacy=0x7fceac038500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 10 monclient: shutdown 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 >> v1:172.21.15.76:6789/0 conn(0x7fcec006cf50 legacy=0x7fcec01c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec49f2700 1 -- 172.21.15.76:0/1063295098 reap_dead start 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec49f2700 1 -- 172.21.15.76:0/1063295098 reap_dead start 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 shutdown_connections 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 >> 172.21.15.76:0/1063295098 conn(0x7fcec00a41c0 msgr2=0x7fcec00a6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 shutdown_connections 2024-09-16T06:40:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:06.918+0000 7fcec51f3700 1 -- 172.21.15.76:0/1063295098 wait complete. 2024-09-16T06:40:06.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:40:06.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:40:07.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.402+0000 7fc4daffd700 1 -- 172.21.15.189:0/417073816 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v36) v1 ==== 122+0+0 (unknown 3848321630 0 0) 0x7fc4cc0145b0 con 0x7fc4dc09d7a0 2024-09-16T06:40:07.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.402+0000 7fc4daffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T06:40:07.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.402+0000 7fc4daffd700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-16T06:40:07.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.418+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc4c8036540 legacy=0x7fc4c8038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:07.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.418+0000 7fc4e3360700 10 monclient: shutdown 2024-09-16T06:40:07.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.418+0000 7fc4e3360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:07.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.418+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 >> v1:172.21.15.189:6789/0 conn(0x7fc4dc09d7a0 legacy=0x7fc4dc1377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.422+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 shutdown_connections 2024-09-16T06:40:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.422+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 >> 172.21.15.189:0/417073816 conn(0x7fc4dc0a43d0 msgr2=0x7fc4dc134380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:07.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.422+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 shutdown_connections 2024-09-16T06:40:07.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.422+0000 7fc4e3360700 1 -- 172.21.15.189:0/417073816 wait complete. 2024-09-16T06:40:07.426 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 4 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 2 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=2 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T06:40:07.445 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T06:40:07.446 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T06:40:07.458 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T06:40:07.873 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 2' 2024-09-16T06:40:07.873 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T06:40:07.873 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 2 2024-09-16T06:40:07.873 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T06:40:07.876 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:2 2024-09-16T06:40:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:2 2024-09-16T06:40:07.877 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T06:40:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=2 2024-09-16T06:40:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 2 2024-09-16T06:40:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T06:40:07.879 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=2 2024-09-16T06:40:07.880 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 2 != \2 ]] 2024-09-16T06:40:07.880 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 10 2024-09-16T06:40:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: get_monmap_and_config 2024-09-16T06:40:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: build_initial_monmap 2024-09-16T06:40:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: monmap: 2024-09-16T06:40:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:07.981547+0000 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:07.981547+0000 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:07.981 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 1 Processor -- start 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 1 -- start start 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: init 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-b con 0x7f52641352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-a con 0x7f5264135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-c con 0x7f52640a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526409b050 con 0x7f5264135db0 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526406ab20 con 0x7f52640a2030 2024-09-16T06:40:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526406a840 con 0x7f52641352c0 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f52693a1700 10 monclient(hunting): authenticate will time out at 960.495095s 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5263fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f52641352c0 legacy=0x7f52641356a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5268ba0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f52640a2030 0x7f526413df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39468/0 (socket says 172.21.15.189:39468) 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5268ba0700 1 -- 172.21.15.189:0/3516469513 learned_addr learned my addr 172.21.15.189:0/3516469513 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:07.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047295935 0 0) 0x7f526406ab20 con 0x7f52640a2030 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient(hunting): my global_id is 94429 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f524c0035d0 con 0x7f52640a2030 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 519493976 0 0) 0x7f524c0035d0 con 0x7f52640a2030 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 >> v1:172.21.15.76:6790/0 conn(0x7f52641352c0 legacy=0x7f52641356a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 >> v1:172.21.15.76:6789/0 conn(0x7f5264135db0 legacy=0x7f526413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:07.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 monclient: found mon.noname-c 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526406c1b0 con 0x7f52640a2030 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.978+0000 7f5262ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 5 monclient: authenticate success, global_id 94429 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f525c001990 con 0x7f52640a2030 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: dump: 2024-09-16T06:40:07.990 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f525c001cb0 con 0x7f52640a2030 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:07.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 1 -- 172.21.15.189:0/3516469513 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f525c002040 con 0x7f52640a2030 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 4 set_mon_vals no callback set 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5262ffd700 10 monclient: dump: 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:07.992 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:07.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: shutdown 2024-09-16T06:40:07.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:07.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- 172.21.15.189:0/3516469513 >> v1:172.21.15.189:6789/0 conn(0x7f52640a2030 legacy=0x7f526413df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:07.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- 172.21.15.189:0/3516469513 shutdown_connections 2024-09-16T06:40:07.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- 172.21.15.189:0/3516469513 >> 172.21.15.189:0/3516469513 conn(0x7f52640a4550 msgr2=0x7f52640a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:07.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- 172.21.15.189:0/3516469513 shutdown_connections 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- 172.21.15.189:0/3516469513 wait complete. 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: build_initial_monmap 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: monmap: 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:07.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 Processor -- start 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- start start 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: init 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-a con 0x7f52640a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-c con 0x7f52641352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient(hunting): picked mon.noname-b con 0x7f5264135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5264130ce0 con 0x7f52640a2030 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526412e1c0 con 0x7f5264135db0 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526412f3e0 con 0x7f52641352c0 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:07.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52693a1700 10 monclient(hunting): authenticate will time out at 960.499791s 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f52637fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f52641352c0 legacy=0x7f526412dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5268ba0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5264135db0 0x7f52641d25f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:39480/0 (socket says 172.21.15.189:39480) 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.982+0000 7f5268ba0700 1 -- 172.21.15.189:0/2249957371 learned_addr learned my addr 172.21.15.189:0/2249957371 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369836310 0 0) 0x7f526412e1c0 con 0x7f5264135db0 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): my global_id is 94432 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52500035d0 con 0x7f5264135db0 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981757936 0 0) 0x7f5264130ce0 con 0x7f52640a2030 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:08.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): my global_id is 104226 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526412e1c0 con 0x7f52640a2030 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134827087 0 0) 0x7f52500035d0 con 0x7f5264135db0 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5264130ce0 con 0x7f5264135db0 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f525c0017b0 con 0x7f5264135db0 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:08.001 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2091566325 0 0) 0x7f5264130ce0 con 0x7f5264135db0 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 >> v1:172.21.15.76:6790/0 conn(0x7f52641352c0 legacy=0x7f526412dab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 >> v1:172.21.15.76:6789/0 conn(0x7f52640a2030 legacy=0x7f526412d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 monclient: found mon.b 2024-09-16T06:40:08.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52641312a0 con 0x7f5264135db0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 5 monclient: authenticate success, global_id 94432 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _renew_subs 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52640a66b0 con 0x7f5264135db0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _renew_subs 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f526412f620 con 0x7f5264135db0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f525c005070 con 0x7f5264135db0 2024-09-16T06:40:08.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5261ffb700 4 set_mon_vals no callback set 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f525c005380 con 0x7f5264135db0 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 10 monclient: dump: 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:08.004 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f525c00fdf0 con 0x7f5264135db0 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5244005100 con 0x7f5264135db0 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(36..36 src has 1..36) v4 ==== 4634+0+0 (unknown 27273792 0 0) 0x7f525c043c10 con 0x7f5264135db0 2024-09-16T06:40:08.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.986+0000 7f5263fff700 10 monclient: get_auth_request con 0x7f5250036140 auth_method 0 2024-09-16T06:40:08.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.998+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f525c014bd0 con 0x7f5264135db0 2024-09-16T06:40:08.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.998+0000 7f5260ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:08.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:07.998+0000 7f5260ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.150+0000 7f52693a1700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.150+0000 7f52693a1700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.150+0000 7f52693a1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.150+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f5244005930 con 0x7f5264135db0 2024-09-16T06:40:08.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.422+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v37) v1 ==== 124+0+0 (unknown 1012676041 0 0) 0x7f525c013270 con 0x7f5264135db0 2024-09-16T06:40:08.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.422+0000 7f5260ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:08.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.422+0000 7f5260ff9700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-16T06:40:08.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.554+0000 7f52693a1700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:08.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.554+0000 7f52693a1700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:08.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.554+0000 7f52693a1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:08.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:08.554+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7f52440053b0 con 0x7f5264135db0 2024-09-16T06:40:09.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.422+0000 7f5260ff9700 1 -- 172.21.15.189:0/2249957371 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v38) v1 ==== 124+0+0 (unknown 2690976392 0 0) 0x7f525c0149f0 con 0x7f5264135db0 2024-09-16T06:40:09.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.422+0000 7f5260ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T06:40:09.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.422+0000 7f5260ff9700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-16T06:40:09.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.434+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5250036140 legacy=0x7f5250038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:09.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.434+0000 7f52693a1700 10 monclient: shutdown 2024-09-16T06:40:09.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.434+0000 7f52693a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:09.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.434+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 >> v1:172.21.15.189:6789/0 conn(0x7f5264135db0 legacy=0x7f52641d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:09.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.442+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 shutdown_connections 2024-09-16T06:40:09.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.442+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 >> 172.21.15.189:0/2249957371 conn(0x7f52640a4550 msgr2=0x7f52640a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:09.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.442+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 shutdown_connections 2024-09-16T06:40:09.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:09.442+0000 7f52693a1700 1 -- 172.21.15.189:0/2249957371 wait complete. 2024-09-16T06:40:09.448 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 4 size to 10 2024-09-16T06:40:09.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_config_value foo min_size 5 2024-09-16T06:40:09.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ local pool_name config_opt expected_val val 2024-09-16T06:40:09.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ pool_name=foo 2024-09-16T06:40:09.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ config_opt=min_size 2024-09-16T06:40:09.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_val=5 2024-09-16T06:40:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:++ get_config_value_or_die foo min_size 2024-09-16T06:40:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:++ local pool_name config_opt raw val 2024-09-16T06:40:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:++ pool_name=foo 2024-09-16T06:40:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:++ config_opt=min_size 2024-09-16T06:40:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:+++ ceph osd pool get foo min_size 2024-09-16T06:40:09.905 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw='min_size: 5' 2024-09-16T06:40:09.906 INFO:tasks.workunit.client.0.smithi189.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T06:40:09.906 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size: 5 2024-09-16T06:40:09.906 INFO:tasks.workunit.client.0.smithi189.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T06:40:09.909 INFO:tasks.workunit.client.0.smithi189.stderr:++ raw=min_size:5 2024-09-16T06:40:09.910 INFO:tasks.workunit.client.0.smithi189.stderr:+++ echo min_size:5 2024-09-16T06:40:09.910 INFO:tasks.workunit.client.0.smithi189.stderr:+++ cut -f2 -d: 2024-09-16T06:40:09.911 INFO:tasks.workunit.client.0.smithi189.stderr:++ val=5 2024-09-16T06:40:09.911 INFO:tasks.workunit.client.0.smithi189.stderr:++ echo 5 2024-09-16T06:40:09.911 INFO:tasks.workunit.client.0.smithi189.stderr:++ return 0 2024-09-16T06:40:09.912 INFO:tasks.workunit.client.0.smithi189.stderr:+ val=5 2024-09-16T06:40:09.912 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 5 != \5 ]] 2024-09-16T06:40:09.912 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-16T06:40:09.912 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:09.912 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 0 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: get_monmap_and_config 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: monmap: 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:10.022439+0000 2024-09-16T06:40:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:10.022439+0000 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:10.022 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 1 Processor -- start 2024-09-16T06:40:10.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 1 -- start start 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: init 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c4132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c4138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c413bf70 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c406abd0 con 0x7fa9c4138460 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c406b170 con 0x7fa9c4132080 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c4132b70 con 0x7fa9c413bf70 2024-09-16T06:40:10.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c9b7d700 10 monclient(hunting): authenticate will time out at 962.535697s 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c937c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fa9c413bf70 legacy=0x7fa9c413e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c8b7b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9c4132080 0x7fa9c4132460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44188/0 (socket says 172.21.15.189:44188) 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.018+0000 7fa9c8b7b700 1 -- 172.21.15.189:0/354723202 learned_addr learned my addr 172.21.15.189:0/354723202 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257214712 0 0) 0x7fa9c406b170 con 0x7fa9c4132080 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): my global_id is 94441 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b00035d0 con 0x7fa9c4132080 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435854225 0 0) 0x7fa9c406abd0 con 0x7fa9c4138460 2024-09-16T06:40:10.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient(hunting): my global_id is 104235 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c406b170 con 0x7fa9c4138460 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716031600 0 0) 0x7fa9b00035d0 con 0x7fa9c4132080 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 >> v1:172.21.15.76:6790/0 conn(0x7fa9c413bf70 legacy=0x7fa9c413e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 >> v1:172.21.15.76:6789/0 conn(0x7fa9c4138460 legacy=0x7fa9c413a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40a38b0 con 0x7fa9c4132080 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9b4001990 con 0x7fa9c4132080 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 5 monclient: authenticate success, global_id 94441 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: dump: 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b4001c70 con 0x7fa9c4132080 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 1 -- 172.21.15.189:0/354723202 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9b4002000 con 0x7fa9c4132080 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:10.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 4 set_mon_vals no callback set 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c37fe700 10 monclient: dump: 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.030 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 10 monclient: shutdown 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/354723202 >> v1:172.21.15.189:6789/0 conn(0x7fa9c4132080 legacy=0x7fa9c4132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/354723202 shutdown_connections 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/354723202 >> 172.21.15.189:0/354723202 conn(0x7fa9c40a4660 msgr2=0x7fa9c4099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/354723202 shutdown_connections 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/354723202 wait complete. 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 10 monclient: monmap: 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 Processor -- start 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 1 -- start start 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.022+0000 7fa9c9b7d700 10 monclient: init 2024-09-16T06:40:10.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c4132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c4138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c413bf70 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c409c520 con 0x7fa9c4138460 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c41caf10 con 0x7fa9c413bf70 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c413bb00 con 0x7fa9c4132080 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient(hunting): authenticate will time out at 962.540493s 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c8b7b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fa9c4132080 legacy=0x7fa9c41ca0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c937c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9c413bf70 0x7fa9c41d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44202/0 (socket says 172.21.15.189:44202) 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c937c700 1 -- 172.21.15.189:0/3348436498 learned_addr learned my addr 172.21.15.189:0/3348436498 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175431677 0 0) 0x7fa9c41caf10 con 0x7fa9c413bf70 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): my global_id is 94444 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a40035d0 con 0x7fa9c413bf70 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485495439 0 0) 0x7fa9c409c520 con 0x7fa9c4138460 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): my global_id is 104238 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c41caf10 con 0x7fa9c4138460 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707224603 0 0) 0x7fa9a40035d0 con 0x7fa9c413bf70 2024-09-16T06:40:10.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c409c520 con 0x7fa9c413bf70 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9bc004170 con 0x7fa9c413bf70 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): dump: 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965176417 0 0) 0x7fa9c41caf10 con 0x7fa9c4138460 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9a40035d0 con 0x7fa9c4138460 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9ac0018b0 con 0x7fa9c4138460 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient(hunting): dump: 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 337837987 0 0) 0x7fa9c409c520 con 0x7fa9c413bf70 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 >> v1:172.21.15.76:6790/0 conn(0x7fa9c4132080 legacy=0x7fa9c41ca0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 >> v1:172.21.15.76:6789/0 conn(0x7fa9c4138460 legacy=0x7fa9c41ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 monclient: found mon.b 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4138120 con 0x7fa9c413bf70 2024-09-16T06:40:10.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 5 monclient: authenticate success, global_id 94444 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _renew_subs 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c409e720 con 0x7fa9c413bf70 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _renew_subs 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c409cdc0 con 0x7fa9c413bf70 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9bc004450 con 0x7fa9c413bf70 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9bc0047e0 con 0x7fa9c413bf70 2024-09-16T06:40:10.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 10 monclient: dump: 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c2ffd700 4 set_mon_vals no callback set 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa9bc004a40 con 0x7fa9c413bf70 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.026+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40666e0 con 0x7fa9c413bf70 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.030+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7fa9bc043550 con 0x7fa9c413bf70 2024-09-16T06:40:10.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.030+0000 7fa9c8b7b700 10 monclient: get_auth_request con 0x7fa9a40362e0 auth_method 0 2024-09-16T06:40:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.034+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa9bc005290 con 0x7fa9c413bf70 2024-09-16T06:40:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.034+0000 7fa9c17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.034+0000 7fa9c17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:10.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c9b7d700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T06:40:10.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c9b7d700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T06:40:10.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c9b7d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c9b7d700 1 -- 172.21.15.189:0/3348436498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7fa9c40a6150 con 0x7fa9c413bf70 2024-09-16T06:40:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c17fa700 1 -- 172.21.15.189:0/3348436498 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v38) v1 ==== 136+0+0 (unknown 2759850101 0 0) 0x7fa9bc01d070 con 0x7fa9c413bf70 2024-09-16T06:40:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T06:40:10.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.202+0000 7fa9c17fa700 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T06:40:10.206 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T06:40:10.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:10.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-16T06:40:10.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:10.210 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set foo size 20 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: get_monmap_and_config 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: monmap: 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.313 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:10.314475+0000 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:10.314475+0000 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:10.314 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 1 Processor -- start 2024-09-16T06:40:10.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 1 -- start start 2024-09-16T06:40:10.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: init 2024-09-16T06:40:10.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-c con 0x7fa48406a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-b con 0x7fa48409d010 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-a con 0x7fa48406ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4840a16e0 con 0x7fa48406ae30 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48406c100 con 0x7fa48406a340 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48409cb40 con 0x7fa48409d010 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa48b5b4700 10 monclient(hunting): authenticate will time out at 962.827775s 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa489350700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa48406a340 0x7fa48406a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44212/0 (socket says 172.21.15.189:44212) 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa489350700 1 -- 172.21.15.189:0/3005778983 learned_addr learned my addr 172.21.15.189:0/3005778983 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.310+0000 7fa488b4f700 1 -- 172.21.15.189:0/3005778983 >> v1:172.21.15.76:6790/0 conn(0x7fa48409d010 legacy=0x7fa48413a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181218883 0 0) 0x7fa48406c100 con 0x7fa48406a340 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient(hunting): my global_id is 94447 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4700035d0 con 0x7fa48406a340 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779454602 0 0) 0x7fa4700035d0 con 0x7fa48406a340 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 >> v1:172.21.15.76:6790/0 conn(0x7fa48409d010 legacy=0x7fa48413a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 >> v1:172.21.15.76:6789/0 conn(0x7fa48406ae30 legacy=0x7fa48413df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 monclient: found mon.noname-c 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48406b260 con 0x7fa48406a340 2024-09-16T06:40:10.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 5 monclient: authenticate success, global_id 94447 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa474001610 con 0x7fa48406a340 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: dump: 2024-09-16T06:40:10.320 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa474001930 con 0x7fa48406a340 2024-09-16T06:40:10.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 1 -- 172.21.15.189:0/3005778983 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa474001cc0 con 0x7fa48406a340 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 4 set_mon_vals no callback set 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa47bfff700 10 monclient: dump: 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 10 monclient: shutdown 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3005778983 >> v1:172.21.15.189:6789/0 conn(0x7fa48406a340 legacy=0x7fa48406a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3005778983 shutdown_connections 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3005778983 >> 172.21.15.189:0/3005778983 conn(0x7fa4840a2180 msgr2=0x7fa4840a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3005778983 shutdown_connections 2024-09-16T06:40:10.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3005778983 wait complete. 2024-09-16T06:40:10.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 10 monclient: monmap: 2024-09-16T06:40:10.327 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.327 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 Processor -- start 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 1 -- start start 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.314+0000 7fa48b5b4700 10 monclient: init 2024-09-16T06:40:10.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-b con 0x7fa48406a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-a con 0x7fa48406ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient(hunting): picked mon.noname-c con 0x7fa48409d010 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4841346e0 con 0x7fa48406ae30 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa484131aa0 con 0x7fa48406a340 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4841349c0 con 0x7fa48409d010 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient(hunting): authenticate will time out at 962.832503s 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa489350700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa48406a340 0x7fa484130c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44216/0 (socket says 172.21.15.189:44216) 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa489350700 1 -- 172.21.15.189:0/3880694405 learned_addr learned my addr 172.21.15.189:0/3880694405 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa489b51700 1 -- 172.21.15.189:0/3880694405 >> v1:172.21.15.76:6790/0 conn(0x7fa48409d010 legacy=0x7fa4841d25e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041798686 0 0) 0x7fa484131aa0 con 0x7fa48406a340 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): my global_id is 94450 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4680035d0 con 0x7fa48406a340 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537079296 0 0) 0x7fa4841346e0 con 0x7fa48406ae30 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): my global_id is 104244 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484131aa0 con 0x7fa48406ae30 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668858808 0 0) 0x7fa4680035d0 con 0x7fa48406a340 2024-09-16T06:40:10.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4841346e0 con 0x7fa48406a340 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4740015d0 con 0x7fa48406a340 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568214392 0 0) 0x7fa4841346e0 con 0x7fa48406a340 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 >> v1:172.21.15.76:6790/0 conn(0x7fa48409d010 legacy=0x7fa4841d25e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 >> v1:172.21.15.76:6789/0 conn(0x7fa48406ae30 legacy=0x7fa484131390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 monclient: found mon.b 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48409d510 con 0x7fa48406a340 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 5 monclient: authenticate success, global_id 94450 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _renew_subs 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4841355a0 con 0x7fa48406a340 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _renew_subs 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa484134400 con 0x7fa48406a340 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4740010a0 con 0x7fa48406a340 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa474004e90 con 0x7fa48406a340 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa47b7fe700 4 set_mon_vals no callback set 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 10 monclient: dump: 2024-09-16T06:40:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.334 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa464005100 con 0x7fa48406a340 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa47400f900 con 0x7fa48406a340 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7fa4740436b0 con 0x7fa48406a340 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.318+0000 7fa488b4f700 10 monclient: get_auth_request con 0x7fa468036220 auth_method 0 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.326+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa474014720 con 0x7fa48406a340 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.326+0000 7fa479ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.326+0000 7fa479ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:10.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa48b5b4700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T06:40:10.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa48b5b4700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T06:40:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa48b5b4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa48b5b4700 1 -- 172.21.15.189:0/3880694405 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7fa464005930 con 0x7fa48406a340 2024-09-16T06:40:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa479ffb700 1 -- 172.21.15.189:0/3880694405 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v38) v1 ==== 137+0+0 (unknown 2571951620 0 0) 0x7fa474012dc0 con 0x7fa48406a340 2024-09-16T06:40:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa479ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T06:40:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.482+0000 7fa479ffb700 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T06:40:10.488 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T06:40:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo 2024-09-16T06:40:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:10.492 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo 2024-09-16T06:40:10.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: get_monmap_and_config 2024-09-16T06:40:10.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: monmap: 2024-09-16T06:40:10.592 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:10.593430+0000 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:10.593430+0000 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:10.593 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 1 Processor -- start 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 1 -- start start 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: init 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:10.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-c con 0x7f9c380a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-b con 0x7f9c3809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-a con 0x7f9c380a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3809a040 con 0x7f9c380a3ca0 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c380a33d0 con 0x7f9c380a2350 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c38099d60 con 0x7f9c3809aa20 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient(hunting): authenticate will time out at 963.106508s 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3659c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9c3809aa20 legacy=0x7f9c3813a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c36d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c380a2350 0x7f9c380a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44228/0 (socket says 172.21.15.189:44228) 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c36d9d700 1 -- 172.21.15.189:0/3444342275 learned_addr learned my addr 172.21.15.189:0/3444342275 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801681810 0 0) 0x7f9c380a33d0 con 0x7f9c380a2350 2024-09-16T06:40:10.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): my global_id is 94453 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c200035d0 con 0x7f9c380a2350 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583009971 0 0) 0x7f9c3809a040 con 0x7f9c380a3ca0 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient(hunting): my global_id is 104247 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c380a33d0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396591458 0 0) 0x7f9c200035d0 con 0x7f9c380a2350 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 >> v1:172.21.15.76:6790/0 conn(0x7f9c3809aa20 legacy=0x7f9c3813a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 >> v1:172.21.15.76:6789/0 conn(0x7f9c380a3ca0 legacy=0x7f9c3813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 monclient: found mon.noname-c 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380a3920 con 0x7f9c380a2350 2024-09-16T06:40:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 5 monclient: authenticate success, global_id 94453 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c28001610 con 0x7f9c380a2350 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: dump: 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.601 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c280018f0 con 0x7f9c380a2350 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:10.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c3d05d700 4 set_mon_vals no callback set 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 1 -- 172.21.15.189:0/3444342275 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c28001c80 con 0x7f9c380a2350 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.590+0000 7f9c35d9b700 10 monclient: dump: 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: shutdown 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- 172.21.15.189:0/3444342275 >> v1:172.21.15.189:6789/0 conn(0x7f9c380a2350 legacy=0x7f9c380a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- 172.21.15.189:0/3444342275 shutdown_connections 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- 172.21.15.189:0/3444342275 >> 172.21.15.189:0/3444342275 conn(0x7f9c380a7d70 msgr2=0x7f9c380a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- 172.21.15.189:0/3444342275 shutdown_connections 2024-09-16T06:40:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- 172.21.15.189:0/3444342275 wait complete. 2024-09-16T06:40:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: monmap: 2024-09-16T06:40:10.608 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:10.609 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 Processor -- start 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- start start 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: init 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-a con 0x7f9c3809aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-c con 0x7f9c380a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient(hunting): picked mon.noname-b con 0x7f9c380a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c381cab10 con 0x7f9c3809aa20 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3809b5a0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3806bed0 con 0x7f9c380a2350 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3d05d700 10 monclient(hunting): authenticate will time out at 963.110686s 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3659c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9c380a2350 legacy=0x7f9c381cef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3759e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c380a3ca0 0x7f9c381d2650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44236/0 (socket says 172.21.15.189:44236) 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c3759e700 1 -- 172.21.15.189:0/2399316721 learned_addr learned my addr 172.21.15.189:0/2399316721 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671980968 0 0) 0x7f9c3809b5a0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): my global_id is 94456 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c240035d0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241003320 0 0) 0x7f9c381cab10 con 0x7f9c3809aa20 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): my global_id is 104250 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c3809b5a0 con 0x7f9c3809aa20 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1835586301 0 0) 0x7f9c240035d0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c381cab10 con 0x7f9c380a3ca0 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c30001c50 con 0x7f9c380a3ca0 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:10.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:10.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.594+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4290539109 0 0) 0x7f9c381cab10 con 0x7f9c380a3ca0 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 >> v1:172.21.15.76:6790/0 conn(0x7f9c380a2350 legacy=0x7f9c381cef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 >> v1:172.21.15.76:6789/0 conn(0x7f9c3809aa20 legacy=0x7f9c38135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 monclient: found mon.b 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c38122de0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 5 monclient: authenticate success, global_id 94456 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _renew_subs 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38123120 con 0x7f9c380a3ca0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _renew_subs 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c38135e10 con 0x7f9c380a3ca0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c30001fe0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c30003e90 con 0x7f9c380a3ca0 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3559a700 4 set_mon_vals no callback set 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 10 monclient: dump: 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9c30004070 con 0x7f9c380a3ca0 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7f9c30043930 con 0x7f9c380a3ca0 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c3d05d700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c18005100 con 0x7f9c380a3ca0 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.598+0000 7f9c36d9d700 10 monclient: get_auth_request con 0x7f9c24036180 auth_method 0 2024-09-16T06:40:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.610+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9c300149e0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.610+0000 7f9c1f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.610+0000 7f9c1f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c3d05d700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c3d05d700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c3d05d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c3d05d700 1 -- 172.21.15.189:0/2399316721 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7f9c180053b0 con 0x7f9c380a3ca0 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c1f7fe700 1 -- 172.21.15.189:0/2399316721 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v38) v1 ==== 274+0+0 (unknown 1273930891 0 0) 0x7f9c30014800 con 0x7f9c380a3ca0 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c1f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T06:40:10.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.774+0000 7f9c1f7fe700 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T06:40:10.777 INFO:tasks.workunit.client.0.smithi189.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T06:40:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-16T06:40:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:10.781 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.882+0000 7f57b4d79700 10 monclient: get_monmap_and_config 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.882+0000 7f57b4d79700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.882+0000 7f57b4d79700 10 monclient: monmap: 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:10.887359+0000 2024-09-16T06:40:10.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:10.887359+0000 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:10.887 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 Processor -- start 2024-09-16T06:40:10.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- start start 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: init 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-c con 0x7f57b0099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-a con 0x7f57b0099e40 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-b con 0x7f57b009a250 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b009c460 con 0x7f57b0099e40 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b009c1e0 con 0x7f57b0099a60 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b009bd60 con 0x7f57b009a250 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient(hunting): authenticate will time out at 963.400849s 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57aed9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f57b009a250 legacy=0x7f57b013a090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57ae59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57b0099a60 0x7f57b0098070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44238/0 (socket says 172.21.15.189:44238) 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57ae59c700 1 -- 172.21.15.189:0/3416201831 learned_addr learned my addr 172.21.15.189:0/3416201831 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510426656 0 0) 0x7f57b009c1e0 con 0x7f57b0099a60 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): my global_id is 94459 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c0046e0 con 0x7f57b0099a60 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368935838 0 0) 0x7f57b009c460 con 0x7f57b0099e40 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient(hunting): my global_id is 104253 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b009c1e0 con 0x7f57b0099e40 2024-09-16T06:40:10.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027629672 0 0) 0x7f579c0046e0 con 0x7f57b0099a60 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 >> v1:172.21.15.76:6790/0 conn(0x7f57b009a250 legacy=0x7f57b013a090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 >> v1:172.21.15.76:6789/0 conn(0x7f57b0099e40 legacy=0x7f57b0136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 monclient: found mon.noname-c 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b009d490 con 0x7f57b0099a60 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57a0001610 con 0x7f57b0099a60 2024-09-16T06:40:10.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 5 monclient: authenticate success, global_id 94459 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: dump: 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.895 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57a00018f0 con 0x7f57b0099a60 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 1 -- 172.21.15.189:0/3416201831 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57a0001c80 con 0x7f57b0099a60 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 4 set_mon_vals no callback set 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57add9b700 10 monclient: dump: 2024-09-16T06:40:10.896 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: shutdown 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:10.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- 172.21.15.189:0/3416201831 >> v1:172.21.15.189:6789/0 conn(0x7f57b0099a60 legacy=0x7f57b0098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- 172.21.15.189:0/3416201831 shutdown_connections 2024-09-16T06:40:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- 172.21.15.189:0/3416201831 >> 172.21.15.189:0/3416201831 conn(0x7f57b00a03e0 msgr2=0x7f57b00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- 172.21.15.189:0/3416201831 shutdown_connections 2024-09-16T06:40:10.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 1 -- 172.21.15.189:0/3416201831 wait complete. 2024-09-16T06:40:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: build_initial_monmap 2024-09-16T06:40:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.886+0000 7f57b4d79700 10 monclient: monmap: 2024-09-16T06:40:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:10.903 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 Processor -- start 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- start start 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: init 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-c con 0x7f57b0099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:10.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-a con 0x7f57b0099e40 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient(hunting): picked mon.noname-b con 0x7f57b009a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b00a15f0 con 0x7f57b0099e40 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b0092fb0 con 0x7f57b009a250 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b0090be0 con 0x7f57b0099a60 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient(hunting): authenticate will time out at 963.405300s 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57ae59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f57b0099a60 legacy=0x7f57b0096500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57aed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57b009a250 0x7f57b01d0250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44240/0 (socket says 172.21.15.189:44240) 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57aed9d700 1 -- 172.21.15.189:0/1326163376 learned_addr learned my addr 172.21.15.189:0/1326163376 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069528753 0 0) 0x7f57b0092fb0 con 0x7f57b009a250 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): my global_id is 94462 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57900035d0 con 0x7f57b009a250 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551556711 0 0) 0x7f57b009c1e0 con 0x7f57b0099e40 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): my global_id is 104256 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b0092fb0 con 0x7f57b0099e40 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 945035493 0 0) 0x7f57900035d0 con 0x7f57b009a250 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57b009c1e0 con 0x7f57b009a250 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57a80044f0 con 0x7f57b009a250 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:10.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3954792766 0 0) 0x7f57b009c1e0 con 0x7f57b009a250 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 >> v1:172.21.15.76:6790/0 conn(0x7f57b0099a60 legacy=0x7f57b0096500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 >> v1:172.21.15.76:6789/0 conn(0x7f57b0099e40 legacy=0x7f57b01ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 monclient: found mon.b 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b009eb40 con 0x7f57b009a250 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 5 monclient: authenticate success, global_id 94462 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _renew_subs 2024-09-16T06:40:10.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b009a6e0 con 0x7f57b009a250 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _renew_subs 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57b4d79700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b0092cd0 con 0x7f57b009a250 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57a80047d0 con 0x7f57b009a250 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57acd99700 4 set_mon_vals no callback set 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57a8004b60 con 0x7f57b009a250 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.890+0000 7f57a6ffd700 10 monclient: dump: 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:10.909 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57b4d79700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f57a8004cc0 con 0x7f57b009a250 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57b4d79700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57b4d79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:10.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57b4d79700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794005100 con 0x7f57b009a250 2024-09-16T06:40:10.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7f57a8043820 con 0x7f57b009a250 2024-09-16T06:40:10.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.894+0000 7f57ae59c700 10 monclient: get_auth_request con 0x7f5790035fd0 auth_method 0 2024-09-16T06:40:10.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.902+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f57a8005510 con 0x7f57b009a250 2024-09-16T06:40:10.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.902+0000 7f57a6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:10.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:10.902+0000 7f57a6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:11.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57b4d79700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T06:40:11.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57b4d79700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T06:40:11.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57b4d79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57b4d79700 1 -- 172.21.15.189:0/1326163376 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7f5794005930 con 0x7f57b009a250 2024-09-16T06:40:11.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57a6ffd700 1 -- 172.21.15.189:0/1326163376 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v38) v1 ==== 290+0+0 (unknown 3836209161 0 0) 0x7f57a8012f70 con 0x7f57b009a250 2024-09-16T06:40:11.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57a6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T06:40:11.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.058+0000 7f57a6ffd700 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T06:40:11.066 INFO:tasks.workunit.client.0.smithi189.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T06:40:11.067 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:11.067 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-16T06:40:11.067 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:11.067 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --force 2024-09-16T06:40:11.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 10 monclient: monmap: 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:11.174800+0000 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:11.174800+0000 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.174 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.175 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:11.175 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:11.175 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 1 Processor -- start 2024-09-16T06:40:11.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 1 -- start start 2024-09-16T06:40:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.170+0000 7f7f3ffff700 10 monclient: init 2024-09-16T06:40:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f7f400a10a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f7f400710c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f7f40071550 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400a1480 con 0x7f7f40071550 2024-09-16T06:40:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f40133400 con 0x7f7f400710c0 2024-09-16T06:40:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4009d360 con 0x7f7f400a10a0 2024-09-16T06:40:11.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient(hunting): authenticate will time out at 963.689144s 2024-09-16T06:40:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f37fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7f400710c0 0x7f7f40137f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44242/0 (socket says 172.21.15.189:44242) 2024-09-16T06:40:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f37fff700 1 -- 172.21.15.189:0/1159195982 learned_addr learned my addr 172.21.15.189:0/1159195982 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3effd700 1 -- 172.21.15.189:0/1159195982 >> v1:172.21.15.76:6790/0 conn(0x7f7f400a10a0 legacy=0x7f7f40132cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511871224 0 0) 0x7f7f40133400 con 0x7f7f400710c0 2024-09-16T06:40:11.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient(hunting): my global_id is 94465 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c0035d0 con 0x7f7f400710c0 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2049418766 0 0) 0x7f7f2c0035d0 con 0x7f7f400710c0 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 >> v1:172.21.15.76:6790/0 conn(0x7f7f400a10a0 legacy=0x7f7f40132cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 >> v1:172.21.15.76:6789/0 conn(0x7f7f40071550 legacy=0x7f7f4013b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:40:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4009d640 con 0x7f7f400710c0 2024-09-16T06:40:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 5 monclient: authenticate success, global_id 94465 2024-09-16T06:40:11.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:11.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f28001610 con 0x7f7f400710c0 2024-09-16T06:40:11.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:11.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: dump: 2024-09-16T06:40:11.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.198 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.198 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:11.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f28001930 con 0x7f7f400710c0 2024-09-16T06:40:11.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 1 -- 172.21.15.189:0/1159195982 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f28001cc0 con 0x7f7f400710c0 2024-09-16T06:40:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 4 set_mon_vals no callback set 2024-09-16T06:40:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3e7fc700 10 monclient: dump: 2024-09-16T06:40:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 10 monclient: shutdown 2024-09-16T06:40:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- 172.21.15.189:0/1159195982 >> v1:172.21.15.189:6789/0 conn(0x7f7f400710c0 legacy=0x7f7f40137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- 172.21.15.189:0/1159195982 shutdown_connections 2024-09-16T06:40:11.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- 172.21.15.189:0/1159195982 >> 172.21.15.189:0/1159195982 conn(0x7f7f400a0580 msgr2=0x7f7f400a0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- 172.21.15.189:0/1159195982 shutdown_connections 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.174+0000 7f7f3ffff700 1 -- 172.21.15.189:0/1159195982 wait complete. 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: monmap: 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.204 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 Processor -- start 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- start start 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: init 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f7f400710c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f7f40071550 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f7f400a10a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f40134b40 con 0x7f7f400710c0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f40134750 con 0x7f7f40071550 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400a2f80 con 0x7f7f400a10a0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3f7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f7f400a10a0 legacy=0x7f7f40070300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient(hunting): authenticate will time out at 963.694140s 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f37fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7f40071550 0x7f7f4006fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44250/0 (socket says 172.21.15.189:44250) 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f37fff700 1 -- 172.21.15.189:0/428384279 learned_addr learned my addr 172.21.15.189:0/428384279 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586519151 0 0) 0x7f7f40134750 con 0x7f7f40071550 2024-09-16T06:40:11.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): my global_id is 94468 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f200035d0 con 0x7f7f40071550 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684988027 0 0) 0x7f7f40134b40 con 0x7f7f400710c0 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): my global_id is 104262 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f40134750 con 0x7f7f400710c0 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 828206111 0 0) 0x7f7f200035d0 con 0x7f7f40071550 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f40134b40 con 0x7f7f40071550 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f280015d0 con 0x7f7f40071550 2024-09-16T06:40:11.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): dump: 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 751076699 0 0) 0x7f7f40134b40 con 0x7f7f40071550 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 >> v1:172.21.15.76:6790/0 conn(0x7f7f400a10a0 legacy=0x7f7f40070300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 >> v1:172.21.15.76:6789/0 conn(0x7f7f400710c0 legacy=0x7f7f4006f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 monclient: found mon.b 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f401360b0 con 0x7f7f40071550 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 5 monclient: authenticate success, global_id 94468 2024-09-16T06:40:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _renew_subs 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f40131c80 con 0x7f7f40071550 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _renew_subs 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3ffff700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f40135dd0 con 0x7f7f40071550 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f28004ca0 con 0x7f7f40071550 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f28004e10 con 0x7f7f40071550 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3dffb700 4 set_mon_vals no callback set 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.178+0000 7f7f3cff9700 10 monclient: dump: 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7f2800f880 con 0x7f7f40071550 2024-09-16T06:40:11.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3ffff700 1 -- 172.21.15.189:0/428384279 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f24005100 con 0x7f7f40071550 2024-09-16T06:40:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7f7f28043730 con 0x7f7f40071550 2024-09-16T06:40:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.182+0000 7f7f3effd700 10 monclient: get_auth_request con 0x7f7f20036220 auth_method 0 2024-09-16T06:40:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.190+0000 7f7f3cff9700 1 -- 172.21.15.189:0/428384279 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7f280147c0 con 0x7f7f40071550 2024-09-16T06:40:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.190+0000 7f7f3cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.190+0000 7f7f3cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-16T06:40:11.364 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['--force'] 2024-09-16T06:40:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T06:40:11.365 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T06:40:11.367 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T06:40:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: get_monmap_and_config 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: monmap: 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:11.474447+0000 2024-09-16T06:40:11.473 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:11.474447+0000 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:11.474 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 1 Processor -- start 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 1 -- start start 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: init 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-a con 0x7fada006a340 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-c con 0x7fada009d5b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-b con 0x7fada013bb20 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada006b880 con 0x7fada006a340 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada006be90 con 0x7fada009d5b0 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada00a1a40 con 0x7fada013bb20 2024-09-16T06:40:11.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fada4c91700 10 monclient(hunting): authenticate will time out at 963.987569s 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fad9ed9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fada013bb20 legacy=0x7fada013df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fad9dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fada009d5b0 0x7fada006ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44266/0 (socket says 172.21.15.189:44266) 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.470+0000 7fad9dd9b700 1 -- 172.21.15.189:0/2337887783 learned_addr learned my addr 172.21.15.189:0/2337887783 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705678843 0 0) 0x7fada006be90 con 0x7fada009d5b0 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): my global_id is 94471 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad880035d0 con 0x7fada009d5b0 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249716300 0 0) 0x7fada006b880 con 0x7fada006a340 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient(hunting): my global_id is 104265 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada006be90 con 0x7fada006a340 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511467940 0 0) 0x7fad880035d0 con 0x7fada009d5b0 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 >> v1:172.21.15.76:6790/0 conn(0x7fada013bb20 legacy=0x7fada013df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 >> v1:172.21.15.76:6789/0 conn(0x7fada006a340 legacy=0x7fada006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 monclient: found mon.noname-c 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00a16f0 con 0x7fada009d5b0 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 5 monclient: authenticate success, global_id 94471 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad90001610 con 0x7fada009d5b0 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: dump: 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad900018f0 con 0x7fada009d5b0 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 1 -- 172.21.15.189:0/2337887783 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad90001c80 con 0x7fada009d5b0 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 4 set_mon_vals no callback set 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9d59a700 10 monclient: dump: 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.481 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: shutdown 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- 172.21.15.189:0/2337887783 >> v1:172.21.15.189:6789/0 conn(0x7fada009d5b0 legacy=0x7fada006ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- 172.21.15.189:0/2337887783 shutdown_connections 2024-09-16T06:40:11.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- 172.21.15.189:0/2337887783 >> 172.21.15.189:0/2337887783 conn(0x7fada00a21f0 msgr2=0x7fada00a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- 172.21.15.189:0/2337887783 shutdown_connections 2024-09-16T06:40:11.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- 172.21.15.189:0/2337887783 wait complete. 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: monmap: 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.487 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 Processor -- start 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- start start 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: init 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-c con 0x7fada006a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-b con 0x7fada009d5b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient(hunting): picked mon.noname-a con 0x7fada013bb20 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada0136b60 con 0x7fada013bb20 2024-09-16T06:40:11.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada009c700 con 0x7fada009d5b0 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada01390d0 con 0x7fada006a340 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fada4c91700 10 monclient(hunting): authenticate will time out at 963.991832s 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9e59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fada006a340 legacy=0x7fada01c9f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fada009d5b0 0x7fada01cf0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44276/0 (socket says 172.21.15.189:44276) 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.474+0000 7fad9dd9b700 1 -- 172.21.15.189:0/2711694100 learned_addr learned my addr 172.21.15.189:0/2711694100 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397640125 0 0) 0x7fada009c700 con 0x7fada009d5b0 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): my global_id is 94474 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad800035d0 con 0x7fada009d5b0 2024-09-16T06:40:11.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287984065 0 0) 0x7fada0136b60 con 0x7fada013bb20 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): my global_id is 104268 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada009c700 con 0x7fada013bb20 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 939310167 0 0) 0x7fad800035d0 con 0x7fada009d5b0 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada0136b60 con 0x7fada009d5b0 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad900015d0 con 0x7fada009d5b0 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): dump: 2024-09-16T06:40:11.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3376352025 0 0) 0x7fada0136b60 con 0x7fada009d5b0 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 >> v1:172.21.15.76:6790/0 conn(0x7fada006a340 legacy=0x7fada01c9f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 >> v1:172.21.15.76:6789/0 conn(0x7fada013bb20 legacy=0x7fada01d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 monclient: found mon.b 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0136d10 con 0x7fada009d5b0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 5 monclient: authenticate success, global_id 94474 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _renew_subs 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada01345e0 con 0x7fada009d5b0 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _renew_subs 2024-09-16T06:40:11.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fada0071e90 con 0x7fada009d5b0 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad90004ca0 con 0x7fada009d5b0 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad90004fb0 con 0x7fada009d5b0 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad9cd99700 4 set_mon_vals no callback set 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 10 monclient: dump: 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.493 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fad9000fa20 con 0x7fada009d5b0 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7fad900438b0 con 0x7fada009d5b0 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fada4c91700 1 -- 172.21.15.189:0/2711694100 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada009f9f0 con 0x7fada009d5b0 2024-09-16T06:40:11.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.478+0000 7fad9e59c700 10 monclient: get_auth_request con 0x7fad80036220 auth_method 0 2024-09-16T06:40:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.486+0000 7fad8effd700 1 -- 172.21.15.189:0/2711694100 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fad900148e0 con 0x7fada009d5b0 2024-09-16T06:40:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.486+0000 7fad8effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.486+0000 7fad8effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-16T06:40:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-16T06:40:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T06:40:11.659 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T06:40:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T06:40:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:11.663 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T06:40:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: get_monmap_and_config 2024-09-16T06:40:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: monmap: 2024-09-16T06:40:11.773 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:11.774603+0000 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:11.774603+0000 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:11.774 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 1 Processor -- start 2024-09-16T06:40:11.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 1 -- start start 2024-09-16T06:40:11.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: init 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-b con 0x7fe84409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-c con 0x7fe844099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-a con 0x7fe8441396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84409f520 con 0x7fe8441396a0 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84409bc10 con 0x7fe844099700 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84409b810 con 0x7fe84409f140 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe84b2a6700 10 monclient(hunting): authenticate will time out at 964.287708s 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe849042700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe84409f140 legacy=0x7fe84409a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.770+0000 7fe848841700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe844099700 0x7fe84409b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44292/0 (socket says 172.21.15.189:44292) 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe848841700 1 -- 172.21.15.189:0/933134916 learned_addr learned my addr 172.21.15.189:0/933134916 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903462112 0 0) 0x7fe84409f520 con 0x7fe8441396a0 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): my global_id is 104271 2024-09-16T06:40:11.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8300035d0 con 0x7fe8441396a0 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465631916 0 0) 0x7fe84409bc10 con 0x7fe844099700 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient(hunting): my global_id is 94477 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84409f520 con 0x7fe844099700 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747076384 0 0) 0x7fe84409f520 con 0x7fe844099700 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 >> v1:172.21.15.76:6790/0 conn(0x7fe84409f140 legacy=0x7fe84409a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 >> v1:172.21.15.76:6789/0 conn(0x7fe8441396a0 legacy=0x7fe84413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 monclient: found mon.noname-c 2024-09-16T06:40:11.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84409ceb0 con 0x7fe844099700 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 5 monclient: authenticate success, global_id 94477 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe8340015f0 con 0x7fe844099700 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: dump: 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe834001890 con 0x7fe844099700 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:11.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 1 -- 172.21.15.189:0/933134916 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe834001c60 con 0x7fe844099700 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 4 set_mon_vals no callback set 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe83bfff700 10 monclient: dump: 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.789 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: shutdown 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- 172.21.15.189:0/933134916 >> v1:172.21.15.189:6789/0 conn(0x7fe844099700 legacy=0x7fe84409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- 172.21.15.189:0/933134916 shutdown_connections 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- 172.21.15.189:0/933134916 >> 172.21.15.189:0/933134916 conn(0x7fe8440a03e0 msgr2=0x7fe8440a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- 172.21.15.189:0/933134916 shutdown_connections 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- 172.21.15.189:0/933134916 wait complete. 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: build_initial_monmap 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: monmap: 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:11.790 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 Processor -- start 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- start start 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: init 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-c con 0x7fe844099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-b con 0x7fe84409f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient(hunting): picked mon.noname-a con 0x7fe8441396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe844136840 con 0x7fe8441396a0 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8440a2c70 con 0x7fe84409f140 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8440a31a0 con 0x7fe844099700 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe84b2a6700 10 monclient(hunting): authenticate will time out at 964.291463s 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe849042700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe844099700 legacy=0x7fe8441c73b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe848841700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe84409f140 0x7fe844138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44302/0 (socket says 172.21.15.189:44302) 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe848841700 1 -- 172.21.15.189:0/2838937591 learned_addr learned my addr 172.21.15.189:0/2838937591 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.774+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039845672 0 0) 0x7fe8440a2c70 con 0x7fe84409f140 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): my global_id is 94480 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8280035d0 con 0x7fe84409f140 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503932519 0 0) 0x7fe844136840 con 0x7fe8441396a0 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): my global_id is 104274 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8440a2c70 con 0x7fe8441396a0 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684479321 0 0) 0x7fe8280035d0 con 0x7fe84409f140 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe844136840 con 0x7fe84409f140 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe8340014f0 con 0x7fe84409f140 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:11.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3138422950 0 0) 0x7fe8440a2c70 con 0x7fe8441396a0 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8280035d0 con 0x7fe8441396a0 2024-09-16T06:40:11.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe83c001d70 con 0x7fe8441396a0 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3906647741 0 0) 0x7fe844136840 con 0x7fe84409f140 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 >> v1:172.21.15.76:6790/0 conn(0x7fe844099700 legacy=0x7fe8441c73b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 >> v1:172.21.15.76:6789/0 conn(0x7fe8441396a0 legacy=0x7fe8441cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 monclient: found mon.b 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84409a6e0 con 0x7fe84409f140 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 5 monclient: authenticate success, global_id 94480 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _renew_subs 2024-09-16T06:40:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe844139490 con 0x7fe84409f140 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _renew_subs 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8441344e0 con 0x7fe84409f140 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe834004ca0 con 0x7fe84409f140 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe83affd700 4 set_mon_vals no callback set 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe834004fb0 con 0x7fe84409f140 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 10 monclient: dump: 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe84b2a6700 1 -- 172.21.15.189:0/2838937591 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824005100 con 0x7fe84409f140 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe83400fa20 con 0x7fe84409f140 2024-09-16T06:40:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7fe834043820 con 0x7fe84409f140 2024-09-16T06:40:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.778+0000 7fe849042700 10 monclient: get_auth_request con 0x7fe8280362f0 auth_method 0 2024-09-16T06:40:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.790+0000 7fe839ffb700 1 -- 172.21.15.189:0/2838937591 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe834014830 con 0x7fe84409f140 2024-09-16T06:40:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.790+0000 7fe839ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:11.790+0000 7fe839ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-16T06:40:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:Invalid command: unused arguments: ['foo'] 2024-09-16T06:40:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T06:40:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: invalid command 2024-09-16T06:40:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:11.963 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-16T06:40:12.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: get_monmap_and_config 2024-09-16T06:40:12.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: build_initial_monmap 2024-09-16T06:40:12.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: monmap: 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:12.069561+0000 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:12.069561+0000 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:12.069 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 1 Processor -- start 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 1 -- start start 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: init 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:12.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc09cca0 con 0x7fc7dc138da0 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc09f750 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc097620 con 0x7fc7dc09bb00 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 10 monclient(hunting): authenticate will time out at 964.582590s 2024-09-16T06:40:12.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc7dc09c5f0 0x7fc7dc137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44308/0 (socket says 172.21.15.189:44308) 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7da59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fc7dc09bb00 legacy=0x7fc7dc09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d9d9b700 1 -- 172.21.15.189:0/937095687 learned_addr learned my addr 172.21.15.189:0/937095687 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918249775 0 0) 0x7fc7dc09f750 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): my global_id is 94483 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c80035d0 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994299000 0 0) 0x7fc7dc09cca0 con 0x7fc7dc138da0 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient(hunting): my global_id is 104277 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc09f750 con 0x7fc7dc138da0 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397720222 0 0) 0x7fc7c80035d0 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 >> v1:172.21.15.76:6790/0 conn(0x7fc7dc09bb00 legacy=0x7fc7dc09bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 >> v1:172.21.15.76:6789/0 conn(0x7fc7dc138da0 legacy=0x7fc7dc13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 monclient: found mon.noname-c 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc097900 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: _finish_auth 0 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7c4001610 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7d959a700 10 monclient: dump: 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.078 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 5 monclient: authenticate success, global_id 94483 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.066+0000 7fc7e0969700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7d959a700 1 -- 172.21.15.189:0/937095687 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7c40018f0 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7d959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 4 set_mon_vals no callback set 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: shutdown 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:12.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- 172.21.15.189:0/937095687 >> v1:172.21.15.189:6789/0 conn(0x7fc7dc09c5f0 legacy=0x7fc7dc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- 172.21.15.189:0/937095687 shutdown_connections 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- 172.21.15.189:0/937095687 >> 172.21.15.189:0/937095687 conn(0x7fc7dc0a03e0 msgr2=0x7fc7dc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- 172.21.15.189:0/937095687 shutdown_connections 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- 172.21.15.189:0/937095687 wait complete. 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: build_initial_monmap 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: monmap: 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:12.080 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 Processor -- start 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- start start 2024-09-16T06:40:12.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: init 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc095050 con 0x7fc7dc09bb00 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc08f4c0 con 0x7fc7dc138da0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7d9d9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fc7dc09c5f0 legacy=0x7fc7dc08e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0a2300 con 0x7fc7dc09c5f0 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7e0969700 10 monclient(hunting): authenticate will time out at 964.587893s 2024-09-16T06:40:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7dad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc7dc138da0 0x7fc7dc1d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44310/0 (socket says 172.21.15.189:44310) 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.070+0000 7fc7dad9d700 1 -- 172.21.15.189:0/2766634000 learned_addr learned my addr 172.21.15.189:0/2766634000 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774313323 0 0) 0x7fc7dc08f4c0 con 0x7fc7dc138da0 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): my global_id is 94486 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7bc0035d0 con 0x7fc7dc138da0 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685626859 0 0) 0x7fc7dc095050 con 0x7fc7dc09bb00 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): my global_id is 104280 2024-09-16T06:40:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc08f4c0 con 0x7fc7dc09bb00 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412474644 0 0) 0x7fc7bc0035d0 con 0x7fc7dc138da0 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc095050 con 0x7fc7dc138da0 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d4001c30 con 0x7fc7dc138da0 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.088 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882490336 0 0) 0x7fc7dc08f4c0 con 0x7fc7dc09bb00 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7bc0035d0 con 0x7fc7dc09bb00 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7cc0041a0 con 0x7fc7dc09bb00 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2420485058 0 0) 0x7fc7dc095050 con 0x7fc7dc138da0 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 >> v1:172.21.15.76:6790/0 conn(0x7fc7dc09c5f0 legacy=0x7fc7dc08e3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 >> v1:172.21.15.76:6789/0 conn(0x7fc7dc09bb00 legacy=0x7fc7dc08dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 monclient: found mon.b 2024-09-16T06:40:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc1386b0 con 0x7fc7dc138da0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 5 monclient: authenticate success, global_id 94486 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _renew_subs 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc094d70 con 0x7fc7dc138da0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _renew_subs 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc09b650 con 0x7fc7dc138da0 2024-09-16T06:40:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7d4001fe0 con 0x7fc7dc138da0 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d4003e90 con 0x7fc7dc138da0 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d8d99700 4 set_mon_vals no callback set 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 10 monclient: dump: 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0005100 con 0x7fc7dc138da0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc7d4004070 con 0x7fc7dc138da0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(38..38 src has 1..38) v4 ==== 4634+0+0 (unknown 1987267958 0 0) 0x7fc7d40438a0 con 0x7fc7dc138da0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.074+0000 7fc7da59c700 10 monclient: get_auth_request con 0x7fc7bc0362a0 auth_method 0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.082+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc7d4014ba0 con 0x7fc7dc138da0 2024-09-16T06:40:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.082+0000 7fc7d2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.082+0000 7fc7d2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.254+0000 7fc7e0969700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.254+0000 7fc7e0969700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.254+0000 7fc7e0969700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.254+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc7c0005930 con 0x7fc7dc138da0 2024-09-16T06:40:12.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.450+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v39) v1 ==== 155+0+0 (unknown 3765254449 0 0) 0x7fc7d4013010 con 0x7fc7dc138da0 2024-09-16T06:40:12.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.450+0000 7fc7d2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.450+0000 7fc7d2ffd700 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-16T06:40:12.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7e0969700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7e0969700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7e0969700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc7c0005420 con 0x7fc7dc138da0 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7d2ffd700 1 -- 172.21.15.189:0/2766634000 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v39) v1 ==== 162+0+0 (unknown 1102853297 0 0) 0x7fc7d4014790 con 0x7fc7dc138da0 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7d2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.558+0000 7fc7d2ffd700 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7bc0362a0 legacy=0x7fc7bc038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 10 monclient: shutdown 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 >> v1:172.21.15.189:6789/0 conn(0x7fc7dc138da0 legacy=0x7fc7dc1d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 shutdown_connections 2024-09-16T06:40:12.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 >> 172.21.15.189:0/2766634000 conn(0x7fc7dc0a03e0 msgr2=0x7fc7dc06db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:12.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 shutdown_connections 2024-09-16T06:40:12.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.562+0000 7fc7e0969700 1 -- 172.21.15.189:0/2766634000 wait complete. 2024-09-16T06:40:12.565 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foooo' does not exist 2024-09-16T06:40:12.573 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T06:40:12.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: get_monmap_and_config 2024-09-16T06:40:12.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: build_initial_monmap 2024-09-16T06:40:12.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: monmap: 2024-09-16T06:40:12.681 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:12.682189+0000 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:12.682189+0000 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:12.682 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 1 Processor -- start 2024-09-16T06:40:12.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 1 -- start start 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: init 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient(hunting): picked mon.noname-c con 0x7faa5c132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient(hunting): picked mon.noname-b con 0x7faa5c138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient(hunting): picked mon.noname-a con 0x7faa5c13bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c06abd0 con 0x7faa5c13bf70 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c06b170 con 0x7faa5c132080 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c132b70 con 0x7faa5c138460 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa62183700 10 monclient(hunting): authenticate will time out at 965.195779s 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa60980700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7faa5c138460 legacy=0x7faa5c13a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:12.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.678+0000 7faa61181700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa5c132080 0x7faa5c132460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44320/0 (socket says 172.21.15.189:44320) 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa61181700 1 -- 172.21.15.189:0/1995045109 learned_addr learned my addr 172.21.15.189:0/1995045109 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502483637 0 0) 0x7faa5c06b170 con 0x7faa5c132080 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): my global_id is 94489 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480035d0 con 0x7faa5c132080 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641990606 0 0) 0x7faa5c06abd0 con 0x7faa5c13bf70 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient(hunting): my global_id is 104283 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c06b170 con 0x7faa5c13bf70 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 40161687 0 0) 0x7faa480035d0 con 0x7faa5c132080 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 >> v1:172.21.15.76:6790/0 conn(0x7faa5c138460 legacy=0x7faa5c13a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 >> v1:172.21.15.76:6789/0 conn(0x7faa5c13bf70 legacy=0x7faa5c13e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 monclient: found mon.noname-c 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c0a38b0 con 0x7faa5c132080 2024-09-16T06:40:12.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: _finish_auth 0 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 5 monclient: authenticate success, global_id 94489 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa58001610 con 0x7faa5c132080 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: dump: 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa580018f0 con 0x7faa5c132080 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 4 set_mon_vals no callback set 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 1 -- 172.21.15.189:0/1995045109 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa58001c80 con 0x7faa5c132080 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:12.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa4ffff700 10 monclient: dump: 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.692 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 10 monclient: shutdown 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 -- 172.21.15.189:0/1995045109 >> v1:172.21.15.189:6789/0 conn(0x7faa5c132080 legacy=0x7faa5c132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 -- 172.21.15.189:0/1995045109 shutdown_connections 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 -- 172.21.15.189:0/1995045109 >> 172.21.15.189:0/1995045109 conn(0x7faa5c0a4660 msgr2=0x7faa5c099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 -- 172.21.15.189:0/1995045109 shutdown_connections 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 -- 172.21.15.189:0/1995045109 wait complete. 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 10 monclient: build_initial_monmap 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 10 monclient: monmap: 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:12.693 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:12.694 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:12.694 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:12.694 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:12.694 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:12.694 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:12.695 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.682+0000 7faa62183700 1 Processor -- start 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- start start 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: init 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient(hunting): picked mon.noname-b con 0x7faa5c132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient(hunting): picked mon.noname-c con 0x7faa5c138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient(hunting): picked mon.noname-a con 0x7faa5c13bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c09cb50 con 0x7faa5c13bf70 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c138200 con 0x7faa5c132080 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c09e720 con 0x7faa5c138460 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient(hunting): authenticate will time out at 965.201119s 2024-09-16T06:40:12.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa60980700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7faa5c138460 legacy=0x7faa5c09c440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa61181700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa5c132080 0x7faa5c09bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44330/0 (socket says 172.21.15.189:44330) 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa61181700 1 -- 172.21.15.189:0/2245173446 learned_addr learned my addr 172.21.15.189:0/2245173446 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326631396 0 0) 0x7faa5c138200 con 0x7faa5c132080 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): my global_id is 94492 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa400035d0 con 0x7faa5c132080 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608076353 0 0) 0x7faa5c09cb50 con 0x7faa5c13bf70 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:12.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): my global_id is 104286 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c138200 con 0x7faa5c13bf70 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515852329 0 0) 0x7faa400035d0 con 0x7faa5c132080 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa5c09cb50 con 0x7faa5c132080 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa580015d0 con 0x7faa5c132080 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): dump: 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.705 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2100837548 0 0) 0x7faa5c138200 con 0x7faa5c13bf70 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa400035d0 con 0x7faa5c13bf70 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa54001bb0 con 0x7faa5c13bf70 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient(hunting): dump: 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.706 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1982423191 0 0) 0x7faa5c09cb50 con 0x7faa5c132080 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 >> v1:172.21.15.76:6790/0 conn(0x7faa5c138460 legacy=0x7faa5c09c440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 >> v1:172.21.15.76:6789/0 conn(0x7faa5c13bf70 legacy=0x7faa5c1d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 monclient: found mon.b 2024-09-16T06:40:12.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c0a0170 con 0x7faa5c132080 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 5 monclient: authenticate success, global_id 94492 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _renew_subs 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c09f8d0 con 0x7faa5c132080 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa58004ca0 con 0x7faa5c132080 2024-09-16T06:40:12.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa58005030 con 0x7faa5c132080 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 10 monclient: dump: 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:12.709 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4f7fe700 4 set_mon_vals no callback set 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _renew_subs 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c09f5a0 con 0x7faa5c132080 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faa580052a0 con 0x7faa5c132080 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.686+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c0666e0 con 0x7faa5c132080 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.690+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 4290+0+0 (unknown 563664783 0 0) 0x7faa580437c0 con 0x7faa5c132080 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.690+0000 7faa60980700 10 monclient: get_auth_request con 0x7faa400362a0 auth_method 0 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.694+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faa580149e0 con 0x7faa5c132080 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.694+0000 7faa4dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:12.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.694+0000 7faa4dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.862+0000 7faa62183700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.862+0000 7faa62183700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.862+0000 7faa62183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:12.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:12.862+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa5c09fac0 con 0x7faa5c132080 2024-09-16T06:40:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.466+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v40) v1 ==== 152+0+0 (unknown 1878772362 0 0) 0x7faa58013080 con 0x7faa5c132080 2024-09-16T06:40:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.466+0000 7faa4dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.466+0000 7faa4dffb700 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-16T06:40:13.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa62183700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa62183700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa62183700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa5c09e1f0 con 0x7faa5c132080 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa4dffb700 1 -- 172.21.15.189:0/2245173446 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v40) v1 ==== 159+0+0 (unknown 2860897059 0 0) 0x7faa58014800 con 0x7faa5c132080 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa4dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.626+0000 7faa4dffb700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 >> v1:172.21.15.189:6812/3098810674 conn(0x7faa400362a0 legacy=0x7faa40038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 10 monclient: shutdown 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 >> v1:172.21.15.189:6789/0 conn(0x7faa5c132080 legacy=0x7faa5c09bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 shutdown_connections 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 >> 172.21.15.189:0/2245173446 conn(0x7faa5c0a4660 msgr2=0x7faa5c136a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 shutdown_connections 2024-09-16T06:40:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.630+0000 7faa62183700 1 -- 172.21.15.189:0/2245173446 wait complete. 2024-09-16T06:40:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T06:40:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: get_monmap_and_config 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: build_initial_monmap 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: monmap: 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:13.751397+0000 2024-09-16T06:40:13.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:13.751397+0000 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:13.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 1 Processor -- start 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 1 -- start start 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: init 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-c con 0x7f9e24099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.746+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-a con 0x7f9e2409ada0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-b con 0x7f9e2409b1b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e240998c0 con 0x7f9e2409ada0 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2409bf60 con 0x7f9e24099c80 2024-09-16T06:40:13.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2409d410 con 0x7f9e2409b1b0 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient(hunting): authenticate will time out at 966.264075s 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2ab07700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9e2409b1b0 legacy=0x7f9e2413aa50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2a306700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e24099c80 0x7f9e2409a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44334/0 (socket says 172.21.15.189:44334) 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2a306700 1 -- 172.21.15.189:0/3732270794 learned_addr learned my addr 172.21.15.189:0/3732270794 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556175360 0 0) 0x7f9e2409bf60 con 0x7f9e24099c80 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _negotiate 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): my global_id is 94495 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e100035d0 con 0x7f9e24099c80 2024-09-16T06:40:13.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015659165 0 0) 0x7f9e240998c0 con 0x7f9e2409ada0 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _negotiate 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient(hunting): my global_id is 104289 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2409bf60 con 0x7f9e2409ada0 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110768997 0 0) 0x7f9e100035d0 con 0x7f9e24099c80 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 >> v1:172.21.15.76:6790/0 conn(0x7f9e2409b1b0 legacy=0x7f9e2413aa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 >> v1:172.21.15.76:6789/0 conn(0x7f9e2409ada0 legacy=0x7f9e24137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: _finish_hunting 0 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 monclient: found mon.noname-c 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2409c160 con 0x7f9e24099c80 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: _finish_auth 0 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: _check_auth_tickets 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 5 monclient: authenticate success, global_id 94495 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e14001610 con 0x7f9e24099c80 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: dump: 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.762 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e14001870 con 0x7f9e24099c80 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 1 -- 172.21.15.189:0/3732270794 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e14001c40 con 0x7f9e24099c80 2024-09-16T06:40:13.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 4 set_mon_vals no callback set 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e29304700 10 monclient: dump: 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.764 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient: shutdown 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- 172.21.15.189:0/3732270794 >> v1:172.21.15.189:6789/0 conn(0x7f9e24099c80 legacy=0x7f9e2409a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- 172.21.15.189:0/3732270794 shutdown_connections 2024-09-16T06:40:13.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- 172.21.15.189:0/3732270794 >> 172.21.15.189:0/3732270794 conn(0x7f9e2409fe70 msgr2=0x7f9e240a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- 172.21.15.189:0/3732270794 shutdown_connections 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- 172.21.15.189:0/3732270794 wait complete. 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient: build_initial_monmap 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient: monmap: 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:13.766 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 Processor -- start 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 1 -- start start 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.750+0000 7f9e2b308700 10 monclient: init 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-a con 0x7f9e24099c80 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-c con 0x7f9e2409ada0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient(hunting): picked mon.noname-b con 0x7f9e2409b1b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2408ebc0 con 0x7f9e24099c80 2024-09-16T06:40:13.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e240906f0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e24090cb0 con 0x7f9e2409ada0 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient(hunting): authenticate will time out at 966.268249s 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e29b05700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9e2409ada0 legacy=0x7f9e24096500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2ab07700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e2409b1b0 0x7f9e241cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44348/0 (socket says 172.21.15.189:44348) 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2ab07700 1 -- 172.21.15.189:0/2809492389 learned_addr learned my addr 172.21.15.189:0/2809492389 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251890258 0 0) 0x7f9e240906f0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:13.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): my global_id is 94498 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e080035d0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271457318 0 0) 0x7f9e2408ebc0 con 0x7f9e24099c80 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): my global_id is 104292 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e240906f0 con 0x7f9e24099c80 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3613971133 0 0) 0x7f9e080035d0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e2408ebc0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e1c004460 con 0x7f9e2409b1b0 2024-09-16T06:40:13.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): dump: 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.770 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 911445741 0 0) 0x7f9e240906f0 con 0x7f9e24099c80 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e080035d0 con 0x7f9e24099c80 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e14001b80 con 0x7f9e24099c80 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient(hunting): dump: 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.771 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3439648720 0 0) 0x7f9e2408ebc0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 >> v1:172.21.15.76:6790/0 conn(0x7f9e2409ada0 legacy=0x7f9e24096500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 >> v1:172.21.15.76:6789/0 conn(0x7f9e24099c80 legacy=0x7f9e241c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 monclient: found mon.b 2024-09-16T06:40:13.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e24090f90 con 0x7f9e2409b1b0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: _finish_auth 0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 5 monclient: authenticate success, global_id 94498 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _renew_subs 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e240a2e30 con 0x7f9e2409b1b0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _renew_subs 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2408f790 con 0x7f9e2409b1b0 2024-09-16T06:40:13.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e1c0046d0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e1c004a60 con 0x7f9e2409b1b0 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e28b03700 4 set_mon_vals no callback set 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 10 monclient: dump: 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:13.774 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c005100 con 0x7f9e2409b1b0 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9e1c004cc0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(40..40 src has 1..40) v4 ==== 3940+0+0 (unknown 2247657360 0 0) 0x7f9e1c043520 con 0x7f9e2409b1b0 2024-09-16T06:40:13.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.754+0000 7f9e2a306700 10 monclient: get_auth_request con 0x7f9e080362c0 auth_method 0 2024-09-16T06:40:13.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.762+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9e1c014ab0 con 0x7f9e2409b1b0 2024-09-16T06:40:13.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.762+0000 7f9e1affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:13.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.762+0000 7f9e1affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:13.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.930+0000 7f9e2b308700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.930+0000 7f9e2b308700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:13.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.930+0000 7f9e2b308700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:13.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:13.930+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e0c005930 con 0x7f9e2409b1b0 2024-09-16T06:40:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.478+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v41) v1 ==== 149+0+0 (unknown 3998687224 0 0) 0x7f9e1c01d0a0 con 0x7f9e2409b1b0 2024-09-16T06:40:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.478+0000 7f9e1affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.478+0000 7f9e1affd700 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.598+0000 7f9e2b308700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.598+0000 7f9e2b308700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.598+0000 7f9e2b308700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.598+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e0c005420 con 0x7f9e2409b1b0 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e1affd700 1 -- 172.21.15.189:0/2809492389 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v41) v1 ==== 156+0+0 (unknown 1496118214 0 0) 0x7f9e1c012f20 con 0x7f9e2409b1b0 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e1affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e1affd700 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9e080362c0 legacy=0x7f9e08038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 10 monclient: shutdown 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 >> v1:172.21.15.189:6789/0 conn(0x7f9e2409b1b0 legacy=0x7f9e241cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 shutdown_connections 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 >> 172.21.15.189:0/2809492389 conn(0x7f9e2409fe70 msgr2=0x7f9e2409a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 shutdown_connections 2024-09-16T06:40:14.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.602+0000 7f9e2b308700 1 -- 172.21.15.189:0/2809492389 wait complete. 2024-09-16T06:40:14.607 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' does not exist 2024-09-16T06:40:14.615 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T06:40:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 10 monclient: get_monmap_and_config 2024-09-16T06:40:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 10 monclient: build_initial_monmap 2024-09-16T06:40:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 10 monclient: monmap: 2024-09-16T06:40:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:14.715456+0000 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:14.715456+0000 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:14.716 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 1 Processor -- start 2024-09-16T06:40:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 1 -- start start 2024-09-16T06:40:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.710+0000 7ff4dabbd700 10 monclient: init 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-a con 0x7ff4d409eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-c con 0x7ff4d409beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d4099ed0 con 0x7ff4d409eba0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d409d2e0 con 0x7ff4d409beb0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d409ef80 con 0x7ff4d409c2e0 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient(hunting): authenticate will time out at 967.228296s 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d93ba700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff4d409beb0 0x7ff4d4137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44362/0 (socket says 172.21.15.189:44362) 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d93ba700 1 -- 172.21.15.189:0/3998072034 learned_addr learned my addr 172.21.15.189:0/3998072034 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:14.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4da3bc700 1 -- 172.21.15.189:0/3998072034 >> v1:172.21.15.76:6790/0 conn(0x7ff4d409c2e0 legacy=0x7ff4d413ac60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031228110 0 0) 0x7ff4d409d2e0 con 0x7ff4d409beb0 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient(hunting): my global_id is 94501 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c00035d0 con 0x7ff4d409beb0 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1142967076 0 0) 0x7ff4c00035d0 con 0x7ff4d409beb0 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 >> v1:172.21.15.76:6790/0 conn(0x7ff4d409c2e0 legacy=0x7ff4d413ac60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 >> v1:172.21.15.76:6789/0 conn(0x7ff4d409eba0 legacy=0x7ff4d409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 monclient: found mon.noname-c 2024-09-16T06:40:14.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d409cc70 con 0x7ff4d409beb0 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: _finish_auth 0 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4d0001610 con 0x7ff4d409beb0 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 5 monclient: authenticate success, global_id 94501 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: dump: 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:14.727 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4d0001900 con 0x7ff4d409beb0 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 1 -- 172.21.15.189:0/3998072034 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4d0001c50 con 0x7ff4d409beb0 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:14.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 4 set_mon_vals no callback set 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4d8bb9700 10 monclient: dump: 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:14.729 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: shutdown 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- 172.21.15.189:0/3998072034 >> v1:172.21.15.189:6789/0 conn(0x7ff4d409beb0 legacy=0x7ff4d4137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- 172.21.15.189:0/3998072034 shutdown_connections 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- 172.21.15.189:0/3998072034 >> 172.21.15.189:0/3998072034 conn(0x7ff4d409fe70 msgr2=0x7ff4d40a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- 172.21.15.189:0/3998072034 shutdown_connections 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- 172.21.15.189:0/3998072034 wait complete. 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: build_initial_monmap 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: monmap: 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 Processor -- start 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 1 -- start start 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.714+0000 7ff4dabbd700 10 monclient: init 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-b con 0x7ff4d409beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-c con 0x7ff4d409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient(hunting): picked mon.noname-a con 0x7ff4d409eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d412c350 con 0x7ff4d409eba0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d409e830 con 0x7ff4d409beb0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d412e160 con 0x7ff4d409c2e0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient(hunting): authenticate will time out at 967.232317s 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4d93ba700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff4d409c2e0 legacy=0x7ff4d41ca990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4d9bbb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff4d409beb0 0x7ff4d41326c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44372/0 (socket says 172.21.15.189:44372) 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4d9bbb700 1 -- 172.21.15.189:0/2834327698 learned_addr learned my addr 172.21.15.189:0/2834327698 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096182804 0 0) 0x7ff4d409e830 con 0x7ff4d409beb0 2024-09-16T06:40:14.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): my global_id is 94504 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b80035d0 con 0x7ff4d409beb0 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040752321 0 0) 0x7ff4d412c350 con 0x7ff4d409eba0 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): my global_id is 104295 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d409e830 con 0x7ff4d409eba0 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176572762 0 0) 0x7ff4b80035d0 con 0x7ff4d409beb0 2024-09-16T06:40:14.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d412c350 con 0x7ff4d409beb0 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4c40018c0 con 0x7ff4d409beb0 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): dump: 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:14.734 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2132380387 0 0) 0x7ff4d409e830 con 0x7ff4d409eba0 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b80035d0 con 0x7ff4d409eba0 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4cc004460 con 0x7ff4d409eba0 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient(hunting): dump: 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.735 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3468843546 0 0) 0x7ff4d412c350 con 0x7ff4d409beb0 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 >> v1:172.21.15.76:6790/0 conn(0x7ff4d409c2e0 legacy=0x7ff4d41ca990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 >> v1:172.21.15.76:6789/0 conn(0x7ff4d409eba0 legacy=0x7ff4d41ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 monclient: found mon.b 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:14.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40a1270 con 0x7ff4d409beb0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 5 monclient: authenticate success, global_id 94504 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _renew_subs 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d409a380 con 0x7ff4d409beb0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _renew_subs 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d412be00 con 0x7ff4d409beb0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c4001c50 con 0x7ff4d409beb0 2024-09-16T06:40:14.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4c4003af0 con 0x7ff4d409beb0 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4cbfff700 4 set_mon_vals no callback set 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 10 monclient: dump: 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:14.738 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff4c4003c50 con 0x7ff4d409beb0 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc005100 con 0x7ff4d409beb0 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 3598+0+0 (unknown 3211008311 0 0) 0x7ff4c40431e0 con 0x7ff4d409beb0 2024-09-16T06:40:14.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.718+0000 7ff4d93ba700 10 monclient: get_auth_request con 0x7ff4b80362a0 auth_method 0 2024-09-16T06:40:14.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.726+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff4c4014780 con 0x7ff4d409beb0 2024-09-16T06:40:14.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.726+0000 7ff4ca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:14.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.726+0000 7ff4ca7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:14.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4dabbd700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4dabbd700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4dabbd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff4bc005930 con 0x7ff4d409beb0 2024-09-16T06:40:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v41) v1 ==== 156+0+0 (unknown 1496118214 0 0) 0x7ff4c4012bf0 con 0x7ff4d409beb0 2024-09-16T06:40:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4ca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:14.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:14.914+0000 7ff4ca7fc700 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-16T06:40:15.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff4bc005b20 con 0x7ff4d409beb0 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4ca7fc700 1 -- 172.21.15.189:0/2834327698 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v41) v1 ==== 156+0+0 (unknown 1496118214 0 0) 0x7ff4c4012bf0 con 0x7ff4d409beb0 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4ca7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4ca7fc700 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T06:40:15.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff4b80362a0 legacy=0x7ff4b8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 10 monclient: shutdown 2024-09-16T06:40:15.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.014+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 >> v1:172.21.15.189:6789/0 conn(0x7ff4d409beb0 legacy=0x7ff4d41326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.018+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 shutdown_connections 2024-09-16T06:40:15.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.018+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 >> 172.21.15.189:0/2834327698 conn(0x7ff4d409fe70 msgr2=0x7ff4d406d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.018+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 shutdown_connections 2024-09-16T06:40:15.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.018+0000 7ff4dabbd700 1 -- 172.21.15.189:0/2834327698 wait complete. 2024-09-16T06:40:15.020 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'foo' does not exist 2024-09-16T06:40:15.029 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T06:40:15.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 10 monclient: get_monmap_and_config 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 10 monclient: monmap: 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:15.142758+0000 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:15.142758+0000 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:15.142 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 1 Processor -- start 2024-09-16T06:40:15.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 1 -- start start 2024-09-16T06:40:15.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 10 monclient: init 2024-09-16T06:40:15.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.138+0000 7f053906d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient(hunting): picked mon.noname-b con 0x7f053409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient(hunting): picked mon.noname-a con 0x7f0534099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient(hunting): picked mon.noname-c con 0x7f05341396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053409f520 con 0x7f0534099700 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053409bc10 con 0x7f05341396a0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053409b810 con 0x7f053409f140 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient(hunting): authenticate will time out at 967.656164s 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0532d9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f053409f140 legacy=0x7f053409a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053359e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05341396a0 0x7f053413ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44374/0 (socket says 172.21.15.189:44374) 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053359e700 1 -- 172.21.15.189:0/1977420377 learned_addr learned my addr 172.21.15.189:0/1977420377 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021564519 0 0) 0x7f053409bc10 con 0x7f05341396a0 2024-09-16T06:40:15.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): my global_id is 94507 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05200035d0 con 0x7f05341396a0 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408077766 0 0) 0x7f053409f520 con 0x7f0534099700 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient(hunting): my global_id is 104298 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053409bc10 con 0x7f0534099700 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819146006 0 0) 0x7f05200035d0 con 0x7f05341396a0 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 >> v1:172.21.15.76:6790/0 conn(0x7f053409f140 legacy=0x7f053409a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:15.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 >> v1:172.21.15.76:6789/0 conn(0x7f0534099700 legacy=0x7f053409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 monclient: found mon.noname-c 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053409ceb0 con 0x7f05341396a0 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:15.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f052c001990 con 0x7f05341396a0 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 5 monclient: authenticate success, global_id 94507 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: dump: 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.150 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f052c001c70 con 0x7f05341396a0 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 1 -- 172.21.15.189:0/1977420377 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f052c002000 con 0x7f05341396a0 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 4 set_mon_vals no callback set 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f0531d9b700 10 monclient: dump: 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.151 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: shutdown 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- 172.21.15.189:0/1977420377 >> v1:172.21.15.189:6789/0 conn(0x7f05341396a0 legacy=0x7f053413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- 172.21.15.189:0/1977420377 shutdown_connections 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- 172.21.15.189:0/1977420377 >> 172.21.15.189:0/1977420377 conn(0x7f05340a03e0 msgr2=0x7f05340a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- 172.21.15.189:0/1977420377 shutdown_connections 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- 172.21.15.189:0/1977420377 wait complete. 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: monmap: 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:15.153 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 Processor -- start 2024-09-16T06:40:15.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 1 -- start start 2024-09-16T06:40:15.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.142+0000 7f053906d700 10 monclient: init 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient(hunting): picked mon.noname-a con 0x7f0534099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient(hunting): picked mon.noname-c con 0x7f053409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient(hunting): picked mon.noname-b con 0x7f05341396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053409a6e0 con 0x7f0534099700 2024-09-16T06:40:15.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05341394d0 con 0x7f05341396a0 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340926a0 con 0x7f053409f140 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient(hunting): authenticate will time out at 967.660726s 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053259c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f053409f140 legacy=0x7f05341cc850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053359e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05341396a0 0x7f05341cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44382/0 (socket says 172.21.15.189:44382) 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053359e700 1 -- 172.21.15.189:0/3793423628 learned_addr learned my addr 172.21.15.189:0/3793423628 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360462391 0 0) 0x7f05341394d0 con 0x7f05341396a0 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): my global_id is 94510 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05140035d0 con 0x7f05341396a0 2024-09-16T06:40:15.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709612381 0 0) 0x7f053409a6e0 con 0x7f0534099700 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): my global_id is 104301 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05341394d0 con 0x7f0534099700 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3152919815 0 0) 0x7f05140035d0 con 0x7f05341396a0 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f053409a6e0 con 0x7f05341396a0 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f052c001950 con 0x7f05341396a0 2024-09-16T06:40:15.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.162 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796334088 0 0) 0x7f05341394d0 con 0x7f0534099700 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05140035d0 con 0x7f0534099700 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f05240041a0 con 0x7f0534099700 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.163 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601385818 0 0) 0x7f053409a6e0 con 0x7f05341396a0 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 >> v1:172.21.15.76:6790/0 conn(0x7f053409f140 legacy=0x7f05341cc850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 >> v1:172.21.15.76:6789/0 conn(0x7f0534099700 legacy=0x7f05340963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 monclient: found mon.b 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534071470 con 0x7f05341396a0 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 5 monclient: authenticate success, global_id 94510 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _renew_subs 2024-09-16T06:40:15.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05340a2e40 con 0x7f05341396a0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _renew_subs 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0534092980 con 0x7f05341396a0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f052c005020 con 0x7f05341396a0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f052c0053b0 con 0x7f05341396a0 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053159a700 4 set_mon_vals no callback set 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f052b7fe700 10 monclient: dump: 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.166 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.146+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518005100 con 0x7f05341396a0 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.150+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f052c005510 con 0x7f05341396a0 2024-09-16T06:40:15.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.150+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 3598+0+0 (unknown 3211008311 0 0) 0x7f052c0435b0 con 0x7f05341396a0 2024-09-16T06:40:15.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.150+0000 7f0532d9d700 10 monclient: get_auth_request con 0x7f05140362a0 auth_method 0 2024-09-16T06:40:15.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.154+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f052c014cc0 con 0x7f05341396a0 2024-09-16T06:40:15.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.154+0000 7f052b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.154+0000 7f052b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:15.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f053906d700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f053906d700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f053906d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05180053b0 con 0x7f05341396a0 2024-09-16T06:40:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) v1 ==== 159+0+0 (unknown 3902454184 0 0) 0x7f052c0148b0 con 0x7f05341396a0 2024-09-16T06:40:15.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f052b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.330+0000 7f052b7fe700 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T06:40:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0518005e00 con 0x7f05341396a0 2024-09-16T06:40:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f052b7fe700 1 -- 172.21.15.189:0/3793423628 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) v1 ==== 159+0+0 (unknown 3902454184 0 0) 0x7f052c014a90 con 0x7f05341396a0 2024-09-16T06:40:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f052b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f052b7fe700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 >> v1:172.21.15.189:6812/3098810674 conn(0x7f05140362a0 legacy=0x7f0514038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 10 monclient: shutdown 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 >> v1:172.21.15.189:6789/0 conn(0x7f05341396a0 legacy=0x7f05341cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 shutdown_connections 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 >> 172.21.15.189:0/3793423628 conn(0x7f05340a03e0 msgr2=0x7f05340a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 shutdown_connections 2024-09-16T06:40:15.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.430+0000 7f053906d700 1 -- 172.21.15.189:0/3793423628 wait complete. 2024-09-16T06:40:15.436 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T06:40:15.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: get_monmap_and_config 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: monmap: 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:15.554317+0000 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:15.554317+0000 2024-09-16T06:40:15.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:15.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:15.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 1 Processor -- start 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 1 -- start start 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: init 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient(hunting): picked mon.noname-c con 0x7ff4141352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient(hunting): picked mon.noname-a con 0x7ff414135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient(hunting): picked mon.noname-b con 0x7ff4140a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41409b050 con 0x7ff414135db0 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41406ab20 con 0x7ff4141352c0 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41406a840 con 0x7ff4140a2030 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff419531700 10 monclient(hunting): authenticate will time out at 968.067284s 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff413fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff4141352c0 0x7ff4141356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44396/0 (socket says 172.21.15.189:44396) 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff413fff700 1 -- 172.21.15.189:0/2635673368 learned_addr learned my addr 172.21.15.189:0/2635673368 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff418d30700 1 -- 172.21.15.189:0/2635673368 >> v1:172.21.15.76:6790/0 conn(0x7ff4140a2030 legacy=0x7ff41413df90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014371362 0 0) 0x7ff41406ab20 con 0x7ff4141352c0 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 10 monclient(hunting): my global_id is 94513 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.550+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc0035d0 con 0x7ff4141352c0 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269192058 0 0) 0x7ff3fc0035d0 con 0x7ff4141352c0 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 >> v1:172.21.15.76:6790/0 conn(0x7ff4140a2030 legacy=0x7ff41413df90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 >> v1:172.21.15.76:6789/0 conn(0x7ff414135db0 legacy=0x7ff41413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 monclient: found mon.noname-c 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41406c1b0 con 0x7ff4141352c0 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 5 monclient: authenticate success, global_id 94513 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff408001610 con 0x7ff4141352c0 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: dump: 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff408001930 con 0x7ff4141352c0 2024-09-16T06:40:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 4 set_mon_vals no callback set 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 1 -- 172.21.15.189:0/2635673368 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff408001cc0 con 0x7ff4141352c0 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff412ffd700 10 monclient: dump: 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: shutdown 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- 172.21.15.189:0/2635673368 >> v1:172.21.15.189:6789/0 conn(0x7ff4141352c0 legacy=0x7ff4141356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- 172.21.15.189:0/2635673368 shutdown_connections 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- 172.21.15.189:0/2635673368 >> 172.21.15.189:0/2635673368 conn(0x7ff4140a4550 msgr2=0x7ff4140a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- 172.21.15.189:0/2635673368 shutdown_connections 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- 172.21.15.189:0/2635673368 wait complete. 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: monmap: 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 Processor -- start 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- start start 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: init 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient(hunting): picked mon.noname-a con 0x7ff4140a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient(hunting): picked mon.noname-c con 0x7ff4141352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient(hunting): picked mon.noname-b con 0x7ff414135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4141394d0 con 0x7ff4140a2030 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41413ae30 con 0x7ff414135db0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4140a6510 con 0x7ff4141352c0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff419531700 10 monclient(hunting): authenticate will time out at 968.071666s 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff4137fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff4141352c0 legacy=0x7ff41413e360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff418d30700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff414135db0 0x7ff4141d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44412/0 (socket says 172.21.15.189:44412) 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.554+0000 7ff418d30700 1 -- 172.21.15.189:0/2144647034 learned_addr learned my addr 172.21.15.189:0/2144647034 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568574 0 0) 0x7ff41413ae30 con 0x7ff414135db0 2024-09-16T06:40:15.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): my global_id is 94516 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4000035d0 con 0x7ff414135db0 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619836915 0 0) 0x7ff4141394d0 con 0x7ff4140a2030 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): my global_id is 104307 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41413ae30 con 0x7ff4140a2030 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1631768413 0 0) 0x7ff4000035d0 con 0x7ff414135db0 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4141394d0 con 0x7ff414135db0 2024-09-16T06:40:15.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c004460 con 0x7ff414135db0 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621974546 0 0) 0x7ff41413ae30 con 0x7ff4140a2030 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4000035d0 con 0x7ff4140a2030 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff408001430 con 0x7ff4140a2030 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.576 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3387290009 0 0) 0x7ff4141394d0 con 0x7ff414135db0 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 >> v1:172.21.15.76:6790/0 conn(0x7ff4141352c0 legacy=0x7ff41413e360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 >> v1:172.21.15.76:6789/0 conn(0x7ff4140a2030 legacy=0x7ff4141c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 monclient: found mon.b 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41409e820 con 0x7ff414135db0 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:15.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 5 monclient: authenticate success, global_id 94516 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _renew_subs 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4140a60e0 con 0x7ff414135db0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _renew_subs 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff414138ad0 con 0x7ff414135db0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff40c004740 con 0x7ff414135db0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff4127fc700 4 set_mon_vals no callback set 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c004ad0 con 0x7ff414135db0 2024-09-16T06:40:15.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 10 monclient: dump: 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff40c004c30 con 0x7ff414135db0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f4005100 con 0x7ff414135db0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 3598+0+0 (unknown 3211008311 0 0) 0x7ff40c043360 con 0x7ff414135db0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.558+0000 7ff413fff700 10 monclient: get_auth_request con 0x7ff400036280 auth_method 0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.566+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff40c005480 con 0x7ff414135db0 2024-09-16T06:40:15.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.566+0000 7ff410ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.566+0000 7ff410ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:15.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff419531700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff419531700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff419531700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3f4005930 con 0x7ff414135db0 2024-09-16T06:40:15.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) v1 ==== 159+0+0 (unknown 3902454184 0 0) 0x7ff40c01d040 con 0x7ff414135db0 2024-09-16T06:40:15.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff410ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.742+0000 7ff410ff9700 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T06:40:15.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff419531700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff419531700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff419531700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3f4005b20 con 0x7ff414135db0 2024-09-16T06:40:15.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff410ff9700 1 -- 172.21.15.189:0/2144647034 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) v1 ==== 159+0+0 (unknown 3902454184 0 0) 0x7ff40c012ec0 con 0x7ff414135db0 2024-09-16T06:40:15.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff410ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:15.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.838+0000 7ff410ff9700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff400036280 legacy=0x7ff400038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 10 monclient: shutdown 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 >> v1:172.21.15.189:6789/0 conn(0x7ff414135db0 legacy=0x7ff4141d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 shutdown_connections 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 >> 172.21.15.189:0/2144647034 conn(0x7ff4140a4550 msgr2=0x7ff4140a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 shutdown_connections 2024-09-16T06:40:15.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.842+0000 7ff419531700 1 -- 172.21.15.189:0/2144647034 wait complete. 2024-09-16T06:40:15.846 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooo' does not exist 2024-09-16T06:40:15.862 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-16T06:40:15.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: get_monmap_and_config 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: monmap: 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:15.976789+0000 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:15.976789+0000 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.976 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:15.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 Processor -- start 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- start start 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: init 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-c con 0x7fbc440a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-a con 0x7fbc4409aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-b con 0x7fbc440a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4409a040 con 0x7fbc4409aa20 2024-09-16T06:40:15.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440a33d0 con 0x7fbc440a2350 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc44099d60 con 0x7fbc440a3ca0 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient(hunting): authenticate will time out at 968.489834s 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc49652700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc440a2350 0x7fbc440a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44414/0 (socket says 172.21.15.189:44414) 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc49652700 1 -- 172.21.15.189:0/3752949648 learned_addr learned my addr 172.21.15.189:0/3752949648 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc49e53700 1 -- 172.21.15.189:0/3752949648 >> v1:172.21.15.76:6790/0 conn(0x7fbc440a3ca0 legacy=0x7fbc4413dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187051407 0 0) 0x7fbc440a33d0 con 0x7fbc440a2350 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient(hunting): my global_id is 94519 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc300035d0 con 0x7fbc440a2350 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177810564 0 0) 0x7fbc300035d0 con 0x7fbc440a2350 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 >> v1:172.21.15.76:6790/0 conn(0x7fbc440a3ca0 legacy=0x7fbc4413dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 >> v1:172.21.15.76:6789/0 conn(0x7fbc4409aa20 legacy=0x7fbc4413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 monclient: found mon.noname-c 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:15.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440a3920 con 0x7fbc440a2350 2024-09-16T06:40:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: _finish_auth 0 2024-09-16T06:40:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:15.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc40001610 con 0x7fbc440a2350 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 5 monclient: authenticate success, global_id 94519 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: dump: 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:15.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc40001930 con 0x7fbc440a2350 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 1 -- 172.21.15.189:0/3752949648 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc40001cc0 con 0x7fbc440a2350 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 4 set_mon_vals no callback set 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc3bfff700 10 monclient: dump: 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 10 monclient: shutdown 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/3752949648 >> v1:172.21.15.189:6789/0 conn(0x7fbc440a2350 legacy=0x7fbc440a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/3752949648 shutdown_connections 2024-09-16T06:40:15.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/3752949648 >> 172.21.15.189:0/3752949648 conn(0x7fbc440a7d70 msgr2=0x7fbc440a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/3752949648 shutdown_connections 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.974+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/3752949648 wait complete. 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: build_initial_monmap 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: monmap: 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:15.993 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 Processor -- start 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- start start 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: init 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:15.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-c con 0x7fbc4409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-a con 0x7fbc440a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient(hunting): picked mon.noname-b con 0x7fbc440a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc44121470 con 0x7fbc440a2350 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4409b640 con 0x7fbc440a3ca0 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4406bed0 con 0x7fbc4409aa20 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient(hunting): authenticate will time out at 968.494214s 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc49652700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fbc4409aa20 legacy=0x7fbc44135480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc49e53700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc440a3ca0 0x7fbc441d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44420/0 (socket says 172.21.15.189:44420) 2024-09-16T06:40:15.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc49e53700 1 -- 172.21.15.189:0/1866359534 learned_addr learned my addr 172.21.15.189:0/1866359534 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211625254 0 0) 0x7fbc4409b640 con 0x7fbc440a3ca0 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): my global_id is 94522 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc280035d0 con 0x7fbc440a3ca0 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486608275 0 0) 0x7fbc44121470 con 0x7fbc440a2350 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:15.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): my global_id is 104313 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4409b640 con 0x7fbc440a2350 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676383518 0 0) 0x7fbc280035d0 con 0x7fbc440a3ca0 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc44121470 con 0x7fbc440a3ca0 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc3c004460 con 0x7fbc440a3ca0 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.997 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1323498923 0 0) 0x7fbc4409b640 con 0x7fbc440a2350 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc280035d0 con 0x7fbc440a2350 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc34001830 con 0x7fbc440a2350 2024-09-16T06:40:15.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:15.999 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 607914794 0 0) 0x7fbc44121470 con 0x7fbc440a3ca0 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 >> v1:172.21.15.76:6790/0 conn(0x7fbc4409aa20 legacy=0x7fbc44135480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 >> v1:172.21.15.76:6789/0 conn(0x7fbc440a2350 legacy=0x7fbc441ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 monclient: found mon.b 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44122e40 con 0x7fbc440a3ca0 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:16.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 5 monclient: authenticate success, global_id 94522 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _renew_subs 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc441225a0 con 0x7fbc440a3ca0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _renew_subs 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.978+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc441360f0 con 0x7fbc440a3ca0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc3c004740 con 0x7fbc440a3ca0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc3c004a50 con 0x7fbc440a3ca0 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc3affd700 4 set_mon_vals no callback set 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 10 monclient: dump: 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.002 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbc3c004c30 con 0x7fbc440a3ca0 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7fbc3c043500 con 0x7fbc440a3ca0 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc4b8b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc4b8b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc4b8b6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc24005100 con 0x7fbc440a3ca0 2024-09-16T06:40:16.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.982+0000 7fbc49652700 10 monclient: get_auth_request con 0x7fbc280362c0 auth_method 0 2024-09-16T06:40:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.990+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbc3c014a90 con 0x7fbc440a3ca0 2024-09-16T06:40:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.990+0000 7fbc39ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:15.990+0000 7fbc39ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:16.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc4b8b6700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc4b8b6700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc4b8b6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbc24005930 con 0x7fbc440a3ca0 2024-09-16T06:40:16.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v42) v1 ==== 162+0+0 (unknown 1346102702 0 0) 0x7fbc3c01d080 con 0x7fbc440a3ca0 2024-09-16T06:40:16.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc39ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.158+0000 7fbc39ffb700 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc4b8b6700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc4b8b6700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc4b8b6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbc24005420 con 0x7fbc440a3ca0 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc39ffb700 1 -- 172.21.15.189:0/1866359534 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v42) v1 ==== 162+0+0 (unknown 1346102702 0 0) 0x7fbc3c012f00 con 0x7fbc440a3ca0 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc39ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:16.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.250+0000 7fbc39ffb700 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-16T06:40:16.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbc280362c0 legacy=0x7fbc28038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 10 monclient: shutdown 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 >> v1:172.21.15.189:6789/0 conn(0x7fbc440a3ca0 legacy=0x7fbc441d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 shutdown_connections 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 >> 172.21.15.189:0/1866359534 conn(0x7fbc440a7d70 msgr2=0x7fbc440a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 shutdown_connections 2024-09-16T06:40:16.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.254+0000 7fbc4b8b6700 1 -- 172.21.15.189:0/1866359534 wait complete. 2024-09-16T06:40:16.257 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fuggg' does not exist 2024-09-16T06:40:16.265 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T06:40:16.266 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T06:40:16.266 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:40:16.266 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:40:16.281 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-16T06:40:16.283 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-16T06:40:16.283 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-16T06:40:16.293 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush dump 2024-09-16T06:40:16.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 10 monclient: get_monmap_and_config 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 10 monclient: build_initial_monmap 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 10 monclient: monmap: 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:16.395602+0000 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:16.395602+0000 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:16.395 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:16.396 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:16.396 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 1 Processor -- start 2024-09-16T06:40:16.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 1 -- start start 2024-09-16T06:40:16.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.390+0000 7fac290d7700 10 monclient: init 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-a con 0x7fac2409eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-b con 0x7fac2409beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-c con 0x7fac2409c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac24099ed0 con 0x7fac2409eba0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2409d2e0 con 0x7fac2409c2e0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2409ef80 con 0x7fac2409beb0 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:16.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient(hunting): authenticate will time out at 968.908182s 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac1bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fac2409beb0 legacy=0x7fac24137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac288d6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac2409c2e0 0x7fac2413ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44436/0 (socket says 172.21.15.189:44436) 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac288d6700 1 -- 172.21.15.189:0/3819270948 learned_addr learned my addr 172.21.15.189:0/3819270948 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284515114 0 0) 0x7fac2409d2e0 con 0x7fac2409c2e0 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): my global_id is 94525 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac100035d0 con 0x7fac2409c2e0 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685945014 0 0) 0x7fac24099ed0 con 0x7fac2409eba0 2024-09-16T06:40:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient(hunting): my global_id is 104316 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2409d2e0 con 0x7fac2409eba0 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519868461 0 0) 0x7fac100035d0 con 0x7fac2409c2e0 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 >> v1:172.21.15.76:6790/0 conn(0x7fac2409beb0 legacy=0x7fac24137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 >> v1:172.21.15.76:6789/0 conn(0x7fac2409eba0 legacy=0x7fac2409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2409cc70 con 0x7fac2409c2e0 2024-09-16T06:40:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 5 monclient: authenticate success, global_id 94525 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fac1c001990 con 0x7fac2409c2e0 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: dump: 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac1c001c00 con 0x7fac2409c2e0 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 4 set_mon_vals no callback set 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 1 -- 172.21.15.189:0/3819270948 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fac1c001f90 con 0x7fac2409c2e0 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac237fe700 10 monclient: dump: 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 10 monclient: shutdown 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- 172.21.15.189:0/3819270948 >> v1:172.21.15.189:6789/0 conn(0x7fac2409c2e0 legacy=0x7fac2413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- 172.21.15.189:0/3819270948 shutdown_connections 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- 172.21.15.189:0/3819270948 >> 172.21.15.189:0/3819270948 conn(0x7fac2409fe70 msgr2=0x7fac240a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- 172.21.15.189:0/3819270948 shutdown_connections 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.394+0000 7fac290d7700 1 -- 172.21.15.189:0/3819270948 wait complete. 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: build_initial_monmap 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: monmap: 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:16.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:16.407 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 Processor -- start 2024-09-16T06:40:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- start start 2024-09-16T06:40:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: init 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-a con 0x7fac2409beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-c con 0x7fac2409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient(hunting): picked mon.noname-b con 0x7fac2409eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2409b190 con 0x7fac2409beb0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2412bb10 con 0x7fac2409eba0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac240a27f0 con 0x7fac2409c2e0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient(hunting): authenticate will time out at 968.913810s 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac1bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fac2409c2e0 legacy=0x7fac2412ace0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:16.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac288d6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fac2409eba0 0x7fac241d2390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44452/0 (socket says 172.21.15.189:44452) 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac288d6700 1 -- 172.21.15.189:0/2502131875 learned_addr learned my addr 172.21.15.189:0/2502131875 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237873563 0 0) 0x7fac2412bb10 con 0x7fac2409eba0 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): my global_id is 94528 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac040035d0 con 0x7fac2409eba0 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849571855 0 0) 0x7fac040035d0 con 0x7fac2409eba0 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac2412bb10 con 0x7fac2409eba0 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fac1c001f10 con 0x7fac2409eba0 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): dump: 2024-09-16T06:40:16.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259185226 0 0) 0x7fac2409b190 con 0x7fac2409beb0 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient(hunting): my global_id is 104319 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac040035d0 con 0x7fac2409beb0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 654408178 0 0) 0x7fac2412bb10 con 0x7fac2409eba0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 >> v1:172.21.15.76:6790/0 conn(0x7fac2409c2e0 legacy=0x7fac2412ace0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 >> v1:172.21.15.76:6789/0 conn(0x7fac2409beb0 legacy=0x7fac2412a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 monclient: found mon.b 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac240a22a0 con 0x7fac2409eba0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:16.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 5 monclient: authenticate success, global_id 94528 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _renew_subs 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac241315d0 con 0x7fac2409eba0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _renew_subs 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2412b3f0 con 0x7fac2409eba0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac1c004f40 con 0x7fac2409eba0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fac1c001590 con 0x7fac2409eba0 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac22ffd700 4 set_mon_vals no callback set 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 10 monclient: dump: 2024-09-16T06:40:16.414 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.398+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fac1c0052d0 con 0x7fac2409eba0 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac290d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac290d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac290d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08005100 con 0x7fac2409eba0 2024-09-16T06:40:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7fac1c0437d0 con 0x7fac2409eba0 2024-09-16T06:40:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.402+0000 7fac23fff700 10 monclient: get_auth_request con 0x7fac04036180 auth_method 0 2024-09-16T06:40:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.406+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fac1c014dc0 con 0x7fac2409eba0 2024-09-16T06:40:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.406+0000 7fac217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.406+0000 7fac217fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.574+0000 7fac290d7700 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.574+0000 7fac290d7700 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.574+0000 7fac290d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.574+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7fac08005930 con 0x7fac2409eba0 2024-09-16T06:40:16.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.578+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v42) v1 ==== 62+0+7238 (unknown 3275263029 0 1903872936) 0x7fac1c01c050 con 0x7fac2409eba0 2024-09-16T06:40:16.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.578+0000 7fac217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.578+0000 7fac217fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:16.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac290d7700 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac290d7700 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac290d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7fac08005650 con 0x7fac2409eba0 2024-09-16T06:40:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac217fa700 1 -- 172.21.15.189:0/2502131875 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v42) v1 ==== 62+0+7238 (unknown 3275263029 0 1903872936) 0x7fac1c01ccb0 con 0x7fac2409eba0 2024-09-16T06:40:16.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac217fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.678+0000 7fac217fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: "devices": [ 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.0", 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.682 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.1", 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.2", 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.3", 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.683 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.4", 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd.5", 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "class": "ssd" 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "types": [ 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 0, 2024-09-16T06:40:16.684 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "osd" 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "host" 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 2, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "chassis" 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 3, 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "rack" 2024-09-16T06:40:16.685 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 4, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "row" 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 5, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pdu" 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 6, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "pod" 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.686 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 7, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "room" 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 8, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "datacenter" 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 9, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "zone" 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 10, 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "region" 2024-09-16T06:40:16.687 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "root" 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "buckets": [ 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -1, 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default", 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 34560, 2024-09-16T06:40:16.688 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -5, 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.689 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -2, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "default~ssd", 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 11, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "root", 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 34560, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.690 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -6, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -3, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189", 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.691 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.692 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -4, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi189~ssd", 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.693 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 3, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 5, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.694 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 4, 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -5, 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi076", 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T06:40:16.695 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.696 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "id": -6, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "name": "smithi076~ssd", 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "type_id": 1, 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "type_name": "host", 2024-09-16T06:40:16.697 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 17280, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "alg": "straw2", 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "hash": "rjenkins1", 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "items": [ 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 1, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 0 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 2, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 1 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.698 INFO:tasks.workunit.client.0.smithi189.stdout: "id": 0, 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "weight": 5760, 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "pos": 2 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "rules": [ 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.699 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.700 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "erasure-code", 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2024-09-16T06:40:16.701 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T06:40:16.702 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "tunables": { 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_tries": 0, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_local_fallback_tries": 0, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_total_tries": 50, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_descend_once": 1, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_vary_r": 1, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "chooseleaf_stable": 1, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "straw_calc_version": 1, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "allowed_bucket_algs": 54, 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "profile": "jewel", 2024-09-16T06:40:16.703 INFO:tasks.workunit.client.0.smithi189.stdout: "optimal_tunables": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "legacy_tunables": 0, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "minimum_required_version": "jewel", 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables2": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v2_rules": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables3": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v3_rules": 0, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v4_buckets": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "require_feature_tunables5": 1, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "has_v5_rules": 0 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout: "choose_args": {} 2024-09-16T06:40:16.704 INFO:tasks.workunit.client.0.smithi189.stdout:} 2024-09-16T06:40:16.705 INFO:tasks.workunit.client.0.smithi189.stdout: 2024-09-16T06:40:16.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 >> v1:172.21.15.189:6812/3098810674 conn(0x7fac04036180 legacy=0x7fac04038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 10 monclient: shutdown 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 >> v1:172.21.15.189:6789/0 conn(0x7fac2409eba0 legacy=0x7fac241d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 shutdown_connections 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 >> 172.21.15.189:0/2502131875 conn(0x7fac2409fe70 msgr2=0x7fac240a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 shutdown_connections 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.682+0000 7fac290d7700 1 -- 172.21.15.189:0/2502131875 wait complete. 2024-09-16T06:40:16.706 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule dump 2024-09-16T06:40:16.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: get_monmap_and_config 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: build_initial_monmap 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: monmap: 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:16.800750+0000 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:16.800750+0000 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:16.800 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:16.801 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:16.801 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 Processor -- start 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 -- start start 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: init 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-c con 0x7fe64009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-b con 0x7fe64009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:16.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-a con 0x7fe640138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64009cca0 con 0x7fe640138da0 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64009f750 con 0x7fe64009bb00 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640097620 con 0x7fe64009c5f0 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient(hunting): authenticate will time out at 969.313671s 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe645599700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe64009bb00 0x7fe64009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44456/0 (socket says 172.21.15.189:44456) 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe644d98700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe64009c5f0 legacy=0x7fe640137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe645599700 1 -- 172.21.15.189:0/2008419581 learned_addr learned my addr 172.21.15.189:0/2008419581 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399911749 0 0) 0x7fe64009f750 con 0x7fe64009bb00 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): my global_id is 94531 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe62c0057e0 con 0x7fe64009bb00 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377229309 0 0) 0x7fe64009cca0 con 0x7fe640138da0 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient(hunting): my global_id is 104322 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64009f750 con 0x7fe640138da0 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679546489 0 0) 0x7fe62c0057e0 con 0x7fe64009bb00 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 >> v1:172.21.15.76:6790/0 conn(0x7fe64009c5f0 legacy=0x7fe640137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:16.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 >> v1:172.21.15.76:6789/0 conn(0x7fe640138da0 legacy=0x7fe64013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 monclient: found mon.noname-c 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640097900 con 0x7fe64009bb00 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 5 monclient: authenticate success, global_id 94531 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe634001610 con 0x7fe64009bb00 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:16.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: dump: 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.809 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6340018f0 con 0x7fe64009bb00 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 1 -- 172.21.15.189:0/2008419581 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe634001c80 con 0x7fe64009bb00 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 4 set_mon_vals no callback set 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe633fff700 10 monclient: dump: 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.810 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.811 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 10 monclient: shutdown 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.798+0000 7fe6477fd700 1 -- 172.21.15.189:0/2008419581 >> v1:172.21.15.189:6789/0 conn(0x7fe64009bb00 legacy=0x7fe64009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- 172.21.15.189:0/2008419581 shutdown_connections 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- 172.21.15.189:0/2008419581 >> 172.21.15.189:0/2008419581 conn(0x7fe6400a03e0 msgr2=0x7fe6400a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- 172.21.15.189:0/2008419581 shutdown_connections 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- 172.21.15.189:0/2008419581 wait complete. 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient: build_initial_monmap 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient: monmap: 2024-09-16T06:40:16.812 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 Processor -- start 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- start start 2024-09-16T06:40:16.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient: init 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-b con 0x7fe64009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-c con 0x7fe64009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient(hunting): picked mon.noname-a con 0x7fe640138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64008f330 con 0x7fe640138da0 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6400a08a0 con 0x7fe64009bb00 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64008d950 con 0x7fe64009c5f0 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:16.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe6477fd700 10 monclient(hunting): authenticate will time out at 969.318871s 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe644d98700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe64009c5f0 legacy=0x7fe64008d240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe645599700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe64009bb00 0x7fe64008cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44468/0 (socket says 172.21.15.189:44468) 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe645599700 1 -- 172.21.15.189:0/2347431548 learned_addr learned my addr 172.21.15.189:0/2347431548 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717099875 0 0) 0x7fe6400a08a0 con 0x7fe64009bb00 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): my global_id is 94534 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6240035d0 con 0x7fe64009bb00 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926733565 0 0) 0x7fe64008f330 con 0x7fe640138da0 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:16.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 10 monclient(hunting): my global_id is 104325 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6400a08a0 con 0x7fe640138da0 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155480967 0 0) 0x7fe6240035d0 con 0x7fe64009bb00 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.802+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe64008f330 con 0x7fe64009bb00 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe6340015d0 con 0x7fe64009bb00 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.821 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369427237 0 0) 0x7fe6400a08a0 con 0x7fe640138da0 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6240035d0 con 0x7fe640138da0 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe638001c30 con 0x7fe640138da0 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.822 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1725044936 0 0) 0x7fe64008f330 con 0x7fe64009bb00 2024-09-16T06:40:16.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 >> v1:172.21.15.76:6790/0 conn(0x7fe64009c5f0 legacy=0x7fe64008d240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 >> v1:172.21.15.76:6789/0 conn(0x7fe640138da0 legacy=0x7fe6401cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 monclient: found mon.b 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6400a2cb0 con 0x7fe64009bb00 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 5 monclient: authenticate success, global_id 94534 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _renew_subs 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64008e6f0 con 0x7fe64009bb00 2024-09-16T06:40:16.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _renew_subs 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe640090770 con 0x7fe64009bb00 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64008f010 con 0x7fe64009bb00 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe634004ca0 con 0x7fe64009bb00 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe634004fb0 con 0x7fe64009bb00 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:16.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 10 monclient: dump: 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe6337fe700 4 set_mon_vals no callback set 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe63400fa20 con 0x7fe64009bb00 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7fe634043340 con 0x7fe64009bb00 2024-09-16T06:40:16.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.806+0000 7fe644d98700 10 monclient: get_auth_request con 0x7fe6240362a0 auth_method 0 2024-09-16T06:40:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.814+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe6340147c0 con 0x7fe64009bb00 2024-09-16T06:40:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.814+0000 7fe631ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:16.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.814+0000 7fe631ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:16.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe6477fd700 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:16.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe6477fd700 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:16.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe6477fd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:16.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7fe640066480 con 0x7fe64009bb00 2024-09-16T06:40:16.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v42) v1 ==== 67+0+1058 (unknown 2429497187 0 3068570633) 0x7fe6340011d0 con 0x7fe64009bb00 2024-09-16T06:40:16.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe631ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:16.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:16.974+0000 7fe631ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:17.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.066+0000 7fe6477fd700 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.066+0000 7fe6477fd700 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.066+0000 7fe6477fd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.066+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7fe640090960 con 0x7fe64009bb00 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe631ffb700 1 -- 172.21.15.189:0/2347431548 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v42) v1 ==== 67+0+1058 (unknown 2429497187 0 3068570633) 0x7fe634012e60 con 0x7fe64009bb00 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe631ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T06:40:17.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe631ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout:[ 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 0, 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "replicated_rule", 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 1, 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.072 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_firstn", 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_id": 1, 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "rule_name": "erasure-code", 2024-09-16T06:40:17.073 INFO:tasks.workunit.client.0.smithi189.stdout: "type": 3, 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "steps": [ 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_chooseleaf_tries", 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 5 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "set_choose_tries", 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 100 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "take", 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "item": -1, 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: "item_name": "default" 2024-09-16T06:40:17.074 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "choose_indep", 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: "num": 0, 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: "type": "osd" 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: "op": "emit" 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout:] 2024-09-16T06:40:17.075 INFO:tasks.workunit.client.0.smithi189.stdout: 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe6240362a0 legacy=0x7fe624038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe6477fd700 10 monclient: shutdown 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe6477fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.070+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 >> v1:172.21.15.189:6789/0 conn(0x7fe64009bb00 legacy=0x7fe64008cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.074+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 shutdown_connections 2024-09-16T06:40:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.074+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 >> 172.21.15.189:0/2347431548 conn(0x7fe6400a03e0 msgr2=0x7fe64009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:17.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.074+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 shutdown_connections 2024-09-16T06:40:17.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.074+0000 7fe6477fd700 1 -- 172.21.15.189:0/2347431548 wait complete. 2024-09-16T06:40:17.088 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.182+0000 7f7841730700 10 monclient: get_monmap_and_config 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.182+0000 7f7841730700 10 monclient: build_initial_monmap 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.182+0000 7f7841730700 10 monclient: monmap: 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:17.188463+0000 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:17.188463+0000 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 1 Processor -- start 2024-09-16T06:40:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 1 -- start start 2024-09-16T06:40:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient: init 2024-09-16T06:40:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient(hunting): picked mon.noname-b con 0x7f783c066520 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient(hunting): picked mon.noname-c con 0x7f783c09a290 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient(hunting): picked mon.noname-a con 0x7f783c133920 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c099f50 con 0x7f783c133920 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c066900 con 0x7f783c09a290 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c09d1a0 con 0x7f783c066520 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient(hunting): authenticate will time out at 969.702704s 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7833fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f783c09a290 0x7f783c138080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44484/0 (socket says 172.21.15.189:44484) 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7833fff700 1 -- 172.21.15.189:0/2091502247 learned_addr learned my addr 172.21.15.189:0/2091502247 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783bfff700 1 -- 172.21.15.189:0/2091502247 >> v1:172.21.15.76:6790/0 conn(0x7f783c066520 legacy=0x7f783c133210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907144824 0 0) 0x7f783c066900 con 0x7f783c09a290 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient(hunting): my global_id is 94537 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78280035d0 con 0x7f783c09a290 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931580692 0 0) 0x7f78280035d0 con 0x7f783c09a290 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 >> v1:172.21.15.76:6790/0 conn(0x7f783c066520 legacy=0x7f783c133210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 >> v1:172.21.15.76:6789/0 conn(0x7f783c133920 legacy=0x7f783c13b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c09d480 con 0x7f783c09a290 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:17.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 5 monclient: authenticate success, global_id 94537 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7824001610 con 0x7f783c09a290 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: dump: 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.195 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7824001930 con 0x7f783c09a290 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 1 -- 172.21.15.189:0/2091502247 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7824001cc0 con 0x7f783c09a290 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f7841730700 4 set_mon_vals no callback set 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.186+0000 7f783b7fe700 10 monclient: dump: 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: shutdown 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- 172.21.15.189:0/2091502247 >> v1:172.21.15.189:6789/0 conn(0x7f783c09a290 legacy=0x7f783c138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- 172.21.15.189:0/2091502247 shutdown_connections 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- 172.21.15.189:0/2091502247 >> 172.21.15.189:0/2091502247 conn(0x7f783c0a4320 msgr2=0x7f783c0a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- 172.21.15.189:0/2091502247 shutdown_connections 2024-09-16T06:40:17.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- 172.21.15.189:0/2091502247 wait complete. 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: build_initial_monmap 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: monmap: 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 Processor -- start 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- start start 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: init 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient(hunting): picked mon.noname-b con 0x7f783c066520 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient(hunting): picked mon.noname-a con 0x7f783c09a290 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient(hunting): picked mon.noname-c con 0x7f783c133920 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c129c30 con 0x7f783c09a290 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c129f10 con 0x7f783c066520 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c12a1f0 con 0x7f783c133920 2024-09-16T06:40:17.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7841730700 10 monclient(hunting): authenticate will time out at 969.707801s 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f7840f2f700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f783c133920 legacy=0x7f783c1cfdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f783bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f783c066520 0x7f783c1c4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44494/0 (socket says 172.21.15.189:44494) 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.190+0000 7f783bfff700 1 -- 172.21.15.189:0/1894061540 learned_addr learned my addr 172.21.15.189:0/1894061540 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982196655 0 0) 0x7f783c129f10 con 0x7f783c066520 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): my global_id is 94540 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c0035d0 con 0x7f783c066520 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57722204 0 0) 0x7f783c129c30 con 0x7f783c09a290 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): my global_id is 104331 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c129f10 con 0x7f783c09a290 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101895584 0 0) 0x7f781c0035d0 con 0x7f783c066520 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f783c129c30 con 0x7f783c066520 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f782c004280 con 0x7f783c066520 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): dump: 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.205 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2175216223 0 0) 0x7f783c129c30 con 0x7f783c066520 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 >> v1:172.21.15.76:6790/0 conn(0x7f783c133920 legacy=0x7f783c1cfdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 >> v1:172.21.15.76:6789/0 conn(0x7f783c09a290 legacy=0x7f783c1cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 monclient: found mon.b 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c12d0d0 con 0x7f783c066520 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 5 monclient: authenticate success, global_id 94540 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _renew_subs 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783c128d40 con 0x7f783c066520 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _renew_subs 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f783c1c4a50 con 0x7f783c066520 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f782c0044a0 con 0x7f783c066520 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f782c0047b0 con 0x7f783c066520 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f783affd700 4 set_mon_vals no callback set 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 10 monclient: dump: 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7820005100 con 0x7f783c066520 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f782c004990 con 0x7f783c066520 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7f782c043130 con 0x7f783c066520 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.194+0000 7f7833fff700 10 monclient: get_auth_request con 0x7f781c035fa0 auth_method 0 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.202+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f782c0051b0 con 0x7f783c066520 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.202+0000 7f78397fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.202+0000 7f78397fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f7841730700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f7841730700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f7841730700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f7820005930 con 0x7f783c066520 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v42) v1 ==== 65+0+29 (unknown 2266644849 0 852635902) 0x7f782c012c10 con 0x7f783c066520 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f78397fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.374+0000 7f78397fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f7820005420 con 0x7f783c066520 2024-09-16T06:40:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f78397fa700 1 -- 172.21.15.189:0/1894061540 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v42) v1 ==== 65+0+29 (unknown 2266644849 0 852635902) 0x7f782c014390 con 0x7f783c066520 2024-09-16T06:40:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f78397fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:17.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f78397fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:17.477 INFO:tasks.workunit.client.0.smithi189.stdout:replicated_rule 2024-09-16T06:40:17.477 INFO:tasks.workunit.client.0.smithi189.stdout:erasure-code 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 >> v1:172.21.15.189:6812/3098810674 conn(0x7f781c035fa0 legacy=0x7f781c038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 10 monclient: shutdown 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 >> v1:172.21.15.189:6789/0 conn(0x7f783c066520 legacy=0x7f783c1c4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 shutdown_connections 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.474+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 >> 172.21.15.189:0/1894061540 conn(0x7f783c0a4320 msgr2=0x7f783c130930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:17.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.478+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 shutdown_connections 2024-09-16T06:40:17.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.478+0000 7f7841730700 1 -- 172.21.15.189:0/1894061540 wait complete. 2024-09-16T06:40:17.489 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule list 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.586+0000 7f9216458700 10 monclient: get_monmap_and_config 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.586+0000 7f9216458700 10 monclient: build_initial_monmap 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.586+0000 7f9216458700 10 monclient: monmap: 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:17.591538+0000 2024-09-16T06:40:17.590 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:17.591538+0000 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 Processor -- start 2024-09-16T06:40:17.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- start start 2024-09-16T06:40:17.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient: init 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient(hunting): picked mon.noname-a con 0x7f921009e6d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient(hunting): picked mon.noname-c con 0x7f921009f9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient(hunting): picked mon.noname-b con 0x7f921009f1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921009ff50 con 0x7f921009e6d0 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921009b8f0 con 0x7f921009f9d0 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f921009f9d0 0x7f9210136aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44504/0 (socket says 172.21.15.189:44504) 2024-09-16T06:40:17.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920f7fe700 1 -- 172.21.15.189:0/3468630536 learned_addr learned my addr 172.21.15.189:0/3468630536 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921009e030 con 0x7f921009f1c0 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f92149f5700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f921009f1c0 legacy=0x7f921013a1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient(hunting): authenticate will time out at 970.106122s 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686297637 0 0) 0x7f921009b8f0 con 0x7f921009f9d0 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): my global_id is 94543 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f80035d0 con 0x7f921009f9d0 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936129735 0 0) 0x7f921009ff50 con 0x7f921009e6d0 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient(hunting): my global_id is 104334 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f921009b8f0 con 0x7f921009e6d0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343416020 0 0) 0x7f91f80035d0 con 0x7f921009f9d0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 >> v1:172.21.15.76:6790/0 conn(0x7f921009f1c0 legacy=0x7f921013a1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 >> v1:172.21.15.76:6789/0 conn(0x7f921009e6d0 legacy=0x7f921009eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 monclient: found mon.noname-c 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100a1150 con 0x7f921009f9d0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: _finish_auth 0 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:17.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 5 monclient: authenticate success, global_id 94543 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9204001610 con 0x7f921009f9d0 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: dump: 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92040018f0 con 0x7f921009f9d0 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 4 set_mon_vals no callback set 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 1 -- 172.21.15.189:0/3468630536 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9204001c80 con 0x7f921009f9d0 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f920effd700 10 monclient: dump: 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.599 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 10 monclient: shutdown 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 >> v1:172.21.15.189:6789/0 conn(0x7f921009f9d0 legacy=0x7f9210136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 shutdown_connections 2024-09-16T06:40:17.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 >> 172.21.15.189:0/3468630536 conn(0x7f92100a4410 msgr2=0x7f92100a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 shutdown_connections 2024-09-16T06:40:17.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.590+0000 7f9216458700 1 -- 172.21.15.189:0/3468630536 wait complete. 2024-09-16T06:40:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: build_initial_monmap 2024-09-16T06:40:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: monmap: 2024-09-16T06:40:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:17.604 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 Processor -- start 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- start start 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: init 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:17.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient(hunting): picked mon.noname-c con 0x7f921009e6d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient(hunting): picked mon.noname-a con 0x7f921009f1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient(hunting): picked mon.noname-b con 0x7f921009f9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92100a2360 con 0x7f921009f1c0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9210070f80 con 0x7f921009f9d0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921006b210 con 0x7f921009e6d0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient(hunting): authenticate will time out at 970.110088s 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920ffff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f921009e6d0 legacy=0x7f921013cc40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f92149f5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f921009f9d0 0x7f92101cf140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44510/0 (socket says 172.21.15.189:44510) 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f92149f5700 1 -- 172.21.15.189:0/3543988098 learned_addr learned my addr 172.21.15.189:0/3543988098 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405230079 0 0) 0x7f9210070f80 con 0x7f921009f9d0 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): my global_id is 94546 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc0035d0 con 0x7f921009f9d0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69593647 0 0) 0x7f92100a1150 con 0x7f921009f1c0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): my global_id is 104337 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9210070f80 con 0x7f921009f1c0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358887276 0 0) 0x7f91fc0035d0 con 0x7f921009f9d0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92100a1150 con 0x7f921009f9d0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9208004460 con 0x7f921009f9d0 2024-09-16T06:40:17.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): dump: 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:17.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068967471 0 0) 0x7f92100a1150 con 0x7f921009f9d0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 >> v1:172.21.15.76:6790/0 conn(0x7f921009e6d0 legacy=0x7f921013cc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 >> v1:172.21.15.76:6789/0 conn(0x7f921009f1c0 legacy=0x7f92101cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 monclient: found mon.b 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92101396f0 con 0x7f921009f9d0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 5 monclient: authenticate success, global_id 94546 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _renew_subs 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9210138e50 con 0x7f921009f9d0 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _renew_subs 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9210137390 con 0x7f921009f9d0 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9208004740 con 0x7f921009f9d0 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920e7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9208004ad0 con 0x7f921009f9d0 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.594+0000 7f920cff9700 10 monclient: dump: 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:17.610 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9208004cb0 con 0x7f921009f9d0 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f9216458700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f9216458700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f9216458700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f0005100 con 0x7f921009f9d0 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7f9208043580 con 0x7f921009f9d0 2024-09-16T06:40:17.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.598+0000 7f920ffff700 10 monclient: get_auth_request con 0x7f91fc0361b0 auth_method 0 2024-09-16T06:40:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.606+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9208014b10 con 0x7f921009f9d0 2024-09-16T06:40:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.606+0000 7f920cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:17.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.606+0000 7f920cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:17.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f9216458700 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f9216458700 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f9216458700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f91f0005930 con 0x7f921009f9d0 2024-09-16T06:40:17.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v42) v1 ==== 67+0+29 (unknown 527599421 0 852635902) 0x7f9208012f80 con 0x7f921009f9d0 2024-09-16T06:40:17.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f920cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.778+0000 7f920cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:17.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.874+0000 7f9216458700 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:17.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7f91f0005650 con 0x7f921009f9d0 2024-09-16T06:40:17.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f920cff9700 1 -- 172.21.15.189:0/3543988098 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v42) v1 ==== 67+0+29 (unknown 527599421 0 852635902) 0x7f9208014700 con 0x7f921009f9d0 2024-09-16T06:40:17.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f920cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-16T06:40:17.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f920cff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:17.880 INFO:tasks.workunit.client.0.smithi189.stdout:replicated_rule 2024-09-16T06:40:17.880 INFO:tasks.workunit.client.0.smithi189.stdout:erasure-code 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 >> v1:172.21.15.189:6812/3098810674 conn(0x7f91fc0361b0 legacy=0x7f91fc038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 10 monclient: shutdown 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 >> v1:172.21.15.189:6789/0 conn(0x7f921009f9d0 legacy=0x7f92101cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 shutdown_connections 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 >> 172.21.15.189:0/3543988098 conn(0x7f92100a4410 msgr2=0x7f9210099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:17.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 shutdown_connections 2024-09-16T06:40:17.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:17.878+0000 7f9216458700 1 -- 172.21.15.189:0/3543988098 wait complete. 2024-09-16T06:40:17.896 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: get_monmap_and_config 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: build_initial_monmap 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: monmap: 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:18.014369+0000 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:18.014369+0000 2024-09-16T06:40:18.013 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:18.014 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:18.014 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 1 Processor -- start 2024-09-16T06:40:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 1 -- start start 2024-09-16T06:40:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: init 2024-09-16T06:40:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:18.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-c con 0x7f72c009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-b con 0x7f72c009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-a con 0x7f72c0138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c009cca0 con 0x7f72c0138da0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c009f750 con 0x7f72c009bb00 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c0097620 con 0x7f72c009c5f0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72c6781700 10 monclient(hunting): authenticate will time out at 970.527410s 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72bffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72c009bb00 0x7f72c009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44512/0 (socket says 172.21.15.189:44512) 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72bffff700 1 -- 172.21.15.189:0/2345226671 learned_addr learned my addr 172.21.15.189:0/2345226671 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.010+0000 7f72bf7fe700 1 -- 172.21.15.189:0/2345226671 >> v1:172.21.15.76:6790/0 conn(0x7f72c009c5f0 legacy=0x7f72c0137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010328324 0 0) 0x7f72c009f750 con 0x7f72c009bb00 2024-09-16T06:40:18.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): my global_id is 94549 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a40035d0 con 0x7f72c009bb00 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060684029 0 0) 0x7f72c009cca0 con 0x7f72c0138da0 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient(hunting): my global_id is 104340 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c009f750 con 0x7f72c0138da0 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997978205 0 0) 0x7f72a40035d0 con 0x7f72c009bb00 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 >> v1:172.21.15.76:6790/0 conn(0x7f72c009c5f0 legacy=0x7f72c0137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 >> v1:172.21.15.76:6789/0 conn(0x7f72c0138da0 legacy=0x7f72c013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 monclient: found mon.noname-c 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c0097900 con 0x7f72c009bb00 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72ac001610 con 0x7f72c009bb00 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 5 monclient: authenticate success, global_id 94549 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: dump: 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.025 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72ac0018f0 con 0x7f72c009bb00 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 1 -- 172.21.15.189:0/2345226671 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72ac001c80 con 0x7f72c009bb00 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:18.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 4 set_mon_vals no callback set 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72beffd700 10 monclient: dump: 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: shutdown 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- 172.21.15.189:0/2345226671 >> v1:172.21.15.189:6789/0 conn(0x7f72c009bb00 legacy=0x7f72c009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- 172.21.15.189:0/2345226671 shutdown_connections 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- 172.21.15.189:0/2345226671 >> 172.21.15.189:0/2345226671 conn(0x7f72c00a03e0 msgr2=0x7f72c00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- 172.21.15.189:0/2345226671 shutdown_connections 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- 172.21.15.189:0/2345226671 wait complete. 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: build_initial_monmap 2024-09-16T06:40:18.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: monmap: 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 Processor -- start 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- start start 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: init 2024-09-16T06:40:18.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-a con 0x7f72c009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-b con 0x7f72c009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient(hunting): picked mon.noname-c con 0x7f72c0138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c00924b0 con 0x7f72c009bb00 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c009a6e0 con 0x7f72c009c5f0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c00a2d60 con 0x7f72c0138da0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c6781700 10 monclient(hunting): authenticate will time out at 970.531185s 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72c4d1e700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f72c0138da0 legacy=0x7f72c01d28c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72c009c5f0 0x7f72c01cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44526/0 (socket says 172.21.15.189:44526) 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bf7fe700 1 -- 172.21.15.189:0/1825639299 learned_addr learned my addr 172.21.15.189:0/1825639299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496260156 0 0) 0x7f72c009a6e0 con 0x7f72c009c5f0 2024-09-16T06:40:18.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): my global_id is 94552 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a80035d0 con 0x7f72c009c5f0 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860864394 0 0) 0x7f72c00924b0 con 0x7f72c009bb00 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.014+0000 7f72bcff9700 10 monclient(hunting): my global_id is 104343 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c009a6e0 con 0x7f72c009bb00 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4003894300 0 0) 0x7f72a80035d0 con 0x7f72c009c5f0 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72c00924b0 con 0x7f72c009c5f0 2024-09-16T06:40:18.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72b0004230 con 0x7f72c009c5f0 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient(hunting): dump: 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1271632850 0 0) 0x7f72c00924b0 con 0x7f72c009c5f0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 >> v1:172.21.15.76:6790/0 conn(0x7f72c0138da0 legacy=0x7f72c01d28c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 >> v1:172.21.15.76:6789/0 conn(0x7f72c009bb00 legacy=0x7f72c0137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 monclient: found mon.b 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c0093e10 con 0x7f72c009c5f0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 5 monclient: authenticate success, global_id 94552 2024-09-16T06:40:18.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _renew_subs 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c0093570 con 0x7f72c009c5f0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _renew_subs 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c0093240 con 0x7f72c009c5f0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72b0004510 con 0x7f72c009c5f0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72b00048a0 con 0x7f72c009c5f0 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bdffb700 4 set_mon_vals no callback set 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 10 monclient: dump: 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.034 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c005100 con 0x7f72c009c5f0 2024-09-16T06:40:18.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f72b0004a80 con 0x7f72c009c5f0 2024-09-16T06:40:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(42..42 src has 1..42) v4 ==== 3598+0+0 (unknown 325265261 0 0) 0x7f72b00431d0 con 0x7f72c009c5f0 2024-09-16T06:40:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.018+0000 7f72bffff700 10 monclient: get_auth_request con 0x7f72a80361b0 auth_method 0 2024-09-16T06:40:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.022+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f72b00148a0 con 0x7f72c009c5f0 2024-09-16T06:40:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.022+0000 7f72bcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.022+0000 7f72bcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:18.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.182+0000 7f72c6781700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.182+0000 7f72c6781700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.182+0000 7f72c6781700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.182+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f729c005930 con 0x7f72c009c5f0 2024-09-16T06:40:18.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.506+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v43) v1 ==== 126+0+0 (unknown 2269662458 0 0) 0x7f72b0012d10 con 0x7f72c009c5f0 2024-09-16T06:40:18.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.506+0000 7f72bcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.506+0000 7f72bcff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:18.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72c6781700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72c6781700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72c6781700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f729c005420 con 0x7f72c009c5f0 2024-09-16T06:40:18.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72bcff9700 1 -- 172.21.15.189:0/1825639299 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) v1 ==== 149+0+0 (unknown 4013494279 0 0) 0x7f72b0014490 con 0x7f72c009c5f0 2024-09-16T06:40:18.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72bcff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.610+0000 7f72bcff9700 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T06:40:18.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 >> v1:172.21.15.189:6812/3098810674 conn(0x7f72a80361b0 legacy=0x7f72a8038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 10 monclient: shutdown 2024-09-16T06:40:18.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 >> v1:172.21.15.189:6789/0 conn(0x7f72c009c5f0 legacy=0x7f72c01cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 shutdown_connections 2024-09-16T06:40:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 >> 172.21.15.189:0/1825639299 conn(0x7f72c00a03e0 msgr2=0x7f72c00968a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 shutdown_connections 2024-09-16T06:40:18.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.614+0000 7f72c6781700 1 -- 172.21.15.189:0/1825639299 wait complete. 2024-09-16T06:40:18.619 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo already exists 2024-09-16T06:40:18.627 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: get_monmap_and_config 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: build_initial_monmap 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: monmap: 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:18.737200+0000 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:18.737200+0000 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:18.736 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:18.737 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 1 Processor -- start 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 1 -- start start 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: init 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-a con 0x7f38c809f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-c con 0x7f38c8099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-b con 0x7f38c81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:18.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c809f520 con 0x7f38c809f140 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c809bc10 con 0x7f38c8099700 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c809b810 con 0x7f38c81396a0 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient(hunting): authenticate will time out at 971.250133s 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c6d9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f38c81396a0 legacy=0x7f38c813ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c5d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f38c8099700 0x7f38c809b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34092/0 (socket says 172.21.15.189:34092) 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c5d9b700 1 -- 172.21.15.189:0/676696517 learned_addr learned my addr 172.21.15.189:0/676696517 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439106359 0 0) 0x7f38c809f520 con 0x7f38c809f140 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): my global_id is 104346 2024-09-16T06:40:18.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b40035d0 con 0x7f38c809f140 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223587442 0 0) 0x7f38c809bc10 con 0x7f38c8099700 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient(hunting): my global_id is 94555 2024-09-16T06:40:18.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c809f520 con 0x7f38c8099700 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54269835 0 0) 0x7f38c809f520 con 0x7f38c8099700 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 >> v1:172.21.15.76:6790/0 conn(0x7f38c81396a0 legacy=0x7f38c813ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 >> v1:172.21.15.76:6789/0 conn(0x7f38c809f140 legacy=0x7f38c809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 monclient: found mon.noname-c 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c809ceb0 con 0x7f38c8099700 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: _finish_auth 0 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:18.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 5 monclient: authenticate success, global_id 94555 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38b00015f0 con 0x7f38c8099700 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: dump: 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b0001910 con 0x7f38c8099700 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38cc8c7700 4 set_mon_vals no callback set 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 1 -- 172.21.15.189:0/676696517 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38b0001ce0 con 0x7f38c8099700 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.734+0000 7f38c559a700 10 monclient: dump: 2024-09-16T06:40:18.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: shutdown 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/676696517 >> v1:172.21.15.189:6789/0 conn(0x7f38c8099700 legacy=0x7f38c809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/676696517 shutdown_connections 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/676696517 >> 172.21.15.189:0/676696517 conn(0x7f38c80a03e0 msgr2=0x7f38c80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/676696517 shutdown_connections 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/676696517 wait complete. 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: build_initial_monmap 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: monmap: 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:18.749 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 Processor -- start 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- start start 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: init 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-b con 0x7f38c8099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-c con 0x7f38c809f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient(hunting): picked mon.noname-a con 0x7f38c81396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80a13f0 con 0x7f38c81396a0 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c81394d0 con 0x7f38c8099700 2024-09-16T06:40:18.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c813ccb0 con 0x7f38c809f140 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient(hunting): authenticate will time out at 971.254115s 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38c5d9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f38c809f140 legacy=0x7f38c8138dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38c659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f38c8099700 0x7f38c81386b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34100/0 (socket says 172.21.15.189:34100) 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38c659c700 1 -- 172.21.15.189:0/2052122232 learned_addr learned my addr 172.21.15.189:0/2052122232 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186235042 0 0) 0x7f38c81394d0 con 0x7f38c8099700 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): my global_id is 94558 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a80035d0 con 0x7f38c8099700 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774622822 0 0) 0x7f38c0003e20 con 0x7f38c81396a0 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:18.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): my global_id is 104349 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c81394d0 con 0x7f38c81396a0 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020122499 0 0) 0x7f38a80035d0 con 0x7f38c8099700 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c0003e20 con 0x7f38c8099700 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38b80019f0 con 0x7f38c8099700 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): dump: 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.752 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 234054879 0 0) 0x7f38c81394d0 con 0x7f38c81396a0 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a80035d0 con 0x7f38c81396a0 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1987572720 0 0) 0x7f38c0003e20 con 0x7f38c8099700 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 >> v1:172.21.15.76:6790/0 conn(0x7f38c809f140 legacy=0x7f38c8138dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 >> v1:172.21.15.76:6789/0 conn(0x7f38c81396a0 legacy=0x7f38c81d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:18.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 monclient: found mon.b 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c809a6e0 con 0x7f38c8099700 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 5 monclient: authenticate success, global_id 94558 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _renew_subs 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c808fd60 con 0x7f38c8099700 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _renew_subs 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c8090bb0 con 0x7f38c8099700 2024-09-16T06:40:18.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.738+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b8001ce0 con 0x7f38c8099700 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38b8003b70 con 0x7f38c8099700 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38bffff700 4 set_mon_vals no callback set 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 10 monclient: dump: 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:18.755 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f38b8003d50 con 0x7f38c8099700 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38cc8c7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38cc8c7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38cc8c7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac005100 con 0x7f38c8099700 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(43..43 src has 1..43) v4 ==== 3657+0+0 (unknown 536647824 0 0) 0x7f38b80432e0 con 0x7f38c8099700 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.742+0000 7f38c5d9b700 10 monclient: get_auth_request con 0x7f38a8036250 auth_method 0 2024-09-16T06:40:18.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.750+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f38b8014660 con 0x7f38c8099700 2024-09-16T06:40:18.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.750+0000 7f38beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:18.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.750+0000 7f38beffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38cc8c7700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38cc8c7700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38cc8c7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:18.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f38ac005930 con 0x7f38c8099700 2024-09-16T06:40:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) v1 ==== 149+0+0 (unknown 4013494279 0 0) 0x7f38b8014480 con 0x7f38c8099700 2024-09-16T06:40:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:18.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:18.918+0000 7f38beffd700 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-16T06:40:19.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38cc8c7700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:19.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38cc8c7700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38cc8c7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f38ac005420 con 0x7f38c8099700 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38beffd700 1 -- 172.21.15.189:0/2052122232 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) v1 ==== 149+0+0 (unknown 4013494279 0 0) 0x7f38b8014ac0 con 0x7f38c8099700 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38beffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.030+0000 7f38beffd700 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 >> v1:172.21.15.189:6812/3098810674 conn(0x7f38a8036250 legacy=0x7f38a8038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 10 monclient: shutdown 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 >> v1:172.21.15.189:6789/0 conn(0x7f38c8099700 legacy=0x7f38c81386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 shutdown_connections 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 >> 172.21.15.189:0/2052122232 conn(0x7f38c80a03e0 msgr2=0x7f38c8098c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:19.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 shutdown_connections 2024-09-16T06:40:19.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.034+0000 7f38cc8c7700 1 -- 172.21.15.189:0/2052122232 wait complete. 2024-09-16T06:40:19.039 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo already exists 2024-09-16T06:40:19.047 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: get_monmap_and_config 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: build_initial_monmap 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: monmap: 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.161 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:19.162579+0000 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:19.162579+0000 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:19.162 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 1 Processor -- start 2024-09-16T06:40:19.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 1 -- start start 2024-09-16T06:40:19.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: init 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-c con 0x7fef9c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-a con 0x7fef9c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-b con 0x7fef9c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09cca0 con 0x7fef9c09c5f0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09f750 con 0x7fef9c09bb00 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c097620 con 0x7fef9c138da0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fefa2139700 10 monclient(hunting): authenticate will time out at 971.675562s 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fef9bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fef9c138da0 legacy=0x7fef9c13b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fef9b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef9c09bb00 0x7fef9c09bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34116/0 (socket says 172.21.15.189:34116) 2024-09-16T06:40:19.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.158+0000 7fef9b7fe700 1 -- 172.21.15.189:0/2380773527 learned_addr learned my addr 172.21.15.189:0/2380773527 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480689815 0 0) 0x7fef9c09f750 con 0x7fef9c09bb00 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): my global_id is 94561 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef840035d0 con 0x7fef9c09bb00 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472176260 0 0) 0x7fef9c09cca0 con 0x7fef9c09c5f0 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient(hunting): my global_id is 104352 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c09f750 con 0x7fef9c09c5f0 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662703638 0 0) 0x7fef840035d0 con 0x7fef9c09bb00 2024-09-16T06:40:19.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 >> v1:172.21.15.76:6790/0 conn(0x7fef9c138da0 legacy=0x7fef9c13b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 >> v1:172.21.15.76:6789/0 conn(0x7fef9c09c5f0 legacy=0x7fef9c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 monclient: found mon.noname-c 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c097900 con 0x7fef9c09bb00 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 5 monclient: authenticate success, global_id 94561 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef90001610 con 0x7fef9c09bb00 2024-09-16T06:40:19.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: dump: 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:19.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef900018f0 con 0x7fef9c09bb00 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 4 set_mon_vals no callback set 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 1 -- 172.21.15.189:0/2380773527 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef90001c80 con 0x7fef9c09bb00 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9a7fc700 10 monclient: dump: 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: shutdown 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- 172.21.15.189:0/2380773527 >> v1:172.21.15.189:6789/0 conn(0x7fef9c09bb00 legacy=0x7fef9c09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- 172.21.15.189:0/2380773527 shutdown_connections 2024-09-16T06:40:19.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- 172.21.15.189:0/2380773527 >> 172.21.15.189:0/2380773527 conn(0x7fef9c0a03e0 msgr2=0x7fef9c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- 172.21.15.189:0/2380773527 shutdown_connections 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- 172.21.15.189:0/2380773527 wait complete. 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: build_initial_monmap 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: monmap: 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:19.176 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 Processor -- start 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- start start 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: init 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-c con 0x7fef9c09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-a con 0x7fef9c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient(hunting): picked mon.noname-b con 0x7fef9c138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09a6e0 con 0x7fef9c09c5f0 2024-09-16T06:40:19.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c099580 con 0x7fef9c138da0 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c094030 con 0x7fef9c09bb00 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fefa2139700 10 monclient(hunting): authenticate will time out at 971.679024s 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9b7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fef9c09bb00 legacy=0x7fef9c137a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef9c138da0 0x7fef9c1d2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34126/0 (socket says 172.21.15.189:34126) 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef9bfff700 1 -- 172.21.15.189:0/2919591362 learned_addr learned my addr 172.21.15.189:0/2919591362 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205165147 0 0) 0x7fef9c099580 con 0x7fef9c138da0 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): my global_id is 94564 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef880035d0 con 0x7fef9c138da0 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241788356 0 0) 0x7fef9c09a6e0 con 0x7fef9c09c5f0 2024-09-16T06:40:19.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): my global_id is 104355 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c099580 con 0x7fef9c09c5f0 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 552842443 0 0) 0x7fef880035d0 con 0x7fef9c138da0 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c09a6e0 con 0x7fef9c138da0 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef940044c0 con 0x7fef9c138da0 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): dump: 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.162+0000 7fef83fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3788694033 0 0) 0x7fef9c09a6e0 con 0x7fef9c138da0 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 >> v1:172.21.15.76:6790/0 conn(0x7fef9c09bb00 legacy=0x7fef9c137a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 >> v1:172.21.15.76:6789/0 conn(0x7fef9c09c5f0 legacy=0x7fef9c1cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 monclient: found mon.b 2024-09-16T06:40:19.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c093790 con 0x7fef9c138da0 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 5 monclient: authenticate success, global_id 94564 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _renew_subs 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c093a70 con 0x7fef9c138da0 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _renew_subs 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c091cd0 con 0x7fef9c138da0 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef940047a0 con 0x7fef9c138da0 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef94004b30 con 0x7fef9c138da0 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef99ffb700 4 set_mon_vals no callback set 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 10 monclient: dump: 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.182 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fef94004d10 con 0x7fef9c138da0 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(43..43 src has 1..43) v4 ==== 3657+0+0 (unknown 536647824 0 0) 0x7fef940435e0 con 0x7fef9c138da0 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fef9b7fe700 10 monclient: get_auth_request con 0x7fef88036200 auth_method 0 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.166+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c005100 con 0x7fef9c138da0 2024-09-16T06:40:19.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.174+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fef94014b30 con 0x7fef9c138da0 2024-09-16T06:40:19.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.174+0000 7fef83fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.174+0000 7fef83fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.346+0000 7fefa2139700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.346+0000 7fefa2139700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.346+0000 7fefa2139700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.346+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7fef7c005930 con 0x7fef9c138da0 2024-09-16T06:40:19.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.518+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v44) v1 ==== 126+0+0 (unknown 3441966178 0 0) 0x7fef94012fa0 con 0x7fef9c138da0 2024-09-16T06:40:19.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.518+0000 7fef83fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.518+0000 7fef83fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:19.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7fef7c005420 con 0x7fef9c138da0 2024-09-16T06:40:19.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fef83fff700 1 -- 172.21.15.189:0/2919591362 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v44) v1 ==== 149+0+0 (unknown 145563413 0 0) 0x7fef94014720 con 0x7fef9c138da0 2024-09-16T06:40:19.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fef83fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T06:40:19.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fef83fff700 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 >> v1:172.21.15.189:6812/3098810674 conn(0x7fef88036200 legacy=0x7fef880386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 10 monclient: shutdown 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.630+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 >> v1:172.21.15.189:6789/0 conn(0x7fef9c138da0 legacy=0x7fef9c1d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.634+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 shutdown_connections 2024-09-16T06:40:19.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.634+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 >> 172.21.15.189:0/2919591362 conn(0x7fef9c0a03e0 msgr2=0x7fef9c0a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:19.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.634+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 shutdown_connections 2024-09-16T06:40:19.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.634+0000 7fefa2139700 1 -- 172.21.15.189:0/2919591362 wait complete. 2024-09-16T06:40:19.637 INFO:tasks.workunit.client.0.smithi189.stderr:rule bar already exists 2024-09-16T06:40:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class all 2024-09-16T06:40:19.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:19.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: build_initial_monmap 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: monmap: 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:19.749701+0000 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:19.749701+0000 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:19.749 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 1 Processor -- start 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 1 -- start start 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: init 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-c con 0x7f5894132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-b con 0x7f5894138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-a con 0x7f589413bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:19.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589406abd0 con 0x7f589413bf70 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589406b170 con 0x7f5894132080 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5894132b70 con 0x7f5894138460 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 10 monclient(hunting): authenticate will time out at 972.262799s 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5893fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f5894138460 legacy=0x7f589413a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5898a8c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5894132080 0x7f5894132460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34128/0 (socket says 172.21.15.189:34128) 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5898a8c700 1 -- 172.21.15.189:0/3393335133 learned_addr learned my addr 172.21.15.189:0/3393335133 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229344750 0 0) 0x7f589406b170 con 0x7f5894132080 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): my global_id is 94567 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c0035d0 con 0x7f5894132080 2024-09-16T06:40:19.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957439800 0 0) 0x7f589406abd0 con 0x7f589413bf70 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient(hunting): my global_id is 104358 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589406b170 con 0x7f589413bf70 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081267210 0 0) 0x7f587c0035d0 con 0x7f5894132080 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 >> v1:172.21.15.76:6790/0 conn(0x7f5894138460 legacy=0x7f589413a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 >> v1:172.21.15.76:6789/0 conn(0x7f589413bf70 legacy=0x7f589413e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940a38b0 con 0x7f5894132080 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:19.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 5 monclient: authenticate success, global_id 94567 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5888001610 con 0x7f5894132080 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f58937fe700 10 monclient: dump: 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.756 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.746+0000 7f5899a8e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58880018f0 con 0x7f5894132080 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 1 -- 172.21.15.189:0/3393335133 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5888001c80 con 0x7f5894132080 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 4 set_mon_vals no callback set 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58937fe700 10 monclient: dump: 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: shutdown 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- 172.21.15.189:0/3393335133 >> v1:172.21.15.189:6789/0 conn(0x7f5894132080 legacy=0x7f5894132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- 172.21.15.189:0/3393335133 shutdown_connections 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- 172.21.15.189:0/3393335133 >> 172.21.15.189:0/3393335133 conn(0x7f58940a4660 msgr2=0x7f5894099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:19.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- 172.21.15.189:0/3393335133 shutdown_connections 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- 172.21.15.189:0/3393335133 wait complete. 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: build_initial_monmap 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: monmap: 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 Processor -- start 2024-09-16T06:40:19.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- start start 2024-09-16T06:40:19.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: init 2024-09-16T06:40:19.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:19.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:19.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-c con 0x7f5894132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:19.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-b con 0x7f5894138460 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:19.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient(hunting): picked mon.noname-a con 0x7f589413bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589413bb00 con 0x7f589413bf70 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5894138090 con 0x7f5894138460 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589413b750 con 0x7f5894132080 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5898a8c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f5894132080 legacy=0x7f58940a02c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5899a8e700 10 monclient(hunting): authenticate will time out at 972.267156s 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5893fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5894138460 0x7f58941ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34140/0 (socket says 172.21.15.189:34140) 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f5893fff700 1 -- 172.21.15.189:0/899639028 learned_addr learned my addr 172.21.15.189:0/899639028 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088122455 0 0) 0x7f5894138090 con 0x7f5894138460 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): my global_id is 94570 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58800035d0 con 0x7f5894138460 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027695164 0 0) 0x7f589406b170 con 0x7f589413bf70 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 10 monclient(hunting): my global_id is 104361 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5894138090 con 0x7f589413bf70 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711242451 0 0) 0x7f58800035d0 con 0x7f5894138460 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f589406b170 con 0x7f5894138460 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.750+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f58840045b0 con 0x7f5894138460 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:19.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient(hunting): dump: 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3864600822 0 0) 0x7f589406b170 con 0x7f5894138460 2024-09-16T06:40:19.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 >> v1:172.21.15.76:6790/0 conn(0x7f5894132080 legacy=0x7f58940a02c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 >> v1:172.21.15.76:6789/0 conn(0x7f589413bf70 legacy=0x7f58941d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 monclient: found mon.b 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589409f600 con 0x7f5894138460 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 5 monclient: authenticate success, global_id 94570 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _renew_subs 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589413ab80 con 0x7f5894138460 2024-09-16T06:40:19.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _renew_subs 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941d2f60 con 0x7f5894138460 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58840047d0 con 0x7f5894138460 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5892ffd700 4 set_mon_vals no callback set 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5884004b60 con 0x7f5894138460 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 10 monclient: dump: 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.768 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5884004d40 con 0x7f5894138460 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(44..44 src has 1..44) v4 ==== 3716+0+0 (unknown 4195441208 0 0) 0x7f58840435e0 con 0x7f5894138460 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5874005100 con 0x7f5894138460 2024-09-16T06:40:19.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.754+0000 7f5898a8c700 10 monclient: get_auth_request con 0x7f5880036200 auth_method 0 2024-09-16T06:40:19.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.762+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5884005560 con 0x7f5894138460 2024-09-16T06:40:19.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.762+0000 7f58917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:19.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.762+0000 7f58917fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.934+0000 7f5899a8e700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.934+0000 7f5899a8e700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.934+0000 7f5899a8e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:19.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:19.934+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f58740053b0 con 0x7f5894138460 2024-09-16T06:40:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.526+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v45) v1 ==== 131+0+0 (unknown 2832017534 0 0) 0x7f5884012fc0 con 0x7f5894138460 2024-09-16T06:40:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.526+0000 7f58917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:20.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.526+0000 7f58917fa700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-16T06:40:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.634+0000 7f5899a8e700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.634+0000 7f5899a8e700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.634+0000 7f5899a8e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.634+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f58740023c0 con 0x7f5894138460 2024-09-16T06:40:20.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.894+0000 7f58917fa700 1 -- 172.21.15.189:0/899639028 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v46) v1 ==== 282+0+0 (unknown 1309378230 0 0) 0x7f5884014740 con 0x7f5894138460 2024-09-16T06:40:20.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.894+0000 7f58917fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T06:40:20.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.894+0000 7f58917fa700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T06:40:20.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5880036200 legacy=0x7f58800386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:20.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 10 monclient: shutdown 2024-09-16T06:40:20.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:20.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 >> v1:172.21.15.189:6789/0 conn(0x7f5894138460 legacy=0x7f58941ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:20.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 shutdown_connections 2024-09-16T06:40:20.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 >> 172.21.15.189:0/899639028 conn(0x7f58940a4660 msgr2=0x7f58940a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:20.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 shutdown_connections 2024-09-16T06:40:20.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:20.898+0000 7f5899a8e700 1 -- 172.21.15.189:0/899639028 wait complete. 2024-09-16T06:40:20.904 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T06:40:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-16T06:40:21.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: get_monmap_and_config 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: build_initial_monmap 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: monmap: 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:21.025893+0000 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:21.025893+0000 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:21.025 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:21.026 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:21.026 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 1 Processor -- start 2024-09-16T06:40:21.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 1 -- start start 2024-09-16T06:40:21.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: init 2024-09-16T06:40:21.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-a con 0x7fba6c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-b con 0x7fba6c09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-c con 0x7fba6c138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c09cca0 con 0x7fba6c09bb00 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c09f750 con 0x7fba6c138da0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c097620 con 0x7fba6c09c5f0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 10 monclient(hunting): authenticate will time out at 973.538875s 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba71e09700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fba6c138da0 0x7fba6c13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34142/0 (socket says 172.21.15.189:34142) 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba71e09700 1 -- 172.21.15.189:0/4259854283 learned_addr learned my addr 172.21.15.189:0/4259854283 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba70e07700 1 -- 172.21.15.189:0/4259854283 >> v1:172.21.15.76:6790/0 conn(0x7fba6c09c5f0 legacy=0x7fba6c137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:21.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826001934 0 0) 0x7fba6c09f750 con 0x7fba6c138da0 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): my global_id is 94573 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba580035d0 con 0x7fba6c138da0 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537486779 0 0) 0x7fba6c09cca0 con 0x7fba6c09bb00 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient(hunting): my global_id is 104364 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c09f750 con 0x7fba6c09bb00 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119292173 0 0) 0x7fba580035d0 con 0x7fba6c138da0 2024-09-16T06:40:21.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 >> v1:172.21.15.76:6790/0 conn(0x7fba6c09c5f0 legacy=0x7fba6c137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 >> v1:172.21.15.76:6789/0 conn(0x7fba6c09bb00 legacy=0x7fba6c09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 monclient: found mon.noname-c 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c097900 con 0x7fba6c138da0 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba64001990 con 0x7fba6c138da0 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 5 monclient: authenticate success, global_id 94573 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.022+0000 7fba7386c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:21.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: dump: 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.034 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba64001bb0 con 0x7fba6c138da0 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 1 -- 172.21.15.189:0/4259854283 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba64001f40 con 0x7fba6c138da0 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba63fff700 10 monclient: dump: 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 4 set_mon_vals no callback set 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: shutdown 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- 172.21.15.189:0/4259854283 >> v1:172.21.15.189:6789/0 conn(0x7fba6c138da0 legacy=0x7fba6c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- 172.21.15.189:0/4259854283 shutdown_connections 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- 172.21.15.189:0/4259854283 >> 172.21.15.189:0/4259854283 conn(0x7fba6c0a03e0 msgr2=0x7fba6c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:21.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- 172.21.15.189:0/4259854283 shutdown_connections 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- 172.21.15.189:0/4259854283 wait complete. 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: build_initial_monmap 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: monmap: 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:21.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 Processor -- start 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- start start 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: init 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-b con 0x7fba6c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-c con 0x7fba6c09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient(hunting): picked mon.noname-a con 0x7fba6c138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c099510 con 0x7fba6c138da0 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c094490 con 0x7fba6c09bb00 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c09a6e0 con 0x7fba6c09c5f0 2024-09-16T06:40:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba7386c700 10 monclient(hunting): authenticate will time out at 973.542829s 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba70e07700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fba6c09c5f0 legacy=0x7fba6c1cf190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba71608700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fba6c09bb00 0x7fba6c1ca1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34154/0 (socket says 172.21.15.189:34154) 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba71608700 1 -- 172.21.15.189:0/2658459 learned_addr learned my addr 172.21.15.189:0/2658459 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251143854 0 0) 0x7fba6c094490 con 0x7fba6c09bb00 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): my global_id is 94576 2024-09-16T06:40:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba500035d0 con 0x7fba6c09bb00 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048775374 0 0) 0x7fba6c097900 con 0x7fba6c138da0 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): my global_id is 104367 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c094490 con 0x7fba6c138da0 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541162888 0 0) 0x7fba500035d0 con 0x7fba6c09bb00 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba6c097900 con 0x7fba6c09bb00 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba5c001970 con 0x7fba6c09bb00 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.026+0000 7fba61ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2065016872 0 0) 0x7fba6c094490 con 0x7fba6c138da0 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba500035d0 con 0x7fba6c138da0 2024-09-16T06:40:21.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba640016f0 con 0x7fba6c138da0 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:21.044 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1716065094 0 0) 0x7fba6c097900 con 0x7fba6c09bb00 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 >> v1:172.21.15.76:6790/0 conn(0x7fba6c09c5f0 legacy=0x7fba6c1cf190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 >> v1:172.21.15.76:6789/0 conn(0x7fba6c138da0 legacy=0x7fba6c1d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 monclient: found mon.b 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0937f0 con 0x7fba6c09bb00 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 5 monclient: authenticate success, global_id 94576 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _renew_subs 2024-09-16T06:40:21.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c094ff0 con 0x7fba6c09bb00 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _renew_subs 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c06d9e0 con 0x7fba6c09bb00 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba5c001ce0 con 0x7fba6c09bb00 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c068180 con 0x7fba6c09bb00 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba5c003b10 con 0x7fba6c09bb00 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:21.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba637fe700 4 set_mon_vals no callback set 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 10 monclient: dump: 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fba5c003cb0 con 0x7fba6c09bb00 2024-09-16T06:40:21.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(46..46 src has 1..46) v4 ==== 3412+0+0 (unknown 509815128 0 0) 0x7fba5c043170 con 0x7fba6c09bb00 2024-09-16T06:40:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.030+0000 7fba70e07700 10 monclient: get_auth_request con 0x7fba50036310 auth_method 0 2024-09-16T06:40:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.038+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fba5c014810 con 0x7fba6c09bb00 2024-09-16T06:40:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.038+0000 7fba61ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:21.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.038+0000 7fba61ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:21.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.210+0000 7fba7386c700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:21.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.210+0000 7fba7386c700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:21.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.210+0000 7fba7386c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:21.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.210+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7fba6c094750 con 0x7fba6c09bb00 2024-09-16T06:40:21.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.898+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v47) v1 ==== 135+0+0 (unknown 2203644516 0 0) 0x7fba5c012c80 con 0x7fba6c09bb00 2024-09-16T06:40:21.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.898+0000 7fba61ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:21.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:21.898+0000 7fba61ffb700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-16T06:40:22.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.010+0000 7fba7386c700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:22.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.010+0000 7fba7386c700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:22.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.010+0000 7fba7386c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:22.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.010+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7fba6c094b30 con 0x7fba6c09bb00 2024-09-16T06:40:22.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.898+0000 7fba61ffb700 1 -- 172.21.15.189:0/2658459 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v48) v1 ==== 237+0+0 (unknown 911157329 0 0) 0x7fba5c014400 con 0x7fba6c09bb00 2024-09-16T06:40:22.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.898+0000 7fba61ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T06:40:22.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.898+0000 7fba61ffb700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 >> v1:172.21.15.189:6812/3098810674 conn(0x7fba50036310 legacy=0x7fba500387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 10 monclient: shutdown 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 >> v1:172.21.15.189:6789/0 conn(0x7fba6c09bb00 legacy=0x7fba6c1ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 shutdown_connections 2024-09-16T06:40:22.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.902+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 >> 172.21.15.189:0/2658459 conn(0x7fba6c0a03e0 msgr2=0x7fba6c09a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:22.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.906+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 shutdown_connections 2024-09-16T06:40:22.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:22.906+0000 7fba7386c700 1 -- 172.21.15.189:0/2658459 wait complete. 2024-09-16T06:40:22.909 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T06:40:22.924 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-16T06:40:23.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: get_monmap_and_config 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: build_initial_monmap 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: monmap: 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:23.033845+0000 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:23.033845+0000 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:23.033 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 1 Processor -- start 2024-09-16T06:40:23.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 1 -- start start 2024-09-16T06:40:23.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: init 2024-09-16T06:40:23.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:23.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:23.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-a con 0x7f39cc133380 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:23.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-b con 0x7f39cc09f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-c con 0x7f39cc0a1190 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.030+0000 7f39d3e65700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc132ec0 con 0x7f39cc133380 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc09e4c0 con 0x7f39cc0a1190 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc09e7a0 con 0x7f39cc09f760 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient(hunting): authenticate will time out at 975.548113s 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d1400700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f39cc09f760 legacy=0x7f39cc138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d2402700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f39cc0a1190 0x7f39cc13b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34162/0 (socket says 172.21.15.189:34162) 2024-09-16T06:40:23.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d2402700 1 -- 172.21.15.189:0/2754733814 learned_addr learned my addr 172.21.15.189:0/2754733814 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112504989 0 0) 0x7f39cc09e4c0 con 0x7f39cc0a1190 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): my global_id is 94579 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b80035d0 con 0x7f39cc0a1190 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319005377 0 0) 0x7f39cc132ec0 con 0x7f39cc133380 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient(hunting): my global_id is 104370 2024-09-16T06:40:23.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39cc09e4c0 con 0x7f39cc133380 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3411793044 0 0) 0x7f39b80035d0 con 0x7f39cc0a1190 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 >> v1:172.21.15.76:6790/0 conn(0x7f39cc09f760 legacy=0x7f39cc138240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 >> v1:172.21.15.76:6789/0 conn(0x7f39cc133380 legacy=0x7f39cc133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 monclient: found mon.noname-c 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0a1cb0 con 0x7f39cc0a1190 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: _finish_auth 0 2024-09-16T06:40:23.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 5 monclient: authenticate success, global_id 94579 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39c4001990 con 0x7f39cc0a1190 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: dump: 2024-09-16T06:40:23.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39c4001c70 con 0x7f39cc0a1190 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:23.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 4 set_mon_vals no callback set 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 1 -- 172.21.15.189:0/2754733814 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39c4002000 con 0x7f39cc0a1190 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d0bff700 10 monclient: dump: 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:23.044 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient: shutdown 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- 172.21.15.189:0/2754733814 >> v1:172.21.15.189:6789/0 conn(0x7f39cc0a1190 legacy=0x7f39cc13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- 172.21.15.189:0/2754733814 shutdown_connections 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- 172.21.15.189:0/2754733814 >> 172.21.15.189:0/2754733814 conn(0x7f39cc0a21f0 msgr2=0x7f39cc0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- 172.21.15.189:0/2754733814 shutdown_connections 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- 172.21.15.189:0/2754733814 wait complete. 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient: build_initial_monmap 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient: monmap: 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:23.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:23.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 Processor -- start 2024-09-16T06:40:23.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 1 -- start start 2024-09-16T06:40:23.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.034+0000 7f39d3e65700 10 monclient: init 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-c con 0x7f39cc09f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-b con 0x7f39cc0a1190 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient(hunting): picked mon.noname-a con 0x7f39cc133380 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc06f4b0 con 0x7f39cc133380 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc06ed60 con 0x7f39cc0a1190 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc09c700 con 0x7f39cc09f760 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient(hunting): authenticate will time out at 975.552316s 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d1400700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f39cc0a1190 0x7f39cc12c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34166/0 (socket says 172.21.15.189:34166) 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d1400700 1 -- 172.21.15.189:0/1144543321 learned_addr learned my addr 172.21.15.189:0/1144543321 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:23.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d1c01700 1 -- 172.21.15.189:0/1144543321 >> v1:172.21.15.76:6790/0 conn(0x7f39cc09f760 legacy=0x7f39cc12bb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550181809 0 0) 0x7f39cc06ed60 con 0x7f39cc0a1190 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): my global_id is 94582 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b00035d0 con 0x7f39cc0a1190 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967897246 0 0) 0x7f39cc06f4b0 con 0x7f39cc133380 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): my global_id is 104373 2024-09-16T06:40:23.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39cc06ed60 con 0x7f39cc133380 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437662788 0 0) 0x7f39b00035d0 con 0x7f39cc0a1190 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39cc06f4b0 con 0x7f39cc0a1190 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39bc004240 con 0x7f39cc0a1190 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): dump: 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:23.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 283094045 0 0) 0x7f39cc06f4b0 con 0x7f39cc0a1190 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 >> v1:172.21.15.76:6790/0 conn(0x7f39cc09f760 legacy=0x7f39cc12bb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 >> v1:172.21.15.76:6789/0 conn(0x7f39cc133380 legacy=0x7f39cc1d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 monclient: found mon.b 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc070620 con 0x7f39cc0a1190 2024-09-16T06:40:23.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 5 monclient: authenticate success, global_id 94582 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _renew_subs 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc12c920 con 0x7f39cc0a1190 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _renew_subs 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc0a5020 con 0x7f39cc0a1190 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39bc004460 con 0x7f39cc0a1190 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39bc0047f0 con 0x7f39cc0a1190 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:23.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c3fff700 4 set_mon_vals no callback set 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 10 monclient: dump: 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:23.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f39bc0049d0 con 0x7f39cc0a1190 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b4005100 con 0x7f39cc0a1190 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(48..48 src has 1..48) v4 ==== 3636+0+0 (unknown 594575816 0 0) 0x7f39bc043230 con 0x7f39cc0a1190 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.038+0000 7f39d1c01700 10 monclient: get_auth_request con 0x7f39b00361d0 auth_method 0 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.046+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f39bc0147f0 con 0x7f39cc0a1190 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.046+0000 7f39c27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:23.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.046+0000 7f39c27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:23.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.222+0000 7f39d3e65700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:23.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.222+0000 7f39d3e65700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:23.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.222+0000 7f39d3e65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:23.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.222+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f39b4005930 con 0x7f39cc0a1190 2024-09-16T06:40:23.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.918+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v49) v1 ==== 135+0+0 (unknown 3857376242 0 0) 0x7f39bc012c60 con 0x7f39cc0a1190 2024-09-16T06:40:23.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.918+0000 7f39c27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:23.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:23.918+0000 7f39c27fc700 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-16T06:40:24.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.030+0000 7f39d3e65700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:24.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.030+0000 7f39d3e65700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:24.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.030+0000 7f39d3e65700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:24.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.030+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f39b40053f0 con 0x7f39cc0a1190 2024-09-16T06:40:24.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.922+0000 7f39c27fc700 1 -- 172.21.15.189:0/1144543321 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v50) v1 ==== 237+0+0 (unknown 1931645736 0 0) 0x7f39bc0143e0 con 0x7f39cc0a1190 2024-09-16T06:40:24.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.922+0000 7f39c27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T06:40:24.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.922+0000 7f39c27fc700 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T06:40:24.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 >> v1:172.21.15.189:6812/3098810674 conn(0x7f39b00361d0 legacy=0x7f39b0038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:24.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 10 monclient: shutdown 2024-09-16T06:40:24.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 >> v1:172.21.15.189:6789/0 conn(0x7f39cc0a1190 legacy=0x7f39cc12c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 shutdown_connections 2024-09-16T06:40:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 >> 172.21.15.189:0/1144543321 conn(0x7f39cc0a21f0 msgr2=0x7f39cc12ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 shutdown_connections 2024-09-16T06:40:24.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:24.926+0000 7f39d3e65700 1 -- 172.21.15.189:0/1144543321 wait complete. 2024-09-16T06:40:24.931 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T06:40:24.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 10 monclient: get_monmap_and_config 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 10 monclient: build_initial_monmap 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 10 monclient: monmap: 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:25.058113+0000 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:25.058113+0000 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:25.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:25.058 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 1 Processor -- start 2024-09-16T06:40:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 1 -- start start 2024-09-16T06:40:25.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.054+0000 7f22d239c700 10 monclient: init 2024-09-16T06:40:25.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-c con 0x7f22cc0a27a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-b con 0x7f22cc0a3d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-a con 0x7f22cc0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc099a40 con 0x7f22cc0a2350 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc099ef0 con 0x7f22cc0a27a0 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc09a1d0 con 0x7f22cc0a3d10 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient(hunting): authenticate will time out at 977.572534s 2024-09-16T06:40:25.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22c3fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f22cc0a3d10 legacy=0x7f22cc13a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f22cc0a27a0 0x7f22cc0a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34172/0 (socket says 172.21.15.189:34172) 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cbfff700 1 -- 172.21.15.189:0/3102834891 learned_addr learned my addr 172.21.15.189:0/3102834891 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415853563 0 0) 0x7f22cc099ef0 con 0x7f22cc0a27a0 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): my global_id is 94585 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b80035d0 con 0x7f22cc0a27a0 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545437422 0 0) 0x7f22cc099a40 con 0x7f22cc0a2350 2024-09-16T06:40:25.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient(hunting): my global_id is 104376 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc099ef0 con 0x7f22cc0a2350 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635215282 0 0) 0x7f22b80035d0 con 0x7f22cc0a27a0 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 >> v1:172.21.15.76:6790/0 conn(0x7f22cc0a3d10 legacy=0x7f22cc13a5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 >> v1:172.21.15.76:6789/0 conn(0x7f22cc0a2350 legacy=0x7f22cc13dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:25.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc0a3290 con 0x7f22cc0a27a0 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 5 monclient: authenticate success, global_id 94585 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22bc001610 con 0x7f22cc0a27a0 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:25.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: dump: 2024-09-16T06:40:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.067 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22bc0018f0 con 0x7f22cc0a27a0 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 4 set_mon_vals no callback set 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 1 -- 172.21.15.189:0/3102834891 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22bc001c80 con 0x7f22cc0a27a0 2024-09-16T06:40:25.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22cb7fe700 10 monclient: dump: 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:25.069 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: shutdown 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- 172.21.15.189:0/3102834891 >> v1:172.21.15.189:6789/0 conn(0x7f22cc0a27a0 legacy=0x7f22cc0a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- 172.21.15.189:0/3102834891 shutdown_connections 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- 172.21.15.189:0/3102834891 >> 172.21.15.189:0/3102834891 conn(0x7f22cc0a7d70 msgr2=0x7f22cc0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- 172.21.15.189:0/3102834891 shutdown_connections 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- 172.21.15.189:0/3102834891 wait complete. 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: build_initial_monmap 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: monmap: 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:25.070 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 Processor -- start 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 1 -- start start 2024-09-16T06:40:25.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.058+0000 7f22d239c700 10 monclient: init 2024-09-16T06:40:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:25.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-a con 0x7f22cc0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-c con 0x7f22cc0a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient(hunting): picked mon.noname-b con 0x7f22cc135820 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc09b510 con 0x7f22cc0a2350 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc06bea0 con 0x7f22cc135820 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22cc13d530 con 0x7f22cc0a27a0 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient(hunting): authenticate will time out at 977.576415s 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c3fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f22cc0a27a0 legacy=0x7f22cc135110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d0939700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f22cc135820 0x7f22cc1d1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34176/0 (socket says 172.21.15.189:34176) 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d0939700 1 -- 172.21.15.189:0/2572612344 learned_addr learned my addr 172.21.15.189:0/2572612344 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216464868 0 0) 0x7f22cc06bea0 con 0x7f22cc135820 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:25.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): my global_id is 94588 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ac0035d0 con 0x7f22cc135820 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116793416 0 0) 0x7f22cc09b510 con 0x7f22cc0a2350 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:25.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): my global_id is 104379 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc06bea0 con 0x7f22cc0a2350 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796189394 0 0) 0x7f22ac0035d0 con 0x7f22cc135820 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22cc09b510 con 0x7f22cc135820 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22c4001c50 con 0x7f22cc135820 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): dump: 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:25.077 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 627608353 0 0) 0x7f22cc06bea0 con 0x7f22cc0a2350 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ac0035d0 con 0x7f22cc0a2350 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22bc0015d0 con 0x7f22cc0a2350 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:25.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient(hunting): dump: 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:25.079 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902279084 0 0) 0x7f22cc09b510 con 0x7f22cc135820 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 >> v1:172.21.15.76:6790/0 conn(0x7f22cc0a27a0 legacy=0x7f22cc135110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 >> v1:172.21.15.76:6789/0 conn(0x7f22cc0a2350 legacy=0x7f22cc13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 monclient: found mon.b 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc13c160 con 0x7f22cc135820 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 5 monclient: authenticate success, global_id 94588 2024-09-16T06:40:25.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _renew_subs 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc135cb0 con 0x7f22cc135820 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _renew_subs 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc135f60 con 0x7f22cc135820 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22c4001fe0 con 0x7f22cc135820 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22ca7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22c4003e90 con 0x7f22cc135820 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 10 monclient: dump: 2024-09-16T06:40:25.081 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f22c40040b0 con 0x7f22cc135820 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.062+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b0005100 con 0x7f22cc135820 2024-09-16T06:40:25.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.066+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(50..50 src has 1..50) v4 ==== 3868+0+0 (unknown 2170589261 0 0) 0x7f22c4043760 con 0x7f22cc135820 2024-09-16T06:40:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.066+0000 7f22cbfff700 10 monclient: get_auth_request con 0x7f22ac0362a0 auth_method 0 2024-09-16T06:40:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.070+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f22c40149b0 con 0x7f22cc135820 2024-09-16T06:40:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.070+0000 7f22c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:25.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.070+0000 7f22c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.246+0000 7f22d239c700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.246+0000 7f22d239c700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.246+0000 7f22d239c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.246+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f22b00053b0 con 0x7f22cc135820 2024-09-16T06:40:25.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.898+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v51) v1 ==== 150+0+0 (unknown 1418137206 0 0) 0x7f22c4013050 con 0x7f22cc135820 2024-09-16T06:40:25.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.898+0000 7f22c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:25.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:25.898+0000 7f22c97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22d239c700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22d239c700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22d239c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f22b0005650 con 0x7f22cc135820 2024-09-16T06:40:26.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22c97fa700 1 -- 172.21.15.189:0/2572612344 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v51) v1 ==== 177+0+0 (unknown 1365241453 0 0) 0x7f22c40147d0 con 0x7f22cc135820 2024-09-16T06:40:26.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22c97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T06:40:26.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.010+0000 7f22c97fa700 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 >> v1:172.21.15.189:6812/3098810674 conn(0x7f22ac0362a0 legacy=0x7f22ac038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 10 monclient: shutdown 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 >> v1:172.21.15.189:6789/0 conn(0x7f22cc135820 legacy=0x7f22cc1d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 shutdown_connections 2024-09-16T06:40:26.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 >> 172.21.15.189:0/2572612344 conn(0x7f22cc0a7d70 msgr2=0x7f22cc0a0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:26.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 shutdown_connections 2024-09-16T06:40:26.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.014+0000 7f22d239c700 1 -- 172.21.15.189:0/2572612344 wait complete. 2024-09-16T06:40:26.018 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-ssd already exists 2024-09-16T06:40:26.027 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-16T06:40:26.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: get_monmap_and_config 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: build_initial_monmap 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: monmap: 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:26.140899+0000 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:26.140899+0000 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:26.140 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 Processor -- start 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- start start 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: init 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-b con 0x7fb90809bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:26.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-a con 0x7fb90809c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-c con 0x7fb908138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90809cca0 con 0x7fb90809c5f0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90809f750 con 0x7fb908138da0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb908097620 con 0x7fb90809bb00 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient(hunting): authenticate will time out at 978.653952s 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90ce2b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fb90809bb00 legacy=0x7fb90809bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90d62c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb908138da0 0x7fb90813b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34188/0 (socket says 172.21.15.189:34188) 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90d62c700 1 -- 172.21.15.189:0/621511656 learned_addr learned my addr 172.21.15.189:0/621511656 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631522732 0 0) 0x7fb90809f750 con 0x7fb908138da0 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:26.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient(hunting): my global_id is 94591 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f00035d0 con 0x7fb908138da0 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025821466 0 0) 0x7fb8f00035d0 con 0x7fb908138da0 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 >> v1:172.21.15.76:6790/0 conn(0x7fb90809bb00 legacy=0x7fb90809bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 >> v1:172.21.15.76:6789/0 conn(0x7fb90809c5f0 legacy=0x7fb908137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 monclient: found mon.noname-c 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908097900 con 0x7fb908138da0 2024-09-16T06:40:26.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 5 monclient: authenticate success, global_id 94591 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb900001990 con 0x7fb908138da0 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: dump: 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.147 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb900001cb0 con 0x7fb908138da0 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 1 -- 172.21.15.189:0/621511656 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb900002040 con 0x7fb908138da0 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 4 set_mon_vals no callback set 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:26.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb9077fe700 10 monclient: dump: 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 10 monclient: shutdown 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- 172.21.15.189:0/621511656 >> v1:172.21.15.189:6789/0 conn(0x7fb908138da0 legacy=0x7fb90813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:26.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- 172.21.15.189:0/621511656 shutdown_connections 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- 172.21.15.189:0/621511656 >> 172.21.15.189:0/621511656 conn(0x7fb9080a03e0 msgr2=0x7fb9080a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- 172.21.15.189:0/621511656 shutdown_connections 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.138+0000 7fb90f08f700 1 -- 172.21.15.189:0/621511656 wait complete. 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: build_initial_monmap 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: monmap: 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:26.150 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:26.151 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 Processor -- start 2024-09-16T06:40:26.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- start start 2024-09-16T06:40:26.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: init 2024-09-16T06:40:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-b con 0x7fb90809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-a con 0x7fb90809c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient(hunting): picked mon.noname-c con 0x7fb908138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb908092a60 con 0x7fb90809c5f0 2024-09-16T06:40:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb908090c00 con 0x7fb90809bb00 2024-09-16T06:40:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90ce2b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb90809bb00 0x7fb90808fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34196/0 (socket says 172.21.15.189:34196) 2024-09-16T06:40:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90809a6e0 con 0x7fb908138da0 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90ce2b700 1 -- 172.21.15.189:0/3845308712 learned_addr learned my addr 172.21.15.189:0/3845308712 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient(hunting): authenticate will time out at 978.658100s 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90d62c700 1 -- 172.21.15.189:0/3845308712 >> v1:172.21.15.76:6790/0 conn(0x7fb908138da0 legacy=0x7fb9081cff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842901112 0 0) 0x7fb908090c00 con 0x7fb90809bb00 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): my global_id is 94594 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f40035d0 con 0x7fb90809bb00 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674652792 0 0) 0x7fb908092a60 con 0x7fb90809c5f0 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:26.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): my global_id is 104385 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb908090c00 con 0x7fb90809c5f0 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1280295492 0 0) 0x7fb8f40035d0 con 0x7fb90809bb00 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb908092a60 con 0x7fb90809bb00 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8f8004230 con 0x7fb90809bb00 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): dump: 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.160 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 690114148 0 0) 0x7fb908090c00 con 0x7fb90809c5f0 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f40035d0 con 0x7fb90809c5f0 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8fc001830 con 0x7fb90809c5f0 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient(hunting): dump: 2024-09-16T06:40:26.161 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 987790991 0 0) 0x7fb908092a60 con 0x7fb90809bb00 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 >> v1:172.21.15.76:6790/0 conn(0x7fb908138da0 legacy=0x7fb9081cff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 >> v1:172.21.15.76:6789/0 conn(0x7fb90809c5f0 legacy=0x7fb9080904f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:26.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 monclient: found mon.b 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908092780 con 0x7fb90809bb00 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb9057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 5 monclient: authenticate success, global_id 94594 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _renew_subs 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9080a2300 con 0x7fb90809bb00 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _renew_subs 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.142+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90809b6f0 con 0x7fb90809bb00 2024-09-16T06:40:26.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8f8004510 con 0x7fb90809bb00 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8f80048a0 con 0x7fb90809bb00 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb906ffd700 4 set_mon_vals no callback set 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 10 monclient: dump: 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:26.164 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb90f08f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb90f08f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb90f08f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8005100 con 0x7fb90809bb00 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb8f8004a80 con 0x7fb90809bb00 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(51..51 src has 1..51) v4 ==== 3931+0+0 (unknown 2832754057 0 0) 0x7fb8f8043390 con 0x7fb90809bb00 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.146+0000 7fb907fff700 10 monclient: get_auth_request con 0x7fb8f40362c0 auth_method 0 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.154+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb8f8014830 con 0x7fb90809bb00 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.154+0000 7fb9057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:26.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.154+0000 7fb9057fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:26.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.326+0000 7fb90f08f700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:26.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.326+0000 7fb90f08f700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:26.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.326+0000 7fb90f08f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:26.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.326+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7fb8e80053b0 con 0x7fb90809bb00 2024-09-16T06:40:26.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.942+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v52) v1 ==== 150+0+0 (unknown 2876392965 0 0) 0x7fb8f8012ca0 con 0x7fb90809bb00 2024-09-16T06:40:26.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.942+0000 7fb9057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:26.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:26.942+0000 7fb9057fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:26.948 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:40:26.948 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:40:26.948 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:40:26.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:40:26.951 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:40:26.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:40:26.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:40:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb90f08f700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb90f08f700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb90f08f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7fb8e8005650 con 0x7fb90809bb00 2024-09-16T06:40:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb9057fa700 1 -- 172.21.15.189:0/3845308712 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v52) v1 ==== 177+0+0 (unknown 1392463029 0 0) 0x7fb8f8014420 con 0x7fb90809bb00 2024-09-16T06:40:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb9057fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T06:40:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.050+0000 7fb9057fa700 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 10 monclient: monmap: 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:27.057945+0000 2024-09-16T06:40:27.056 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:27.057945+0000 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.057 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 1 Processor -- start 2024-09-16T06:40:27.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 1 -- start start 2024-09-16T06:40:27.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.053+0000 7f7604296700 10 monclient: init 2024-09-16T06:40:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb8f40362c0 legacy=0x7fb8f4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 10 monclient: shutdown 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 >> v1:172.21.15.189:6789/0 conn(0x7fb90809bb00 legacy=0x7fb90808fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 shutdown_connections 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 >> 172.21.15.189:0/3845308712 conn(0x7fb9080a03e0 msgr2=0x7fb9080a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 shutdown_connections 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.054+0000 7fb90f08f700 1 -- 172.21.15.189:0/3845308712 wait complete. 2024-09-16T06:40:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-hdd already exists 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient(hunting): picked mon.noname-a con 0x7f75fc135500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient(hunting): picked mon.noname-c con 0x7f75fc09a9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient(hunting): picked mon.noname-b con 0x7f75fc13b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc0a67b0 con 0x7f75fc135500 2024-09-16T06:40:27.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc0a7320 con 0x7f75fc09a9d0 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7602032700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f75fc135500 0x7f75fc1358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33478/0 (socket says 172.21.15.76:33478) 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7602032700 1 -- 172.21.15.76:0/2504740766 learned_addr learned my addr 172.21.15.76:0/2504740766 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7602833700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f75fc13b980 0x7f75fc13dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38270/0 (socket says 172.21.15.76:38270) 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc09ade0 con 0x7f75fc13b980 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient(hunting): authenticate will time out at 974.192730s 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076197517 0 0) 0x7f75fc0a67b0 con 0x7f75fc135500 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): my global_id is 104388 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e80035d0 con 0x7f75fc135500 2024-09-16T06:40:27.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317629569 0 0) 0x7f75fc0a7320 con 0x7f75fc09a9d0 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient(hunting): my global_id is 94597 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc0a67b0 con 0x7f75fc09a9d0 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2642170426 0 0) 0x7f75e80035d0 con 0x7f75fc135500 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 >> v1:172.21.15.76:6790/0 conn(0x7f75fc13b980 legacy=0x7f75fc13dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 >> v1:172.21.15.189:6789/0 conn(0x7f75fc09a9d0 legacy=0x7f75fc13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 monclient: found mon.noname-a 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc099d10 con 0x7f75fc135500 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 5 monclient: authenticate success, global_id 104388 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75f8001610 con 0x7f75fc135500 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: dump: 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75f80018f0 con 0x7f75fc135500 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 1 -- 172.21.15.76:0/2504740766 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75f8001c80 con 0x7f75fc135500 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 4 set_mon_vals no callback set 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7601030700 10 monclient: dump: 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 10 monclient: shutdown 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 >> v1:172.21.15.76:6789/0 conn(0x7f75fc135500 legacy=0x7f75fc1358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 shutdown_connections 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 >> 172.21.15.76:0/2504740766 conn(0x7f75fc0a42f0 msgr2=0x7f75fc0a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 shutdown_connections 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.057+0000 7f7604296700 1 -- 172.21.15.76:0/2504740766 wait complete. 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: monmap: 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 Processor -- start 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- start start 2024-09-16T06:40:27.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: init 2024-09-16T06:40:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T06:40:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-ssd 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient(hunting): picked mon.noname-c con 0x7f75fc09a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient(hunting): picked mon.noname-a con 0x7f75fc135500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient(hunting): picked mon.noname-b con 0x7f75fc13b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc0a3c80 con 0x7f75fc135500 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc09c240 con 0x7f75fc13b980 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc134b00 con 0x7f75fc09a9d0 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient(hunting): authenticate will time out at 974.197930s 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7602032700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f75fc09a9d0 0x7f75fc13a0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38286/0 (socket says 172.21.15.76:38286) 2024-09-16T06:40:27.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7602032700 1 -- 172.21.15.76:0/4144621577 learned_addr learned my addr 172.21.15.76:0/4144621577 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120434692 0 0) 0x7f75fc0a3c80 con 0x7f75fc135500 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): my global_id is 104391 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e00035d0 con 0x7f75fc135500 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105238899 0 0) 0x7f75fc09c240 con 0x7f75fc13b980 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): my global_id is 94600 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc0a3c80 con 0x7f75fc13b980 2024-09-16T06:40:27.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344923883 0 0) 0x7f75e00035d0 con 0x7f75fc135500 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75fc09c240 con 0x7f75fc135500 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75ec0018b0 con 0x7f75fc135500 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.077 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 882249982 0 0) 0x7f75fc09c240 con 0x7f75fc135500 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 >> v1:172.21.15.76:6790/0 conn(0x7f75fc09a9d0 legacy=0x7f75fc13a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 >> v1:172.21.15.189:6789/0 conn(0x7f75fc13b980 legacy=0x7f75fc13aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 monclient: found mon.a 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc06d430 con 0x7f75fc135500 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 5 monclient: authenticate success, global_id 104391 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _renew_subs 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75ec0036c0 con 0x7f75fc135500 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc09d6a0 con 0x7f75fc135500 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75ec0039d0 con 0x7f75fc135500 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f75f2ffd700 10 monclient: dump: 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f760082f700 4 set_mon_vals no callback set 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _renew_subs 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.061+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc1d3590 con 0x7f75fc135500 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.065+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f75ec00e440 con 0x7f75fc135500 2024-09-16T06:40:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.065+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f75ec042020 con 0x7f75fc135500 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.065+0000 7f7602032700 10 monclient: get_auth_request con 0x7f75e0036060 auth_method 0 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.069+0000 7f7604296700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.069+0000 7f7604296700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.069+0000 7f7604296700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.069+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc09d140 con 0x7f75fc135500 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.073+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f75fc09d140 con 0x7f75fc135500 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.073+0000 7f75f2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.073+0000 7f75f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: monmap: 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.169 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:27.170400+0000 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:27.170400+0000 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.170 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 1 Processor -- start 2024-09-16T06:40:27.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 1 -- start start 2024-09-16T06:40:27.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: init 2024-09-16T06:40:27.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient(hunting): picked mon.noname-b con 0x7f334409fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient(hunting): picked mon.noname-a con 0x7f33440717e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient(hunting): picked mon.noname-c con 0x7f33440a07d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344074a10 con 0x7f33440717e0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344098450 con 0x7f33440a07d0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33440986f0 con 0x7f334409fce0 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f334359e700 10 monclient(hunting): authenticate will time out at 979.683423s 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f3342d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f33440a07d0 0x7f334413d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34212/0 (socket says 172.21.15.189:34212) 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.166+0000 7f3342d9d700 1 -- 172.21.15.189:0/930619471 learned_addr learned my addr 172.21.15.189:0/930619471 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850188576 0 0) 0x7f3344098450 con 0x7f33440a07d0 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): my global_id is 94603 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c0035d0 con 0x7f33440a07d0 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554423721 0 0) 0x7f3344074a10 con 0x7f33440717e0 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient(hunting): my global_id is 104394 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3344098450 con 0x7f33440717e0 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868637207 0 0) 0x7f332c0035d0 con 0x7f33440a07d0 2024-09-16T06:40:27.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 >> v1:172.21.15.76:6790/0 conn(0x7f334409fce0 legacy=0x7f3344098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 >> v1:172.21.15.76:6789/0 conn(0x7f33440717e0 legacy=0x7f33440a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 monclient: found mon.noname-c 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440746f0 con 0x7f33440a07d0 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f333c001990 con 0x7f33440a07d0 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: dump: 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.176 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 5 monclient: authenticate success, global_id 94603 2024-09-16T06:40:27.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f333c001c70 con 0x7f33440a07d0 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 1 -- 172.21.15.189:0/930619471 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f333c002000 con 0x7f33440a07d0 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 4 set_mon_vals no callback set 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334159a700 10 monclient: dump: 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 10 monclient: shutdown 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 1 -- 172.21.15.189:0/930619471 >> v1:172.21.15.189:6789/0 conn(0x7f33440a07d0 legacy=0x7f334413d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 1 -- 172.21.15.189:0/930619471 shutdown_connections 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 1 -- 172.21.15.189:0/930619471 >> 172.21.15.189:0/930619471 conn(0x7f33440a0fb0 msgr2=0x7f33440a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 1 -- 172.21.15.189:0/930619471 shutdown_connections 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 1 -- 172.21.15.189:0/930619471 wait complete. 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.170+0000 7f334359e700 10 monclient: monmap: 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 Processor -- start 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 -- start start 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient: init 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient(hunting): picked mon.noname-b con 0x7f33440717e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient(hunting): picked mon.noname-c con 0x7f334409fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient(hunting): picked mon.noname-a con 0x7f33440a07d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334409b4c0 con 0x7f33440a07d0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334409ad30 con 0x7f33440717e0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344129cd0 con 0x7f334409fce0 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f33440717e0 0x7f33441ca4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34216/0 (socket says 172.21.15.189:34216) 2024-09-16T06:40:27.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334259c700 1 -- 172.21.15.189:0/1969830203 learned_addr learned my addr 172.21.15.189:0/1969830203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient(hunting): authenticate will time out at 979.689530s 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723214605 0 0) 0x7f334409ad30 con 0x7f33440717e0 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): my global_id is 94606 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33240035d0 con 0x7f33440717e0 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86445647 0 0) 0x7f334409b4c0 con 0x7f33440a07d0 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): my global_id is 104397 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334409ad30 con 0x7f33440a07d0 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470535368 0 0) 0x7f33240035d0 con 0x7f33440717e0 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f334409b4c0 con 0x7f33440717e0 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3338001910 con 0x7f33440717e0 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696045565 0 0) 0x7f334409ad30 con 0x7f33440a07d0 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33240035d0 con 0x7f33440a07d0 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f333c0017b0 con 0x7f33440a07d0 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1064834800 0 0) 0x7f334409b4c0 con 0x7f33440717e0 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 >> v1:172.21.15.76:6790/0 conn(0x7f334409fce0 legacy=0x7f33441cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 >> v1:172.21.15.76:6789/0 conn(0x7f33440a07d0 legacy=0x7f33441d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 monclient: found mon.b 2024-09-16T06:40:27.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344129430 con 0x7f33440717e0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f3332ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 5 monclient: authenticate success, global_id 94606 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient: _renew_subs 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.174+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334413e5d0 con 0x7f33440717e0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 10 monclient: _renew_subs 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f334413c3c0 con 0x7f33440717e0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3338001ca0 con 0x7f33440717e0 2024-09-16T06:40:27.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3338003b10 con 0x7f33440717e0 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3340d99700 4 set_mon_vals no callback set 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 10 monclient: dump: 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.191 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3328005100 con 0x7f33440717e0 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3338003cf0 con 0x7f33440717e0 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f3338043420 con 0x7f33440717e0 2024-09-16T06:40:27.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.178+0000 7f3341d9b700 10 monclient: get_auth_request con 0x7f33240362a0 auth_method 0 2024-09-16T06:40:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.186+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3338014820 con 0x7f33440717e0 2024-09-16T06:40:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.186+0000 7f3332ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.186+0000 7f3332ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.305+0000 7f7604296700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.305+0000 7f7604296700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.305+0000 7f7604296700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.305+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f75fc09d140 con 0x7f75fc135500 2024-09-16T06:40:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f75f2ffd700 1 -- 172.21.15.76:0/4144621577 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 407106245) 0x7f75fc09d140 con 0x7f75fc135500 2024-09-16T06:40:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f75f2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f75f2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.312 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 >> v1:172.21.15.189:6812/3098810674 conn(0x7f75e0036060 legacy=0x7f75e0038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 10 monclient: shutdown 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 >> v1:172.21.15.76:6789/0 conn(0x7f75fc135500 legacy=0x7f75fc13a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 shutdown_connections 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 >> 172.21.15.76:0/4144621577 conn(0x7f75fc0a42f0 msgr2=0x7f75fc13dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 shutdown_connections 2024-09-16T06:40:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.309+0000 7f7604296700 1 -- 172.21.15.76:0/4144621577 wait complete. 2024-09-16T06:40:27.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:40:27.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:40:27.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:40:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f334359e700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f334359e700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f334359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3328005930 con 0x7f33440717e0 2024-09-16T06:40:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) v1 ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7f33380145f0 con 0x7f33440717e0 2024-09-16T06:40:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f3332ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.358+0000 7f3332ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: monmap: 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:27.437322+0000 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:27.437322+0000 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.436 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.437 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.437 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 1 Processor -- start 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 1 -- start start 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: init 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c098990 con 0x7f3f6c139500 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c097f40 con 0x7f3f6c097b60 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c09d610 con 0x7f3f6c099090 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 10 monclient(hunting): authenticate will time out at 974.570679s 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f7158f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3f6c099090 0x7f3f6c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38302/0 (socket says 172.21.15.76:38302) 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f7158f700 1 -- 172.21.15.76:0/2274221367 learned_addr learned my addr 172.21.15.76:0/2274221367 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424618005 0 0) 0x7f3f6c09d610 con 0x7f3f6c099090 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): my global_id is 124103 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f580035d0 con 0x7f3f6c099090 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026802221 0 0) 0x7f3f6c098990 con 0x7f3f6c139500 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): my global_id is 104400 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c09d610 con 0x7f3f6c139500 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392547156 0 0) 0x7f3f6c097f40 con 0x7f3f6c097b60 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient(hunting): my global_id is 94609 2024-09-16T06:40:27.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c098990 con 0x7f3f6c097b60 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 386210446 0 0) 0x7f3f6c09d610 con 0x7f3f6c139500 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 >> v1:172.21.15.76:6790/0 conn(0x7f3f6c099090 legacy=0x7f3f6c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 >> v1:172.21.15.189:6789/0 conn(0x7f3f6c097b60 legacy=0x7f3f6c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 monclient: found mon.noname-a 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0986b0 con 0x7f3f6c139500 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f64001970 con 0x7f3f6c139500 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f70d8e700 10 monclient: dump: 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.445 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 5 monclient: authenticate success, global_id 104400 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.433+0000 7f3f72d92700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f64001c10 con 0x7f3f6c139500 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 1 -- 172.21.15.76:0/2274221367 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f64001fa0 con 0x7f3f6c139500 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 4 set_mon_vals no callback set 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f70d8e700 10 monclient: dump: 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: shutdown 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/2274221367 >> v1:172.21.15.76:6789/0 conn(0x7f3f6c139500 legacy=0x7f3f6c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/2274221367 shutdown_connections 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/2274221367 >> 172.21.15.76:0/2274221367 conn(0x7f3f6c0a0030 msgr2=0x7f3f6c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/2274221367 shutdown_connections 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/2274221367 wait complete. 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: monmap: 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.448 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 Processor -- start 2024-09-16T06:40:27.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- start start 2024-09-16T06:40:27.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: init 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c070f30 con 0x7f3f6c099090 2024-09-16T06:40:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f71d90700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3f6c097b60 0x7f3f6c06f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38316/0 (socket says 172.21.15.76:38316) 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f71d90700 1 -- 172.21.15.76:0/378793353 learned_addr learned my addr 172.21.15.76:0/378793353 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c1330d0 con 0x7f3f6c139500 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c133d20 con 0x7f3f6c097b60 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.437+0000 7f3f72d92700 10 monclient(hunting): authenticate will time out at 974.575839s 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832413624 0 0) 0x7f3f6c133d20 con 0x7f3f6c097b60 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): my global_id is 124106 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f500035d0 con 0x7f3f6c097b60 2024-09-16T06:40:27.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283285088 0 0) 0x7f3f6c070f30 con 0x7f3f6c099090 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): my global_id is 104403 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c133d20 con 0x7f3f6c099090 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705270571 0 0) 0x7f3f6c1330d0 con 0x7f3f6c139500 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): my global_id is 94612 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c070f30 con 0x7f3f6c139500 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660398615 0 0) 0x7f3f500035d0 con 0x7f3f6c097b60 2024-09-16T06:40:27.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c1330d0 con 0x7f3f6c097b60 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f5c001990 con 0x7f3f6c097b60 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): dump: 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.456 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545711325 0 0) 0x7f3f6c133d20 con 0x7f3f6c099090 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f500035d0 con 0x7f3f6c099090 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f680010e0 con 0x7f3f6c099090 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): dump: 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.457 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215306141 0 0) 0x7f3f6c070f30 con 0x7f3f6c139500 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c133d20 con 0x7f3f6c139500 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f64001f20 con 0x7f3f6c139500 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): dump: 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.458 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3813876191 0 0) 0x7f3f6c1330d0 con 0x7f3f6c097b60 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.189:6789/0 conn(0x7f3f6c139500 legacy=0x7f3f6c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6789/0 conn(0x7f3f6c099090 legacy=0x7f3f6c06f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 monclient: found mon.c 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c070010 con 0x7f3f6c097b60 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 5 monclient: authenticate success, global_id 124106 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 10 monclient: _renew_subs 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c133900 con 0x7f3f6c097b60 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 10 monclient: _renew_subs 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c138f30 con 0x7f3f6c097b60 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f5c001d40 con 0x7f3f6c097b60 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f5c003af0 con 0x7f3f6c097b60 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f637fe700 4 set_mon_vals no callback set 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: dump: 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.461 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3f54006590 con 0x7f3f54002830 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3f5c003cd0 con 0x7f3f6c097b60 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f3f5c0433b0 con 0x7f3f6c097b60 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150659755 0 0) 0x7f3f54006590 con 0x7f3f54002830 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient(hunting): my global_id is 104406 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c1330d0 con 0x7f3f54002830 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f72591700 10 monclient: get_auth_request con 0x7f3f500365c0 auth_method 0 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1012155297 0 0) 0x7f3f6c1330d0 con 0x7f3f54002830 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f54006590 con 0x7f3f54002830 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f68004890 con 0x7f3f54002830 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 10 monclient: dump: 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.463 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2292408876 0 0) 0x7f3f54006590 con 0x7f3f54002830 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.441+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f5400a190 con 0x7f3f54002830 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.445+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3f68006b80 con 0x7f3f54002830 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.445+0000 7f3f627fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.445+0000 7f3f627fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.445+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6789/0 conn(0x7f3f54002830 legacy=0x7f3f54004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.445+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6789/0 conn(0x7f3f54002830 legacy=0x7f3f54004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3328005bc0 con 0x7f33440717e0 2024-09-16T06:40:27.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f3332ffd700 1 -- 172.21.15.189:0/1969830203 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) v1 ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7f33380145f0 con 0x7f33440717e0 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f3332ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f3332ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 >> v1:172.21.15.189:6812/3098810674 conn(0x7f33240362a0 legacy=0x7f3324038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 10 monclient: shutdown 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 >> v1:172.21.15.189:6789/0 conn(0x7f33440717e0 legacy=0x7f33441ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 shutdown_connections 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 >> 172.21.15.189:0/1969830203 conn(0x7f33440a0fb0 msgr2=0x7f334409e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 shutdown_connections 2024-09-16T06:40:27.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.458+0000 7f334359e700 1 -- 172.21.15.189:0/1969830203 wait complete. 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3f54009e30 con 0x7f3f54009150 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408767093 0 0) 0x7f3f54009e30 con 0x7f3f54009150 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient(hunting): my global_id is 104409 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f54006590 con 0x7f3f54009150 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4251569885 0 0) 0x7f3f54006590 con 0x7f3f54009150 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f54009e30 con 0x7f3f54009150 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f68004fd0 con 0x7f3f54009150 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient: dump: 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.469 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2546044058 0 0) 0x7f3f54009e30 con 0x7f3f54009150 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f5400c340 con 0x7f3f54009150 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3989135769) 0x7f3f680063b0 con 0x7f3f54009150 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6789/0 conn(0x7f3f54009150 legacy=0x7f3f540095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f627fc700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6789/0 conn(0x7f3f54009150 legacy=0x7f3f540095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:27.471 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.472 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.473 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:27.474 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.475 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.476 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.477 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.478 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.479 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.481 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.482 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:27.483 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:27.484 INFO:tasks.workunit.client.0.smithi189.stdout:foo-ssd 2024-09-16T06:40:27.485 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T06:40:27.485 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-hdd 2024-09-16T06:40:27.485 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:27.485 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-hdd 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3f500365c0 legacy=0x7f3f50038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 10 monclient: shutdown 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 >> v1:172.21.15.76:6790/0 conn(0x7f3f6c097b60 legacy=0x7f3f6c06f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72591700 1 -- 172.21.15.76:0/378793353 reap_dead start 2024-09-16T06:40:27.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72591700 1 -- 172.21.15.76:0/378793353 reap_dead start 2024-09-16T06:40:27.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 shutdown_connections 2024-09-16T06:40:27.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 >> 172.21.15.76:0/378793353 conn(0x7f3f6c0a0030 msgr2=0x7f3f6c12f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 shutdown_connections 2024-09-16T06:40:27.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.453+0000 7f3f72d92700 1 -- 172.21.15.76:0/378793353 wait complete. 2024-09-16T06:40:27.486 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:27.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:40:27.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:40:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: monmap: 2024-09-16T06:40:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:27.581343+0000 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:27.581343+0000 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.581 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 1 Processor -- start 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 1 -- start start 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: init 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-b con 0x7f3fe0098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-a con 0x7f3fe0099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-c con 0x7f3fe0138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe009c460 con 0x7f3fe0099a60 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe009c1e0 con 0x7f3fe0138a80 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe009bd60 con 0x7f3fe0098070 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient(hunting): authenticate will time out at 980.094366s 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fded9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3fe0138a80 0x7f3fe013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34218/0 (socket says 172.21.15.189:34218) 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fded9d700 1 -- 172.21.15.189:0/3781166353 learned_addr learned my addr 172.21.15.189:0/3781166353 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834766468 0 0) 0x7f3fe009c1e0 con 0x7f3fe0138a80 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient(hunting): my global_id is 94615 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc80057e0 con 0x7f3fe0138a80 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046227084 0 0) 0x7f3fc80057e0 con 0x7f3fe0138a80 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 >> v1:172.21.15.76:6790/0 conn(0x7f3fe0098070 legacy=0x7f3fe009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 >> v1:172.21.15.76:6789/0 conn(0x7f3fe0099a60 legacy=0x7f3fe0099e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 monclient: found mon.noname-c 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe009d490 con 0x7f3fe0138a80 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fd8001990 con 0x7f3fe0138a80 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 5 monclient: authenticate success, global_id 94615 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: dump: 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.586 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd8001c40 con 0x7f3fe0138a80 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 1 -- 172.21.15.189:0/3781166353 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fd8001f90 con 0x7f3fe0138a80 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fdd59a700 10 monclient: dump: 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.587 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.588 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.588 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.578+0000 7f3fe4b4b700 4 set_mon_vals no callback set 2024-09-16T06:40:27.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: shutdown 2024-09-16T06:40:27.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/3781166353 >> v1:172.21.15.189:6789/0 conn(0x7f3fe0138a80 legacy=0x7f3fe013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/3781166353 shutdown_connections 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/3781166353 >> 172.21.15.189:0/3781166353 conn(0x7f3fe00a03e0 msgr2=0x7f3fe00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/3781166353 shutdown_connections 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/3781166353 wait complete. 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: monmap: 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:27.594 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 Processor -- start 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- start start 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: init 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-c con 0x7f3fe0098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-b con 0x7f3fe0099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient(hunting): picked mon.noname-a con 0x7f3fe0138a80 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe0090010 con 0x7f3fe0138a80 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe009a6e0 con 0x7f3fe0099a60 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fddd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3fe0099a60 0x7f3fe01caf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34230/0 (socket says 172.21.15.189:34230) 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fddd9b700 1 -- 172.21.15.189:0/305686766 learned_addr learned my addr 172.21.15.189:0/305686766 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00a2ef0 con 0x7f3fe0098070 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.582+0000 7f3fe4b4b700 10 monclient(hunting): authenticate will time out at 980.099719s 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177639130 0 0) 0x7f3fe009a6e0 con 0x7f3fe0099a60 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): my global_id is 94618 2024-09-16T06:40:27.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc00035d0 con 0x7f3fe0099a60 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071887818 0 0) 0x7f3fe00a2ef0 con 0x7f3fe0098070 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): my global_id is 124112 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe009a6e0 con 0x7f3fe0098070 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955605876 0 0) 0x7f3fe0090010 con 0x7f3fe0138a80 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): my global_id is 104415 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe00a2ef0 con 0x7f3fe0138a80 2024-09-16T06:40:27.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395962517 0 0) 0x7f3fc00035d0 con 0x7f3fe0099a60 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe0090010 con 0x7f3fe0099a60 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fd00018b0 con 0x7f3fe0099a60 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887536866 0 0) 0x7f3fe009a6e0 con 0x7f3fe0098070 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fc00035d0 con 0x7f3fe0098070 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fd40018b0 con 0x7f3fe0098070 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.599 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557101622 0 0) 0x7f3fe0090010 con 0x7f3fe0099a60 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 >> v1:172.21.15.76:6790/0 conn(0x7f3fe0098070 legacy=0x7f3fe01c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 >> v1:172.21.15.76:6789/0 conn(0x7f3fe0138a80 legacy=0x7f3fe01ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 monclient: found mon.b 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe0090b70 con 0x7f3fe0099a60 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 5 monclient: authenticate success, global_id 94618 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _renew_subs 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe008e810 con 0x7f3fe0099a60 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _renew_subs 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe008dbd0 con 0x7f3fe0099a60 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd0001c60 con 0x7f3fe0099a60 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fd0003b10 con 0x7f3fe0099a60 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 10 monclient: dump: 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fdcd99700 4 set_mon_vals no callback set 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3fd0003c70 con 0x7f3fe0099a60 2024-09-16T06:40:27.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f3fd0043240 con 0x7f3fe0099a60 2024-09-16T06:40:27.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fde59c700 10 monclient: get_auth_request con 0x7f3fc0036230 auth_method 0 2024-09-16T06:40:27.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.586+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc4005100 con 0x7f3fe0099a60 2024-09-16T06:40:27.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.594+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3fd0004490 con 0x7f3fe0099a60 2024-09-16T06:40:27.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.594+0000 7f3fceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.594+0000 7f3fceffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: monmap: 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:27.602023+0000 2024-09-16T06:40:27.613 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:27.602023+0000 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.614 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 1 Processor -- start 2024-09-16T06:40:27.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 1 -- start start 2024-09-16T06:40:27.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: init 2024-09-16T06:40:27.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-a con 0x7f8b5009f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-b con 0x7f8b5009d520 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-c con 0x7f8b5009fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5009b520 con 0x7f8b5009f230 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b50071f30 con 0x7f8b5009fd20 2024-09-16T06:40:27.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5009b0e0 con 0x7f8b5009d520 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b56c55700 10 monclient(hunting): authenticate will time out at 974.735507s 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b549f1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8b5009f230 0x7f8b5009f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33528/0 (socket says 172.21.15.76:33528) 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.597+0000 7f8b549f1700 1 -- 172.21.15.76:0/1027942661 learned_addr learned my addr 172.21.15.76:0/1027942661 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370895885 0 0) 0x7f8b50071f30 con 0x7f8b5009fd20 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): my global_id is 94621 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b380035d0 con 0x7f8b5009fd20 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995859788 0 0) 0x7f8b5009b0e0 con 0x7f8b5009d520 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): my global_id is 124115 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b50071f30 con 0x7f8b5009d520 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626322063 0 0) 0x7f8b5009b520 con 0x7f8b5009f230 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient(hunting): my global_id is 104418 2024-09-16T06:40:27.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5009b0e0 con 0x7f8b5009f230 2024-09-16T06:40:27.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297985041 0 0) 0x7f8b5009b0e0 con 0x7f8b5009f230 2024-09-16T06:40:27.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 >> v1:172.21.15.76:6790/0 conn(0x7f8b5009d520 legacy=0x7f8b50137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 >> v1:172.21.15.189:6789/0 conn(0x7f8b5009fd20 legacy=0x7f8b5013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5009c7d0 con 0x7f8b5009f230 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 5 monclient: authenticate success, global_id 104418 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b44001530 con 0x7f8b5009f230 2024-09-16T06:40:27.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: dump: 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b440017a0 con 0x7f8b5009f230 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 1 -- 172.21.15.76:0/1027942661 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b44001af0 con 0x7f8b5009f230 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 4 set_mon_vals no callback set 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b4f7fe700 10 monclient: dump: 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.623 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 10 monclient: shutdown 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 1 -- 172.21.15.76:0/1027942661 >> v1:172.21.15.76:6789/0 conn(0x7f8b5009f230 legacy=0x7f8b5009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 1 -- 172.21.15.76:0/1027942661 shutdown_connections 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 1 -- 172.21.15.76:0/1027942661 >> 172.21.15.76:0/1027942661 conn(0x7f8b500a3e50 msgr2=0x7f8b500a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 1 -- 172.21.15.76:0/1027942661 shutdown_connections 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 1 -- 172.21.15.76:0/1027942661 wait complete. 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.601+0000 7f8b56c55700 10 monclient: monmap: 2024-09-16T06:40:27.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 Processor -- start 2024-09-16T06:40:27.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- start start 2024-09-16T06:40:27.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: init 2024-09-16T06:40:27.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-c con 0x7f8b5009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-b con 0x7f8b5009fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient(hunting): picked mon.noname-a con 0x7f8b50130be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5011e4a0 con 0x7f8b50130be0 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5011e720 con 0x7f8b5009fd20 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b500a28e0 con 0x7f8b5009f230 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient(hunting): authenticate will time out at 974.741504s 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b551f2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8b50130be0 0x7f8b501ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33536/0 (socket says 172.21.15.76:33536) 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b551f2700 1 -- 172.21.15.76:0/111055340 learned_addr learned my addr 172.21.15.76:0/111055340 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190229301 0 0) 0x7f8b500a28e0 con 0x7f8b5009f230 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): my global_id is 124118 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c0035d0 con 0x7f8b5009f230 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272627584 0 0) 0x7f8b5011e720 con 0x7f8b5009fd20 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): my global_id is 94624 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b500a28e0 con 0x7f8b5009fd20 2024-09-16T06:40:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691419016 0 0) 0x7f8b5011e4a0 con 0x7f8b50130be0 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): my global_id is 104421 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5011e720 con 0x7f8b50130be0 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107176635 0 0) 0x7f8b3c0035d0 con 0x7f8b5009f230 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b5011e4a0 con 0x7f8b5009f230 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b440014f0 con 0x7f8b5009f230 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160125420 0 0) 0x7f8b500a28e0 con 0x7f8b5009fd20 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b3c0035d0 con 0x7f8b5009fd20 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1058074795 0 0) 0x7f8b5011e720 con 0x7f8b50130be0 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b500a28e0 con 0x7f8b50130be0 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b400017b0 con 0x7f8b5009fd20 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.632 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b48001d70 con 0x7f8b50130be0 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.633 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4207278083 0 0) 0x7f8b5011e4a0 con 0x7f8b5009f230 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.189:6789/0 conn(0x7f8b5009fd20 legacy=0x7f8b501ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6789/0 conn(0x7f8b50130be0 legacy=0x7f8b501ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 monclient: found mon.c 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5011fbd0 con 0x7f8b5009f230 2024-09-16T06:40:27.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 5 monclient: authenticate success, global_id 124118 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _renew_subs 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5011db50 con 0x7f8b5009f230 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _renew_subs 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b501c5ab0 con 0x7f8b5009f230 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b44004b40 con 0x7f8b5009f230 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b44004e50 con 0x7f8b5009f230 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4effd700 4 set_mon_vals no callback set 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.605+0000 7f8b4d7fa700 10 monclient: dump: 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8b44004fb0 con 0x7f8b5009f230 2024-09-16T06:40:27.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f8b44043420 con 0x7f8b5009f230 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4ffff700 10 monclient: get_auth_request con 0x7f8b3c0366a0 auth_method 0 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b56c55700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8b30006590 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858370824 0 0) 0x7f8b30006590 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient(hunting): my global_id is 124121 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5011e4a0 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2271459285 0 0) 0x7f8b5011e4a0 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b30006590 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b48003bc0 con 0x7f8b30002830 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient: dump: 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 560146914 0 0) 0x7f8b30006590 con 0x7f8b30002830 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b3000a190 con 0x7f8b30002830 2024-09-16T06:40:27.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8b48005bb0 con 0x7f8b30002830 2024-09-16T06:40:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6790/0 conn(0x7f8b30002830 legacy=0x7f8b30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.609+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6790/0 conn(0x7f8b30002830 legacy=0x7f8b30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.617+0000 7f8b56c55700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.617+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8b30009e10 con 0x7f8b30009150 2024-09-16T06:40:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739564422 0 0) 0x7f8b30009e10 con 0x7f8b30009150 2024-09-16T06:40:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient(hunting): my global_id is 124124 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b30006590 con 0x7f8b30009150 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2773465830 0 0) 0x7f8b30006590 con 0x7f8b30009150 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b30009e10 con 0x7f8b30009150 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b48004310 con 0x7f8b30009150 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient: dump: 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2785092413 0 0) 0x7f8b30009e10 con 0x7f8b30009150 2024-09-16T06:40:27.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8b3000c340 con 0x7f8b30009150 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 604409129) 0x7f8b48004690 con 0x7f8b30009150 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6790/0 conn(0x7f8b30009150 legacy=0x7f8b300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b4d7fa700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6790/0 conn(0x7f8b30009150 legacy=0x7f8b300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:27.643 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:27.644 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.645 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.646 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:27.647 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.648 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.649 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:27.650 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:27.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:27.655 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:27.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8b3c0366a0 legacy=0x7f8b3c038b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 10 monclient: shutdown 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 >> v1:172.21.15.76:6790/0 conn(0x7f8b5009f230 legacy=0x7f8b501304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b551f2700 1 -- 172.21.15.76:0/111055340 reap_dead start 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 shutdown_connections 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 >> 172.21.15.76:0/111055340 conn(0x7f8b500a3e50 msgr2=0x7f8b501323d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 shutdown_connections 2024-09-16T06:40:27.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.621+0000 7f8b56c55700 1 -- 172.21.15.76:0/111055340 wait complete. 2024-09-16T06:40:27.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:27.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:40:27.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:40:27.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fe4b4b700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fe4b4b700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fe4b4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3fc4005930 con 0x7f3fe0099a60 2024-09-16T06:40:27.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) v1 ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7f3fd0012bf0 con 0x7f3fe0099a60 2024-09-16T06:40:27.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.762+0000 7f3fceffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: monmap: 2024-09-16T06:40:27.776 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.776 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:27.777999+0000 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:27.777999+0000 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 1 Processor -- start 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 1 -- start start 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: init 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:27.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-c con 0x7f4d5409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-b con 0x7f4d54098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-a con 0x7f4d54099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5409d560 con 0x7f4d54099030 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5409f140 con 0x7f4d5409ed60 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5409d790 con 0x7f4d54098be0 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.773+0000 7f4d594b8700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient(hunting): authenticate will time out at 974.912050s 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d537fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d54098be0 0x7f4d541374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38348/0 (socket says 172.21.15.76:38348) 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d537fe700 1 -- 172.21.15.76:0/123606513 learned_addr learned my addr 172.21.15.76:0/123606513 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d58cb7700 1 --1- 172.21.15.76:0/123606513 >> v1:172.21.15.76:6789/0 conn(0x7f4d54099030 0x7f4d5413ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33550/0 (socket says 172.21.15.76:33550) 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822627040 0 0) 0x7f4d5409d560 con 0x7f4d54099030 2024-09-16T06:40:27.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): my global_id is 104424 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d3c0035d0 con 0x7f4d54099030 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116909683 0 0) 0x7f4d5409d790 con 0x7f4d54098be0 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): my global_id is 124127 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d5409d560 con 0x7f4d54098be0 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759873798 0 0) 0x7f4d5409f140 con 0x7f4d5409ed60 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient(hunting): my global_id is 94627 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d5409d790 con 0x7f4d5409ed60 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365865157 0 0) 0x7f4d3c0035d0 con 0x7f4d54099030 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 >> v1:172.21.15.76:6790/0 conn(0x7f4d54098be0 legacy=0x7f4d541374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 >> v1:172.21.15.189:6789/0 conn(0x7f4d5409ed60 legacy=0x7f4d540984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 monclient: found mon.noname-a 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5409f620 con 0x7f4d54099030 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d4c001990 con 0x7f4d54099030 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 5 monclient: authenticate success, global_id 104424 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: dump: 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d4c001c70 con 0x7f4d54099030 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 4 set_mon_vals no callback set 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 1 -- 172.21.15.76:0/123606513 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d4c002000 con 0x7f4d54099030 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d52ffd700 10 monclient: dump: 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient: shutdown 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- 172.21.15.76:0/123606513 >> v1:172.21.15.76:6789/0 conn(0x7f4d54099030 legacy=0x7f4d5413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- 172.21.15.76:0/123606513 shutdown_connections 2024-09-16T06:40:27.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- 172.21.15.76:0/123606513 >> 172.21.15.76:0/123606513 conn(0x7f4d540a0030 msgr2=0x7f4d540a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- 172.21.15.76:0/123606513 shutdown_connections 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- 172.21.15.76:0/123606513 wait complete. 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient: monmap: 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 Processor -- start 2024-09-16T06:40:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 1 -- start start 2024-09-16T06:40:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.777+0000 7f4d594b8700 10 monclient: init 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-a con 0x7f4d54098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-b con 0x7f4d54099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient(hunting): picked mon.noname-c con 0x7f4d5409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d54090fe0 con 0x7f4d54098be0 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d540987b0 con 0x7f4d54099030 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d540a13e0 con 0x7f4d5409ed60 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient(hunting): authenticate will time out at 974.916619s 2024-09-16T06:40:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d58cb7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d5409ed60 0x7f4d541ce140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38362/0 (socket says 172.21.15.76:38362) 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d53fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4d54098be0 0x7f4d54096550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33562/0 (socket says 172.21.15.76:33562) 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d58cb7700 1 -- 172.21.15.76:0/2845768547 learned_addr learned my addr 172.21.15.76:0/2845768547 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479663894 0 0) 0x7f4d54090fe0 con 0x7f4d54098be0 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): my global_id is 104427 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d400035d0 con 0x7f4d54098be0 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645943611 0 0) 0x7f4d540a13e0 con 0x7f4d5409ed60 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): my global_id is 124130 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d54090fe0 con 0x7f4d5409ed60 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376962205 0 0) 0x7f4d540987b0 con 0x7f4d54099030 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): my global_id is 94630 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d540a13e0 con 0x7f4d54099030 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504993134 0 0) 0x7f4d54090fe0 con 0x7f4d5409ed60 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d540987b0 con 0x7f4d5409ed60 2024-09-16T06:40:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d4c001950 con 0x7f4d5409ed60 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200513417 0 0) 0x7f4d400035d0 con 0x7f4d54098be0 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d54090fe0 con 0x7f4d54098be0 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d480017d0 con 0x7f4d54098be0 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.800 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471867794 0 0) 0x7f4d540987b0 con 0x7f4d5409ed60 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6789/0 conn(0x7f4d54099030 legacy=0x7f4d541caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.76:6789/0 conn(0x7f4d54098be0 legacy=0x7f4d54096550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 monclient: found mon.c 2024-09-16T06:40:27.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5409a540 con 0x7f4d5409ed60 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 5 monclient: authenticate success, global_id 124130 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _renew_subs 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d54090d00 con 0x7f4d5409ed60 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _renew_subs 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5408ec80 con 0x7f4d5409ed60 2024-09-16T06:40:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d540a1230 con 0x7f4d54096c60 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d4c004f60 con 0x7f4d5409ed60 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d4c0052f0 con 0x7f4d5409ed60 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d50ff9700 10 monclient: dump: 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.781+0000 7f4d527fc700 4 set_mon_vals no callback set 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604371611 0 0) 0x7f4d540a1230 con 0x7f4d54096c60 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient(hunting): my global_id is 94633 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d540987b0 con 0x7f4d54096c60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4d4c005510 con 0x7f4d5409ed60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f4d4c0438d0 con 0x7f4d5409ed60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 45612007 0 0) 0x7f4d540987b0 con 0x7f4d54096c60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d540a1230 con 0x7f4d54096c60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d480036a0 con 0x7f4d54096c60 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient: dump: 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.805 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d537fe700 10 monclient: get_auth_request con 0x7f4d40036400 auth_method 0 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 443173810 0 0) 0x7f4d540a1230 con 0x7f4d54096c60 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d54090b90 con 0x7f4d54096c60 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4d48005870 con 0x7f4d54096c60 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6789/0 conn(0x7f4d54096c60 legacy=0x7f4d5406f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.785+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6789/0 conn(0x7f4d54096c60 legacy=0x7f4d5406f300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d540672a0 con 0x7f4d541c5690 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874363313 0 0) 0x7f4d540672a0 con 0x7f4d541c5690 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient(hunting): my global_id is 94636 2024-09-16T06:40:27.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d540a1230 con 0x7f4d541c5690 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1814332294 0 0) 0x7f4d540a1230 con 0x7f4d541c5690 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d540672a0 con 0x7f4d541c5690 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d48003e10 con 0x7f4d541c5690 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient: dump: 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1359504681 0 0) 0x7f4d540672a0 con 0x7f4d541c5690 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d5409aaa0 con 0x7f4d541c5690 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1774574594) 0x7f4d48004180 con 0x7f4d541c5690 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6789/0 conn(0x7f4d541c5690 legacy=0x7f4d5402c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d50ff9700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6789/0 conn(0x7f4d541c5690 legacy=0x7f4d5402c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.810 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.811 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.812 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:27.814 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:27.815 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:27.816 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.820 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.821 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:27.822 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:27.823 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:27.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4d40036400 legacy=0x7f4d400388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 10 monclient: shutdown 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d58cb7700 1 -- 172.21.15.76:0/2845768547 reap_dead start 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 >> v1:172.21.15.76:6790/0 conn(0x7f4d5409ed60 legacy=0x7f4d541ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 shutdown_connections 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 >> 172.21.15.76:0/2845768547 conn(0x7f4d540a0030 msgr2=0x7f4d540972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 shutdown_connections 2024-09-16T06:40:27.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.797+0000 7f4d594b8700 1 -- 172.21.15.76:0/2845768547 wait complete. 2024-09-16T06:40:27.825 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:27.826 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:40:27.826 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:40:27.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3fc4005bc0 con 0x7f3fe0099a60 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fceffd700 1 -- 172.21.15.189:0/305686766 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) v1 ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7f3fd0014370 con 0x7f3fe0099a60 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fceffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T06:40:27.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fceffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:27.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3fc0036230 legacy=0x7f3fc00386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 10 monclient: shutdown 2024-09-16T06:40:27.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.862+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 >> v1:172.21.15.189:6789/0 conn(0x7f3fe0099a60 legacy=0x7f3fe01caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.866+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 shutdown_connections 2024-09-16T06:40:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.866+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 >> 172.21.15.189:0/305686766 conn(0x7f3fe00a03e0 msgr2=0x7f3fe009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.866+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 shutdown_connections 2024-09-16T06:40:27.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.866+0000 7f3fe4b4b700 1 -- 172.21.15.189:0/305686766 wait complete. 2024-09-16T06:40:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:27.877 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T06:40:27.878 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-hdd 2024-09-16T06:40:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: monmap: 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:27.944213+0000 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:27.944213+0000 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.944 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 Processor -- start 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 -- start start 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: init 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fa2e8097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fa2e8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fa2e8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8098990 con 0x7fa2e8139500 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8097f40 con 0x7fa2e8097b60 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e809d610 con 0x7fa2e8099090 2024-09-16T06:40:27.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient(hunting): authenticate will time out at 975.077509s 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e6d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa2e8139500 0x7fa2e813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33568/0 (socket says 172.21.15.76:33568) 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e6d9d700 1 -- 172.21.15.76:0/4149484577 learned_addr learned my addr 172.21.15.76:0/4149484577 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038578713 0 0) 0x7fa2e8098990 con 0x7fa2e8139500 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): my global_id is 104430 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d40035d0 con 0x7fa2e8139500 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927541949 0 0) 0x7fa2e809d610 con 0x7fa2e8099090 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): my global_id is 124133 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e8098990 con 0x7fa2e8099090 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283189476 0 0) 0x7fa2e8097f40 con 0x7fa2e8097b60 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient(hunting): my global_id is 94639 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e809d610 con 0x7fa2e8097b60 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 957740186 0 0) 0x7fa2d40035d0 con 0x7fa2e8139500 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 >> v1:172.21.15.76:6790/0 conn(0x7fa2e8099090 legacy=0x7fa2e809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 >> v1:172.21.15.189:6789/0 conn(0x7fa2e8097b60 legacy=0x7fa2e809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 monclient: found mon.noname-a 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80986b0 con 0x7fa2e8139500 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 5 monclient: authenticate success, global_id 104430 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2e0001990 con 0x7fa2e8139500 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: dump: 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.953 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e0001c70 con 0x7fa2e8139500 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 1 -- 172.21.15.76:0/4149484577 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2e0002000 con 0x7fa2e8139500 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 4 set_mon_vals no callback set 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e5d9b700 10 monclient: dump: 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 10 monclient: shutdown 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.941+0000 7fa2e759e700 1 -- 172.21.15.76:0/4149484577 >> v1:172.21.15.76:6789/0 conn(0x7fa2e8139500 legacy=0x7fa2e813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- 172.21.15.76:0/4149484577 shutdown_connections 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- 172.21.15.76:0/4149484577 >> 172.21.15.76:0/4149484577 conn(0x7fa2e80a0030 msgr2=0x7fa2e80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- 172.21.15.76:0/4149484577 shutdown_connections 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- 172.21.15.76:0/4149484577 wait complete. 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient: monmap: 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 Processor -- start 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- start start 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient: init 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fa2e8097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fa2e8099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fa2e8139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e813c230 con 0x7fa2e8099090 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80a2c90 con 0x7fa2e8139500 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8091750 con 0x7fa2e8097b60 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e759e700 10 monclient(hunting): authenticate will time out at 975.081843s 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa2e8097b60 0x7fa2e8138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38380/0 (socket says 172.21.15.76:38380) 2024-09-16T06:40:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.945+0000 7fa2e659c700 1 -- 172.21.15.76:0/1508743819 learned_addr learned my addr 172.21.15.76:0/1508743819 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787408263 0 0) 0x7fa2e8098990 con 0x7fa2e8099090 2024-09-16T06:40:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): my global_id is 104433 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c80035d0 con 0x7fa2e8099090 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434921797 0 0) 0x7fa2e8091750 con 0x7fa2e8097b60 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): my global_id is 124136 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e8098990 con 0x7fa2e8097b60 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111493819 0 0) 0x7fa2e80a2c90 con 0x7fa2e8139500 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): my global_id is 94642 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e8091750 con 0x7fa2e8139500 2024-09-16T06:40:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2576698488 0 0) 0x7fa2e8098990 con 0x7fa2e8097b60 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e80a2c90 con 0x7fa2e8097b60 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2d8001970 con 0x7fa2e8097b60 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.963 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87262849 0 0) 0x7fa2c80035d0 con 0x7fa2e8099090 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e8098990 con 0x7fa2e8099090 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2d0001690 con 0x7fa2e8099090 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3954791919 0 0) 0x7fa2e8098990 con 0x7fa2e8099090 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 >> v1:172.21.15.76:6790/0 conn(0x7fa2e8097b60 legacy=0x7fa2e8138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 >> v1:172.21.15.189:6789/0 conn(0x7fa2e8139500 legacy=0x7fa2e81d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 monclient: found mon.a 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8096f10 con 0x7fa2e8099090 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 5 monclient: authenticate success, global_id 104433 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 10 monclient: _renew_subs 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e809a540 con 0x7fa2e8099090 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 10 monclient: _renew_subs 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8091b70 con 0x7fa2e8099090 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d0001b60 con 0x7fa2e8099090 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2d00036c0 con 0x7fa2e8099090 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2e559a700 4 set_mon_vals no callback set 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.949+0000 7fa2df7fe700 10 monclient: dump: 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.967 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa2d000e130 con 0x7fa2e8099090 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2e759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2e759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc005100 con 0x7fa2e8099090 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7fa2d0041da0 con 0x7fa2e8099090 2024-09-16T06:40:27.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.953+0000 7fa2e659c700 10 monclient: get_auth_request con 0x7fa2c80362a0 auth_method 0 2024-09-16T06:40:27.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.957+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa2d0013130 con 0x7fa2e8099090 2024-09-16T06:40:27.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.957+0000 7fa2df7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:27.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:27.957+0000 7fa2df7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: get_monmap_and_config 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: monmap: 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:27.978658+0000 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:27.978658+0000 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:27.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 1 Processor -- start 2024-09-16T06:40:27.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 1 -- start start 2024-09-16T06:40:27.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: init 2024-09-16T06:40:27.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-c con 0x7ffb1809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-b con 0x7ffb1809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-a con 0x7ffb18138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1809cca0 con 0x7ffb18138da0 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1809f750 con 0x7ffb1809bb00 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18097620 con 0x7ffb1809c5f0 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.974+0000 7ffb1cafd700 10 monclient(hunting): authenticate will time out at 980.491844s 2024-09-16T06:40:27.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb16d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffb18138da0 0x7ffb1813b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:38184/0 (socket says 172.21.15.189:38184) 2024-09-16T06:40:27.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb16d9d700 1 -- 172.21.15.189:0/2180891569 learned_addr learned my addr 172.21.15.189:0/2180891569 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb1809bb00 0x7ffb1809bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34236/0 (socket says 172.21.15.189:34236) 2024-09-16T06:40:27.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb15d9b700 1 --1- 172.21.15.189:0/2180891569 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 0x7ffb18137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:38250/0 (socket says 172.21.15.189:38250) 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021026880 0 0) 0x7ffb1809cca0 con 0x7ffb18138da0 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): my global_id is 104436 2024-09-16T06:40:27.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb000035d0 con 0x7ffb18138da0 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030295795 0 0) 0x7ffb1809f750 con 0x7ffb1809bb00 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): my global_id is 94645 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1809cca0 con 0x7ffb1809bb00 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245393670 0 0) 0x7ffb18097620 con 0x7ffb1809c5f0 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient(hunting): my global_id is 124139 2024-09-16T06:40:27.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1809f750 con 0x7ffb1809c5f0 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781606346 0 0) 0x7ffb1809cca0 con 0x7ffb1809bb00 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 legacy=0x7ffb18137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 >> v1:172.21.15.76:6789/0 conn(0x7ffb18138da0 legacy=0x7ffb1813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 monclient: found mon.noname-c 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18097900 con 0x7ffb1809bb00 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 5 monclient: authenticate success, global_id 94645 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb0c0015f0 con 0x7ffb1809bb00 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: dump: 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb0c001910 con 0x7ffb1809bb00 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 1 -- 172.21.15.189:0/2180891569 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb0c001ce0 con 0x7ffb1809bb00 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 4 set_mon_vals no callback set 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1559a700 10 monclient: dump: 2024-09-16T06:40:27.990 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: shutdown 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- 172.21.15.189:0/2180891569 >> v1:172.21.15.189:6789/0 conn(0x7ffb1809bb00 legacy=0x7ffb1809bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- 172.21.15.189:0/2180891569 shutdown_connections 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- 172.21.15.189:0/2180891569 >> 172.21.15.189:0/2180891569 conn(0x7ffb180a03e0 msgr2=0x7ffb180a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- 172.21.15.189:0/2180891569 shutdown_connections 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- 172.21.15.189:0/2180891569 wait complete. 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: build_initial_monmap 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: monmap: 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:27.992 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 Processor -- start 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- start start 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: init 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-a con 0x7ffb1809bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:27.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-c con 0x7ffb1809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient(hunting): picked mon.noname-b con 0x7ffb18138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18096c90 con 0x7ffb1809bb00 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18090cf0 con 0x7ffb18138da0 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180a2300 con 0x7ffb1809c5f0 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb1cafd700 10 monclient(hunting): authenticate will time out at 980.495694s 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb16d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb18138da0 0x7ffb181d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34252/0 (socket says 172.21.15.189:34252) 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.978+0000 7ffb16d9d700 1 -- 172.21.15.189:0/1250373581 learned_addr learned my addr 172.21.15.189:0/1250373581 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269987835 0 0) 0x7ffb18090cf0 con 0x7ffb18138da0 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): my global_id is 94648 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf80035d0 con 0x7ffb18138da0 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb15d9b700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 legacy=0x7ffb180905e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022384547 0 0) 0x7ffb18096c90 con 0x7ffb1809bb00 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): my global_id is 104439 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb15d9b700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 legacy=0x7ffb180905e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb15d9b700 1 --1- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 0x7ffb180905e0 :-1 s=OPENED pgs=284 cs=1 l=1).handle_message read tag failed 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb15d9b700 1 --1- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 0x7ffb180905e0 :-1 s=OPENED pgs=284 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18090cf0 con 0x7ffb1809bb00 2024-09-16T06:40:27.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 928202818 0 0) 0x7ffaf80035d0 con 0x7ffb18138da0 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb18096c90 con 0x7ffb18138da0 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb10001d70 con 0x7ffb18138da0 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 345668863 0 0) 0x7ffb18096c90 con 0x7ffb18138da0 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6790/0 conn(0x7ffb1809c5f0 legacy=0x7ffb180905e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.76:6789/0 conn(0x7ffb1809bb00 legacy=0x7ffb1808fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 monclient: found mon.b 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18071440 con 0x7ffb18138da0 2024-09-16T06:40:27.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 5 monclient: authenticate success, global_id 94648 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _renew_subs 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18092ac0 con 0x7ffb18138da0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _renew_subs 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb100018c0 con 0x7ffb18138da0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb18091390 con 0x7ffb18138da0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffb10004db0 con 0x7ffb18138da0 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:27.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 10 monclient: dump: 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:27.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb14d99700 4 set_mon_vals no callback set 2024-09-16T06:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18066480 con 0x7ffb18138da0 2024-09-16T06:40:28.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ffb10006000 con 0x7ffb18138da0 2024-09-16T06:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7ffb100435b0 con 0x7ffb18138da0 2024-09-16T06:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.982+0000 7ffb1659c700 10 monclient: get_auth_request con 0x7ffaf8035fd0 auth_method 0 2024-09-16T06:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.986+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffb10014950 con 0x7ffb18138da0 2024-09-16T06:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.986+0000 7ffb06ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:27.986+0000 7ffb06ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:28.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb1cafd700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb1cafd700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb1cafd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7ffb1809b660 con 0x7ffb18138da0 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) v1 ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7ffb10012dc0 con 0x7ffb18138da0 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb06ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.150+0000 7ffb06ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:28.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.193+0000 7fa2e759e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:40:28.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.193+0000 7fa2e759e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:40:28.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.193+0000 7fa2e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:28.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.193+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa2cc005bc0 con 0x7fa2e8099090 2024-09-16T06:40:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.197+0000 7fa2df7fe700 1 -- 172.21.15.76:0/1508743819 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fa2d00117d0 con 0x7fa2e8099090 2024-09-16T06:40:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.197+0000 7fa2df7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:40:28.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.197+0000 7fa2df7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3631228251,config=1097430493,health=3172947059,kv=2418047910,logm=3075800639}) 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=2293843192,config=1097430493,health=3897446751,kv=4212347341,logm=1630063934}) 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3631228251,config=1097430493,health=3172947059,kv=2418047910,logm=3075800639}) 2024-09-16T06:40:28.202 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.198+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3414705576,config=1097430493,health=2374691465,kv=4212347341,logm=193108035}) 2024-09-16T06:40:28.203 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.203 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972476039}) 2024-09-16T06:40:28.204 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534236094}) 2024-09-16T06:40:28.204 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.204 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972476039}) 2024-09-16T06:40:28.204 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=858673673}) 2024-09-16T06:40:28.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa2c80362a0 legacy=0x7fa2c8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 10 monclient: shutdown 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 >> v1:172.21.15.76:6789/0 conn(0x7fa2e8099090 legacy=0x7fa2e81cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 shutdown_connections 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 >> 172.21.15.76:0/1508743819 conn(0x7fa2e80a0030 msgr2=0x7fa2e80716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 shutdown_connections 2024-09-16T06:40:28.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:28.201+0000 7fa2e759e700 1 -- 172.21.15.76:0/1508743819 wait complete. 2024-09-16T06:40:28.207 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.207 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133524765}) 2024-09-16T06:40:28.207 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231432133}) 2024-09-16T06:40:28.207 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.207 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133524765}) 2024-09-16T06:40:28.208 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.202+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061067339}) 2024-09-16T06:40:28.209 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.209 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015447023}) 2024-09-16T06:40:28.209 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219378143}) 2024-09-16T06:40:28.210 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.210 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015447023}) 2024-09-16T06:40:28.210 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.206+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2148678562}) 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1538375291}) 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790116884}) 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1538375291}) 2024-09-16T06:40:28.212 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1620678761}) 2024-09-16T06:40:28.214 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.214 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911042028}) 2024-09-16T06:40:28.214 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780275060}) 2024-09-16T06:40:28.214 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.215 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911042028}) 2024-09-16T06:40:28.215 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.210+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127068841}) 2024-09-16T06:40:28.215 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.216 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787507055}) 2024-09-16T06:40:28.216 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659565149}) 2024-09-16T06:40:28.216 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.216 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787507055}) 2024-09-16T06:40:28.216 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=233434099}) 2024-09-16T06:40:28.217 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.217 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=2418948738,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4274837763,mon_config_key=3994339013,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1161151261,osd_snap=2033388227,osdmap=2451406516}) 2024-09-16T06:40:28.217 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=2238276586,mdsmap=1531151339,mgr=1605231956,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4274837763,mon_config_key=590061470,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1161151261,osd_snap=2033388227,osdmap=1444955774}) 2024-09-16T06:40:28.218 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.218 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=2418948738,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4274837763,mon_config_key=3994339013,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1161151261,osd_snap=2033388227,osdmap=2451406516}) 2024-09-16T06:40:28.218 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.214+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=17,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=1524545168,mdsmap=3140514009,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4274837763,mon_config_key=3333735536,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=1161151261,osd_snap=2033388227,osdmap=3951458070}) 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=60} crc {osdmap=2729309552}) 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=3028319388}) 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=60} crc {osdmap=2729309552}) 2024-09-16T06:40:28.221 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:40:28.218+0000 7fdc77413700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=2641500258}) 2024-09-16T06:40:28.223 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:40:28.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb1cafd700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb1cafd700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb1cafd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7ffb1809ac90 con 0x7ffb18138da0 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb06ffd700 1 -- 172.21.15.189:0/1250373581 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) v1 ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7ffb10014540 con 0x7ffb18138da0 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb06ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.246+0000 7ffb06ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.189:6812/3098810674 conn(0x7ffaf8035fd0 legacy=0x7ffaf8038490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 10 monclient: shutdown 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 >> v1:172.21.15.189:6789/0 conn(0x7ffb18138da0 legacy=0x7ffb181d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 shutdown_connections 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 >> 172.21.15.189:0/1250373581 conn(0x7ffb180a03e0 msgr2=0x7ffb1806db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 shutdown_connections 2024-09-16T06:40:28.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.250+0000 7ffb1cafd700 1 -- 172.21.15.189:0/1250373581 wait complete. 2024-09-16T06:40:28.261 INFO:tasks.workunit.client.0.smithi189.stdout:foo-hdd 2024-09-16T06:40:28.262 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T06:40:28.262 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-ssd 2024-09-16T06:40:28.262 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:28.262 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-ssd 2024-09-16T06:40:28.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: get_monmap_and_config 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: build_initial_monmap 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: monmap: 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:28.364633+0000 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:28.364633+0000 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.364 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:28.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:28.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:28.365 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 Processor -- start 2024-09-16T06:40:28.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- start start 2024-09-16T06:40:28.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: init 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient(hunting): picked mon.noname-b con 0x7f6908132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient(hunting): picked mon.noname-c con 0x7f6908138460 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient(hunting): picked mon.noname-a con 0x7f690813bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690806abd0 con 0x7f690813bf70 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690806b170 con 0x7f6908138460 2024-09-16T06:40:28.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6908132b70 con 0x7f6908132080 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient(hunting): authenticate will time out at 980.877483s 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f69077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6908138460 0x7f690813a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34268/0 (socket says 172.21.15.189:34268) 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f69077fe700 1 -- 172.21.15.189:0/4288741182 learned_addr learned my addr 172.21.15.189:0/4288741182 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424025931 0 0) 0x7f690806b170 con 0x7f6908138460 2024-09-16T06:40:28.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient(hunting): my global_id is 94651 2024-09-16T06:40:28.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f00035d0 con 0x7f6908138460 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3092341288 0 0) 0x7f68f00035d0 con 0x7f6908138460 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 >> v1:172.21.15.76:6790/0 conn(0x7f6908132080 legacy=0x7f6908132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 >> v1:172.21.15.76:6789/0 conn(0x7f690813bf70 legacy=0x7f690813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 monclient: found mon.noname-c 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69080a38b0 con 0x7f6908138460 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:28.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68f8001610 con 0x7f6908138460 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 5 monclient: authenticate success, global_id 94651 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: dump: 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.377 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68f80018b0 con 0x7f6908138460 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 1 -- 172.21.15.189:0/4288741182 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68f8001c40 con 0x7f6908138460 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 4 set_mon_vals no callback set 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:28.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f6906ffd700 10 monclient: dump: 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: shutdown 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- 172.21.15.189:0/4288741182 >> v1:172.21.15.189:6789/0 conn(0x7f6908138460 legacy=0x7f690813a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- 172.21.15.189:0/4288741182 shutdown_connections 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- 172.21.15.189:0/4288741182 >> 172.21.15.189:0/4288741182 conn(0x7f69080a4660 msgr2=0x7f6908099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- 172.21.15.189:0/4288741182 shutdown_connections 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 1 -- 172.21.15.189:0/4288741182 wait complete. 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: build_initial_monmap 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.362+0000 7f690d476700 10 monclient: monmap: 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:28.380 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 Processor -- start 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- start start 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: init 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:28.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient(hunting): picked mon.noname-c con 0x7f6908132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient(hunting): picked mon.noname-b con 0x7f6908138460 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient(hunting): picked mon.noname-a con 0x7f690813bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690809e310 con 0x7f690813bf70 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690809cae0 con 0x7f6908138460 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690809de20 con 0x7f6908132080 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient(hunting): authenticate will time out at 980.881134s 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f69077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6908138460 0x7f690809c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:34280/0 (socket says 172.21.15.189:34280) 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f69077fe700 1 -- 172.21.15.189:0/3823746223 learned_addr learned my addr 172.21.15.189:0/3823746223 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:28.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966279069 0 0) 0x7f690809cae0 con 0x7f6908138460 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): my global_id is 94654 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f40035d0 con 0x7f6908138460 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666161699 0 0) 0x7f690809de20 con 0x7f6908132080 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): my global_id is 124148 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690809cae0 con 0x7f6908132080 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130303658 0 0) 0x7f690809e310 con 0x7f690813bf70 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): my global_id is 104445 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690809de20 con 0x7f690813bf70 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863939087 0 0) 0x7f68f40035d0 con 0x7f6908138460 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f690809e310 con 0x7f6908138460 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68f8001bc0 con 0x7f6908138460 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4165467774 0 0) 0x7f690809cae0 con 0x7f6908132080 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68f40035d0 con 0x7f6908132080 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68fc001830 con 0x7f6908132080 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:28.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801471571 0 0) 0x7f690809e310 con 0x7f6908138460 2024-09-16T06:40:28.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 >> v1:172.21.15.76:6790/0 conn(0x7f6908132080 legacy=0x7f690809bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 >> v1:172.21.15.76:6789/0 conn(0x7f690813bf70 legacy=0x7f69081d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 monclient: found mon.b 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690813bc50 con 0x7f6908138460 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 5 monclient: authenticate success, global_id 94654 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _renew_subs 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690813b870 con 0x7f6908138460 2024-09-16T06:40:28.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _renew_subs 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69080a1920 con 0x7f6908138460 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68f8004ca0 con 0x7f6908138460 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68f8004fb0 con 0x7f6908138460 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6904ff9700 10 monclient: dump: 2024-09-16T06:40:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.366+0000 7f6905ffb700 4 set_mon_vals no callback set 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f690d476700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f690d476700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f690d476700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69080666e0 con 0x7f6908138460 2024-09-16T06:40:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f68f800fa20 con 0x7f6908138460 2024-09-16T06:40:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f68f8043360 con 0x7f6908138460 2024-09-16T06:40:28.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.370+0000 7f6907fff700 10 monclient: get_auth_request con 0x7f68f4036430 auth_method 0 2024-09-16T06:40:28.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.374+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f68f80146f0 con 0x7f6908138460 2024-09-16T06:40:28.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.374+0000 7f6904ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.374+0000 7f6904ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:28.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f690d476700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f690d476700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f690d476700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f690809d9d0 con 0x7f6908138460 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) v1 ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7f690809d9d0 con 0x7f6908138460 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f6904ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.550+0000 7f6904ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:28.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f690d476700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f690d476700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f690d476700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f690809d9d0 con 0x7f6908138460 2024-09-16T06:40:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f6904ff9700 1 -- 172.21.15.189:0/3823746223 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) v1 ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7f690809d9d0 con 0x7f6908138460 2024-09-16T06:40:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f6904ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T06:40:28.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.654+0000 7f6904ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 >> v1:172.21.15.189:6812/3098810674 conn(0x7f68f4036430 legacy=0x7f68f40388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 10 monclient: shutdown 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 >> v1:172.21.15.189:6789/0 conn(0x7f6908138460 legacy=0x7f690809c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 shutdown_connections 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 >> 172.21.15.189:0/3823746223 conn(0x7f69080a4660 msgr2=0x7f69080a5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 shutdown_connections 2024-09-16T06:40:28.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.658+0000 7f690d476700 1 -- 172.21.15.189:0/3823746223 wait complete. 2024-09-16T06:40:28.670 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:28.670 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: get_monmap_and_config 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: build_initial_monmap 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: monmap: 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:28.786058+0000 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:28.786058+0000 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.785 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:28.786 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:28.786 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 1 Processor -- start 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 1 -- start start 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: init 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-a con 0x7f864009eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-b con 0x7f864009beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:28.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-c con 0x7f864009c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640099ed0 con 0x7f864009eba0 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864009d2e0 con 0x7f864009c2e0 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864009ef80 con 0x7f864009beb0 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8646ccd700 10 monclient(hunting): authenticate will time out at 981.298997s 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f86464cc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f864009c2e0 0x7f864013ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57274/0 (socket says 172.21.15.189:57274) 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f86464cc700 1 -- 172.21.15.189:0/150444581 learned_addr learned my addr 172.21.15.189:0/150444581 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500118519 0 0) 0x7f864009d2e0 con 0x7f864009c2e0 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): my global_id is 94657 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c0035d0 con 0x7f864009c2e0 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359221561 0 0) 0x7f864009ef80 con 0x7f864009beb0 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient(hunting): my global_id is 124151 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864009d2e0 con 0x7f864009beb0 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275100994 0 0) 0x7f862c0035d0 con 0x7f864009c2e0 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 >> v1:172.21.15.76:6790/0 conn(0x7f864009beb0 legacy=0x7f8640137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 >> v1:172.21.15.76:6789/0 conn(0x7f864009eba0 legacy=0x7f864009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 monclient: found mon.noname-c 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864009cc70 con 0x7f864009c2e0 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient: _finish_auth 0 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8638001990 con 0x7f864009c2e0 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.782+0000 7f8644cc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: dump: 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.794 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8638001c00 con 0x7f864009c2e0 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 5 monclient: authenticate success, global_id 94657 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 1 -- 172.21.15.189:0/150444581 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8638001fd0 con 0x7f864009c2e0 2024-09-16T06:40:28.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 4 set_mon_vals no callback set 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8644cc9700 10 monclient: dump: 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.796 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: shutdown 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- 172.21.15.189:0/150444581 >> v1:172.21.15.189:6789/0 conn(0x7f864009c2e0 legacy=0x7f864013ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- 172.21.15.189:0/150444581 shutdown_connections 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- 172.21.15.189:0/150444581 >> 172.21.15.189:0/150444581 conn(0x7f864009fe70 msgr2=0x7f86400a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- 172.21.15.189:0/150444581 shutdown_connections 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- 172.21.15.189:0/150444581 wait complete. 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: build_initial_monmap 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: monmap: 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:28.797 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:28.798 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 Processor -- start 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- start start 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: init 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-b con 0x7f864009beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-c con 0x7f864009c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient(hunting): picked mon.noname-a con 0x7f864009eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:28.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864012da80 con 0x7f864009eba0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864012ea00 con 0x7f864009beb0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864012ece0 con 0x7f864009c2e0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8646ccd700 10 monclient(hunting): authenticate will time out at 981.303772s 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8645ccb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f864009beb0 0x7f86401c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57282/0 (socket says 172.21.15.189:57282) 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.786+0000 7f8645ccb700 1 -- 172.21.15.189:0/2884921918 learned_addr learned my addr 172.21.15.189:0/2884921918 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328609115 0 0) 0x7f864012ea00 con 0x7f864009beb0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): my global_id is 94660 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86240035d0 con 0x7f864009beb0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342202729 0 0) 0x7f864009cc70 con 0x7f864009eba0 2024-09-16T06:40:28.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): my global_id is 104451 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864012ea00 con 0x7f864009eba0 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163361775 0 0) 0x7f864012ece0 con 0x7f864009c2e0 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): my global_id is 124154 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f864009cc70 con 0x7f864009c2e0 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200836559 0 0) 0x7f86240035d0 con 0x7f864009beb0 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f864012ece0 con 0x7f864009beb0 2024-09-16T06:40:28.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8630001910 con 0x7f864009beb0 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): dump: 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 944558374 0 0) 0x7f864012ece0 con 0x7f864009beb0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 >> v1:172.21.15.76:6790/0 conn(0x7f864009c2e0 legacy=0x7f86401327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 >> v1:172.21.15.76:6789/0 conn(0x7f864009eba0 legacy=0x7f86401ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 monclient: found mon.b 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8640132f00 con 0x7f864009beb0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:28.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 5 monclient: authenticate success, global_id 94660 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _renew_subs 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864012c610 con 0x7f864009beb0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _renew_subs 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86401d0050 con 0x7f864009beb0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8630001cf0 con 0x7f864009beb0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8630003b10 con 0x7f864009beb0 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 10 monclient: dump: 2024-09-16T06:40:28.809 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8637fff700 4 set_mon_vals no callback set 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8630003c70 con 0x7f864009beb0 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628005100 con 0x7f864009beb0 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f86300433c0 con 0x7f864009beb0 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.790+0000 7f86454ca700 10 monclient: get_auth_request con 0x7f8624036110 auth_method 0 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.798+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f86300044c0 con 0x7f864009beb0 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.798+0000 7f86367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:28.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.798+0000 7f86367fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:28.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.970+0000 7f8646ccd700 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:28.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.970+0000 7f8646ccd700 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:28.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.970+0000 7f8646ccd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:28.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:28.970+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f86280053b0 con 0x7f864009beb0 2024-09-16T06:40:29.225 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:40:29.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: get_monmap_and_config 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: monmap: 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:29.344439+0000 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:29.344439+0000 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.344 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:29.344 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:29.344 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 Processor -- start 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- start start 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: init 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient(hunting): picked mon.noname-a con 0x7f56580973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient(hunting): picked mon.noname-b con 0x7f5658097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient(hunting): picked mon.noname-c con 0x7f56581400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658099350 con 0x7f56580973c0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565809f690 con 0x7f56581400d0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658099bf0 con 0x7f5658097eb0 2024-09-16T06:40:29.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient(hunting): authenticate will time out at 976.477693s 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f56577fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5658097eb0 0x7f565813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38392/0 (socket says 172.21.15.76:38392) 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f56577fe700 1 -- 172.21.15.76:0/2469908787 learned_addr learned my addr 172.21.15.76:0/2469908787 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5657fff700 1 --1- 172.21.15.76:0/2469908787 >> v1:172.21.15.76:6789/0 conn(0x7f56580973c0 0x7f56580977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33592/0 (socket says 172.21.15.76:33592) 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714040264 0 0) 0x7f5658099bf0 con 0x7f5658097eb0 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): my global_id is 124157 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56400035d0 con 0x7f5658097eb0 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052854201 0 0) 0x7f5658099350 con 0x7f56580973c0 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): my global_id is 104454 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5658099bf0 con 0x7f56580973c0 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275653830 0 0) 0x7f565809f690 con 0x7f56581400d0 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient(hunting): my global_id is 94663 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5658099350 con 0x7f56581400d0 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 508313798 0 0) 0x7f56400035d0 con 0x7f5658097eb0 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 >> v1:172.21.15.189:6789/0 conn(0x7f56581400d0 legacy=0x7f56581424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 >> v1:172.21.15.76:6789/0 conn(0x7f56580973c0 legacy=0x7f56580977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 monclient: found mon.noname-b 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658098360 con 0x7f5658097eb0 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 5 monclient: authenticate success, global_id 124157 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5648001990 con 0x7f5658097eb0 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: dump: 2024-09-16T06:40:29.356 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5648001c70 con 0x7f5658097eb0 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 1 -- 172.21.15.76:0/2469908787 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5648002000 con 0x7f5658097eb0 2024-09-16T06:40:29.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 4 set_mon_vals no callback set 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f5656ffd700 10 monclient: dump: 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 10 monclient: shutdown 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- 172.21.15.76:0/2469908787 >> v1:172.21.15.76:6790/0 conn(0x7f5658097eb0 legacy=0x7f565813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- 172.21.15.76:0/2469908787 shutdown_connections 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- 172.21.15.76:0/2469908787 >> 172.21.15.76:0/2469908787 conn(0x7f56580a0030 msgr2=0x7f56580a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- 172.21.15.76:0/2469908787 shutdown_connections 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.341+0000 7f565d287700 1 -- 172.21.15.76:0/2469908787 wait complete. 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: monmap: 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.359 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 Processor -- start 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- start start 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: init 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient(hunting): picked mon.noname-c con 0x7f56580973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient(hunting): picked mon.noname-b con 0x7f5658097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient(hunting): picked mon.noname-a con 0x7f56581400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5658072660 con 0x7f56581400d0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565813fb50 con 0x7f5658097eb0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565809a540 con 0x7f56580973c0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient(hunting): authenticate will time out at 976.482096s 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565ca86700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f56581400d0 0x7f56581d6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33608/0 (socket says 172.21.15.76:33608) 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565ca86700 1 -- 172.21.15.76:0/1082850257 learned_addr learned my addr 172.21.15.76:0/1082850257 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339924132 0 0) 0x7f5658072660 con 0x7f56581400d0 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): my global_id is 104457 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56440035d0 con 0x7f56581400d0 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678146914 0 0) 0x7f565809a540 con 0x7f56580973c0 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): my global_id is 124160 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5658072660 con 0x7f56580973c0 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123323521 0 0) 0x7f565813fb50 con 0x7f5658097eb0 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): my global_id is 94666 2024-09-16T06:40:29.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f565809a540 con 0x7f5658097eb0 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468706847 0 0) 0x7f5658072660 con 0x7f56580973c0 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f565813fb50 con 0x7f56580973c0 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f564c0018b0 con 0x7f56580973c0 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376085732 0 0) 0x7f56440035d0 con 0x7f56581400d0 2024-09-16T06:40:29.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5658072660 con 0x7f56581400d0 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5650001850 con 0x7f56581400d0 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428248479 0 0) 0x7f565809a540 con 0x7f5658097eb0 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56440035d0 con 0x7f5658097eb0 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5648001950 con 0x7f5658097eb0 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 807172600 0 0) 0x7f565813fb50 con 0x7f56580973c0 2024-09-16T06:40:29.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 >> v1:172.21.15.189:6789/0 conn(0x7f5658097eb0 legacy=0x7f56581d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 >> v1:172.21.15.76:6789/0 conn(0x7f56581400d0 legacy=0x7f56581d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 monclient: found mon.c 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565806bdb0 con 0x7f56580973c0 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f5654ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 5 monclient: authenticate success, global_id 124160 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _renew_subs 2024-09-16T06:40:29.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565813eb80 con 0x7f56580973c0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _renew_subs 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.345+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f565806c630 con 0x7f56580973c0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f564c004890 con 0x7f56580973c0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f564c004ba0 con 0x7f56580973c0 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f56567fc700 4 set_mon_vals no callback set 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 10 monclient: dump: 2024-09-16T06:40:29.370 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f564c00f610 con 0x7f56580973c0 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f565d287700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f565d287700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f565d287700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638005100 con 0x7f56580973c0 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f564c043270 con 0x7f56580973c0 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.349+0000 7f56577fe700 10 monclient: get_auth_request con 0x7f5644036500 auth_method 0 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.357+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f564c014440 con 0x7f56580973c0 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.357+0000 7f5654ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.357+0000 7f5654ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:29.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:29.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:29.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5638005930 con 0x7f56580973c0 2024-09-16T06:40:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f5654ff9700 1 -- 172.21.15.76:0/1082850257 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1817000522) 0x7f564c012ae0 con 0x7f56580973c0 2024-09-16T06:40:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f5654ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:29.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f5654ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:29.592 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5644036500 legacy=0x7f56440389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 10 monclient: shutdown 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 >> v1:172.21.15.76:6790/0 conn(0x7f56580973c0 legacy=0x7f56581ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 shutdown_connections 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 >> 172.21.15.76:0/1082850257 conn(0x7f56580a0030 msgr2=0x7f56580a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.589+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 shutdown_connections 2024-09-16T06:40:29.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.593+0000 7f565d287700 1 -- 172.21.15.76:0/1082850257 wait complete. 2024-09-16T06:40:29.606 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:40:29.606 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:40:29.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:40:29.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:40:29.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: monmap: 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:29.717371+0000 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:29.717371+0000 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.716 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:29.717 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:29.717 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 1 Processor -- start 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 1 -- start start 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: init 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient(hunting): picked mon.noname-a con 0x7f101c09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient(hunting): picked mon.noname-c con 0x7f101c0709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient(hunting): picked mon.noname-b con 0x7f101c070de0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c09f140 con 0x7f101c09ed60 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c072530 con 0x7f101c0709d0 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c097f00 con 0x7f101c070de0 2024-09-16T06:40:29.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 10 monclient(hunting): authenticate will time out at 976.850697s 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f101c09ed60 0x7f101c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33610/0 (socket says 172.21.15.76:33610) 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101bfff700 1 -- 172.21.15.76:0/3596864715 learned_addr learned my addr 172.21.15.76:0/3596864715 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f1020b1d700 1 --1- 172.21.15.76:0/3596864715 >> v1:172.21.15.76:6790/0 conn(0x7f101c070de0 0x7f101c13de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38422/0 (socket says 172.21.15.76:38422) 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251809479 0 0) 0x7f101c09f140 con 0x7f101c09ed60 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): my global_id is 104460 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10080035d0 con 0x7f101c09ed60 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659277596 0 0) 0x7f101c097f00 con 0x7f101c070de0 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): my global_id is 124163 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c09f140 con 0x7f101c070de0 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693216192 0 0) 0x7f101c072530 con 0x7f101c0709d0 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient(hunting): my global_id is 94669 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c097f00 con 0x7f101c0709d0 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 271642515 0 0) 0x7f101c09f140 con 0x7f101c070de0 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 >> v1:172.21.15.189:6789/0 conn(0x7f101c0709d0 legacy=0x7f101c13a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 >> v1:172.21.15.76:6789/0 conn(0x7f101c09ed60 legacy=0x7f101c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c09a320 con 0x7f101c070de0 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1014001970 con 0x7f101c070de0 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 5 monclient: authenticate success, global_id 124163 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:29.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f101b7fe700 10 monclient: dump: 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.725 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.713+0000 7f102131e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1014001c10 con 0x7f101c070de0 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 1 -- 172.21.15.76:0/3596864715 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1014001fe0 con 0x7f101c070de0 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f101b7fe700 10 monclient: dump: 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 4 set_mon_vals no callback set 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 10 monclient: shutdown 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- 172.21.15.76:0/3596864715 >> v1:172.21.15.76:6790/0 conn(0x7f101c070de0 legacy=0x7f101c13de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- 172.21.15.76:0/3596864715 shutdown_connections 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- 172.21.15.76:0/3596864715 >> 172.21.15.76:0/3596864715 conn(0x7f101c0a0030 msgr2=0x7f101c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- 172.21.15.76:0/3596864715 shutdown_connections 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- 172.21.15.76:0/3596864715 wait complete. 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 10 monclient: monmap: 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:29.728 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 Processor -- start 2024-09-16T06:40:29.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 1 -- start start 2024-09-16T06:40:29.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.717+0000 7f102131e700 10 monclient: init 2024-09-16T06:40:29.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient(hunting): picked mon.noname-a con 0x7f101c0709d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient(hunting): picked mon.noname-b con 0x7f101c070de0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient(hunting): picked mon.noname-c con 0x7f101c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c09a540 con 0x7f101c0709d0 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0a23f0 con 0x7f101c070de0 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c08ed90 con 0x7f101c09ed60 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient(hunting): authenticate will time out at 976.856488s 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f1020b1d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f101c09ed60 0x7f101c1d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38432/0 (socket says 172.21.15.76:38432) 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f1020b1d700 1 -- 172.21.15.76:0/1674685517 learned_addr learned my addr 172.21.15.76:0/1674685517 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f101bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f101c0709d0 0x7f101c1c9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33616/0 (socket says 172.21.15.76:33616) 2024-09-16T06:40:29.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361302719 0 0) 0x7f101c09a540 con 0x7f101c0709d0 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): my global_id is 104463 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ffc0035d0 con 0x7f101c0709d0 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196591639 0 0) 0x7f101c08ed90 con 0x7f101c09ed60 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): my global_id is 124166 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c09a540 con 0x7f101c09ed60 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1814833999 0 0) 0x7f0ffc0035d0 con 0x7f101c0709d0 2024-09-16T06:40:29.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f101c08ed90 con 0x7f101c0709d0 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f100c001850 con 0x7f101c0709d0 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): dump: 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.736 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395632691 0 0) 0x7f101c0a23f0 con 0x7f101c070de0 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): my global_id is 94672 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ffc0035d0 con 0x7f101c070de0 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250661091 0 0) 0x7f101c09a540 con 0x7f101c09ed60 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f101c0a23f0 con 0x7f101c09ed60 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1014001870 con 0x7f101c09ed60 2024-09-16T06:40:29.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): dump: 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.738 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4215144887 0 0) 0x7f101c08ed90 con 0x7f101c0709d0 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6790/0 conn(0x7f101c09ed60 legacy=0x7f101c1d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.189:6789/0 conn(0x7f101c070de0 legacy=0x7f101c1ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 monclient: found mon.a 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c0908c0 con 0x7f101c0709d0 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 5 monclient: authenticate success, global_id 104463 2024-09-16T06:40:29.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _renew_subs 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c091160 con 0x7f101c0709d0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _renew_subs 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c1d32d0 con 0x7f101c0709d0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f100c004890 con 0x7f101c0709d0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f100c004ba0 con 0x7f101c0709d0 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f101a7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient: dump: 2024-09-16T06:40:29.740 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1000006590 con 0x7f1000002830 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237683701 0 0) 0x7f1000006590 con 0x7f1000002830 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 10 monclient(hunting): my global_id is 104466 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c08ed90 con 0x7f1000002830 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.721+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f100c00f610 con 0x7f101c0709d0 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2996857319 0 0) 0x7f101c08ed90 con 0x7f1000002830 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1000006590 con 0x7f1000002830 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7f100c0433e0 con 0x7f101c0709d0 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10040015e0 con 0x7f1000002830 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 10 monclient: dump: 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 909092672 0 0) 0x7f1000006590 con 0x7f1000002830 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f100000a190 con 0x7f1000002830 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f1020b1d700 10 monclient: get_auth_request con 0x7f0ffc036450 auth_method 0 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1004001950 con 0x7f1000002830 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:29.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6789/0 conn(0x7f1000002830 legacy=0x7f1000004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.725+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6789/0 conn(0x7f1000002830 legacy=0x7f1000004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:29.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:29.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1000009e30 con 0x7f1000009150 2024-09-16T06:40:29.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404452903 0 0) 0x7f1000009e30 con 0x7f1000009150 2024-09-16T06:40:29.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient(hunting): my global_id is 104469 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1000006590 con 0x7f1000009150 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1552667344 0 0) 0x7f1000006590 con 0x7f1000009150 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1000009e30 con 0x7f1000009150 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1004003840 con 0x7f1000009150 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient: dump: 2024-09-16T06:40:29.746 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1462126669 0 0) 0x7f1000009e30 con 0x7f1000009150 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f100000c340 con 0x7f1000009150 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 611867795) 0x7f1004005a20 con 0x7f1000009150 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6789/0 conn(0x7f1000009150 legacy=0x7f10000095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f10197fa700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6789/0 conn(0x7f1000009150 legacy=0x7f10000095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:29.748 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.750 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.751 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.751 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.751 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.751 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.751 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:29.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.754 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.756 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.759 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:29.760 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:29.761 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ffc036450 legacy=0x7f0ffc038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 10 monclient: shutdown 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 >> v1:172.21.15.76:6789/0 conn(0x7f101c0709d0 legacy=0x7f101c1c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f1020b1d700 1 -- 172.21.15.76:0/1674685517 reap_dead start 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f1020b1d700 1 -- 172.21.15.76:0/1674685517 reap_dead start 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 shutdown_connections 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 >> 172.21.15.76:0/1674685517 conn(0x7f101c0a0030 msgr2=0x7f101c0a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 shutdown_connections 2024-09-16T06:40:29.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.733+0000 7f102131e700 1 -- 172.21.15.76:0/1674685517 wait complete. 2024-09-16T06:40:29.764 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:29.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:40:29.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:40:29.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.877+0000 7fef5b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.877+0000 7fef5b59e700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.877+0000 7fef5b59e700 10 monclient: monmap: 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:29.883300+0000 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:29.883300+0000 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:29.882 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 Processor -- start 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- start start 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient: init 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fef5c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fef5c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fef5c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c098990 con 0x7fef5c099090 2024-09-16T06:40:29.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c097f40 con 0x7fef5c139500 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c09d610 con 0x7fef5c097b60 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient(hunting): authenticate will time out at 977.016997s 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5a59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef5c097b60 0x7fef5c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38440/0 (socket says 172.21.15.76:38440) 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5a59c700 1 -- 172.21.15.76:0/1517373635 learned_addr learned my addr 172.21.15.76:0/1517373635 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310803167 0 0) 0x7fef5c098990 con 0x7fef5c099090 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): my global_id is 104472 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef440035d0 con 0x7fef5c099090 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434808840 0 0) 0x7fef5c09d610 con 0x7fef5c097b60 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): my global_id is 124169 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5c098990 con 0x7fef5c097b60 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740053898 0 0) 0x7fef5c097f40 con 0x7fef5c139500 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient(hunting): my global_id is 94675 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5c09d610 con 0x7fef5c139500 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421607567 0 0) 0x7fef5c098990 con 0x7fef5c097b60 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 >> v1:172.21.15.189:6789/0 conn(0x7fef5c139500 legacy=0x7fef5c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 >> v1:172.21.15.76:6789/0 conn(0x7fef5c099090 legacy=0x7fef5c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 monclient: found mon.noname-b 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0986b0 con 0x7fef5c097b60 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c001970 con 0x7fef5c097b60 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 5 monclient: authenticate success, global_id 124169 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: dump: 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef4c001c10 con 0x7fef5c097b60 2024-09-16T06:40:29.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 4 set_mon_vals no callback set 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 1 -- 172.21.15.76:0/1517373635 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c001fe0 con 0x7fef5c097b60 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5959a700 10 monclient: dump: 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.891 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 10 monclient: shutdown 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- 172.21.15.76:0/1517373635 >> v1:172.21.15.76:6790/0 conn(0x7fef5c097b60 legacy=0x7fef5c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- 172.21.15.76:0/1517373635 shutdown_connections 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- 172.21.15.76:0/1517373635 >> 172.21.15.76:0/1517373635 conn(0x7fef5c0a0030 msgr2=0x7fef5c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- 172.21.15.76:0/1517373635 shutdown_connections 2024-09-16T06:40:29.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.881+0000 7fef5b59e700 1 -- 172.21.15.76:0/1517373635 wait complete. 2024-09-16T06:40:29.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: build_initial_monmap 2024-09-16T06:40:29.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: monmap: 2024-09-16T06:40:29.897 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 Processor -- start 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- start start 2024-09-16T06:40:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: init 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fef5c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fef5c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fef5c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c134590 con 0x7fef5c099090 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c133ae0 con 0x7fef5c097b60 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5c136960 con 0x7fef5c139500 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient(hunting): authenticate will time out at 977.021957s 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5ad9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef5c139500 0x7fef5c1cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38456/0 (socket says 172.21.15.76:38456) 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5ad9d700 1 -- 172.21.15.76:0/113312412 learned_addr learned my addr 172.21.15.76:0/113312412 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501098875 0 0) 0x7fef5c134590 con 0x7fef5c099090 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): my global_id is 104475 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef3c0035d0 con 0x7fef5c099090 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157928118 0 0) 0x7fef5c136960 con 0x7fef5c139500 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): my global_id is 124172 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5c134590 con 0x7fef5c139500 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1694950699 0 0) 0x7fef3c0035d0 con 0x7fef5c099090 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef5c136960 con 0x7fef5c099090 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef50001b10 con 0x7fef5c099090 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): dump: 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.901 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916512079 0 0) 0x7fef5c133ae0 con 0x7fef5c097b60 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): my global_id is 94678 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef3c0035d0 con 0x7fef5c097b60 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873612444 0 0) 0x7fef5c134590 con 0x7fef5c139500 2024-09-16T06:40:29.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef5c133ae0 con 0x7fef5c139500 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef54001990 con 0x7fef5c139500 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient(hunting): dump: 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.903 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 526957044 0 0) 0x7fef5c136960 con 0x7fef5c099090 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6790/0 conn(0x7fef5c139500 legacy=0x7fef5c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.189:6789/0 conn(0x7fef5c097b60 legacy=0x7fef5c132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 monclient: found mon.a 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c134d10 con 0x7fef5c099090 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: _finish_auth 0 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 5 monclient: authenticate success, global_id 104475 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _renew_subs 2024-09-16T06:40:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c0a3010 con 0x7fef5c099090 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _renew_subs 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c138550 con 0x7fef5c099090 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef500036a0 con 0x7fef5c099090 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef50003a30 con 0x7fef5c099090 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4bfff700 4 set_mon_vals no callback set 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.885+0000 7fef4affd700 10 monclient: dump: 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef5b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef40006590 con 0x7fef40002830 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fef50003c10 con 0x7fef5c099090 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3994+0+0 (unknown 3068503969 0 0) 0x7fef500420d0 con 0x7fef5c099090 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43453814 0 0) 0x7fef40006590 con 0x7fef40002830 2024-09-16T06:40:29.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient(hunting): my global_id is 124175 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5c136960 con 0x7fef40002830 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef5ad9d700 10 monclient: get_auth_request con 0x7fef3c036450 auth_method 0 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2226339119 0 0) 0x7fef5c136960 con 0x7fef40002830 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef40006590 con 0x7fef40002830 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c0019f0 con 0x7fef40002830 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient: dump: 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4261647362 0 0) 0x7fef40006590 con 0x7fef40002830 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef4000a190 con 0x7fef40002830 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fef4c001790 con 0x7fef40002830 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6790/0 conn(0x7fef40002830 legacy=0x7fef40004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.889+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6790/0 conn(0x7fef40002830 legacy=0x7fef40004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef5b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef40009e30 con 0x7fef40009150 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614613484 0 0) 0x7fef40009e30 con 0x7fef40009150 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:29.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient(hunting): my global_id is 124178 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40006590 con 0x7fef40009150 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1117586005 0 0) 0x7fef40006590 con 0x7fef40009150 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef40009e30 con 0x7fef40009150 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c0054a0 con 0x7fef40009150 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient: dump: 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2520644501 0 0) 0x7fef40009e30 con 0x7fef40009150 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef4000c340 con 0x7fef40009150 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3818155701) 0x7fef4c006f30 con 0x7fef40009150 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6790/0 conn(0x7fef40009150 legacy=0x7fef400095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.901+0000 7fef4affd700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6790/0 conn(0x7fef40009150 legacy=0x7fef400095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.914 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.915 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:29.916 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:29.917 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:29.919 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.920 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:29.922 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:29.923 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.189:6812/3098810674 conn(0x7fef3c036450 legacy=0x7fef3c038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 10 monclient: shutdown 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 >> v1:172.21.15.76:6789/0 conn(0x7fef5c099090 legacy=0x7fef5c1333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5ad9d700 1 -- 172.21.15.76:0/113312412 reap_dead start 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 shutdown_connections 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 >> 172.21.15.76:0/113312412 conn(0x7fef5c0a0030 msgr2=0x7fef5c06be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 shutdown_connections 2024-09-16T06:40:29.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:29.905+0000 7fef5b59e700 1 -- 172.21.15.76:0/113312412 wait complete. 2024-09-16T06:40:29.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:29.926 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:40:29.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:40:29.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:29.958+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v53) v1 ==== 151+0+0 (unknown 3129025207 0 0) 0x7f8630012c90 con 0x7f864009beb0 2024-09-16T06:40:29.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:29.958+0000 7f86367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:29.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:29.958+0000 7f86367fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: get_monmap_and_config 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: monmap: 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:30.049677+0000 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:30.049677+0000 2024-09-16T06:40:30.048 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:30.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:30.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 1 Processor -- start 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 1 -- start start 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: init 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fbbfc0964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fbbfc09b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fbbfc137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc098560 con 0x7fbbfc137e70 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc099ec0 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc09b5f0 con 0x7fbbfc0964f0 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfbfff700 10 monclient(hunting): authenticate will time out at 977.182896s 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfb7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbbfc137e70 0x7fbbfc13a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33672/0 (socket says 172.21.15.76:33672) 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfaffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fbbfc0964f0 0x7fbbfc0968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38492/0 (socket says 172.21.15.76:38492) 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbfb7fe700 1 -- 172.21.15.76:0/1918895836 learned_addr learned my addr 172.21.15.76:0/1918895836 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807212630 0 0) 0x7fbbfc098560 con 0x7fbbfc137e70 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): my global_id is 104478 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbdc0035d0 con 0x7fbbfc137e70 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112707429 0 0) 0x7fbbfc09b5f0 con 0x7fbbfc0964f0 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): my global_id is 124181 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc098560 con 0x7fbbfc0964f0 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469245208 0 0) 0x7fbbfc099ec0 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 10 monclient(hunting): my global_id is 94681 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc09b5f0 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.045+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073747301 0 0) 0x7fbbdc0035d0 con 0x7fbbfc137e70 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 >> v1:172.21.15.76:6790/0 conn(0x7fbbfc0964f0 legacy=0x7fbbfc0968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 >> v1:172.21.15.189:6789/0 conn(0x7fbbfc09b1c0 legacy=0x7fbbfc136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 monclient: found mon.noname-a 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc099be0 con 0x7fbbfc137e70 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 5 monclient: authenticate success, global_id 104478 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbf4001990 con 0x7fbbfc137e70 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: dump: 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf4001c70 con 0x7fbbfc137e70 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 1 -- 172.21.15.76:0/1918895836 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbf4002000 con 0x7fbbfc137e70 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 4 set_mon_vals no callback set 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf9ffb700 10 monclient: dump: 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.062 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: shutdown 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1918895836 >> v1:172.21.15.76:6789/0 conn(0x7fbbfc137e70 legacy=0x7fbbfc13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1918895836 shutdown_connections 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1918895836 >> 172.21.15.76:0/1918895836 conn(0x7fbbfc0a0030 msgr2=0x7fbbfc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1918895836 shutdown_connections 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1918895836 wait complete. 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: monmap: 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:30.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 Processor -- start 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- start start 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: init 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fbbfc0964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fbbfc09b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fbbfc137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc095e60 con 0x7fbbfc0964f0 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc09a540 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc08e700 con 0x7fbbfc137e70 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfaffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbbfc0964f0 0x7fbbfc08c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33684/0 (socket says 172.21.15.76:33684) 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfaffd700 1 -- 172.21.15.76:0/1950509997 learned_addr learned my addr 172.21.15.76:0/1950509997 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbfbfff700 10 monclient(hunting): authenticate will time out at 977.187650s 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791094313 0 0) 0x7fbbfc095e60 con 0x7fbbfc0964f0 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.049+0000 7fbbf37fe700 10 monclient(hunting): my global_id is 104481 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe00035d0 con 0x7fbbfc0964f0 2024-09-16T06:40:30.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938329326 0 0) 0x7fbbfc08e700 con 0x7fbbfc137e70 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): my global_id is 124184 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc095e60 con 0x7fbbfc137e70 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118114693 0 0) 0x7fbbfc09a540 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): my global_id is 94684 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc08e700 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701138286 0 0) 0x7fbbfc095e60 con 0x7fbbfc137e70 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbfc09a540 con 0x7fbbfc137e70 2024-09-16T06:40:30.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbf4001f80 con 0x7fbbfc137e70 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): dump: 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.067 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720095468 0 0) 0x7fbbe00035d0 con 0x7fbbfc0964f0 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbfc095e60 con 0x7fbbfc0964f0 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbec0017f0 con 0x7fbbfc0964f0 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): dump: 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97624489 0 0) 0x7fbbfc08e700 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbe00035d0 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbe40018b0 con 0x7fbbfc09b1c0 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): dump: 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.069 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2272307625 0 0) 0x7fbbfc09a540 con 0x7fbbfc137e70 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6789/0 conn(0x7fbbfc09b1c0 legacy=0x7fbbfc1cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.76:6789/0 conn(0x7fbbfc0964f0 legacy=0x7fbbfc08c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 monclient: found mon.c 2024-09-16T06:40:30.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc08de60 con 0x7fbbfc137e70 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 5 monclient: authenticate success, global_id 124184 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 10 monclient: _renew_subs 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbfc08e140 con 0x7fbbfc137e70 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 10 monclient: _renew_subs 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbfc08c6c0 con 0x7fbbfc137e70 2024-09-16T06:40:30.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf40014f0 con 0x7fbbfc137e70 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbf40051c0 con 0x7fbbfc137e70 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf97fa700 4 set_mon_vals no callback set 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient: dump: 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.072 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbbf400fc30 con 0x7fbbfc137e70 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbd4006590 con 0x7fbbd4002830 2024-09-16T06:40:30.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 4219+0+0 (unknown 3132307126 0 0) 0x7fbbf40438a0 con 0x7fbbfc137e70 2024-09-16T06:40:30.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbfa7fc700 10 monclient: get_auth_request con 0x7fbbe00364c0 auth_method 0 2024-09-16T06:40:30.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021015900 0 0) 0x7fbbd4006590 con 0x7fbbd4002830 2024-09-16T06:40:30.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 10 monclient(hunting): my global_id is 94687 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.053+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc09a540 con 0x7fbbd4002830 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2642340503 0 0) 0x7fbbfc09a540 con 0x7fbbd4002830 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbd4006590 con 0x7fbbd4002830 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbec004890 con 0x7fbbd4002830 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 10 monclient: dump: 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 832010897 0 0) 0x7fbbd4006590 con 0x7fbbd4002830 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbd400a190 con 0x7fbbd4002830 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbbec004c00 con 0x7fbbd4002830 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6789/0 conn(0x7fbbd4002830 legacy=0x7fbbd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.057+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6789/0 conn(0x7fbbd4002830 legacy=0x7fbbd4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.065+0000 7fbbfbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.065+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbd4009e30 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130713951 0 0) 0x7fbbd4009e30 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient(hunting): my global_id is 94690 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd4006590 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2349608572 0 0) 0x7fbbd4006590 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbd4009e30 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbec005180 con 0x7fbbd4009150 2024-09-16T06:40:30.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient: dump: 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1846828040 0 0) 0x7fbbd4009e30 con 0x7fbbd4009150 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbbd400c340 con 0x7fbbd4009150 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1014817712) 0x7fbbec006590 con 0x7fbbd4009150 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6789/0 conn(0x7fbbd4009150 legacy=0x7fbbd40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbf37fe700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6789/0 conn(0x7fbbd4009150 legacy=0x7fbbd40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:30.080 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 30, 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:30.081 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:30.082 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:30.083 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:30.084 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.085 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:30.086 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.087 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:30.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:30.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:30.091 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:30.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:30.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:30.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:30.092 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:30.092 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbbe00364c0 legacy=0x7fbbe0038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 10 monclient: shutdown 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 >> v1:172.21.15.76:6790/0 conn(0x7fbbfc137e70 legacy=0x7fbbfc1cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfb7fe700 1 -- 172.21.15.76:0/1950509997 reap_dead start 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfb7fe700 1 -- 172.21.15.76:0/1950509997 reap_dead start 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 shutdown_connections 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 >> 172.21.15.76:0/1950509997 conn(0x7fbbfc0a0030 msgr2=0x7fbbfc0a19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 shutdown_connections 2024-09-16T06:40:30.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.069+0000 7fbbfbfff700 1 -- 172.21.15.76:0/1950509997 wait complete. 2024-09-16T06:40:30.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:40:30.095 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:40:30.095 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:40:30.095 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:40:30.095 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:40:30.096 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:40:30.096 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:40:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:40:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f8628005650 con 0x7f864009beb0 2024-09-16T06:40:30.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f86367fc700 1 -- 172.21.15.189:0/2884921918 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v53) v1 ==== 151+0+0 (unknown 3129025207 0 0) 0x7f8630014410 con 0x7f864009beb0 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f86367fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f86367fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8624036110 legacy=0x7f86240385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 10 monclient: shutdown 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 >> v1:172.21.15.189:6789/0 conn(0x7f864009beb0 legacy=0x7f86401c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 shutdown_connections 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 >> 172.21.15.189:0/2884921918 conn(0x7f864009fe70 msgr2=0x7f864006e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 shutdown_connections 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.074+0000 7f8646ccd700 1 -- 172.21.15.189:0/2884921918 wait complete. 2024-09-16T06:40:30.097 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-16T06:40:30.149 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:40:30.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:40:30.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:40:30.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:30.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 10 monclient: monmap: 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:30.199605+0000 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:30.199605+0000 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:30.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 1 Processor -- start 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 1 -- start start 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.194+0000 7f84ef59e700 10 monclient: init 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f84f009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f84f009beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f84f009c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f0099ed0 con 0x7f84f009c2e0 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f009d2e0 con 0x7f84f009beb0 2024-09-16T06:40:30.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f009ef80 con 0x7f84f009eba0 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient(hunting): authenticate will time out at 982.712612s 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84edd9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f84f009beb0 legacy=0x7f84f0137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84eed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f84f009c2e0 0x7f84f013ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54354/0 (socket says 172.21.15.189:54354) 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ee59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f84f009eba0 0x7f84f009b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:49362/0 (socket says 172.21.15.189:49362) 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84eed9d700 1 -- 172.21.15.189:0/158074782 learned_addr learned my addr 172.21.15.189:0/158074782 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221259661 0 0) 0x7f84f0099ed0 con 0x7f84f009c2e0 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): my global_id is 104484 2024-09-16T06:40:30.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84dc0035d0 con 0x7f84f009c2e0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051779957 0 0) 0x7f84f009ef80 con 0x7f84f009eba0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient(hunting): my global_id is 124187 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f0099ed0 con 0x7f84f009eba0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971052602 0 0) 0x7f84dc0035d0 con 0x7f84f009c2e0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 >> v1:172.21.15.76:6790/0 conn(0x7f84f009eba0 legacy=0x7f84f009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 >> v1:172.21.15.189:6789/0 conn(0x7f84f009beb0 legacy=0x7f84f0137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 monclient: found mon.noname-a 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f009cc70 con 0x7f84f009c2e0 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 5 monclient: authenticate success, global_id 104484 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84e80018c0 con 0x7f84f009c2e0 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: dump: 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84e8001b30 con 0x7f84f009c2e0 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 1 -- 172.21.15.189:0/158074782 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84e8001ec0 con 0x7f84f009c2e0 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 4 set_mon_vals no callback set 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ed59a700 10 monclient: dump: 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 10 monclient: shutdown 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- 172.21.15.189:0/158074782 >> v1:172.21.15.76:6789/0 conn(0x7f84f009c2e0 legacy=0x7f84f013ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- 172.21.15.189:0/158074782 shutdown_connections 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- 172.21.15.189:0/158074782 >> 172.21.15.189:0/158074782 conn(0x7f84f009fe70 msgr2=0x7f84f00a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- 172.21.15.189:0/158074782 shutdown_connections 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.198+0000 7f84ef59e700 1 -- 172.21.15.189:0/158074782 wait complete. 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: monmap: 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 Processor -- start 2024-09-16T06:40:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- start start 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: init 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f84f009beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f84f009c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f84f009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f012b8c0 con 0x7f84f009c2e0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ee59c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f84f009beb0 legacy=0x7f84f01299d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f006fcd0 con 0x7f84f009beb0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f0070000 con 0x7f84f009eba0 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient(hunting): authenticate will time out at 982.717659s 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84edd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f84f009c2e0 0x7f84f012a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54364/0 (socket says 172.21.15.189:54364) 2024-09-16T06:40:30.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84edd9b700 1 -- 172.21.15.189:0/1730856853 learned_addr learned my addr 172.21.15.189:0/1730856853 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796314188 0 0) 0x7f84f012b8c0 con 0x7f84f009c2e0 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): my global_id is 104487 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d00035d0 con 0x7f84f009c2e0 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207916881 0 0) 0x7f84f0070000 con 0x7f84f009eba0 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): my global_id is 124190 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f012b8c0 con 0x7f84f009eba0 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378030323 0 0) 0x7f84d00035d0 con 0x7f84f009c2e0 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84f0070000 con 0x7f84f009c2e0 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84e0004200 con 0x7f84f009c2e0 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.216 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469816037 0 0) 0x7f84f012b8c0 con 0x7f84f009eba0 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84d00035d0 con 0x7f84f009eba0 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84e80016e0 con 0x7f84f009eba0 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 185500717 0 0) 0x7f84d00035d0 con 0x7f84f009eba0 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 >> v1:172.21.15.189:6789/0 conn(0x7f84f009beb0 legacy=0x7f84f01299d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 >> v1:172.21.15.76:6789/0 conn(0x7f84f009c2e0 legacy=0x7f84f012a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 monclient: found mon.c 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f009a380 con 0x7f84f009eba0 2024-09-16T06:40:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 5 monclient: authenticate success, global_id 124190 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _renew_subs 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f012d430 con 0x7f84f009eba0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _renew_subs 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f012b3b0 con 0x7f84f009eba0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84e8001450 con 0x7f84f009eba0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f84e8004ec0 con 0x7f84f009eba0 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84e6ffd700 10 monclient: dump: 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.202+0000 7f84ecd99700 4 set_mon_vals no callback set 2024-09-16T06:40:30.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84ef59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f84e800f930 con 0x7f84f009eba0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84ef59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84ef59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d4005100 con 0x7f84f009eba0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 4219+0+0 (unknown 3132307126 0 0) 0x7f84e8043670 con 0x7f84f009eba0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.206+0000 7f84ee59c700 10 monclient: get_auth_request con 0x7f84d0036270 auth_method 0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.218+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f84e8010ed0 con 0x7f84f009eba0 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.218+0000 7f84e6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.218+0000 7f84e6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: get_monmap_and_config 2024-09-16T06:40:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: monmap: 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:30.265117+0000 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:30.265117+0000 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:30.264 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 1 Processor -- start 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 1 -- start start 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: init 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-c con 0x7efe4c09b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-b con 0x7efe4c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-a con 0x7efe4c138840 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c09bfb0 con 0x7efe4c138840 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c09d3c0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c09c220 con 0x7efe4c09f230 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe545d0700 10 monclient(hunting): authenticate will time out at 977.399258s 2024-09-16T06:40:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5236c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efe4c09b6a0 legacy=0x7efe4c0a0a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe52b6d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efe4c138840 0x7efe4c13ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33704/0 (socket says 172.21.15.76:33704) 2024-09-16T06:40:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe51b6b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7efe4c09f230 0x7efe4c137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38520/0 (socket says 172.21.15.76:38520) 2024-09-16T06:40:30.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe52b6d700 1 -- 172.21.15.76:0/1842525249 learned_addr learned my addr 172.21.15.76:0/1842525249 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534458107 0 0) 0x7efe4c09c220 con 0x7efe4c09f230 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5136a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5136a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5136a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.261+0000 7efe5136a700 10 monclient(hunting): my global_id is 124193 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe380035d0 con 0x7efe4c09f230 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660698850 0 0) 0x7efe4c09bfb0 con 0x7efe4c138840 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient(hunting): my global_id is 104490 2024-09-16T06:40:30.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c09c220 con 0x7efe4c138840 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598210544 0 0) 0x7efe380035d0 con 0x7efe4c09f230 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 >> v1:172.21.15.189:6789/0 conn(0x7efe4c09b6a0 legacy=0x7efe4c0a0a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 >> v1:172.21.15.76:6789/0 conn(0x7efe4c138840 legacy=0x7efe4c13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 monclient: found mon.noname-b 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c09c500 con 0x7efe4c09f230 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe3c001540 con 0x7efe4c09f230 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 5 monclient: authenticate success, global_id 124193 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: dump: 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe3c001820 con 0x7efe4c09f230 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 1 -- 172.21.15.76:0/1842525249 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe3c001bb0 con 0x7efe4c09f230 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 4 set_mon_vals no callback set 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe5136a700 10 monclient: dump: 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.273 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 10 monclient: shutdown 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 1 -- 172.21.15.76:0/1842525249 >> v1:172.21.15.76:6790/0 conn(0x7efe4c09f230 legacy=0x7efe4c137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 1 -- 172.21.15.76:0/1842525249 shutdown_connections 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 1 -- 172.21.15.76:0/1842525249 >> 172.21.15.76:0/1842525249 conn(0x7efe4c0a3e50 msgr2=0x7efe4c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 1 -- 172.21.15.76:0/1842525249 shutdown_connections 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 1 -- 172.21.15.76:0/1842525249 wait complete. 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 10 monclient: build_initial_monmap 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.265+0000 7efe545d0700 10 monclient: monmap: 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:30.275 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:30.276 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 Processor -- start 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- start start 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: init 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-a con 0x7efe4c09b6a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-c con 0x7efe4c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient(hunting): picked mon.noname-b con 0x7efe4c138840 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe5236c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efe4c09b6a0 0x7efe4c131040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33720/0 (socket says 172.21.15.76:33720) 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe5236c700 1 -- 172.21.15.76:0/1433495806 learned_addr learned my addr 172.21.15.76:0/1433495806 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe52b6d700 1 -- 172.21.15.76:0/1433495806 >> v1:172.21.15.189:6789/0 conn(0x7efe4c138840 legacy=0x7efe4c1cf3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c131de0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c137a70 con 0x7efe4c138840 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0a2910 con 0x7efe4c09f230 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient(hunting): authenticate will time out at 977.406569s 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851426163 0 0) 0x7efe4c131de0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): my global_id is 104493 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe300057e0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637520660 0 0) 0x7efe4c0a2910 con 0x7efe4c09f230 2024-09-16T06:40:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:30.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): my global_id is 124196 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c131de0 con 0x7efe4c09f230 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494582782 0 0) 0x7efe300057e0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4c0a2910 con 0x7efe4c09b6a0 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe48004230 con 0x7efe4c09b6a0 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:30.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:30.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 837635773 0 0) 0x7efe4c131de0 con 0x7efe4c09f230 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe300057e0 con 0x7efe4c09f230 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe3c001b30 con 0x7efe4c09f230 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.286 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805611171 0 0) 0x7efe4c0a2910 con 0x7efe4c09b6a0 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 >> v1:172.21.15.76:6790/0 conn(0x7efe4c09f230 legacy=0x7efe4c1cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 >> v1:172.21.15.189:6789/0 conn(0x7efe4c138840 legacy=0x7efe4c1cf3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 monclient: found mon.a 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c1385c0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:30.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe42ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:30.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 5 monclient: authenticate success, global_id 104493 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _renew_subs 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c11c420 con 0x7efe4c09b6a0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _renew_subs 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.269+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c137140 con 0x7efe4c09b6a0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe48004450 con 0x7efe4c09b6a0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe480047e0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:30.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 10 monclient: dump: 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe545d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe50b69700 4 set_mon_vals no callback set 2024-09-16T06:40:30.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efe48004940 con 0x7efe4c09b6a0 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 4219+0+0 (unknown 3132307126 0 0) 0x7efe48043430 con 0x7efe4c09b6a0 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe545d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe545d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c0678b0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.273+0000 7efe51b6b700 10 monclient: get_auth_request con 0x7efe3003a820 auth_method 0 2024-09-16T06:40:30.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.281+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efe48004bf0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.281+0000 7efe42ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:30.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.281+0000 7efe42ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.394+0000 7f84ef59e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.394+0000 7f84ef59e700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:30.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.394+0000 7f84ef59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:30.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:30.394+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f84d40053b0 con 0x7f84f009eba0 2024-09-16T06:40:30.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe545d0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:30.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe545d0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:30.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe545d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:30.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe4c0a35a0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe42ffd700 1 -- 172.21.15.76:0/1433495806 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3569116533) 0x7efe4801cdc0 con 0x7efe4c09b6a0 2024-09-16T06:40:30.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe42ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:30.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.513+0000 7efe42ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:30.517 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:30.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 >> v1:172.21.15.189:6812/3098810674 conn(0x7efe3003a820 legacy=0x7efe3003cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 10 monclient: shutdown 2024-09-16T06:40:30.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:30.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 >> v1:172.21.15.76:6789/0 conn(0x7efe4c09b6a0 legacy=0x7efe4c131040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:30.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 shutdown_connections 2024-09-16T06:40:30.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 >> 172.21.15.76:0/1433495806 conn(0x7efe4c0a3e50 msgr2=0x7efe4c09a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:30.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 shutdown_connections 2024-09-16T06:40:30.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:30.517+0000 7efe545d0700 1 -- 172.21.15.76:0/1433495806 wait complete. 2024-09-16T06:40:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:33.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:33.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:33.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:33.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:33.202+0000 7f84e5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:33.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: get_monmap_and_config 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: build_initial_monmap 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: monmap: 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:33.654731+0000 2024-09-16T06:40:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:33.654731+0000 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:33.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 1 Processor -- start 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 1 -- start start 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: init 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient(hunting): picked mon.noname-a con 0x7fb6400973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient(hunting): picked mon.noname-c con 0x7fb640097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 10 monclient(hunting): picked mon.noname-b con 0x7fb6401400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640099350 con 0x7fb6400973c0 2024-09-16T06:40:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.649+0000 7fb645952700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64009f690 con 0x7fb640097eb0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640099bf0 con 0x7fb6401400d0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 10 monclient(hunting): authenticate will time out at 980.788018s 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb640097eb0 legacy=0x7fb64013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb644950700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb6400973c0 0x7fb6400977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33728/0 (socket says 172.21.15.76:33728) 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb644950700 1 -- 172.21.15.76:0/343249439 learned_addr learned my addr 172.21.15.76:0/343249439 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240558394 0 0) 0x7fb640099bf0 con 0x7fb6401400d0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): my global_id is 124199 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6280035d0 con 0x7fb6401400d0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555178036 0 0) 0x7fb640099350 con 0x7fb6400973c0 2024-09-16T06:40:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient(hunting): my global_id is 104496 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb640099bf0 con 0x7fb6400973c0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919794299 0 0) 0x7fb6280035d0 con 0x7fb6401400d0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 >> v1:172.21.15.189:6789/0 conn(0x7fb640097eb0 legacy=0x7fb64013e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 >> v1:172.21.15.76:6789/0 conn(0x7fb6400973c0 legacy=0x7fb6400977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640098360 con 0x7fb6401400d0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb630001610 con 0x7fb6401400d0 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: dump: 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 5 monclient: authenticate success, global_id 124199 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb630001870 con 0x7fb6401400d0 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 1 -- 172.21.15.76:0/343249439 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb630001c00 con 0x7fb6401400d0 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb63f7fe700 10 monclient: dump: 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 4 set_mon_vals no callback set 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 10 monclient: shutdown 2024-09-16T06:40:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- 172.21.15.76:0/343249439 >> v1:172.21.15.76:6790/0 conn(0x7fb6401400d0 legacy=0x7fb6401424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- 172.21.15.76:0/343249439 shutdown_connections 2024-09-16T06:40:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- 172.21.15.76:0/343249439 >> 172.21.15.76:0/343249439 conn(0x7fb6400a0030 msgr2=0x7fb6400a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- 172.21.15.76:0/343249439 shutdown_connections 2024-09-16T06:40:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.653+0000 7fb645952700 1 -- 172.21.15.76:0/343249439 wait complete. 2024-09-16T06:40:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient: build_initial_monmap 2024-09-16T06:40:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient: monmap: 2024-09-16T06:40:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 1 Processor -- start 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 1 -- start start 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient: init 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient(hunting): picked mon.noname-a con 0x7fb6400973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient(hunting): picked mon.noname-b con 0x7fb640097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient(hunting): picked mon.noname-c con 0x7fb6401400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640072660 con 0x7fb6400973c0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64009d590 con 0x7fb640097eb0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64009d170 con 0x7fb6401400d0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb645952700 10 monclient(hunting): authenticate will time out at 980.794628s 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb644950700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb6400973c0 0x7fb64009a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33738/0 (socket says 172.21.15.76:33738) 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb644950700 1 -- 172.21.15.76:0/1416527787 learned_addr learned my addr 172.21.15.76:0/1416527787 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63ffff700 1 -- 172.21.15.76:0/1416527787 >> v1:172.21.15.189:6789/0 conn(0x7fb640097eb0 legacy=0x7fb64009ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409568607 0 0) 0x7fb64009d170 con 0x7fb6401400d0 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): my global_id is 124202 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c0035d0 con 0x7fb6401400d0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533240723 0 0) 0x7fb640072660 con 0x7fb6400973c0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): my global_id is 104499 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64009d170 con 0x7fb6400973c0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398042232 0 0) 0x7fb62c0035d0 con 0x7fb6401400d0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb640072660 con 0x7fb6401400d0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb630001b80 con 0x7fb6401400d0 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1467665258 0 0) 0x7fb64009d170 con 0x7fb6400973c0 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb62c0035d0 con 0x7fb6400973c0 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6340011d0 con 0x7fb6400973c0 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.657+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 102471830 0 0) 0x7fb640072660 con 0x7fb6401400d0 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 >> v1:172.21.15.189:6789/0 conn(0x7fb640097eb0 legacy=0x7fb64009ac50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 >> v1:172.21.15.76:6789/0 conn(0x7fb6400973c0 legacy=0x7fb64009a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 monclient: found mon.c 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640071ea0 con 0x7fb6401400d0 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 5 monclient: authenticate success, global_id 124202 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _renew_subs 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64009c8d0 con 0x7fb6401400d0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _renew_subs 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64009b360 con 0x7fb6401400d0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb630001200 con 0x7fb6401400d0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb630003b70 con 0x7fb6401400d0 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63effd700 4 set_mon_vals no callback set 2024-09-16T06:40:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 10 monclient: dump: 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb620005100 con 0x7fb6401400d0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb630003d50 con 0x7fb6401400d0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 4219+0+0 (unknown 3132307126 0 0) 0x7fb630042430 con 0x7fb6401400d0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.661+0000 7fb644950700 10 monclient: get_auth_request con 0x7fb62c0362a0 auth_method 0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.669+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb6300135e0 con 0x7fb6401400d0 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.669+0000 7fb63d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.669+0000 7fb63d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb620005930 con 0x7fb6401400d0 2024-09-16T06:40:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb63d7fa700 1 -- 172.21.15.76:0/1416527787 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2219879252) 0x7fb630011c80 con 0x7fb6401400d0 2024-09-16T06:40:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb63d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb63d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:33.908 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb62c0362a0 legacy=0x7fb62c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 10 monclient: shutdown 2024-09-16T06:40:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 >> v1:172.21.15.76:6790/0 conn(0x7fb6401400d0 legacy=0x7fb6401d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 shutdown_connections 2024-09-16T06:40:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.905+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 >> 172.21.15.76:0/1416527787 conn(0x7fb6400a0030 msgr2=0x7fb640134800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.909+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 shutdown_connections 2024-09-16T06:40:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:33.909+0000 7fb645952700 1 -- 172.21.15.76:0/1416527787 wait complete. 2024-09-16T06:40:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:34.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:34.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:34.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:34.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:35.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:35.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:35.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:35.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:36.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:36.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:36.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:36.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:36.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:36.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:36.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:40:37.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 10 monclient: get_monmap_and_config 2024-09-16T06:40:37.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 10 monclient: build_initial_monmap 2024-09-16T06:40:37.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 10 monclient: monmap: 2024-09-16T06:40:37.040 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:37.041884+0000 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:37.041884+0000 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:37.041 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:37.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 1 Processor -- start 2024-09-16T06:40:37.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 1 -- start start 2024-09-16T06:40:37.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.037+0000 7fdc819d9700 10 monclient: init 2024-09-16T06:40:37.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:37.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:37.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c06c280 con 0x7fdc7c132330 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c06bfa0 con 0x7fdc7c06cf50 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc7bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdc7c06d480 0x7fdc7c13a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39842/0 (socket says 172.21.15.76:39842) 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc7bfff700 1 -- 172.21.15.76:0/2651571447 learned_addr learned my addr 172.21.15.76:0/2651571447 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c09ace0 con 0x7fdc7c06d480 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c13da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc809d7700 1 --1- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6789/0 conn(0x7fdc7c132330 0x7fdc7c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54806/0 (socket says 172.21.15.76:54806) 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:37.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.041+0000 7fdc819d9700 10 monclient(hunting): authenticate will time out at 984.177268s 2024-09-16T06:40:37.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:37.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:37.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:37.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:37.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:37.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:37.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.245+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c13da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:37.649+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c13da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:38.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:38.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:38.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:38.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:38.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:38.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:38.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:38.453+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c13da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:39.206+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:39.206+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:39.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:39.206+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): tick 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6790/0 conn(0x7fdc7c06d480 legacy=0x7fdc7c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c13da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6789/0 conn(0x7fdc7c132330 legacy=0x7fdc7c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:40.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c06bfa0 con 0x7fdc7c132330 2024-09-16T06:40:40.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6000ef00 con 0x7fdc7c06cf50 2024-09-16T06:40:40.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc60010110 con 0x7fdc7c06d480 2024-09-16T06:40:40.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc7a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:40.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.041+0000 7fdc809d7700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:40.202+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:40.202+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:40.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:40.202+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:40.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.245+0000 7fdc809d7700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:40.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:40.649+0000 7fdc809d7700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:41.201+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:41.201+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:41.201+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:41.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:41.453+0000 7fdc809d7700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:42.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:42.205+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:42.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:42.205+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:42.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:42.205+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:43.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): tick 2024-09-16T06:40:43.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:40:43.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6790/0 conn(0x7fdc7c06d480 legacy=0x7fdc60008f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6789/0 conn(0x7fdc7c132330 legacy=0x7fdc6000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fdc6000df90 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc60011320 con 0x7fdc6000df90 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6000ef00 con 0x7fdc7c06cf50 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6000ec60 con 0x7fdc7c06d480 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc7a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:43.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.045+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:43.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:43.201+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:43.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:43.201+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:43.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:43.201+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:43.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:43.201+0000 7f84e5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:43.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.249+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:43.653+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:44.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:44.205+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:44.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:44.205+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:44.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:44.205+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:44.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.457+0000 7fdc811d8700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059498204 0 0) 0x7fdc60011320 con 0x7fdc6000df90 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 10 monclient(hunting): my global_id is 104502 2024-09-16T06:40:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:44.997+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c0035d0 con 0x7fdc6000df90 2024-09-16T06:40:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754122518 0 0) 0x7fdc6c0035d0 con 0x7fdc6000df90 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6790/0 conn(0x7fdc7c06d480 legacy=0x7fdc60008f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06cf50 legacy=0x7fdc60005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6000b6a0 con 0x7fdc6000df90 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 5 monclient: authenticate success, global_id 104502 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc70001640 con 0x7fdc6000df90 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: dump: 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.005 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc70001870 con 0x7fdc6000df90 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 1 -- 172.21.15.76:0/2651571447 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc70001c40 con 0x7fdc6000df90 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 4 set_mon_vals no callback set 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc7b7fe700 10 monclient: dump: 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.006 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.007 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 10 monclient: shutdown 2024-09-16T06:40:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 >> v1:172.21.15.76:6789/0 conn(0x7fdc6000df90 legacy=0x7fdc6000e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 shutdown_connections 2024-09-16T06:40:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 >> 172.21.15.76:0/2651571447 conn(0x7fdc7c0a41c0 msgr2=0x7fdc7c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 shutdown_connections 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 1 -- 172.21.15.76:0/2651571447 wait complete. 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 10 monclient: build_initial_monmap 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.001+0000 7fdc819d9700 10 monclient: monmap: 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 1 Processor -- start 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 1 -- start start 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient: init 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-a con 0x7fdc6000df90 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c131a60 con 0x7fdc6000df90 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c124560 con 0x7fdc7c06d480 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc809d7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdc6000df90 0x7fdc7c1c9e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45358/0 (socket says 172.21.15.76:45358) 2024-09-16T06:40:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc809d7700 1 -- 172.21.15.76:0/4198568821 learned_addr learned my addr 172.21.15.76:0/4198568821 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0a5750 con 0x7fdc7c06cf50 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc819d9700 10 monclient(hunting): authenticate will time out at 992.142621s 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc811d8700 1 -- 172.21.15.76:0/4198568821 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06d480 legacy=0x7fdc7c1d25e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903776321 0 0) 0x7fdc7c131a60 con 0x7fdc6000df90 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): my global_id is 104505 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc60011320 con 0x7fdc6000df90 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1070286683 0 0) 0x7fdc60011320 con 0x7fdc6000df90 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc7c131a60 con 0x7fdc6000df90 2024-09-16T06:40:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc70001600 con 0x7fdc6000df90 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): dump: 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.005+0000 7fdc797fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 315250923 0 0) 0x7fdc7c131a60 con 0x7fdc6000df90 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 >> v1:172.21.15.76:6790/0 conn(0x7fdc7c06cf50 legacy=0x7fdc7c1ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 >> v1:172.21.15.189:6789/0 conn(0x7fdc7c06d480 legacy=0x7fdc7c1d25e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 monclient: found mon.a 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c124280 con 0x7fdc6000df90 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 5 monclient: authenticate success, global_id 104505 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _renew_subs 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c122b50 con 0x7fdc6000df90 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _renew_subs 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c121570 con 0x7fdc6000df90 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc70004bd0 con 0x7fdc6000df90 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc70004f60 con 0x7fdc6000df90 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc7affd700 4 set_mon_vals no callback set 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 10 monclient: dump: 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.018 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdc70005140 con 0x7fdc6000df90 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5c005100 con 0x7fdc6000df90 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 4219+0+0 (unknown 3132307126 0 0) 0x7fdc70043740 con 0x7fdc6000df90 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.009+0000 7fdc7bfff700 10 monclient: get_auth_request con 0x7fdc60036c20 auth_method 0 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.017+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdc70014b80 con 0x7fdc6000df90 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.017+0000 7fdc797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.017+0000 7fdc797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:45.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:45.201+0000 7f84e5ffb700 10 monclient: tick 2024-09-16T06:40:45.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:45.201+0000 7f84e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:45.201+0000 7f84e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc819d9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:40:45.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc819d9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:45.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc819d9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc5c005930 con 0x7fdc6000df90 2024-09-16T06:40:45.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc797fa700 1 -- 172.21.15.76:0/4198568821 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3119551953) 0x7fdc70014950 con 0x7fdc6000df90 2024-09-16T06:40:45.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:40:45.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.273+0000 7fdc797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:45.278 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdc60036c20 legacy=0x7fdc600390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 10 monclient: shutdown 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 >> v1:172.21.15.76:6789/0 conn(0x7fdc6000df90 legacy=0x7fdc7c1c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 shutdown_connections 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 >> 172.21.15.76:0/4198568821 conn(0x7fdc7c0a41c0 msgr2=0x7fdc7c0a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 shutdown_connections 2024-09-16T06:40:45.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.277+0000 7fdc819d9700 1 -- 172.21.15.76:0/4198568821 wait complete. 2024-09-16T06:40:45.292 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:40:45.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:40:45.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:40:45.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:45.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: build_initial_monmap 2024-09-16T06:40:45.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: monmap: 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:45.409453+0000 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:45.409453+0000 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 1 Processor -- start 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 1 -- start start 2024-09-16T06:40:45.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: init 2024-09-16T06:40:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-a con 0x7f20b0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-c con 0x7f20b0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-b con 0x7f20b0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b0098990 con 0x7f20b0097b60 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b0097f40 con 0x7f20b0099090 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b009d610 con 0x7f20b0139500 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 10 monclient(hunting): authenticate will time out at 992.542776s 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b4f6b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f20b0099090 legacy=0x7f20b009f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b5f6d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f20b0139500 0x7f20b013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39732/0 (socket says 172.21.15.76:39732) 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b5f6d700 1 -- 172.21.15.76:0/38337855 learned_addr learned my addr 172.21.15.76:0/38337855 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995385100 0 0) 0x7f20b009d610 con 0x7f20b0139500 2024-09-16T06:40:45.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): my global_id is 124205 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20980035d0 con 0x7f20b0139500 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269846269 0 0) 0x7f20b0098990 con 0x7f20b0097b60 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient(hunting): my global_id is 104508 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b009d610 con 0x7f20b0097b60 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2261421119 0 0) 0x7f20980035d0 con 0x7f20b0139500 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 >> v1:172.21.15.189:6789/0 conn(0x7f20b0099090 legacy=0x7f20b009f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 >> v1:172.21.15.76:6789/0 conn(0x7f20b0097b60 legacy=0x7f20b009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 monclient: found mon.noname-b 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00986b0 con 0x7f20b0139500 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a0001610 con 0x7f20b0139500 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 5 monclient: authenticate success, global_id 124205 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20b676e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.405+0000 7f20a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: dump: 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.417 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20a00018f0 con 0x7f20b0139500 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 1 -- 172.21.15.76:0/38337855 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a0001c80 con 0x7f20b0139500 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20a7fff700 10 monclient: dump: 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 4 set_mon_vals no callback set 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: shutdown 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- 172.21.15.76:0/38337855 >> v1:172.21.15.76:6790/0 conn(0x7f20b0139500 legacy=0x7f20b013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- 172.21.15.76:0/38337855 shutdown_connections 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- 172.21.15.76:0/38337855 >> 172.21.15.76:0/38337855 conn(0x7f20b00a0030 msgr2=0x7f20b00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- 172.21.15.76:0/38337855 shutdown_connections 2024-09-16T06:40:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- 172.21.15.76:0/38337855 wait complete. 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: build_initial_monmap 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: monmap: 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 Processor -- start 2024-09-16T06:40:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- start start 2024-09-16T06:40:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: init 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-a con 0x7f20b0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-c con 0x7f20b0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient(hunting): picked mon.noname-b con 0x7f20b0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b00a2c00 con 0x7f20b0097b60 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b006ccb0 con 0x7f20b0139500 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b013c120 con 0x7f20b0099090 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:45.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b4f6b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f20b0099090 0x7f20b006c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39746/0 (socket says 172.21.15.76:39746) 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b4f6b700 1 -- 172.21.15.76:0/3176329261 learned_addr learned my addr 172.21.15.76:0/3176329261 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.409+0000 7f20b676e700 10 monclient(hunting): authenticate will time out at 992.547938s 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b5f6d700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.189:6789/0 conn(0x7f20b0139500 legacy=0x7f20b01cfef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547041756 0 0) 0x7f20b013c120 con 0x7f20b0099090 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): my global_id is 124208 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209c0035d0 con 0x7f20b0099090 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950817221 0 0) 0x7f20b00a2c00 con 0x7f20b0097b60 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): my global_id is 104511 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b013c120 con 0x7f20b0097b60 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1247926375 0 0) 0x7f209c0035d0 con 0x7f20b0099090 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20b00a2c00 con 0x7f20b0099090 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20ac0045b0 con 0x7f20b0099090 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350529701 0 0) 0x7f20b013c120 con 0x7f20b0097b60 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f209c0035d0 con 0x7f20b0097b60 2024-09-16T06:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a8001830 con 0x7f20b0097b60 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): dump: 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 930492775 0 0) 0x7f20b00a2c00 con 0x7f20b0099090 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.189:6789/0 conn(0x7f20b0139500 legacy=0x7f20b01cfef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6789/0 conn(0x7f20b0097b60 legacy=0x7f20b006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 monclient: found mon.c 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b009a540 con 0x7f20b0099090 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 5 monclient: authenticate success, global_id 124208 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 10 monclient: _renew_subs 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b006f150 con 0x7f20b0099090 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 10 monclient: _renew_subs 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b006eb90 con 0x7f20b0099090 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20ac0047d0 con 0x7f20b0099090 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20ac004b60 con 0x7f20b0099090 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a77fe700 4 set_mon_vals no callback set 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: dump: 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2090006590 con 0x7f2090002830 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509352535 0 0) 0x7f2090006590 con 0x7f2090002830 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient(hunting): my global_id is 104514 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b00a2c00 con 0x7f2090002830 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f20ac004cc0 con 0x7f20b0099090 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(54..54 src has 1..54) v4 ==== 4305+0+0 (unknown 802108153 0 0) 0x7f20ac043780 con 0x7f20b0099090 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1607503366 0 0) 0x7f20b00a2c00 con 0x7f2090002830 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2090006590 con 0x7f2090002830 2024-09-16T06:40:45.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a8003860 con 0x7f2090002830 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 10 monclient: dump: 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20b5f6d700 10 monclient: get_auth_request con 0x7f209c0363d0 auth_method 0 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 398596926 0 0) 0x7f2090006590 con 0x7f2090002830 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.413+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f209000a190 con 0x7f2090002830 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.417+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f20a80039c0 con 0x7f2090002830 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.417+0000 7f20a5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.417+0000 7f20a5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.417+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6789/0 conn(0x7f2090002830 legacy=0x7f2090004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.417+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6789/0 conn(0x7f2090002830 legacy=0x7f2090004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2090009e30 con 0x7f2090009150 2024-09-16T06:40:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236856233 0 0) 0x7f2090009e30 con 0x7f2090009150 2024-09-16T06:40:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient(hunting): my global_id is 104517 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2090006590 con 0x7f2090009150 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 486165538 0 0) 0x7f2090006590 con 0x7f2090009150 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2090009e30 con 0x7f2090009150 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a8003f40 con 0x7f2090009150 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient: dump: 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2283073967 0 0) 0x7f2090009e30 con 0x7f2090009150 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f209000c340 con 0x7f2090009150 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 1480725510) 0x7f20a8004270 con 0x7f2090009150 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6789/0 conn(0x7f2090009150 legacy=0x7f20900095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20a5ffb700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6789/0 conn(0x7f2090009150 legacy=0x7f20900095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 32, 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:45.439 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.440 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.441 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:45.442 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.443 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:45.444 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:45.445 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.446 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.447 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.448 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:45.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:45.450 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:45.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.189:6812/3098810674 conn(0x7f209c0363d0 legacy=0x7f209c038890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 10 monclient: shutdown 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 >> v1:172.21.15.76:6790/0 conn(0x7f20b0099090 legacy=0x7f20b006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b5f6d700 1 -- 172.21.15.76:0/3176329261 reap_dead start 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b5f6d700 1 -- 172.21.15.76:0/3176329261 reap_dead start 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 shutdown_connections 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.425+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 >> 172.21.15.76:0/3176329261 conn(0x7f20b00a0030 msgr2=0x7f20b013ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.429+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 shutdown_connections 2024-09-16T06:40:45.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.429+0000 7f20b676e700 1 -- 172.21.15.76:0/3176329261 wait complete. 2024-09-16T06:40:45.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:40:45.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:40:45.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:40:45.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.569+0000 7fc63f33f700 10 monclient: get_monmap_and_config 2024-09-16T06:40:45.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.569+0000 7fc63f33f700 10 monclient: build_initial_monmap 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.569+0000 7fc63f33f700 10 monclient: monmap: 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:40:45.575095+0000 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:40:45.575095+0000 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:45.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:45.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.569+0000 7fc63f33f700 1 Processor -- start 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- start start 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: init 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-c con 0x7fc638097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-b con 0x7fc638099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-a con 0x7fc638139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc638098990 con 0x7fc638139500 2024-09-16T06:40:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc638097f40 con 0x7fc638097b60 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63809d610 con 0x7fc638099090 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient(hunting): authenticate will time out at 992.708576s 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63eb3e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc638139500 0x7fc63813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45390/0 (socket says 172.21.15.76:45390) 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63eb3e700 1 -- 172.21.15.76:0/2091927585 learned_addr learned my addr 172.21.15.76:0/2091927585 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63e33d700 1 -- 172.21.15.76:0/2091927585 >> v1:172.21.15.189:6789/0 conn(0x7fc638097b60 legacy=0x7fc63809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198495163 0 0) 0x7fc638098990 con 0x7fc638139500 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): my global_id is 104520 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6240035d0 con 0x7fc638139500 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041671792 0 0) 0x7fc63809d610 con 0x7fc638099090 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient(hunting): my global_id is 124211 2024-09-16T06:40:45.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc638098990 con 0x7fc638099090 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382657254 0 0) 0x7fc6240035d0 con 0x7fc638139500 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 >> v1:172.21.15.76:6790/0 conn(0x7fc638099090 legacy=0x7fc63809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 >> v1:172.21.15.189:6789/0 conn(0x7fc638097b60 legacy=0x7fc63809ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 monclient: found mon.noname-a 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380986b0 con 0x7fc638139500 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 5 monclient: authenticate success, global_id 104520 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc630001990 con 0x7fc638139500 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: dump: 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc630001c70 con 0x7fc638139500 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 1 -- 172.21.15.76:0/2091927585 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc630002000 con 0x7fc638139500 2024-09-16T06:40:45.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 4 set_mon_vals no callback set 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63d33b700 10 monclient: dump: 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.585 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: shutdown 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- 172.21.15.76:0/2091927585 >> v1:172.21.15.76:6789/0 conn(0x7fc638139500 legacy=0x7fc63813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- 172.21.15.76:0/2091927585 shutdown_connections 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- 172.21.15.76:0/2091927585 >> 172.21.15.76:0/2091927585 conn(0x7fc6380a0030 msgr2=0x7fc6380a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- 172.21.15.76:0/2091927585 shutdown_connections 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 1 -- 172.21.15.76:0/2091927585 wait complete. 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: build_initial_monmap 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.573+0000 7fc63f33f700 10 monclient: monmap: 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:45.586 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 Processor -- start 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- start start 2024-09-16T06:40:45.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: init 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-b con 0x7fc638097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-c con 0x7fc638099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient(hunting): picked mon.noname-a con 0x7fc638139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63806dff0 con 0x7fc638139500 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc63806d2d0 con 0x7fc638097b60 2024-09-16T06:40:45.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380a3040 con 0x7fc638099090 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63e33d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc638097b60 legacy=0x7fc63806c4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient(hunting): authenticate will time out at 992.713591s 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63eb3e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc638139500 0x7fc6381cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45398/0 (socket says 172.21.15.76:45398) 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63eb3e700 1 -- 172.21.15.76:0/560979283 learned_addr learned my addr 172.21.15.76:0/560979283 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041491393 0 0) 0x7fc6380a3040 con 0x7fc638099090 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): my global_id is 124214 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc61c0035d0 con 0x7fc638099090 2024-09-16T06:40:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018041968 0 0) 0x7fc63806dff0 con 0x7fc638139500 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): my global_id is 104523 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6380a3040 con 0x7fc638139500 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3466409236 0 0) 0x7fc61c0035d0 con 0x7fc638099090 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc63806dff0 con 0x7fc638099090 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc62c0018b0 con 0x7fc638099090 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): dump: 2024-09-16T06:40:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288443598 0 0) 0x7fc6380a3040 con 0x7fc638139500 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc61c0035d0 con 0x7fc638139500 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc630001950 con 0x7fc638139500 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient(hunting): dump: 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028541008 0 0) 0x7fc63806dff0 con 0x7fc638099090 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.189:6789/0 conn(0x7fc638097b60 legacy=0x7fc63806c4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6789/0 conn(0x7fc638139500 legacy=0x7fc6381cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 monclient: found mon.c 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380a2b10 con 0x7fc638099090 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: _finish_auth 0 2024-09-16T06:40:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 5 monclient: authenticate success, global_id 124214 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _renew_subs 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63809a5a0 con 0x7fc638099090 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _renew_subs 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63f33f700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc638070070 con 0x7fc638099090 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc62c0036c0 con 0x7fc638099090 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc62c0039d0 con 0x7fc638099090 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc62affd700 10 monclient: dump: 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.577+0000 7fc63cb3a700 4 set_mon_vals no callback set 2024-09-16T06:40:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc628ff9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc638071db0 con 0x7fc6380719d0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc62c00e440 con 0x7fc638099090 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(54..54 src has 1..54) v4 ==== 4305+0+0 (unknown 802108153 0 0) 0x7fc62c042120 con 0x7fc638099090 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002993253 0 0) 0x7fc638071db0 con 0x7fc6380719d0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient(hunting): my global_id is 124217 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63806dff0 con 0x7fc6380719d0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc63eb3e700 10 monclient: get_auth_request con 0x7fc61c0362c0 auth_method 0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2398627523 0 0) 0x7fc63806dff0 con 0x7fc6380719d0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc638071db0 con 0x7fc6380719d0 2024-09-16T06:40:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc634003f40 con 0x7fc6380719d0 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient: dump: 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4145365541 0 0) 0x7fc638071db0 con 0x7fc6380719d0 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc63809fb80 con 0x7fc6380719d0 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc634006350 con 0x7fc6380719d0 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6790/0 conn(0x7fc6380719d0 legacy=0x7fc638061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.581+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6790/0 conn(0x7fc6380719d0 legacy=0x7fc638061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:45.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc6380008d0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798729914 0 0) 0x7fc6380008d0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient(hunting): my global_id is 124220 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc638071db0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2571004624 0 0) 0x7fc638071db0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6380008d0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc6340047d0 con 0x7fc6381d4630 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:45.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient: dump: 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2003279722 0 0) 0x7fc6380008d0 con 0x7fc6381d4630 2024-09-16T06:40:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc638099500 con 0x7fc6381d4630 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 3851178641) 0x7fc634005c70 con 0x7fc6381d4630 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6790/0 conn(0x7fc6381d4630 legacy=0x7fc6381d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc62affd700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6790/0 conn(0x7fc6381d4630 legacy=0x7fc6381d6a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:40:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 32, 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:40:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:40:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:40:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:40:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:40:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:40:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:40:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:40:45.619 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:40:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc61c0362c0 legacy=0x7fc61c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 10 monclient: shutdown 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 >> v1:172.21.15.76:6790/0 conn(0x7fc638099090 legacy=0x7fc63806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc63eb3e700 1 -- 172.21.15.76:0/560979283 reap_dead start 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc63eb3e700 1 -- 172.21.15.76:0/560979283 reap_dead start 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 shutdown_connections 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 >> 172.21.15.76:0/560979283 conn(0x7fc6380a0030 msgr2=0x7fc6380a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 shutdown_connections 2024-09-16T06:40:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:40:45.593+0000 7fc628ff9700 1 -- 172.21.15.76:0/560979283 wait complete. 2024-09-16T06:40:45.622 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:40:45.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:40:46.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.069+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v55) v1 ==== 177+0+0 (unknown 459574504 0 0) 0x7f84e8012f70 con 0x7f84f009eba0 2024-09-16T06:40:46.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.069+0000 7f84e6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:46.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.069+0000 7f84e6ffd700 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-16T06:40:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84ef59e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84ef59e700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84ef59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f84d4005c10 con 0x7f84f009eba0 2024-09-16T06:40:46.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84e6ffd700 1 -- 172.21.15.189:0/1730856853 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v55) v1 ==== 184+0+0 (unknown 2382288781 0 0) 0x7f84e80146f0 con 0x7f84f009eba0 2024-09-16T06:40:46.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84e6ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T06:40:46.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.177+0000 7f84e6ffd700 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-16T06:40:46.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 >> v1:172.21.15.189:6812/3098810674 conn(0x7f84d0036270 legacy=0x7f84d0038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 10 monclient: shutdown 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 >> v1:172.21.15.76:6790/0 conn(0x7f84f009eba0 legacy=0x7f84f01d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 shutdown_connections 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 >> 172.21.15.189:0/1730856853 conn(0x7f84f009fe70 msgr2=0x7f84f009a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 shutdown_connections 2024-09-16T06:40:46.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.181+0000 7f84ef59e700 1 -- 172.21.15.189:0/1730856853 wait complete. 2024-09-16T06:40:46.187 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'ec-foo' already exists 2024-09-16T06:40:46.196 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 10 monclient: get_monmap_and_config 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 10 monclient: build_initial_monmap 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 10 monclient: monmap: 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:46.307122+0000 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:46.307122+0000 2024-09-16T06:40:46.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:46.307 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.307 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:46.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:46.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:46.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 1 Processor -- start 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 1 -- start start 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.301+0000 7fea530ff700 10 monclient: init 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-c con 0x7fea4c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-a con 0x7fea4c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:46.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-b con 0x7fea4c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c09cca0 con 0x7fea4c09c5f0 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c09f750 con 0x7fea4c09bb00 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c097620 con 0x7fea4c138da0 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient(hunting): authenticate will time out at 998.820757s 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea50e9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fea4c09bb00 legacy=0x7fea4c09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea5169c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fea4c138da0 0x7fea4c13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35170/0 (socket says 172.21.15.189:35170) 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea5169c700 1 -- 172.21.15.189:0/3700294910 learned_addr learned my addr 172.21.15.189:0/3700294910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362762723 0 0) 0x7fea4c097620 con 0x7fea4c138da0 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:46.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): my global_id is 124223 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c0035d0 con 0x7fea4c138da0 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625442892 0 0) 0x7fea4c09cca0 con 0x7fea4c09c5f0 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient(hunting): my global_id is 104526 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4c097620 con 0x7fea4c09c5f0 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032313998 0 0) 0x7fea3c0035d0 con 0x7fea4c138da0 2024-09-16T06:40:46.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 >> v1:172.21.15.189:6789/0 conn(0x7fea4c09bb00 legacy=0x7fea4c09bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 >> v1:172.21.15.76:6789/0 conn(0x7fea4c09c5f0 legacy=0x7fea4c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c097900 con 0x7fea4c138da0 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fea44001990 con 0x7fea4c138da0 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 5 monclient: authenticate success, global_id 124223 2024-09-16T06:40:46.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: dump: 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:46.316 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea44001bb0 con 0x7fea4c138da0 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 1 -- 172.21.15.189:0/3700294910 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fea44001f40 con 0x7fea4c138da0 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 4 set_mon_vals no callback set 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea4b7fe700 10 monclient: dump: 2024-09-16T06:40:46.317 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 10 monclient: shutdown 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:46.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- 172.21.15.189:0/3700294910 >> v1:172.21.15.76:6790/0 conn(0x7fea4c138da0 legacy=0x7fea4c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- 172.21.15.189:0/3700294910 shutdown_connections 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- 172.21.15.189:0/3700294910 >> 172.21.15.189:0/3700294910 conn(0x7fea4c0a03e0 msgr2=0x7fea4c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- 172.21.15.189:0/3700294910 shutdown_connections 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.305+0000 7fea530ff700 1 -- 172.21.15.189:0/3700294910 wait complete. 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: build_initial_monmap 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: monmap: 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:46.319 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 Processor -- start 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- start start 2024-09-16T06:40:46.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: init 2024-09-16T06:40:46.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-a con 0x7fea4c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-b con 0x7fea4c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient(hunting): picked mon.noname-c con 0x7fea4c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c132990 con 0x7fea4c09bb00 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c099500 con 0x7fea4c09c5f0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c09a6e0 con 0x7fea4c138da0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient(hunting): authenticate will time out at 998.825394s 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea4bfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fea4c09c5f0 legacy=0x7fea4c06f240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:46.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea5169c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fea4c138da0 0x7fea4c1cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35184/0 (socket says 172.21.15.189:35184) 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea5169c700 1 -- 172.21.15.189:0/3466914611 learned_addr learned my addr 172.21.15.189:0/3466914611 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925291785 0 0) 0x7fea4c09a6e0 con 0x7fea4c138da0 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): my global_id is 124226 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea300035d0 con 0x7fea4c138da0 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456613931 0 0) 0x7fea4c132990 con 0x7fea4c09bb00 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:46.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): my global_id is 104529 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4c09a6e0 con 0x7fea4c09bb00 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280397922 0 0) 0x7fea300035d0 con 0x7fea4c138da0 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea4c132990 con 0x7fea4c138da0 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fea44001f80 con 0x7fea4c138da0 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): dump: 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.327 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1516319312 0 0) 0x7fea4c09a6e0 con 0x7fea4c09bb00 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea300035d0 con 0x7fea4c09bb00 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fea40004170 con 0x7fea4c09bb00 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:46.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient(hunting): dump: 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1811274667 0 0) 0x7fea4c132990 con 0x7fea4c138da0 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 >> v1:172.21.15.189:6789/0 conn(0x7fea4c09c5f0 legacy=0x7fea4c06f240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 >> v1:172.21.15.76:6789/0 conn(0x7fea4c09bb00 legacy=0x7fea4c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 monclient: found mon.c 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c06f950 con 0x7fea4c138da0 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:46.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 5 monclient: authenticate success, global_id 124226 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _renew_subs 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c0a2de0 con 0x7fea4c138da0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _renew_subs 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c071570 con 0x7fea4c138da0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea44004ec0 con 0x7fea4c138da0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fea44005250 con 0x7fea4c138da0 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:46.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea4affd700 4 set_mon_vals no callback set 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.309+0000 7fea497fa700 10 monclient: dump: 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:46.332 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea530ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea530ff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea530ff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c005100 con 0x7fea4c138da0 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fea44005470 con 0x7fea4c138da0 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(55..55 src has 1..55) v4 ==== 4660+0+0 (unknown 1883007790 0 0) 0x7fea44043c20 con 0x7fea4c138da0 2024-09-16T06:40:46.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.313+0000 7fea50e9b700 10 monclient: get_auth_request con 0x7fea300362c0 auth_method 0 2024-09-16T06:40:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.321+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fea44014e40 con 0x7fea4c138da0 2024-09-16T06:40:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.321+0000 7fea497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:46.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.321+0000 7fea497fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.493+0000 7fea530ff700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.493+0000 7fea530ff700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.493+0000 7fea530ff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:46.493+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea2c005930 con 0x7fea4c138da0 2024-09-16T06:40:47.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.069+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v56) v1 ==== 154+0+0 (unknown 3521860228 0 0) 0x7fea440132b0 con 0x7fea4c138da0 2024-09-16T06:40:47.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.069+0000 7fea497fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:47.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.069+0000 7fea497fa700 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-16T06:40:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.181+0000 7fea530ff700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.181+0000 7fea530ff700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.181+0000 7fea530ff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.181+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea2c005420 con 0x7fea4c138da0 2024-09-16T06:40:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea497fa700 1 -- 172.21.15.189:0/3466914611 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v56) v1 ==== 161+0+0 (unknown 2470329468 0 0) 0x7fea44014a30 con 0x7fea4c138da0 2024-09-16T06:40:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea497fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea497fa700 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 >> v1:172.21.15.189:6812/3098810674 conn(0x7fea300362c0 legacy=0x7fea30038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 10 monclient: shutdown 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 >> v1:172.21.15.76:6790/0 conn(0x7fea4c138da0 legacy=0x7fea4c1cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 shutdown_connections 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 >> 172.21.15.189:0/3466914611 conn(0x7fea4c0a03e0 msgr2=0x7fea4c09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 shutdown_connections 2024-09-16T06:40:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.185+0000 7fea530ff700 1 -- 172.21.15.189:0/3466914611 wait complete. 2024-09-16T06:40:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'ec-foo' does not exist 2024-09-16T06:40:47.202 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:47.202 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.301+0000 7f51f4916700 10 monclient: get_monmap_and_config 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.301+0000 7f51f4916700 10 monclient: build_initial_monmap 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.301+0000 7f51f4916700 10 monclient: monmap: 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:47.308317+0000 2024-09-16T06:40:47.307 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:47.308317+0000 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:47.308 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 Processor -- start 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- start start 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient: init 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-c con 0x7f51f009ce10 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-b con 0x7f51f009d1f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-a con 0x7f51f0071ee0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f0097fb0 con 0x7f51f0071ee0 2024-09-16T06:40:47.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f009d670 con 0x7f51f009ce10 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f0099690 con 0x7f51f009d1f0 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient(hunting): authenticate will time out at 999.820987s 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51eed9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f51f009ce10 legacy=0x7f51f0097190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51ee59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f51f009d1f0 0x7f51f00978a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35200/0 (socket says 172.21.15.189:35200) 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51ee59c700 1 -- 172.21.15.189:0/1230097368 learned_addr learned my addr 172.21.15.189:0/1230097368 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214270103 0 0) 0x7f51f0099690 con 0x7f51f009d1f0 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): my global_id is 124229 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51dc0035d0 con 0x7f51f009d1f0 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744020908 0 0) 0x7f51f0097fb0 con 0x7f51f0071ee0 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient(hunting): my global_id is 104532 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f0099690 con 0x7f51f0071ee0 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 385626436 0 0) 0x7f51dc0035d0 con 0x7f51f009d1f0 2024-09-16T06:40:47.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 >> v1:172.21.15.189:6789/0 conn(0x7f51f009ce10 legacy=0x7f51f0097190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 >> v1:172.21.15.76:6789/0 conn(0x7f51f0071ee0 legacy=0x7f51f01423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 monclient: found mon.noname-b 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0098660 con 0x7f51f009d1f0 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:47.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 5 monclient: authenticate success, global_id 124229 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51d8001610 con 0x7f51f009d1f0 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: dump: 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.316 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51d8001880 con 0x7f51f009d1f0 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 1 -- 172.21.15.189:0/1230097368 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51d8001c10 con 0x7f51f009d1f0 2024-09-16T06:40:47.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 4 set_mon_vals no callback set 2024-09-16T06:40:47.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:47.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51edd9b700 10 monclient: dump: 2024-09-16T06:40:47.318 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.319 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 10 monclient: shutdown 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- 172.21.15.189:0/1230097368 >> v1:172.21.15.76:6790/0 conn(0x7f51f009d1f0 legacy=0x7f51f00978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- 172.21.15.189:0/1230097368 shutdown_connections 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- 172.21.15.189:0/1230097368 >> 172.21.15.189:0/1230097368 conn(0x7f51f009fe70 msgr2=0x7f51f00a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:47.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- 172.21.15.189:0/1230097368 shutdown_connections 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.305+0000 7f51f4916700 1 -- 172.21.15.189:0/1230097368 wait complete. 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: build_initial_monmap 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: monmap: 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:47.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 Processor -- start 2024-09-16T06:40:47.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- start start 2024-09-16T06:40:47.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: init 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-a con 0x7f51f0071ee0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-c con 0x7f51f009ce10 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient(hunting): picked mon.noname-b con 0x7f51f009d1f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f01ad380 con 0x7f51f0071ee0 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f009a380 con 0x7f51f009d1f0 2024-09-16T06:40:47.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00a28e0 con 0x7f51f009ce10 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient(hunting): authenticate will time out at 999.824950s 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51ef59e700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f51f009d1f0 legacy=0x7f51f01d67c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51eed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f51f0071ee0 0x7f51f01c9dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42150/0 (socket says 172.21.15.189:42150) 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51eed9d700 1 -- 172.21.15.189:0/2230374726 learned_addr learned my addr 172.21.15.189:0/2230374726 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51ee59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f51f009ce10 0x7f51f01d3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35214/0 (socket says 172.21.15.189:35214) 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90898158 0 0) 0x7f51f01ad380 con 0x7f51f0071ee0 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): my global_id is 104535 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d00035d0 con 0x7f51f0071ee0 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550833864 0 0) 0x7f51f00a28e0 con 0x7f51f009ce10 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): my global_id is 124232 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f01ad380 con 0x7f51f009ce10 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568414625 0 0) 0x7f51d00035d0 con 0x7f51f0071ee0 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51f00a28e0 con 0x7f51f0071ee0 2024-09-16T06:40:47.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886267190 0 0) 0x7f51f01ad380 con 0x7f51f009ce10 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51d00035d0 con 0x7f51f009ce10 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51e0004200 con 0x7f51f0071ee0 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): dump: 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.328 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51d80015d0 con 0x7f51f009ce10 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:47.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient(hunting): dump: 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.330 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.331 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3492474841 0 0) 0x7f51d00035d0 con 0x7f51f009ce10 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 >> v1:172.21.15.189:6789/0 conn(0x7f51f009d1f0 legacy=0x7f51f01d67c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 >> v1:172.21.15.76:6789/0 conn(0x7f51f0071ee0 legacy=0x7f51f01c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 monclient: found mon.c 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f01aee70 con 0x7f51f009ce10 2024-09-16T06:40:47.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 5 monclient: authenticate success, global_id 124232 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _renew_subs 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f01ca4e0 con 0x7f51f009ce10 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _renew_subs 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01ae170 con 0x7f51f009ce10 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51d8004bb0 con 0x7f51f009ce10 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51d8004ec0 con 0x7f51f009ce10 2024-09-16T06:40:47.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51ecd99700 4 set_mon_vals no callback set 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.309+0000 7f51e77fe700 10 monclient: dump: 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.334 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51f4916700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51f4916700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51f4916700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d4005100 con 0x7f51f009ce10 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f51d80050e0 con 0x7f51f009ce10 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(56..56 src has 1..56) v4 ==== 4305+0+0 (unknown 1881620634 0 0) 0x7f51d80435b0 con 0x7f51f009ce10 2024-09-16T06:40:47.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.313+0000 7f51eed9d700 10 monclient: get_auth_request con 0x7f51d00362e0 auth_method 0 2024-09-16T06:40:47.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.325+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f51d80148d0 con 0x7f51f009ce10 2024-09-16T06:40:47.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.325+0000 7f51e77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.325+0000 7f51e77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51f4916700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51f4916700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51f4916700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f51d4005930 con 0x7f51f009ce10 2024-09-16T06:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+60 (unknown 1737862046 0 1596542489) 0x7f51d801ce60 con 0x7f51f009ce10 2024-09-16T06:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51e77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.497+0000 7f51e77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51f4916700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51f4916700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51f4916700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f51d4005420 con 0x7f51f009ce10 2024-09-16T06:40:47.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51e77fe700 1 -- 172.21.15.189:0/2230374726 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v56) v1 ==== 65+0+60 (unknown 1737862046 0 1596542489) 0x7f51d8012d40 con 0x7f51f009ce10 2024-09-16T06:40:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51e77fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:47.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.601+0000 7f51e77fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 >> v1:172.21.15.189:6812/3098810674 conn(0x7f51d00362e0 legacy=0x7f51d00387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 10 monclient: shutdown 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 >> v1:172.21.15.76:6790/0 conn(0x7f51f009ce10 legacy=0x7f51f01d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 shutdown_connections 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 >> 172.21.15.189:0/2230374726 conn(0x7f51f009fe70 msgr2=0x7f51f009a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 shutdown_connections 2024-09-16T06:40:47.608 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.605+0000 7f51f4916700 1 -- 172.21.15.189:0/2230374726 wait complete. 2024-09-16T06:40:47.618 INFO:tasks.workunit.client.0.smithi189.stdout:foo 2024-09-16T06:40:47.618 INFO:tasks.workunit.client.0.smithi189.stdout:foo-ssd 2024-09-16T06:40:47.618 INFO:tasks.workunit.client.0.smithi189.stdout:foo-hdd 2024-09-16T06:40:47.618 INFO:tasks.workunit.client.0.smithi189.stdout:ec-foo 2024-09-16T06:40:47.619 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.721+0000 7f11644e3700 10 monclient: get_monmap_and_config 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.721+0000 7f11644e3700 10 monclient: build_initial_monmap 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.721+0000 7f11644e3700 10 monclient: monmap: 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:47.728220+0000 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:47.728220+0000 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.727 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 1 Processor -- start 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 1 -- start start 2024-09-16T06:40:47.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient: init 2024-09-16T06:40:47.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:47.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-c con 0x7f115c0a1280 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-a con 0x7f115c09de70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-b con 0x7f115c09fe50 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c0a1660 con 0x7f115c09de70 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c131b60 con 0x7f115c0a1280 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c131de0 con 0x7f115c09fe50 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116227f700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f115c0a1280 legacy=0x7f115c09d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f11644e3700 10 monclient(hunting): authenticate will time out at 1000.242526s 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f1162a80700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f115c09fe50 0x7f115c13a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35228/0 (socket says 172.21.15.189:35228) 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f1162a80700 1 -- 172.21.15.189:0/3726378501 learned_addr learned my addr 172.21.15.189:0/3726378501 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972831867 0 0) 0x7f115c131de0 con 0x7f115c09fe50 2024-09-16T06:40:47.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): my global_id is 124235 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11480035d0 con 0x7f115c09fe50 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156290711 0 0) 0x7f115c0a1660 con 0x7f115c09de70 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient(hunting): my global_id is 104538 2024-09-16T06:40:47.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f115c131de0 con 0x7f115c09de70 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3581814311 0 0) 0x7f11480035d0 con 0x7f115c09fe50 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 >> v1:172.21.15.189:6789/0 conn(0x7f115c0a1280 legacy=0x7f115c09d760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 >> v1:172.21.15.76:6789/0 conn(0x7f115c09de70 legacy=0x7f115c1369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient: _finish_hunting 0 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 monclient: found mon.noname-b 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c09b490 con 0x7f115c09fe50 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient: _finish_auth 0 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 10 monclient: _check_auth_tickets 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.725+0000 7f116127d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 5 monclient: authenticate success, global_id 124235 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11540019d0 con 0x7f115c09fe50 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: dump: 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1154001bf0 con 0x7f115c09fe50 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 1 -- 172.21.15.189:0/3726378501 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1154001f80 con 0x7f115c09fe50 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 4 set_mon_vals no callback set 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116127d700 10 monclient: dump: 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: shutdown 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:47.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- 172.21.15.189:0/3726378501 >> v1:172.21.15.76:6790/0 conn(0x7f115c09fe50 legacy=0x7f115c13a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- 172.21.15.189:0/3726378501 shutdown_connections 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- 172.21.15.189:0/3726378501 >> 172.21.15.189:0/3726378501 conn(0x7f115c0a0760 msgr2=0x7f115c0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- 172.21.15.189:0/3726378501 shutdown_connections 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- 172.21.15.189:0/3726378501 wait complete. 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: build_initial_monmap 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: monmap: 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:47.742 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 Processor -- start 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- start start 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: init 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-c con 0x7f115c09de70 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-b con 0x7f115c09fe50 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient(hunting): picked mon.noname-a con 0x7f115c0a1280 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c138090 con 0x7f115c0a1280 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c136ab0 con 0x7f115c09fe50 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c137db0 con 0x7f115c09de70 2024-09-16T06:40:47.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f11644e3700 10 monclient(hunting): authenticate will time out at 1000.247380s 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f1161a7e700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f115c09fe50 legacy=0x7f115c1cccc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116227f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f115c09de70 0x7f115c13a5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35236/0 (socket says 172.21.15.189:35236) 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.729+0000 7f116227f700 1 -- 172.21.15.189:0/3964960611 learned_addr learned my addr 172.21.15.189:0/3964960611 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766001338 0 0) 0x7f115c137db0 con 0x7f115c09de70 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): my global_id is 124238 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11400035d0 con 0x7f115c09de70 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904371850 0 0) 0x7f115c138090 con 0x7f115c0a1280 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:47.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): my global_id is 104541 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f115c137db0 con 0x7f115c0a1280 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1895771580 0 0) 0x7f11400035d0 con 0x7f115c09de70 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f115c138090 con 0x7f115c09de70 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1158004200 con 0x7f115c09de70 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.745 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088285724 0 0) 0x7f115c137db0 con 0x7f115c0a1280 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11400035d0 con 0x7f115c0a1280 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11540018d0 con 0x7f115c0a1280 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:47.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1205032850 0 0) 0x7f115c138090 con 0x7f115c09de70 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 >> v1:172.21.15.189:6789/0 conn(0x7f115c09fe50 legacy=0x7f115c1cccc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 >> v1:172.21.15.76:6789/0 conn(0x7f115c0a1280 legacy=0x7f115c1d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:47.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 monclient: found mon.c 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c1377f0 con 0x7f115c09de70 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 5 monclient: authenticate success, global_id 124238 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _renew_subs 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c137ad0 con 0x7f115c09de70 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _renew_subs 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c13c330 con 0x7f115c09de70 2024-09-16T06:40:47.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1158004470 con 0x7f115c09de70 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1158004800 con 0x7f115c09de70 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1160a7c700 4 set_mon_vals no callback set 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 10 monclient: dump: 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:47.749 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144005100 con 0x7f115c09de70 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1158004a20 con 0x7f115c09de70 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(56..56 src has 1..56) v4 ==== 4305+0+0 (unknown 1881620634 0 0) 0x7f1158043560 con 0x7f115c09de70 2024-09-16T06:40:47.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.733+0000 7f1161a7e700 10 monclient: get_auth_request con 0x7f11400362c0 auth_method 0 2024-09-16T06:40:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.741+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f11580147c0 con 0x7f115c09de70 2024-09-16T06:40:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.741+0000 7f1152ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:47.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.741+0000 7f1152ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:47.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.917+0000 7f11644e3700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:47.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.917+0000 7f11644e3700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:47.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.917+0000 7f11644e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:47.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:47.917+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1144005930 con 0x7f115c09de70 2024-09-16T06:40:48.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.085+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v57) v1 ==== 110+0+0 (unknown 3820101190 0 0) 0x7f115801ce10 con 0x7f115c09de70 2024-09-16T06:40:48.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.085+0000 7f1152ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.085+0000 7f1152ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:48.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.197+0000 7f11644e3700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.197+0000 7f11644e3700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.197+0000 7f11644e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.197+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f11440053b0 con 0x7f115c09de70 2024-09-16T06:40:48.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f1152ffd700 1 -- 172.21.15.189:0/3964960611 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v57) v1 ==== 139+0+0 (unknown 2153324670 0 0) 0x7f1158012c30 con 0x7f115c09de70 2024-09-16T06:40:48.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f1152ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f1152ffd700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T06:40:48.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 >> v1:172.21.15.189:6812/3098810674 conn(0x7f11400362c0 legacy=0x7f1140038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 10 monclient: shutdown 2024-09-16T06:40:48.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:48.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 >> v1:172.21.15.76:6790/0 conn(0x7f115c09de70 legacy=0x7f115c13a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 shutdown_connections 2024-09-16T06:40:48.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 >> 172.21.15.189:0/3964960611 conn(0x7f115c0a0760 msgr2=0x7f115c09c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:48.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 shutdown_connections 2024-09-16T06:40:48.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.201+0000 7f11644e3700 1 -- 172.21.15.189:0/3964960611 wait complete. 2024-09-16T06:40:48.207 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo-asdf' 2024-09-16T06:40:48.216 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: get_monmap_and_config 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: build_initial_monmap 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: monmap: 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:48.326442+0000 2024-09-16T06:40:48.325 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:48.326442+0000 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:48.326 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 1 Processor -- start 2024-09-16T06:40:48.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 1 -- start start 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: init 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-b con 0x7fe1600a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-a con 0x7fe16009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.321+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-c con 0x7fe1600a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16009a040 con 0x7fe16009aa20 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15ed9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe1600a3ca0 legacy=0x7fe16013dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600a33d0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160099d60 con 0x7fe1600a2350 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:48.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient(hunting): authenticate will time out at 1000.840087s 2024-09-16T06:40:48.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe155d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe16009aa20 0x7fe16013a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42170/0 (socket says 172.21.15.189:42170) 2024-09-16T06:40:48.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe155d9b700 1 -- 172.21.15.189:0/2534452619 learned_addr learned my addr 172.21.15.189:0/2534452619 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:48.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15e59c700 1 --1- 172.21.15.189:0/2534452619 >> v1:172.21.15.76:6790/0 conn(0x7fe1600a2350 0x7fe1600a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35252/0 (socket says 172.21.15.189:35252) 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850406240 0 0) 0x7fe160099d60 con 0x7fe1600a2350 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): my global_id is 124241 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c0035d0 con 0x7fe1600a2350 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841079440 0 0) 0x7fe16009a040 con 0x7fe16009aa20 2024-09-16T06:40:48.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient(hunting): my global_id is 104544 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160099d60 con 0x7fe16009aa20 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166328807 0 0) 0x7fe160099d60 con 0x7fe16009aa20 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 >> v1:172.21.15.76:6790/0 conn(0x7fe1600a2350 legacy=0x7fe1600a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 >> v1:172.21.15.189:6789/0 conn(0x7fe1600a3ca0 legacy=0x7fe16013dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 monclient: found mon.noname-a 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600a3920 con 0x7fe16009aa20 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:40:48.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe148001570 con 0x7fe16009aa20 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 5 monclient: authenticate success, global_id 104544 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: dump: 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.335 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe148001790 con 0x7fe16009aa20 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 1 -- 172.21.15.189:0/2534452619 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe148001ba0 con 0x7fe16009aa20 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 4 set_mon_vals no callback set 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe15dd9b700 10 monclient: dump: 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.336 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient: shutdown 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- 172.21.15.189:0/2534452619 >> v1:172.21.15.76:6789/0 conn(0x7fe16009aa20 legacy=0x7fe16013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- 172.21.15.189:0/2534452619 shutdown_connections 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- 172.21.15.189:0/2534452619 >> 172.21.15.189:0/2534452619 conn(0x7fe1600a7d70 msgr2=0x7fe1600a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- 172.21.15.189:0/2534452619 shutdown_connections 2024-09-16T06:40:48.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 1 -- 172.21.15.189:0/2534452619 wait complete. 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient: build_initial_monmap 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.325+0000 7fe16492c700 10 monclient: monmap: 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 Processor -- start 2024-09-16T06:40:48.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- start start 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: init 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-c con 0x7fe16009aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-b con 0x7fe1600a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient(hunting): picked mon.noname-a con 0x7fe1600a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160121ba0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1601206f0 con 0x7fe1600a2350 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1601351a0 con 0x7fe16009aa20 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient(hunting): authenticate will time out at 1000.845389s 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe155d9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe1600a2350 legacy=0x7fe1601344c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe15ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe1600a3ca0 0x7fe1601d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42186/0 (socket says 172.21.15.189:42186) 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe15ed9d700 1 -- 172.21.15.189:0/454166104 learned_addr learned my addr 172.21.15.189:0/454166104 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:48.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe15e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe16009aa20 0x7fe1601c9980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35254/0 (socket says 172.21.15.189:35254) 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971872082 0 0) 0x7fe160121ba0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): my global_id is 104547 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1400035d0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463381200 0 0) 0x7fe1601351a0 con 0x7fe16009aa20 2024-09-16T06:40:48.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): my global_id is 124244 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160121ba0 con 0x7fe16009aa20 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926899186 0 0) 0x7fe1400035d0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1601351a0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe158004460 con 0x7fe1600a3ca0 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): dump: 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622270217 0 0) 0x7fe160121ba0 con 0x7fe16009aa20 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1400035d0 con 0x7fe16009aa20 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe150001920 con 0x7fe16009aa20 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:48.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient(hunting): dump: 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3315867014 0 0) 0x7fe1601351a0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 >> v1:172.21.15.76:6790/0 conn(0x7fe16009aa20 legacy=0x7fe1601c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 >> v1:172.21.15.189:6789/0 conn(0x7fe1600a2350 legacy=0x7fe1601344c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 monclient: found mon.a 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1601218c0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 5 monclient: authenticate success, global_id 104547 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _renew_subs 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe160120470 con 0x7fe1600a3ca0 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _renew_subs 2024-09-16T06:40:48.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe160134bd0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe158004680 con 0x7fe1600a3ca0 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe158004990 con 0x7fe1600a3ca0 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe1577fe700 10 monclient: dump: 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.348 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.329+0000 7fe15d59a700 4 set_mon_vals no callback set 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe158004b70 con 0x7fe1600a3ca0 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe16492c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe16492c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe16492c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe144005100 con 0x7fe1600a3ca0 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(57..57 src has 1..57) v4 ==== 4310+0+0 (unknown 4094529160 0 0) 0x7fe158043750 con 0x7fe1600a3ca0 2024-09-16T06:40:48.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.333+0000 7fe15e59c700 10 monclient: get_auth_request con 0x7fe140036310 auth_method 0 2024-09-16T06:40:48.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.341+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe1580053c0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.341+0000 7fe1577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.341+0000 7fe1577fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe16492c700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe16492c700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe16492c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fe144005930 con 0x7fe1600a3ca0 2024-09-16T06:40:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v57) v1 ==== 139+0+0 (unknown 2153324670 0 0) 0x7fe158012e20 con 0x7fe1600a3ca0 2024-09-16T06:40:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe1577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.521+0000 7fe1577fe700 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-16T06:40:48.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fe1440053f0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe1577fe700 1 -- 172.21.15.189:0/454166104 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v57) v1 ==== 139+0+0 (unknown 2153324670 0 0) 0x7fe1580145a0 con 0x7fe1600a3ca0 2024-09-16T06:40:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe1577fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T06:40:48.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe1577fe700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe140036310 legacy=0x7fe1400387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 10 monclient: shutdown 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 >> v1:172.21.15.76:6789/0 conn(0x7fe1600a3ca0 legacy=0x7fe1601d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 shutdown_connections 2024-09-16T06:40:48.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.625+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 >> 172.21.15.189:0/454166104 conn(0x7fe1600a7d70 msgr2=0x7fe1600a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.629+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 shutdown_connections 2024-09-16T06:40:48.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.629+0000 7fe16492c700 1 -- 172.21.15.189:0/454166104 wait complete. 2024-09-16T06:40:48.633 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo-asdf' 2024-09-16T06:40:48.642 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-16T06:40:48.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.733+0000 7fa9058bb700 10 monclient: get_monmap_and_config 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.733+0000 7fa9058bb700 10 monclient: build_initial_monmap 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.733+0000 7fa9058bb700 10 monclient: monmap: 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:48.739733+0000 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:48.739733+0000 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:48.739 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:48.740 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:48.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 1 Processor -- start 2024-09-16T06:40:48.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 1 -- start start 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient: init 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-b con 0x7fa900071370 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-a con 0x7fa900097d90 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-c con 0x7fa90013b8d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa900097690 con 0x7fa900097d90 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa900070db0 con 0x7fa90013b8d0 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9000982b0 con 0x7fa900071370 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa9058bb700 10 monclient(hunting): authenticate will time out at 1001.253094s 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa8ff7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa90013b8d0 legacy=0x7fa90013dcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:48.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa8fe7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa900097d90 0x7fa9000706a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46088/0 (socket says 172.21.15.189:46088) 2024-09-16T06:40:48.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa8fe7fc700 1 -- 172.21.15.189:0/1892996028 learned_addr learned my addr 172.21.15.189:0/1892996028 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:48.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.737+0000 7fa8feffd700 1 --1- 172.21.15.189:0/1892996028 >> v1:172.21.15.76:6790/0 conn(0x7fa900071370 0x7fa90006ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48044/0 (socket says 172.21.15.189:48044) 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278776460 0 0) 0x7fa900097690 con 0x7fa900097d90 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient(hunting): my global_id is 104550 2024-09-16T06:40:48.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e80035d0 con 0x7fa900097d90 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095478122 0 0) 0x7fa8e80035d0 con 0x7fa900097d90 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 >> v1:172.21.15.76:6790/0 conn(0x7fa900071370 legacy=0x7fa90006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 >> v1:172.21.15.189:6789/0 conn(0x7fa90013b8d0 legacy=0x7fa90013dcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 monclient: found mon.noname-a 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:48.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90009f6e0 con 0x7fa900097d90 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 5 monclient: authenticate success, global_id 104550 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa8f4001610 con 0x7fa900097d90 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: dump: 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f4001930 con 0x7fa900097d90 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 1 -- 172.21.15.189:0/1892996028 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa8f4001cc0 con 0x7fa900097d90 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 4 set_mon_vals no callback set 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa8fdffb700 10 monclient: dump: 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 10 monclient: shutdown 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 1 -- 172.21.15.189:0/1892996028 >> v1:172.21.15.76:6789/0 conn(0x7fa900097d90 legacy=0x7fa9000706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 1 -- 172.21.15.189:0/1892996028 shutdown_connections 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 1 -- 172.21.15.189:0/1892996028 >> 172.21.15.189:0/1892996028 conn(0x7fa9000a03e0 msgr2=0x7fa9000a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 1 -- 172.21.15.189:0/1892996028 shutdown_connections 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.741+0000 7fa9058bb700 1 -- 172.21.15.189:0/1892996028 wait complete. 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: build_initial_monmap 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: monmap: 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:48.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 Processor -- start 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- start start 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: init 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-b con 0x7fa900071370 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-a con 0x7fa900097d90 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient(hunting): picked mon.noname-c con 0x7fa90013b8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:48.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa900096940 con 0x7fa900097d90 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90009a6e0 con 0x7fa900071370 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90009d7b0 con 0x7fa90013b8d0 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient(hunting): authenticate will time out at 1001.261554s 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8feffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa900071370 legacy=0x7fa9001358f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8fe7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa900097d90 0x7fa9001cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46102/0 (socket says 172.21.15.189:46102) 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8fe7fc700 1 -- 172.21.15.189:0/2586074677 learned_addr learned my addr 172.21.15.189:0/2586074677 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:48.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8ff7fe700 1 --1- 172.21.15.189:0/2586074677 >> v1:172.21.15.76:6790/0 conn(0x7fa90013b8d0 0x7fa9001d2df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48048/0 (socket says 172.21.15.189:48048) 2024-09-16T06:40:48.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286260121 0 0) 0x7fa90009d7b0 con 0x7fa90013b8d0 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): my global_id is 124247 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec0035d0 con 0x7fa90013b8d0 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826763508 0 0) 0x7fa900096940 con 0x7fa900097d90 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): my global_id is 104553 2024-09-16T06:40:48.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90009d7b0 con 0x7fa900097d90 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415467070 0 0) 0x7fa8ec0035d0 con 0x7fa90013b8d0 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa900096940 con 0x7fa90013b8d0 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa8f80043e0 con 0x7fa90013b8d0 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): dump: 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926571058 0 0) 0x7fa90009d7b0 con 0x7fa900097d90 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8ec0035d0 con 0x7fa900097d90 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa8f40015d0 con 0x7fa900097d90 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient(hunting): dump: 2024-09-16T06:40:48.761 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259117367 0 0) 0x7fa900096940 con 0x7fa90013b8d0 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 >> v1:172.21.15.189:6789/0 conn(0x7fa900071370 legacy=0x7fa9001358f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 >> v1:172.21.15.76:6789/0 conn(0x7fa900097d90 legacy=0x7fa9001cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:48.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 monclient: found mon.c 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90009d2f0 con 0x7fa90013b8d0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 5 monclient: authenticate success, global_id 124247 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _renew_subs 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90008c0c0 con 0x7fa90013b8d0 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _renew_subs 2024-09-16T06:40:48.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90008b160 con 0x7fa90013b8d0 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8f8004600 con 0x7fa90013b8d0 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa8f8004910 con 0x7fa90013b8d0 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8fd7fa700 4 set_mon_vals no callback set 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.745+0000 7fa8e7fff700 10 monclient: dump: 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:48.764 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa9058bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa9058bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa9058bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0005100 con 0x7fa90013b8d0 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa8f8004af0 con 0x7fa90013b8d0 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(57..57 src has 1..57) v4 ==== 4310+0+0 (unknown 4094529160 0 0) 0x7fa8f80436a0 con 0x7fa90013b8d0 2024-09-16T06:40:48.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.749+0000 7fa8feffd700 10 monclient: get_auth_request con 0x7fa8ec0362a0 auth_method 0 2024-09-16T06:40:48.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.757+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa8f8005310 con 0x7fa90013b8d0 2024-09-16T06:40:48.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.757+0000 7fa8e7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:48.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.757+0000 7fa8e7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:48.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.917+0000 7fa9058bb700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:48.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.917+0000 7fa9058bb700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:48.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.917+0000 7fa9058bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:48.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:48.917+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fa8e0005930 con 0x7fa90013b8d0 2024-09-16T06:40:49.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.745+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v58) v1 ==== 110+0+0 (unknown 2250587578 0 0) 0x7fa8f8012d70 con 0x7fa90013b8d0 2024-09-16T06:40:49.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.745+0000 7fa8e7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:49.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.745+0000 7fa8e7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:49.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa9058bb700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:49.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa9058bb700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:49.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa9058bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:49.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fa8e00053b0 con 0x7fa90013b8d0 2024-09-16T06:40:49.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa8e7fff700 1 -- 172.21.15.189:0/2586074677 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v58) v1 ==== 139+0+0 (unknown 1949492088 0 0) 0x7fa8f80144f0 con 0x7fa90013b8d0 2024-09-16T06:40:49.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa8e7fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T06:40:49.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.853+0000 7fa8e7fff700 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-16T06:40:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa8ec0362a0 legacy=0x7fa8ec038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:49.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 10 monclient: shutdown 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 >> v1:172.21.15.76:6790/0 conn(0x7fa90013b8d0 legacy=0x7fa9001d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 shutdown_connections 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 >> 172.21.15.189:0/2586074677 conn(0x7fa9000a03e0 msgr2=0x7fa90009c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 shutdown_connections 2024-09-16T06:40:49.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.857+0000 7fa9058bb700 1 -- 172.21.15.189:0/2586074677 wait complete. 2024-09-16T06:40:49.863 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'bar-asdf' 2024-09-16T06:40:49.872 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:49.872 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-asdf 2024-09-16T06:40:49.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: get_monmap_and_config 2024-09-16T06:40:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: build_initial_monmap 2024-09-16T06:40:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: monmap: 2024-09-16T06:40:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:49.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:49.980612+0000 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:49.980612+0000 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:49.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:49.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:49.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:49.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 1 Processor -- start 2024-09-16T06:40:49.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 1 -- start start 2024-09-16T06:40:49.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: init 2024-09-16T06:40:49.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:49.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-c con 0x7f94f40a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-a con 0x7f94f409aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-b con 0x7f94f40a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f409a040 con 0x7f94f409aa20 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40a33d0 con 0x7f94f40a2350 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f4099d60 con 0x7f94f40a3ca0 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient(hunting): authenticate will time out at 1002.493697s 2024-09-16T06:40:49.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f3fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f94f40a2350 legacy=0x7f94f40a2730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f37fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f94f409aa20 0x7f94f413a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46114/0 (socket says 172.21.15.189:46114) 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f37fe700 1 -- 172.21.15.189:0/533681732 learned_addr learned my addr 172.21.15.189:0/533681732 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120233934 0 0) 0x7f94f409a040 con 0x7f94f409aa20 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:49.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): my global_id is 104556 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e00035d0 con 0x7f94f409aa20 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963126419 0 0) 0x7f94f4099d60 con 0x7f94f40a3ca0 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient(hunting): my global_id is 124250 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f409a040 con 0x7f94f40a3ca0 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1900522697 0 0) 0x7f94e00035d0 con 0x7f94f409aa20 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 >> v1:172.21.15.76:6790/0 conn(0x7f94f40a3ca0 legacy=0x7f94f413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 >> v1:172.21.15.189:6789/0 conn(0x7f94f40a2350 legacy=0x7f94f40a2730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:49.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40a3920 con 0x7f94f409aa20 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f94e4001610 con 0x7f94f409aa20 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 5 monclient: authenticate success, global_id 104556 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: dump: 2024-09-16T06:40:49.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001830 con 0x7f94f409aa20 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:49.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94fa7d7700 4 set_mon_vals no callback set 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 1 -- 172.21.15.189:0/533681732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f94e4001bc0 con 0x7f94f409aa20 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.977+0000 7f94f2ffd700 10 monclient: dump: 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:49.999 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: shutdown 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/533681732 >> v1:172.21.15.76:6789/0 conn(0x7f94f409aa20 legacy=0x7f94f413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/533681732 shutdown_connections 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/533681732 >> 172.21.15.189:0/533681732 conn(0x7f94f40a7d70 msgr2=0x7f94f40a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/533681732 shutdown_connections 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/533681732 wait complete. 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: build_initial_monmap 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: monmap: 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:50.000 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 Processor -- start 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- start start 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: init 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:50.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-b con 0x7f94f409aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-c con 0x7f94f40a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient(hunting): picked mon.noname-a con 0x7f94f40a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f406bed0 con 0x7f94f40a3ca0 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f4120b80 con 0x7f94f409aa20 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f3fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f94f409aa20 legacy=0x7f94f4135520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f409b5c0 con 0x7f94f40a2350 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient(hunting): authenticate will time out at 1002.498030s 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f94f40a2350 0x7f94f41cecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48058/0 (socket says 172.21.15.189:48058) 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f37fe700 1 -- 172.21.15.189:0/503787181 learned_addr learned my addr 172.21.15.189:0/503787181 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417373772 0 0) 0x7f94f409b5c0 con 0x7f94f40a2350 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): my global_id is 124253 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d40035d0 con 0x7f94f40a2350 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598686400 0 0) 0x7f94f406bed0 con 0x7f94f40a3ca0 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): my global_id is 104559 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f409b5c0 con 0x7f94f40a3ca0 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2890466594 0 0) 0x7f94d40035d0 con 0x7f94f40a2350 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94f406bed0 con 0x7f94f40a2350 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f94e40015d0 con 0x7f94f40a2350 2024-09-16T06:40:50.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129594111 0 0) 0x7f94f409b5c0 con 0x7f94f40a3ca0 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94d40035d0 con 0x7f94f40a3ca0 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f94ec001b60 con 0x7f94f40a3ca0 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.005 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2759563737 0 0) 0x7f94f406bed0 con 0x7f94f40a2350 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 >> v1:172.21.15.189:6789/0 conn(0x7f94f409aa20 legacy=0x7f94f4135520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 >> v1:172.21.15.76:6789/0 conn(0x7f94f40a3ca0 legacy=0x7f94f41d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 monclient: found mon.c 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4122c00 con 0x7f94f40a2350 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:50.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94f0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 5 monclient: authenticate success, global_id 124253 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _renew_subs 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f4122ee0 con 0x7f94f40a2350 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _renew_subs 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.981+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f4135c30 con 0x7f94f40a2350 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e40012a0 con 0x7f94f40a2350 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f94e4004de0 con 0x7f94f40a2350 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:50.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 10 monclient: dump: 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f27fc700 4 set_mon_vals no callback set 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f94e400f910 con 0x7f94f40a2350 2024-09-16T06:40:50.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4315+0+0 (unknown 885076031 0 0) 0x7f94e4043650 con 0x7f94f40a2350 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94fa7d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94fa7d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94fa7d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d8005100 con 0x7f94f40a2350 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.985+0000 7f94f3fff700 10 monclient: get_auth_request con 0x7f94d40362a0 auth_method 0 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.993+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f94e4014b90 con 0x7f94f40a2350 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.993+0000 7f94f0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:49.993+0000 7f94f0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:50.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94fa7d7700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94fa7d7700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94fa7d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f94d8005930 con 0x7f94f40a2350 2024-09-16T06:40:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+70 (unknown 2699501919 0 3909614116) 0x7f94e4013000 con 0x7f94f40a2350 2024-09-16T06:40:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94f0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.165+0000 7f94f0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94fa7d7700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94fa7d7700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94fa7d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f94d8005650 con 0x7f94f40a2350 2024-09-16T06:40:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94f0ff9700 1 -- 172.21.15.189:0/503787181 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+70 (unknown 2699501919 0 3909614116) 0x7f94e4014780 con 0x7f94f40a2350 2024-09-16T06:40:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94f0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.265+0000 7f94f0ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 >> v1:172.21.15.189:6812/3098810674 conn(0x7f94d40362a0 legacy=0x7f94d4038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 10 monclient: shutdown 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 >> v1:172.21.15.76:6790/0 conn(0x7f94f40a2350 legacy=0x7f94f41cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 shutdown_connections 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 >> 172.21.15.189:0/503787181 conn(0x7f94f40a7d70 msgr2=0x7f94f40a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 shutdown_connections 2024-09-16T06:40:50.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.269+0000 7f94fa7d7700 1 -- 172.21.15.189:0/503787181 wait complete. 2024-09-16T06:40:50.288 INFO:tasks.workunit.client.0.smithi189.stdout:foo-asdf 2024-09-16T06:40:50.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:50.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep bar-asdf 2024-09-16T06:40:50.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: get_monmap_and_config 2024-09-16T06:40:50.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: build_initial_monmap 2024-09-16T06:40:50.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: monmap: 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:50.389659+0000 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:50.389659+0000 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:50.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 1 Processor -- start 2024-09-16T06:40:50.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 1 -- start start 2024-09-16T06:40:50.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: init 2024-09-16T06:40:50.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:50.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-a con 0x7ff2e4073b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-c con 0x7ff2e409c260 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-b con 0x7ff2e4074650 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e409bdd0 con 0x7ff2e4073b60 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e406b7f0 con 0x7ff2e409c260 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4071950 con 0x7ff2e4074650 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:50.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e94f1700 10 monclient(hunting): authenticate will time out at 1002.902900s 2024-09-16T06:40:50.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e27fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff2e409c260 legacy=0x7ff2e413a640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:50.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff2e4073b60 0x7ff2e4073f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46140/0 (socket says 172.21.15.189:46140) 2024-09-16T06:40:50.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e2ffd700 1 -- 172.21.15.189:0/583039141 learned_addr learned my addr 172.21.15.189:0/583039141 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808242430 0 0) 0x7ff2e409bdd0 con 0x7ff2e4073b60 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): my global_id is 104562 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2cc0035d0 con 0x7ff2e4073b60 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120154829 0 0) 0x7ff2e4071950 con 0x7ff2e4074650 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 10 monclient(hunting): my global_id is 124256 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.385+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e409bdd0 con 0x7ff2e4074650 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3995748021 0 0) 0x7ff2cc0035d0 con 0x7ff2e4073b60 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 >> v1:172.21.15.76:6790/0 conn(0x7ff2e4074650 legacy=0x7ff2e413dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 >> v1:172.21.15.189:6789/0 conn(0x7ff2e409c260 legacy=0x7ff2e413a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 monclient: found mon.noname-a 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4071ae0 con 0x7ff2e4073b60 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:50.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 5 monclient: authenticate success, global_id 104562 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff2d8001610 con 0x7ff2e4073b60 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: dump: 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2d80018f0 con 0x7ff2e4073b60 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 1 -- 172.21.15.189:0/583039141 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff2d8001ae0 con 0x7ff2e4073b60 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 4 set_mon_vals no callback set 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e1ffb700 10 monclient: dump: 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.402 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: shutdown 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- 172.21.15.189:0/583039141 >> v1:172.21.15.76:6789/0 conn(0x7ff2e4073b60 legacy=0x7ff2e4073f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- 172.21.15.189:0/583039141 shutdown_connections 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- 172.21.15.189:0/583039141 >> 172.21.15.189:0/583039141 conn(0x7ff2e40a13b0 msgr2=0x7ff2e40a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- 172.21.15.189:0/583039141 shutdown_connections 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- 172.21.15.189:0/583039141 wait complete. 2024-09-16T06:40:50.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: build_initial_monmap 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: monmap: 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 Processor -- start 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- start start 2024-09-16T06:40:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: init 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-a con 0x7ff2e4073b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-c con 0x7ff2e4074650 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient(hunting): picked mon.noname-b con 0x7ff2e409c260 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40926a0 con 0x7ff2e4073b60 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4094720 con 0x7ff2e409c260 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e4093270 con 0x7ff2e4074650 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e94f1700 10 monclient(hunting): authenticate will time out at 1002.906653s 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e37fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff2e409c260 legacy=0x7ff2e41d1dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e27fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff2e4074650 0x7ff2e41ce6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48080/0 (socket says 172.21.15.189:48080) 2024-09-16T06:40:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2e27fc700 1 -- 172.21.15.189:0/450099255 learned_addr learned my addr 172.21.15.189:0/450099255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015757068 0 0) 0x7ff2e4093270 con 0x7ff2e4074650 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): my global_id is 124259 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d00035d0 con 0x7ff2e4074650 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069332213 0 0) 0x7ff2e40926a0 con 0x7ff2e4073b60 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): my global_id is 104565 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4093270 con 0x7ff2e4073b60 2024-09-16T06:40:50.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3719269788 0 0) 0x7ff2d00035d0 con 0x7ff2e4074650 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e40926a0 con 0x7ff2e4074650 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff2d40044b0 con 0x7ff2e4074650 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.389+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 499733740 0 0) 0x7ff2e4093270 con 0x7ff2e4073b60 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2d00035d0 con 0x7ff2e4073b60 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff2d8001a60 con 0x7ff2e4073b60 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient(hunting): dump: 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.408 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 569912963 0 0) 0x7ff2e40926a0 con 0x7ff2e4074650 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 >> v1:172.21.15.189:6789/0 conn(0x7ff2e409c260 legacy=0x7ff2e41d1dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 >> v1:172.21.15.76:6789/0 conn(0x7ff2e4073b60 legacy=0x7ff2e4097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 monclient: found mon.c 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4094a00 con 0x7ff2e4074650 2024-09-16T06:40:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 5 monclient: authenticate success, global_id 124259 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _renew_subs 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e409f500 con 0x7ff2e4074650 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _renew_subs 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e4094440 con 0x7ff2e4074650 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2d4004720 con 0x7ff2e4074650 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff2d4004ab0 con 0x7ff2e4074650 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e0ff9700 4 set_mon_vals no callback set 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 10 monclient: dump: 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c4005100 con 0x7ff2e4074650 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff2d4004c90 con 0x7ff2e4074650 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4315+0+0 (unknown 885076031 0 0) 0x7ff2d4043800 con 0x7ff2e4074650 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.393+0000 7ff2e2ffd700 10 monclient: get_auth_request con 0x7ff2d0036280 auth_method 0 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.401+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff2d40054e0 con 0x7ff2e4074650 2024-09-16T06:40:50.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.401+0000 7ff2cb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:50.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.401+0000 7ff2cb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:50.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.561+0000 7ff2e94f1700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.561+0000 7ff2e94f1700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.561+0000 7ff2e94f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.561+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff2c4005930 con 0x7ff2e4074650 2024-09-16T06:40:50.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.565+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+70 (unknown 2699501919 0 3909614116) 0x7ff2d4012f30 con 0x7ff2e4074650 2024-09-16T06:40:50.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.565+0000 7ff2cb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.565+0000 7ff2cb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:50.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.653+0000 7ff2e94f1700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.653+0000 7ff2e94f1700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.653+0000 7ff2e94f1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:50.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.653+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff2c4005650 con 0x7ff2e4074650 2024-09-16T06:40:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2cb7fe700 1 -- 172.21.15.189:0/450099255 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+70 (unknown 2699501919 0 3909614116) 0x7ff2d4014890 con 0x7ff2e4074650 2024-09-16T06:40:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2cb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:50.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2cb7fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff2d0036280 legacy=0x7ff2d0038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 10 monclient: shutdown 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 >> v1:172.21.15.76:6790/0 conn(0x7ff2e4074650 legacy=0x7ff2e41ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 shutdown_connections 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 >> 172.21.15.189:0/450099255 conn(0x7ff2e40a13b0 msgr2=0x7ff2e409c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 shutdown_connections 2024-09-16T06:40:50.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:50.657+0000 7ff2e94f1700 1 -- 172.21.15.189:0/450099255 wait complete. 2024-09-16T06:40:50.671 INFO:tasks.workunit.client.0.smithi189.stdout:bar-asdf 2024-09-16T06:40:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'does not exist' 2024-09-16T06:40:50.672 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:50.969+0000 7fd832ffd700 1 -- 172.21.15.189:0/2322447007 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v58) v1 ==== 103+0+0 (unknown 77879010 0 0) 0x7fd838012bd0 con 0x7fd83c135d90 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:50.969+0000 7fd832ffd700 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.065+0000 7fd832ffd700 1 -- 172.21.15.189:0/2322447007 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v58) v1 ==== 103+0+0 (unknown 77879010 0 0) 0x7fd838014350 con 0x7fd83c135d90 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.065+0000 7fd832ffd700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stdout:rule foo does not exist 2024-09-16T06:40:51.083 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm bar 2024-09-16T06:40:51.084 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'does not exist' 2024-09-16T06:40:51.480 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.373+0000 7f8dcffff700 1 -- 172.21.15.189:0/2224601919 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v58) v1 ==== 103+0+0 (unknown 1708714153 0 0) 0x7f8dd0011da0 con 0x7f8dd809b1b0 2024-09-16T06:40:51.480 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.373+0000 7f8dcffff700 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-16T06:40:51.481 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.465+0000 7f8dcffff700 1 -- 172.21.15.189:0/2224601919 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v58) v1 ==== 103+0+0 (unknown 1708714153 0 0) 0x7f8dd0013520 con 0x7f8dd809b1b0 2024-09-16T06:40:51.481 INFO:tasks.workunit.client.0.smithi189.stdout:2024-09-16T06:40:51.465+0000 7f8dcffff700 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T06:40:51.481 INFO:tasks.workunit.client.0.smithi189.stdout:rule bar does not exist 2024-09-16T06:40:51.481 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T06:40:51.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:40:51.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: build_initial_monmap 2024-09-16T06:40:51.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: monmap: 2024-09-16T06:40:51.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:51.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:51.583252+0000 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:51.583252+0000 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:51.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 1 Processor -- start 2024-09-16T06:40:51.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 1 -- start start 2024-09-16T06:40:51.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: init 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.577+0000 7f0a8ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0a9009eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0a9009f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0a90139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:51.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9009ab80 con 0x7f0a90139a50 2024-09-16T06:40:51.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9009a690 con 0x7f0a9009eba0 2024-09-16T06:40:51.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a90072230 con 0x7f0a9009f690 2024-09-16T06:40:51.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:51.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient(hunting): authenticate will time out at 1004.096125s 2024-09-16T06:40:51.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0a9009eba0 legacy=0x7f0a9009ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0a9009f690 0x7f0a90138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48122/0 (socket says 172.21.15.189:48122) 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8e7fc700 1 -- 172.21.15.189:0/710820076 learned_addr learned my addr 172.21.15.189:0/710820076 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194199722 0 0) 0x7f0a9009ab80 con 0x7f0a90139a50 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:51.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): my global_id is 104580 2024-09-16T06:40:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a700035d0 con 0x7f0a90139a50 2024-09-16T06:40:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367969568 0 0) 0x7f0a90072230 con 0x7f0a9009f690 2024-09-16T06:40:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:51.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient(hunting): my global_id is 124274 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9009ab80 con 0x7f0a9009f690 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2350332948 0 0) 0x7f0a700035d0 con 0x7f0a90139a50 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 >> v1:172.21.15.76:6790/0 conn(0x7f0a9009f690 legacy=0x7f0a90138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:51.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 >> v1:172.21.15.189:6789/0 conn(0x7f0a9009eba0 legacy=0x7f0a9009ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 monclient: found mon.noname-a 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9009be90 con 0x7f0a90139a50 2024-09-16T06:40:51.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:51.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:51.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:51.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 5 monclient: authenticate success, global_id 104580 2024-09-16T06:40:51.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a80001610 con 0x7f0a90139a50 2024-09-16T06:40:51.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: dump: 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.604 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.605 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:51.605 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.605 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:51.605 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:51.605 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a800018f0 con 0x7f0a90139a50 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:51.606 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 1 -- 172.21.15.189:0/710820076 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a80001cc0 con 0x7f0a90139a50 2024-09-16T06:40:51.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 4 set_mon_vals no callback set 2024-09-16T06:40:51.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:51.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:51.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8dffb700 10 monclient: dump: 2024-09-16T06:40:51.607 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.608 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient: shutdown 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:51.609 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- 172.21.15.189:0/710820076 >> v1:172.21.15.76:6789/0 conn(0x7f0a90139a50 legacy=0x7f0a9013be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:51.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- 172.21.15.189:0/710820076 shutdown_connections 2024-09-16T06:40:51.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- 172.21.15.189:0/710820076 >> 172.21.15.189:0/710820076 conn(0x7f0a9009fe70 msgr2=0x7f0a900a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:51.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- 172.21.15.189:0/710820076 shutdown_connections 2024-09-16T06:40:51.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- 172.21.15.189:0/710820076 wait complete. 2024-09-16T06:40:51.610 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient: build_initial_monmap 2024-09-16T06:40:51.611 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient: monmap: 2024-09-16T06:40:51.611 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:51.611 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:51.611 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:51.612 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:51.612 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:51.612 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:51.612 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:51.612 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:51.613 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 Processor -- start 2024-09-16T06:40:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 1 -- start start 2024-09-16T06:40:51.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.581+0000 7f0a8ffff700 10 monclient: init 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0a9009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0a9009f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0a90139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:51.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a90092100 con 0x7f0a90139a50 2024-09-16T06:40:51.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a90093ee0 con 0x7f0a9009f690 2024-09-16T06:40:51.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a900711f0 con 0x7f0a9009eba0 2024-09-16T06:40:51.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:51.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient(hunting): authenticate will time out at 1004.100475s 2024-09-16T06:40:51.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8e7fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0a9009f690 legacy=0x7f0a901cf040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0a9009eba0 0x7f0a90135690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48138/0 (socket says 172.21.15.189:48138) 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8effd700 1 -- 172.21.15.189:0/2608329719 learned_addr learned my addr 172.21.15.189:0/2608329719 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530789626 0 0) 0x7f0a900711f0 con 0x7f0a9009eba0 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:51.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): my global_id is 124277 2024-09-16T06:40:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740035d0 con 0x7f0a9009eba0 2024-09-16T06:40:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734098778 0 0) 0x7f0a90092100 con 0x7f0a90139a50 2024-09-16T06:40:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:51.617 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): my global_id is 104583 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a900711f0 con 0x7f0a90139a50 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062028156 0 0) 0x7f0a740035d0 con 0x7f0a9009eba0 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a90092100 con 0x7f0a9009eba0 2024-09-16T06:40:51.618 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a88004580 con 0x7f0a9009eba0 2024-09-16T06:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): dump: 2024-09-16T06:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:51.619 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.620 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:51.621 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187200864 0 0) 0x7f0a900711f0 con 0x7f0a90139a50 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a740035d0 con 0x7f0a90139a50 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a80001c00 con 0x7f0a90139a50 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient(hunting): dump: 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:51.622 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180843723 0 0) 0x7f0a90092100 con 0x7f0a9009eba0 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 >> v1:172.21.15.189:6789/0 conn(0x7f0a9009f690 legacy=0x7f0a901cf040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 >> v1:172.21.15.76:6789/0 conn(0x7f0a90139a50 legacy=0x7f0a901d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:51.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 monclient: found mon.c 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9009e600 con 0x7f0a9009eba0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: _finish_auth 0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 5 monclient: authenticate success, global_id 124277 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _renew_subs 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9009a380 con 0x7f0a9009eba0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _renew_subs 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:51.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a90091bc0 con 0x7f0a9009eba0 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a880047f0 con 0x7f0a9009eba0 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a88004b80 con 0x7f0a9009eba0 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8d7fa700 4 set_mon_vals no callback set 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a87fff700 10 monclient: dump: 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:51.625 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.585+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a68005100 con 0x7f0a9009eba0 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.589+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0a88004da0 con 0x7f0a9009eba0 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.589+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4315+0+0 (unknown 885076031 0 0) 0x7f0a88043780 con 0x7f0a9009eba0 2024-09-16T06:40:51.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.589+0000 7f0a8e7fc700 10 monclient: get_auth_request con 0x7f0a740362a0 auth_method 0 2024-09-16T06:40:51.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.597+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0a88014ba0 con 0x7f0a9009eba0 2024-09-16T06:40:51.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.597+0000 7f0a87fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:51.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.597+0000 7f0a87fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:51.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.773+0000 7f0a8ffff700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:51.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.773+0000 7f0a8ffff700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:51.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.773+0000 7f0a8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:51.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:51.773+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f0a68005930 con 0x7f0a9009eba0 2024-09-16T06:40:52.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.765+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v59) v1 ==== 110+0+0 (unknown 4205848879 0 0) 0x7f0a88013010 con 0x7f0a9009eba0 2024-09-16T06:40:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.765+0000 7f0a87fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:52.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.765+0000 7f0a87fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:52.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.877+0000 7f0a8ffff700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:52.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.877+0000 7f0a8ffff700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:52.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.877+0000 7f0a8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:52.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.877+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f0a680053b0 con 0x7f0a9009eba0 2024-09-16T06:40:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a87fff700 1 -- 172.21.15.189:0/2608329719 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v59) v1 ==== 134+0+0 (unknown 3043379746 0 0) 0x7f0a88014790 con 0x7f0a9009eba0 2024-09-16T06:40:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a87fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:52.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a87fff700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T06:40:52.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0a740362a0 legacy=0x7f0a74038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:52.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a8ffff700 10 monclient: shutdown 2024-09-16T06:40:52.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:52.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.881+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 >> v1:172.21.15.76:6790/0 conn(0x7f0a9009eba0 legacy=0x7f0a90135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:52.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.885+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 shutdown_connections 2024-09-16T06:40:52.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.885+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 >> 172.21.15.189:0/2608329719 conn(0x7f0a9009fe70 msgr2=0x7f0a901390c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:52.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.885+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 shutdown_connections 2024-09-16T06:40:52.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:52.885+0000 7f0a8ffff700 1 -- 172.21.15.189:0/2608329719 wait complete. 2024-09-16T06:40:52.888 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo' 2024-09-16T06:40:52.901 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T06:40:53.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: get_monmap_and_config 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: build_initial_monmap 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: monmap: 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:53.009640+0000 2024-09-16T06:40:53.009 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:53.009640+0000 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:53.010 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 1 Processor -- start 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 1 -- start start 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: init 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-b con 0x7f2260098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-a con 0x7f2260099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-c con 0x7f2260138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:53.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226009c460 con 0x7f2260099a60 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226009c1e0 con 0x7f2260138a80 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226009bd60 con 0x7f2260098070 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f22663a9700 10 monclient(hunting): authenticate will time out at 1005.522787s 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f2264946700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2260138a80 legacy=0x7f226013ae70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2260099a60 0x7f2260099e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46224/0 (socket says 172.21.15.189:46224) 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2260098070 0x7f226009f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48150/0 (socket says 172.21.15.189:48150) 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225f7fe700 1 -- 172.21.15.189:0/4224067434 learned_addr learned my addr 172.21.15.189:0/4224067434 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605938108 0 0) 0x7f226009bd60 con 0x7f2260098070 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): my global_id is 124280 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c0035d0 con 0x7f2260098070 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501320625 0 0) 0x7f226009c460 con 0x7f2260099a60 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 10 monclient(hunting): my global_id is 104586 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.005+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226009bd60 con 0x7f2260099a60 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288758608 0 0) 0x7f224c0035d0 con 0x7f2260098070 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 >> v1:172.21.15.189:6789/0 conn(0x7f2260138a80 legacy=0x7f226013ae70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 >> v1:172.21.15.76:6789/0 conn(0x7f2260099a60 legacy=0x7f2260099e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 monclient: found mon.noname-b 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226009d490 con 0x7f2260098070 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: _finish_auth 0 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2250001990 con 0x7f2260098070 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 5 monclient: authenticate success, global_id 124280 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: dump: 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.026 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.027 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.027 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.027 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2250001b80 con 0x7f2260098070 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 1 -- 172.21.15.189:0/4224067434 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2250001f10 con 0x7f2260098070 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 4 set_mon_vals no callback set 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f225effd700 10 monclient: dump: 2024-09-16T06:40:53.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 10 monclient: shutdown 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 -- 172.21.15.189:0/4224067434 >> v1:172.21.15.76:6790/0 conn(0x7f2260098070 legacy=0x7f226009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 -- 172.21.15.189:0/4224067434 shutdown_connections 2024-09-16T06:40:53.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 -- 172.21.15.189:0/4224067434 >> 172.21.15.189:0/4224067434 conn(0x7f22600a03e0 msgr2=0x7f22600a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:53.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 -- 172.21.15.189:0/4224067434 shutdown_connections 2024-09-16T06:40:53.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 -- 172.21.15.189:0/4224067434 wait complete. 2024-09-16T06:40:53.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 10 monclient: build_initial_monmap 2024-09-16T06:40:53.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 10 monclient: monmap: 2024-09-16T06:40:53.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:53.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:53.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:53.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:53.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.009+0000 7f22663a9700 1 Processor -- start 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- start start 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: init 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:53.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-a con 0x7f2260098070 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-c con 0x7f2260099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient(hunting): picked mon.noname-b con 0x7f2260138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2260129dc0 con 0x7f2260098070 2024-09-16T06:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226009a6e0 con 0x7f2260138a80 2024-09-16T06:40:53.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2260098ee0 con 0x7f2260099a60 2024-09-16T06:40:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f2264946700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2260138a80 legacy=0x7f22601cfe70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient(hunting): authenticate will time out at 1005.529304s 2024-09-16T06:40:53.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2260099a60 0x7f2260133050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48156/0 (socket says 172.21.15.189:48156) 2024-09-16T06:40:53.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225f7fe700 1 -- 172.21.15.189:0/850993384 learned_addr learned my addr 172.21.15.189:0/850993384 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:53.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797973601 0 0) 0x7f2260098ee0 con 0x7f2260099a60 2024-09-16T06:40:53.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): my global_id is 124283 2024-09-16T06:40:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22400035d0 con 0x7f2260099a60 2024-09-16T06:40:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808227516 0 0) 0x7f2260129dc0 con 0x7f2260098070 2024-09-16T06:40:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): my global_id is 104589 2024-09-16T06:40:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260098ee0 con 0x7f2260098070 2024-09-16T06:40:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3183817240 0 0) 0x7f22400035d0 con 0x7f2260099a60 2024-09-16T06:40:53.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2260129dc0 con 0x7f2260099a60 2024-09-16T06:40:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2248001370 con 0x7f2260099a60 2024-09-16T06:40:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): dump: 2024-09-16T06:40:53.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.041 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:53.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924686749 0 0) 0x7f2260098ee0 con 0x7f2260098070 2024-09-16T06:40:53.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22400035d0 con 0x7f2260098070 2024-09-16T06:40:53.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2250001e90 con 0x7f2260098070 2024-09-16T06:40:53.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:53.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient(hunting): dump: 2024-09-16T06:40:53.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.044 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.045 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 50745288 0 0) 0x7f2260129dc0 con 0x7f2260099a60 2024-09-16T06:40:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 >> v1:172.21.15.189:6789/0 conn(0x7f2260138a80 legacy=0x7f22601cfe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:53.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 >> v1:172.21.15.76:6789/0 conn(0x7f2260098070 legacy=0x7f2260132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 monclient: found mon.c 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260133760 con 0x7f2260099a60 2024-09-16T06:40:53.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 5 monclient: authenticate success, global_id 124283 2024-09-16T06:40:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _renew_subs 2024-09-16T06:40:53.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22600717f0 con 0x7f2260099a60 2024-09-16T06:40:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _renew_subs 2024-09-16T06:40:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22600712e0 con 0x7f2260099a60 2024-09-16T06:40:53.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2248001650 con 0x7f2260099a60 2024-09-16T06:40:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2248004ce0 con 0x7f2260099a60 2024-09-16T06:40:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225cff9700 10 monclient: dump: 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.013+0000 7f225e7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:53.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f224800f750 con 0x7f2260099a60 2024-09-16T06:40:53.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f22663a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f22663a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f22663a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2244005100 con 0x7f2260099a60 2024-09-16T06:40:53.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 4310+0+0 (unknown 2537317619 0 0) 0x7f2248043690 con 0x7f2260099a60 2024-09-16T06:40:53.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.017+0000 7f225ffff700 10 monclient: get_auth_request con 0x7f22400362a0 auth_method 0 2024-09-16T06:40:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.029+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2248010d40 con 0x7f2260099a60 2024-09-16T06:40:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.029+0000 7f225cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.029+0000 7f225cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:53.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f22663a9700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f22663a9700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f22663a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f2244005930 con 0x7f2260099a60 2024-09-16T06:40:53.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v59) v1 ==== 134+0+0 (unknown 3043379746 0 0) 0x7f2248012d60 con 0x7f2260099a60 2024-09-16T06:40:53.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f225cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.193+0000 7f225cff9700 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-16T06:40:53.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f22663a9700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f22663a9700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f22663a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f2244005b70 con 0x7f2260099a60 2024-09-16T06:40:53.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f225cff9700 1 -- 172.21.15.189:0/850993384 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v59) v1 ==== 134+0+0 (unknown 3043379746 0 0) 0x7f22480144e0 con 0x7f2260099a60 2024-09-16T06:40:53.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f225cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T06:40:53.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.289+0000 7f225cff9700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T06:40:53.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 >> v1:172.21.15.189:6812/3098810674 conn(0x7f22400362a0 legacy=0x7f2240038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 10 monclient: shutdown 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 >> v1:172.21.15.76:6790/0 conn(0x7f2260099a60 legacy=0x7f2260133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 shutdown_connections 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 >> 172.21.15.189:0/850993384 conn(0x7f22600a03e0 msgr2=0x7f226009b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 shutdown_connections 2024-09-16T06:40:53.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.293+0000 7f22663a9700 1 -- 172.21.15.189:0/850993384 wait complete. 2024-09-16T06:40:53.299 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'foo' 2024-09-16T06:40:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: get_monmap_and_config 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: build_initial_monmap 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: monmap: 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:53.421308+0000 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:53.421308+0000 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:53.421 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 1 Processor -- start 2024-09-16T06:40:53.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 1 -- start start 2024-09-16T06:40:53.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: init 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-c con 0x7f26f0132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-b con 0x7f26f0138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-a con 0x7f26f013bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f006abd0 con 0x7f26f013bf70 2024-09-16T06:40:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f006b170 con 0x7f26f0132080 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f0132b70 con 0x7f26f0138460 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 10 monclient(hunting): authenticate will time out at 1005.934489s 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26effff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f26f0132080 legacy=0x7f26f0132460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f4b04700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f26f013bf70 0x7f26f013e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46246/0 (socket says 172.21.15.189:46246) 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f4b04700 1 -- 172.21.15.189:0/1454757666 learned_addr learned my addr 172.21.15.189:0/1454757666 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639886374 0 0) 0x7f26f006abd0 con 0x7f26f013bf70 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): my global_id is 104592 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d80035d0 con 0x7f26f013bf70 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339152667 0 0) 0x7f26f0132b70 con 0x7f26f0138460 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient(hunting): my global_id is 124286 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f006abd0 con 0x7f26f0138460 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592979516 0 0) 0x7f26d80035d0 con 0x7f26f013bf70 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 >> v1:172.21.15.76:6790/0 conn(0x7f26f0138460 legacy=0x7f26f013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 >> v1:172.21.15.189:6789/0 conn(0x7f26f0132080 legacy=0x7f26f0132460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 monclient: found mon.noname-a 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00a38b0 con 0x7f26f013bf70 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 5 monclient: authenticate success, global_id 104592 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26e8001990 con 0x7f26f013bf70 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26eeffd700 10 monclient: dump: 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.417+0000 7f26f5305700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26e8001c70 con 0x7f26f013bf70 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 1 -- 172.21.15.189:0/1454757666 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26e8002000 con 0x7f26f013bf70 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 4 set_mon_vals no callback set 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26eeffd700 10 monclient: dump: 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.432 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: shutdown 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- 172.21.15.189:0/1454757666 >> v1:172.21.15.76:6789/0 conn(0x7f26f013bf70 legacy=0x7f26f013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- 172.21.15.189:0/1454757666 shutdown_connections 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- 172.21.15.189:0/1454757666 >> 172.21.15.189:0/1454757666 conn(0x7f26f00a4660 msgr2=0x7f26f0099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- 172.21.15.189:0/1454757666 shutdown_connections 2024-09-16T06:40:53.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- 172.21.15.189:0/1454757666 wait complete. 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: build_initial_monmap 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: monmap: 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:53.439 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 Processor -- start 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- start start 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: init 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-b con 0x7f26f0132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-a con 0x7f26f0138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient(hunting): picked mon.noname-c con 0x7f26f013bf70 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f009d4d0 con 0x7f26f0138460 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f009ce20 con 0x7f26f0132080 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f0138130 con 0x7f26f013bf70 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f5305700 10 monclient(hunting): authenticate will time out at 1005.939596s 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26effff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f26f0132080 legacy=0x7f26f009c000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f4b04700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f26f013bf70 0x7f26f01d2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48172/0 (socket says 172.21.15.189:48172) 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26f4b04700 1 -- 172.21.15.189:0/1336576118 learned_addr learned my addr 172.21.15.189:0/1336576118 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.421+0000 7f26ef7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f26f0138460 0x7f26f009c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46258/0 (socket says 172.21.15.189:46258) 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011047476 0 0) 0x7f26f0138130 con 0x7f26f013bf70 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): my global_id is 124289 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc0035d0 con 0x7f26f013bf70 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072102809 0 0) 0x7f26f009d4d0 con 0x7f26f0138460 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): my global_id is 104595 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0138130 con 0x7f26f0138460 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4163678588 0 0) 0x7f26dc0035d0 con 0x7f26f013bf70 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26f009d4d0 con 0x7f26f013bf70 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26e8001950 con 0x7f26f013bf70 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): dump: 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124004595 0 0) 0x7f26f0138130 con 0x7f26f0138460 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26dc0035d0 con 0x7f26f0138460 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26e40018b0 con 0x7f26f0138460 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient(hunting): dump: 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2642731025 0 0) 0x7f26f009d4d0 con 0x7f26f013bf70 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 >> v1:172.21.15.189:6789/0 conn(0x7f26f0132080 legacy=0x7f26f009c000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 >> v1:172.21.15.76:6789/0 conn(0x7f26f0138460 legacy=0x7f26f009c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 monclient: found mon.c 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f013ba20 con 0x7f26f013bf70 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 5 monclient: authenticate success, global_id 124289 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _renew_subs 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f009d9e0 con 0x7f26f013bf70 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _renew_subs 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f009fc20 con 0x7f26f013bf70 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26e8001540 con 0x7f26f013bf70 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26e80051c0 con 0x7f26f013bf70 2024-09-16T06:40:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ee7fc700 4 set_mon_vals no callback set 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 10 monclient: dump: 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:53.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d0005100 con 0x7f26f013bf70 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f26e800fc30 con 0x7f26f013bf70 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 4310+0+0 (unknown 2537317619 0 0) 0x7f26e8043a10 con 0x7f26f013bf70 2024-09-16T06:40:53.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.425+0000 7f26effff700 10 monclient: get_auth_request con 0x7f26dc0362a0 auth_method 0 2024-09-16T06:40:53.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.437+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f26e8014b00 con 0x7f26f013bf70 2024-09-16T06:40:53.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.437+0000 7f26ecff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:53.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.437+0000 7f26ecff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.597+0000 7f26f5305700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.597+0000 7f26f5305700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.597+0000 7f26f5305700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.597+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f26d0005930 con 0x7f26f013bf70 2024-09-16T06:40:53.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.773+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v60) v1 ==== 110+0+0 (unknown 1624877455 0 0) 0x7f26e80131a0 con 0x7f26f013bf70 2024-09-16T06:40:53.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.773+0000 7f26ecff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.773+0000 7f26ecff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:53.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.873+0000 7f26f5305700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.873+0000 7f26f5305700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.873+0000 7f26f5305700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:53.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.873+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7f26d0005b70 con 0x7f26f013bf70 2024-09-16T06:40:53.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.877+0000 7f26ecff9700 1 -- 172.21.15.189:0/1336576118 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v60) v1 ==== 134+0+0 (unknown 162267867 0 0) 0x7f26e8014920 con 0x7f26f013bf70 2024-09-16T06:40:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.877+0000 7f26ecff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T06:40:53.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.877+0000 7f26ecff9700 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-16T06:40:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 >> v1:172.21.15.189:6812/3098810674 conn(0x7f26dc0362a0 legacy=0x7f26dc038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 10 monclient: shutdown 2024-09-16T06:40:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 >> v1:172.21.15.76:6790/0 conn(0x7f26f013bf70 legacy=0x7f26f01d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:53.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 shutdown_connections 2024-09-16T06:40:53.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 >> 172.21.15.189:0/1336576118 conn(0x7f26f00a4660 msgr2=0x7f26f00a6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:53.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 shutdown_connections 2024-09-16T06:40:53.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.881+0000 7f26f5305700 1 -- 172.21.15.189:0/1336576118 wait complete. 2024-09-16T06:40:53.885 INFO:tasks.workunit.client.0.smithi189.stderr:already renamed to 'bar' 2024-09-16T06:40:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep foo-asdf 2024-09-16T06:40:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:53.894 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo-asdf 2024-09-16T06:40:53.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: get_monmap_and_config 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: build_initial_monmap 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: monmap: 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:53.998611+0000 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:53.998611+0000 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:53.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 1 Processor -- start 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 1 -- start start 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: init 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-c con 0x7ff7481359a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-a con 0x7ff748135d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-b con 0x7ff74813c570 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480a3820 con 0x7ff748135d80 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480a3170 con 0x7ff7481359a0 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748099eb0 con 0x7ff74813c570 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74fa0c700 10 monclient(hunting): authenticate will time out at 1006.511675s 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.993+0000 7ff74d7a8700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff7481359a0 legacy=0x7ff7480a2350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74cfa7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff748135d80 0x7ff7480a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46270/0 (socket says 172.21.15.189:46270) 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74dfa9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff74813c570 0x7ff74813e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48174/0 (socket says 172.21.15.189:48174) 2024-09-16T06:40:54.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74cfa7700 1 -- 172.21.15.189:0/3525568329 learned_addr learned my addr 172.21.15.189:0/3525568329 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302133924 0 0) 0x7ff7480a3820 con 0x7ff748135d80 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): my global_id is 104598 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7340035d0 con 0x7ff748135d80 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442258560 0 0) 0x7ff748099eb0 con 0x7ff74813c570 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient(hunting): my global_id is 124292 2024-09-16T06:40:54.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7480a3820 con 0x7ff74813c570 2024-09-16T06:40:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394591728 0 0) 0x7ff7480a3820 con 0x7ff74813c570 2024-09-16T06:40:54.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 >> v1:172.21.15.189:6789/0 conn(0x7ff7481359a0 legacy=0x7ff7480a2350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 >> v1:172.21.15.76:6789/0 conn(0x7ff748135d80 legacy=0x7ff7480a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 monclient: found mon.noname-b 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74809a190 con 0x7ff74813c570 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 5 monclient: authenticate success, global_id 124292 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff740001970 con 0x7ff74813c570 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:54.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: dump: 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff740001c90 con 0x7ff74813c570 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 1 -- 172.21.15.189:0/3525568329 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff740002060 con 0x7ff74813c570 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 4 set_mon_vals no callback set 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:54.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff73ffff700 10 monclient: dump: 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.013 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 10 monclient: shutdown 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- 172.21.15.189:0/3525568329 >> v1:172.21.15.76:6790/0 conn(0x7ff74813c570 legacy=0x7ff74813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- 172.21.15.189:0/3525568329 shutdown_connections 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- 172.21.15.189:0/3525568329 >> 172.21.15.189:0/3525568329 conn(0x7ff7480a7d70 msgr2=0x7ff7480a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- 172.21.15.189:0/3525568329 shutdown_connections 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- 172.21.15.189:0/3525568329 wait complete. 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 10 monclient: build_initial_monmap 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 10 monclient: monmap: 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:54.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 Processor -- start 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 1 -- start start 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:53.997+0000 7ff74fa0c700 10 monclient: init 2024-09-16T06:40:54.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-a con 0x7ff7481359a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-c con 0x7ff748135d80 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient(hunting): picked mon.noname-b con 0x7ff74813c570 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7481373c0 con 0x7ff7481359a0 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74813b360 con 0x7ff74813c570 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748137d10 con 0x7ff748135d80 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient(hunting): authenticate will time out at 1006.516355s 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74dfa9700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff74813c570 legacy=0x7ff7481d2740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74d7a8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff7481359a0 0x7ff7481361b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46278/0 (socket says 172.21.15.189:46278) 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74cfa7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff748135d80 0x7ff7481368c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48182/0 (socket says 172.21.15.189:48182) 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74d7a8700 1 -- 172.21.15.189:0/2286994400 learned_addr learned my addr 172.21.15.189:0/2286994400 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394612164 0 0) 0x7ff7481373c0 con 0x7ff7481359a0 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): my global_id is 104601 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff72c0035d0 con 0x7ff7481359a0 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795522430 0 0) 0x7ff748137d10 con 0x7ff748135d80 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): my global_id is 124295 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7481373c0 con 0x7ff748135d80 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419843935 0 0) 0x7ff72c0035d0 con 0x7ff7481359a0 2024-09-16T06:40:54.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff748137d10 con 0x7ff7481359a0 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff738004200 con 0x7ff7481359a0 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): dump: 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.018 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276289355 0 0) 0x7ff7481373c0 con 0x7ff748135d80 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff72c0035d0 con 0x7ff748135d80 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff744001850 con 0x7ff748135d80 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient(hunting): dump: 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.019 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2352020710 0 0) 0x7ff748137d10 con 0x7ff7481359a0 2024-09-16T06:40:54.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 >> v1:172.21.15.76:6790/0 conn(0x7ff748135d80 legacy=0x7ff7481368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 >> v1:172.21.15.189:6789/0 conn(0x7ff74813c570 legacy=0x7ff7481d2740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 monclient: found mon.a 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748136fd0 con 0x7ff7481359a0 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 5 monclient: authenticate success, global_id 104601 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _renew_subs 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff748137f90 con 0x7ff7481359a0 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _renew_subs 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7480a6ff0 con 0x7ff7481359a0 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff738004470 con 0x7ff7481359a0 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73f7fe700 4 set_mon_vals no callback set 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff738004800 con 0x7ff7481359a0 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:54.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 10 monclient: dump: 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.001+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff738004a20 con 0x7ff7481359a0 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff74fa0c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff74fa0c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff74fa0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff728005100 con 0x7ff7481359a0 2024-09-16T06:40:54.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(60..60 src has 1..60) v4 ==== 4305+0+0 (unknown 1339515136 0 0) 0x7ff738043400 con 0x7ff7481359a0 2024-09-16T06:40:54.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.005+0000 7ff74cfa7700 10 monclient: get_auth_request con 0x7ff72c0362a0 auth_method 0 2024-09-16T06:40:54.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.013+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff7380147c0 con 0x7ff7481359a0 2024-09-16T06:40:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.013+0000 7ff73dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.013+0000 7ff73dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff74fa0c700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff74fa0c700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff74fa0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff728005930 con 0x7ff7481359a0 2024-09-16T06:40:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+60 (unknown 3969698029 0 1596542489) 0x7ff73801ce10 con 0x7ff7481359a0 2024-09-16T06:40:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff73dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.177+0000 7ff73dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:54.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.269+0000 7ff74fa0c700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.269+0000 7ff74fa0c700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.269+0000 7ff74fa0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.269+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7ff728005650 con 0x7ff7481359a0 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff73dffb700 1 -- 172.21.15.189:0/2286994400 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+60 (unknown 3969698029 0 1596542489) 0x7ff738012c30 con 0x7ff7481359a0 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff73dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff73dffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:54.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff72c0362a0 legacy=0x7ff72c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 10 monclient: shutdown 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 >> v1:172.21.15.76:6789/0 conn(0x7ff7481359a0 legacy=0x7ff7481361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 shutdown_connections 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 >> 172.21.15.189:0/2286994400 conn(0x7ff7480a7d70 msgr2=0x7ff7480a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 shutdown_connections 2024-09-16T06:40:54.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.273+0000 7ff74fa0c700 1 -- 172.21.15.189:0/2286994400 wait complete. 2024-09-16T06:40:54.287 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:54.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule ls 2024-09-16T06:40:54.288 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false grep bar-asdf 2024-09-16T06:40:54.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:54.289 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep bar-asdf 2024-09-16T06:40:54.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.389+0000 7fadc806f700 10 monclient: get_monmap_and_config 2024-09-16T06:40:54.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.389+0000 7fadc806f700 10 monclient: build_initial_monmap 2024-09-16T06:40:54.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.389+0000 7fadc806f700 10 monclient: monmap: 2024-09-16T06:40:54.396 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:54.396415+0000 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:54.396415+0000 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 Processor -- start 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- start start 2024-09-16T06:40:54.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: init 2024-09-16T06:40:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-c con 0x7fadc009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-b con 0x7fadc0135d90 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-a con 0x7fadc01361a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc00979e0 con 0x7fadc01361a0 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc0097d90 con 0x7fadc009f140 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc0098010 con 0x7fadc0135d90 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc5e0b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fadc009f140 legacy=0x7fadc0135680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient(hunting): authenticate will time out at 1006.909339s 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc560a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fadc0135d90 0x7fadc013a8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48196/0 (socket says 172.21.15.189:48196) 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc560a700 1 -- 172.21.15.189:0/2607568659 learned_addr learned my addr 172.21.15.189:0/2607568659 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc660c700 1 --1- 172.21.15.189:0/2607568659 >> v1:172.21.15.76:6789/0 conn(0x7fadc01361a0 0x7fadc013dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46284/0 (socket says 172.21.15.189:46284) 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911500597 0 0) 0x7fadc00979e0 con 0x7fadc01361a0 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): my global_id is 104604 2024-09-16T06:40:54.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac0035d0 con 0x7fadc01361a0 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059532483 0 0) 0x7fadc0098010 con 0x7fadc0135d90 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient(hunting): my global_id is 124298 2024-09-16T06:40:54.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc00979e0 con 0x7fadc0135d90 2024-09-16T06:40:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2585149857 0 0) 0x7fadac0035d0 con 0x7fadc01361a0 2024-09-16T06:40:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 >> v1:172.21.15.76:6790/0 conn(0x7fadc0135d90 legacy=0x7fadc013a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 >> v1:172.21.15.189:6789/0 conn(0x7fadc009f140 legacy=0x7fadc0135680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 monclient: found mon.noname-a 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0071870 con 0x7fadc01361a0 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadb8001990 con 0x7fadc01361a0 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 5 monclient: authenticate success, global_id 104604 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: dump: 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.414 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb8001c00 con 0x7fadc01361a0 2024-09-16T06:40:54.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 1 -- 172.21.15.189:0/2607568659 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadb8001f90 con 0x7fadc01361a0 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 4 set_mon_vals no callback set 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:54.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc4e09700 10 monclient: dump: 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: shutdown 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- 172.21.15.189:0/2607568659 >> v1:172.21.15.76:6789/0 conn(0x7fadc01361a0 legacy=0x7fadc013dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- 172.21.15.189:0/2607568659 shutdown_connections 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- 172.21.15.189:0/2607568659 >> 172.21.15.189:0/2607568659 conn(0x7fadc00a03e0 msgr2=0x7fadc00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- 172.21.15.189:0/2607568659 shutdown_connections 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 1 -- 172.21.15.189:0/2607568659 wait complete. 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: build_initial_monmap 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.393+0000 7fadc806f700 10 monclient: monmap: 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:54.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 Processor -- start 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- start start 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: init 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-c con 0x7fadc009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-b con 0x7fadc0135d90 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient(hunting): picked mon.noname-a con 0x7fadc01361a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc0093e80 con 0x7fadc01361a0 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc00911e0 con 0x7fadc0135d90 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc0094160 con 0x7fadc009f140 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient(hunting): authenticate will time out at 1006.912822s 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc560a700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fadc0135d90 legacy=0x7fadc00966a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc5e0b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fadc009f140 0x7fadc0090ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48210/0 (socket says 172.21.15.189:48210) 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc660c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fadc01361a0 0x7fadc01d1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46288/0 (socket says 172.21.15.189:46288) 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc5e0b700 1 -- 172.21.15.189:0/848870000 learned_addr learned my addr 172.21.15.189:0/848870000 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124394936 0 0) 0x7fadc0093e80 con 0x7fadc01361a0 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): my global_id is 104607 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada40035d0 con 0x7fadc01361a0 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252670272 0 0) 0x7fadc0094160 con 0x7fadc009f140 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): my global_id is 124301 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0093e80 con 0x7fadc009f140 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643717580 0 0) 0x7fada40035d0 con 0x7fadc01361a0 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc0094160 con 0x7fadc01361a0 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadb8001950 con 0x7fadc01361a0 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): dump: 2024-09-16T06:40:54.422 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092589636 0 0) 0x7fadc0093e80 con 0x7fadc009f140 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada40035d0 con 0x7fadc009f140 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadb0004200 con 0x7fadc009f140 2024-09-16T06:40:54.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient(hunting): dump: 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165136277 0 0) 0x7fadc0094160 con 0x7fadc01361a0 2024-09-16T06:40:54.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 >> v1:172.21.15.76:6790/0 conn(0x7fadc009f140 legacy=0x7fadc0090ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 >> v1:172.21.15.189:6789/0 conn(0x7fadc0135d90 legacy=0x7fadc00966a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 monclient: found mon.a 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0091f90 con 0x7fadc01361a0 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 5 monclient: authenticate success, global_id 104607 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _renew_subs 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc00a1b80 con 0x7fadc01361a0 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _renew_subs 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc0096db0 con 0x7fadc01361a0 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb8004ec0 con 0x7fadc01361a0 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadb8001420 con 0x7fadc01361a0 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 10 monclient: dump: 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb77fe700 4 set_mon_vals no callback set 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fadb80052d0 con 0x7fadc01361a0 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.397+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(60..60 src has 1..60) v4 ==== 4305+0+0 (unknown 1339515136 0 0) 0x7fadb80438e0 con 0x7fadc01361a0 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.401+0000 7fadc5e0b700 10 monclient: get_auth_request con 0x7fada40362a0 auth_method 0 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.401+0000 7fadc806f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.401+0000 7fadc806f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.401+0000 7fadc806f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.401+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8005100 con 0x7fadc01361a0 2024-09-16T06:40:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.409+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fadb8006960 con 0x7fadc01361a0 2024-09-16T06:40:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.409+0000 7fadb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.409+0000 7fadb67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.577+0000 7fadc806f700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.577+0000 7fadc806f700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.577+0000 7fadc806f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.577+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fada8005930 con 0x7fadc01361a0 2024-09-16T06:40:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.581+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+60 (unknown 3969698029 0 1596542489) 0x7fadb8013230 con 0x7fadc01361a0 2024-09-16T06:40:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.581+0000 7fadb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.581+0000 7fadb67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fada8001140 con 0x7fadc01361a0 2024-09-16T06:40:54.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadb67fc700 1 -- 172.21.15.189:0/848870000 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+60 (unknown 3969698029 0 1596542489) 0x7fadb80149b0 con 0x7fadc01361a0 2024-09-16T06:40:54.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadb67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T06:40:54.693 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadb67fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 >> v1:172.21.15.189:6812/3098810674 conn(0x7fada40362a0 legacy=0x7fada4038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 10 monclient: shutdown 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 >> v1:172.21.15.76:6789/0 conn(0x7fadc01361a0 legacy=0x7fadc01d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 shutdown_connections 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 >> 172.21.15.189:0/848870000 conn(0x7fadc00a03e0 msgr2=0x7fadc009ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:54.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 shutdown_connections 2024-09-16T06:40:54.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.689+0000 7fadc806f700 1 -- 172.21.15.189:0/848870000 wait complete. 2024-09-16T06:40:54.705 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:54.706 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: build_initial_monmap 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: monmap: 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:54.813282+0000 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:54.813282+0000 2024-09-16T06:40:54.812 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:54.813 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.813 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:54.813 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:54.813 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:54.813 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 1 Processor -- start 2024-09-16T06:40:54.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 1 -- start start 2024-09-16T06:40:54.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: init 2024-09-16T06:40:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-b con 0x7f2b5809fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-a con 0x7f2b580717e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-c con 0x7f2b580a07d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58074a10 con 0x7f2b580717e0 2024-09-16T06:40:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58098450 con 0x7f2b580a07d0 2024-09-16T06:40:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580986f0 con 0x7f2b5809fce0 2024-09-16T06:40:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 10 monclient(hunting): authenticate will time out at 1007.326276s 2024-09-16T06:40:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b56d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2b580a07d0 legacy=0x7f2b5813d800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2b5809fce0 0x7f2b58098df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48212/0 (socket says 172.21.15.189:48212) 2024-09-16T06:40:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5659c700 1 -- 172.21.15.189:0/3924214717 learned_addr learned my addr 172.21.15.189:0/3924214717 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408160146 0 0) 0x7f2b58074a10 con 0x7f2b580717e0 2024-09-16T06:40:54.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): my global_id is 104610 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b400035d0 con 0x7f2b580717e0 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722455345 0 0) 0x7f2b580986f0 con 0x7f2b5809fce0 2024-09-16T06:40:54.821 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient(hunting): my global_id is 124304 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58074a10 con 0x7f2b5809fce0 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 175534048 0 0) 0x7f2b400035d0 con 0x7f2b580717e0 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 >> v1:172.21.15.76:6790/0 conn(0x7f2b5809fce0 legacy=0x7f2b58098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 >> v1:172.21.15.189:6789/0 conn(0x7f2b580a07d0 legacy=0x7f2b5813d800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 monclient: found mon.noname-a 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580746f0 con 0x7f2b580717e0 2024-09-16T06:40:54.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b48001610 con 0x7f2b580717e0 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 5 monclient: authenticate success, global_id 104610 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5559a700 10 monclient: dump: 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.823 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.809+0000 7f2b5759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b480018f0 con 0x7f2b580717e0 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 1 -- 172.21.15.189:0/3924214717 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b48001c80 con 0x7f2b580717e0 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 4 set_mon_vals no callback set 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.824 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5559a700 10 monclient: dump: 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: shutdown 2024-09-16T06:40:54.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- 172.21.15.189:0/3924214717 >> v1:172.21.15.76:6789/0 conn(0x7f2b580717e0 legacy=0x7f2b580a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- 172.21.15.189:0/3924214717 shutdown_connections 2024-09-16T06:40:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- 172.21.15.189:0/3924214717 >> 172.21.15.189:0/3924214717 conn(0x7f2b580a0fb0 msgr2=0x7f2b580a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- 172.21.15.189:0/3924214717 shutdown_connections 2024-09-16T06:40:54.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- 172.21.15.189:0/3924214717 wait complete. 2024-09-16T06:40:54.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: build_initial_monmap 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: monmap: 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 Processor -- start 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- start start 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: init 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-a con 0x7f2b580717e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-c con 0x7f2b5809fce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient(hunting): picked mon.noname-b con 0x7f2b580a07d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58099f30 con 0x7f2b580717e0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5808df90 con 0x7f2b580a07d0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5809b4c0 con 0x7f2b5809fce0 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:54.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5759e700 10 monclient(hunting): authenticate will time out at 1007.331137s 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b56d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2b580a07d0 legacy=0x7f2b581d2280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b580717e0 0x7f2b5808c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46294/0 (socket says 172.21.15.189:46294) 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b5659c700 1 -- 172.21.15.189:0/1098633273 learned_addr learned my addr 172.21.15.189:0/1098633273 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b55d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2b5809fce0 0x7f2b5808ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48218/0 (socket says 172.21.15.189:48218) 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310370884 0 0) 0x7f2b58074a10 con 0x7f2b580717e0 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 10 monclient(hunting): my global_id is 104613 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.813+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b380035d0 con 0x7f2b580717e0 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695830596 0 0) 0x7f2b5809b4c0 con 0x7f2b5809fce0 2024-09-16T06:40:54.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): my global_id is 124307 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58074a10 con 0x7f2b5809fce0 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729917878 0 0) 0x7f2b380035d0 con 0x7f2b580717e0 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b5809b4c0 con 0x7f2b580717e0 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b4c001690 con 0x7f2b580717e0 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:54.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.837 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170240170 0 0) 0x7f2b58074a10 con 0x7f2b5809fce0 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b380035d0 con 0x7f2b5809fce0 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b480015d0 con 0x7f2b5809fce0 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient(hunting): dump: 2024-09-16T06:40:54.838 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.839 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1550690478 0 0) 0x7f2b5809b4c0 con 0x7f2b580717e0 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 >> v1:172.21.15.76:6790/0 conn(0x7f2b5809fce0 legacy=0x7f2b5808ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 >> v1:172.21.15.189:6789/0 conn(0x7f2b580a07d0 legacy=0x7f2b581d2280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:54.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 monclient: found mon.a 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58093800 con 0x7f2b580717e0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 5 monclient: authenticate success, global_id 104613 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _renew_subs 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b58097f90 con 0x7f2b580717e0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _renew_subs 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58093e00 con 0x7f2b580717e0 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b4c001cf0 con 0x7f2b580717e0 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b4c004a30 con 0x7f2b580717e0 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b47fff700 4 set_mon_vals no callback set 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 10 monclient: dump: 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:54.842 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2b4c00f4a0 con 0x7f2b580717e0 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c005100 con 0x7f2b580717e0 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(60..60 src has 1..60) v4 ==== 4305+0+0 (unknown 1339515136 0 0) 0x7f2b4c0433b0 con 0x7f2b580717e0 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.817+0000 7f2b55d9b700 10 monclient: get_auth_request con 0x7f2b38036270 auth_method 0 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.825+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b4c010a40 con 0x7f2b580717e0 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.825+0000 7f2b46ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:54.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.825+0000 7f2b46ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.989+0000 7f2b5759e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.989+0000 7f2b5759e700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.989+0000 7f2b5759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:54.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:54.989+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f2b3c0053b0 con 0x7f2b580717e0 2024-09-16T06:40:55.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.781+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v61) v1 ==== 80+0+0 (unknown 1063520273 0 0) 0x7f2b4c012a80 con 0x7f2b580717e0 2024-09-16T06:40:55.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.785+0000 7f2b46ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:55.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.785+0000 7f2b46ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:55.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.889+0000 7f2b5759e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.889+0000 7f2b5759e700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.889+0000 7f2b5759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.889+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f2b3c005900 con 0x7f2b580717e0 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b46ffd700 1 -- 172.21.15.189:0/1098633273 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v61) v1 ==== 103+0+0 (unknown 1012277187 0 0) 0x7f2b4c014200 con 0x7f2b580717e0 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b46ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:55.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b46ffd700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2b38036270 legacy=0x7f2b38038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 10 monclient: shutdown 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 >> v1:172.21.15.76:6789/0 conn(0x7f2b580717e0 legacy=0x7f2b5808c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 shutdown_connections 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 >> 172.21.15.189:0/1098633273 conn(0x7f2b580a0fb0 msgr2=0x7f2b58098cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 shutdown_connections 2024-09-16T06:40:55.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:55.893+0000 7f2b5759e700 1 -- 172.21.15.189:0/1098633273 wait complete. 2024-09-16T06:40:55.899 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo does not exist 2024-09-16T06:40:55.908 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo 2024-09-16T06:40:56.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: get_monmap_and_config 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: build_initial_monmap 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: monmap: 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:56.016752+0000 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:56.016752+0000 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:56.016 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 Processor -- start 2024-09-16T06:40:56.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 -- start start 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: init 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7aac1352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7aac135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7aac0a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac09b050 con 0x7f7aac135db0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac06ab20 con 0x7f7aac1352c0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac06a840 con 0x7f7aac0a2030 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient(hunting): authenticate will time out at 1008.529647s 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaaffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7aac1352c0 legacy=0x7f7aac1356a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:56.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aa3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7aac135db0 0x7f7aac13a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46308/0 (socket says 172.21.15.189:46308) 2024-09-16T06:40:56.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aab7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7aac0a2030 0x7f7aac13df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48228/0 (socket says 172.21.15.189:48228) 2024-09-16T06:40:56.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aa3fff700 1 -- 172.21.15.189:0/3738422587 learned_addr learned my addr 172.21.15.189:0/3738422587 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683694975 0 0) 0x7f7aac09b050 con 0x7f7aac135db0 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): my global_id is 104616 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c0035d0 con 0x7f7aac135db0 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547268271 0 0) 0x7f7aac06a840 con 0x7f7aac0a2030 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient(hunting): my global_id is 124310 2024-09-16T06:40:56.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aac09b050 con 0x7f7aac0a2030 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166019349 0 0) 0x7f7a8c0035d0 con 0x7f7aac135db0 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 >> v1:172.21.15.76:6790/0 conn(0x7f7aac0a2030 legacy=0x7f7aac13df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 >> v1:172.21.15.189:6789/0 conn(0x7f7aac1352c0 legacy=0x7f7aac1356a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 monclient: found mon.noname-a 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac06c1b0 con 0x7f7aac135db0 2024-09-16T06:40:56.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 5 monclient: authenticate success, global_id 104616 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a94001610 con 0x7f7aac135db0 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: dump: 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a940018f0 con 0x7f7aac135db0 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 4 set_mon_vals no callback set 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 1 -- 172.21.15.189:0/3738422587 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a94001c80 con 0x7f7aac135db0 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:56.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aaa7fc700 10 monclient: dump: 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 10 monclient: shutdown 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.013+0000 7f7aabfff700 1 -- 172.21.15.189:0/3738422587 >> v1:172.21.15.76:6789/0 conn(0x7f7aac135db0 legacy=0x7f7aac13a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- 172.21.15.189:0/3738422587 shutdown_connections 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- 172.21.15.189:0/3738422587 >> 172.21.15.189:0/3738422587 conn(0x7f7aac0a4550 msgr2=0x7f7aac0a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- 172.21.15.189:0/3738422587 shutdown_connections 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- 172.21.15.189:0/3738422587 wait complete. 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient: build_initial_monmap 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient: monmap: 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:56.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 Processor -- start 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- start start 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient: init 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7aac0a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:56.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7aac1352c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7aac135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0a65f0 con 0x7f7aac1352c0 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac12dfe0 con 0x7f7aac0a2030 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aaaffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7aac0a2030 legacy=0x7f7aac12d1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac0a6130 con 0x7f7aac135db0 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aabfff700 10 monclient(hunting): authenticate will time out at 1008.534247s 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7aac1352c0 0x7f7aac12d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46324/0 (socket says 172.21.15.189:46324) 2024-09-16T06:40:56.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aab7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7aac135db0 0x7f7aac1d1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48238/0 (socket says 172.21.15.189:48238) 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa3fff700 1 -- 172.21.15.189:0/2709350163 learned_addr learned my addr 172.21.15.189:0/2709350163 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441992570 0 0) 0x7f7aac0a65f0 con 0x7f7aac1352c0 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): my global_id is 104619 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a900035d0 con 0x7f7aac1352c0 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402101188 0 0) 0x7f7aac0a6130 con 0x7f7aac135db0 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): my global_id is 124313 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aac0a65f0 con 0x7f7aac135db0 2024-09-16T06:40:56.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740636450 0 0) 0x7f7aac0a65f0 con 0x7f7aac135db0 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7aac0a6130 con 0x7f7aac135db0 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054232545 0 0) 0x7f7a900035d0 con 0x7f7aac1352c0 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7aac0a65f0 con 0x7f7aac1352c0 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7aa4001c30 con 0x7f7aac135db0 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): dump: 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a940015d0 con 0x7f7aac1352c0 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.017+0000 7f7aa37fe700 10 monclient(hunting): dump: 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536243072 0 0) 0x7f7aac0a6130 con 0x7f7aac135db0 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 >> v1:172.21.15.189:6789/0 conn(0x7f7aac0a2030 legacy=0x7f7aac12d1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 >> v1:172.21.15.76:6789/0 conn(0x7f7aac1352c0 legacy=0x7f7aac12d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 monclient: found mon.c 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:56.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac09e820 con 0x7f7aac135db0 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 5 monclient: authenticate success, global_id 124313 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _renew_subs 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac130d50 con 0x7f7aac135db0 2024-09-16T06:40:56.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _renew_subs 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac132c00 con 0x7f7aac135db0 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7aa4004c10 con 0x7f7aac135db0 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7aa4004f20 con 0x7f7aac135db0 2024-09-16T06:40:56.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa97fa700 4 set_mon_vals no callback set 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 10 monclient: dump: 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.040 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7aa400f990 con 0x7f7aac135db0 2024-09-16T06:40:56.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(61..61 src has 1..61) v4 ==== 4250+0+0 (unknown 28097275 0 0) 0x7f7aa40437a0 con 0x7f7aac135db0 2024-09-16T06:40:56.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aaaffd700 10 monclient: get_auth_request con 0x7f7a900362f0 auth_method 0 2024-09-16T06:40:56.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.021+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a84005100 con 0x7f7aac135db0 2024-09-16T06:40:56.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.033+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7aa4014b00 con 0x7f7aac135db0 2024-09-16T06:40:56.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.033+0000 7f7aa37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.033+0000 7f7aa37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:56.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aabfff700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aabfff700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aabfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f7a840053b0 con 0x7f7aac135db0 2024-09-16T06:40:56.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v61) v1 ==== 103+0+0 (unknown 1012277187 0 0) 0x7f7aa4012f70 con 0x7f7aac135db0 2024-09-16T06:40:56.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aa37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.197+0000 7f7aa37fe700 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T06:40:56.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.289+0000 7f7aabfff700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.289+0000 7f7aabfff700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.289+0000 7f7aabfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.289+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f7a84005b90 con 0x7f7aac135db0 2024-09-16T06:40:56.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aa37fe700 1 -- 172.21.15.189:0/2709350163 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v61) v1 ==== 103+0+0 (unknown 1012277187 0 0) 0x7f7aa40146f0 con 0x7f7aac135db0 2024-09-16T06:40:56.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aa37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T06:40:56.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aa37fe700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T06:40:56.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7a900362f0 legacy=0x7f7a900387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 10 monclient: shutdown 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 >> v1:172.21.15.76:6790/0 conn(0x7f7aac135db0 legacy=0x7f7aac1d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 shutdown_connections 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 >> 172.21.15.189:0/2709350163 conn(0x7f7aac0a4550 msgr2=0x7f7aac0a24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.293+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 shutdown_connections 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.297+0000 7f7aabfff700 1 -- 172.21.15.189:0/2709350163 wait complete. 2024-09-16T06:40:56.302 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo does not exist 2024-09-16T06:40:56.309 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm bar 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.409+0000 7f8f0801e700 10 monclient: get_monmap_and_config 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.409+0000 7f8f0801e700 10 monclient: build_initial_monmap 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.409+0000 7f8f0801e700 10 monclient: monmap: 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:56.416105+0000 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:56.416105+0000 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:56.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 1 Processor -- start 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 1 -- start start 2024-09-16T06:40:56.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient: init 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-b con 0x7f8f000a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-a con 0x7f8f000a3d10 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-c con 0x7f8f000a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:56.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f00099a40 con 0x7f8f000a3d10 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f065bb700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8f000a2350 legacy=0x7f8f0013dcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f00099ef0 con 0x7f8f000a2350 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0009a1d0 con 0x7f8f000a27a0 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 10 monclient(hunting): authenticate will time out at 1008.930717s 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f05dba700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8f000a27a0 0x7f8f000a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48250/0 (socket says 172.21.15.189:48250) 2024-09-16T06:40:56.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f05dba700 1 -- 172.21.15.189:0/2861018331 learned_addr learned my addr 172.21.15.189:0/2861018331 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258815254 0 0) 0x7f8f0009a1d0 con 0x7f8f000a27a0 2024-09-16T06:40:56.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): my global_id is 124316 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eec0035d0 con 0x7f8f000a27a0 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678379290 0 0) 0x7f8f00099a40 con 0x7f8f000a3d10 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient(hunting): my global_id is 104622 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f0009a1d0 con 0x7f8f000a3d10 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694516593 0 0) 0x7f8f0009a1d0 con 0x7f8f000a3d10 2024-09-16T06:40:56.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 >> v1:172.21.15.76:6790/0 conn(0x7f8f000a27a0 legacy=0x7f8f000a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 >> v1:172.21.15.189:6789/0 conn(0x7f8f000a2350 legacy=0x7f8f0013dcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: _finish_hunting 0 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 monclient: found mon.noname-a 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000a3290 con 0x7f8f000a3d10 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: _finish_auth 0 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: _check_auth_tickets 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ef00015f0 con 0x7f8f000a3d10 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f04db8700 10 monclient: dump: 2024-09-16T06:40:56.422 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 5 monclient: authenticate success, global_id 104622 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.413+0000 7f8f0801e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef0001910 con 0x7f8f000a3d10 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 1 -- 172.21.15.189:0/2861018331 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ef0001ce0 con 0x7f8f000a3d10 2024-09-16T06:40:56.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f04db8700 10 monclient: dump: 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 4 set_mon_vals no callback set 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 10 monclient: shutdown 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- 172.21.15.189:0/2861018331 >> v1:172.21.15.76:6789/0 conn(0x7f8f000a3d10 legacy=0x7f8f0013a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- 172.21.15.189:0/2861018331 shutdown_connections 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- 172.21.15.189:0/2861018331 >> 172.21.15.189:0/2861018331 conn(0x7f8f000a7d70 msgr2=0x7f8f000a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- 172.21.15.189:0/2861018331 shutdown_connections 2024-09-16T06:40:56.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- 172.21.15.189:0/2861018331 wait complete. 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 10 monclient: build_initial_monmap 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 10 monclient: monmap: 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:56.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 Processor -- start 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 1 -- start start 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.417+0000 7f8f0801e700 10 monclient: init 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-c con 0x7f8f000a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-a con 0x7f8f000a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient(hunting): picked mon.noname-b con 0x7f8f0013ecd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f001c71a0 con 0x7f8f000a27a0 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0009b400 con 0x7f8f0013ecd0 2024-09-16T06:40:56.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f065bb700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8f0013ecd0 legacy=0x7f8f001d1e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0006bea0 con 0x7f8f000a2350 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient(hunting): authenticate will time out at 1008.936667s 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f05dba700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8f000a2350 0x7f8f001c6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48260/0 (socket says 172.21.15.189:48260) 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f05dba700 1 -- 172.21.15.189:0/1142473184 learned_addr learned my addr 172.21.15.189:0/1142473184 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203287214 0 0) 0x7f8f0006bea0 con 0x7f8f000a2350 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): my global_id is 124319 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee40035d0 con 0x7f8f000a2350 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020044939 0 0) 0x7f8f001c71a0 con 0x7f8f000a27a0 2024-09-16T06:40:56.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): my global_id is 104625 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f0006bea0 con 0x7f8f000a27a0 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2373891943 0 0) 0x7f8ee40035d0 con 0x7f8f000a2350 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f001c71a0 con 0x7f8f000a2350 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8efc0019f0 con 0x7f8f000a2350 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): dump: 2024-09-16T06:40:56.432 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141503258 0 0) 0x7f8f0006bea0 con 0x7f8f000a27a0 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ee40035d0 con 0x7f8f000a27a0 2024-09-16T06:40:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ef0001c20 con 0x7f8f000a27a0 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient(hunting): dump: 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4216457020 0 0) 0x7f8f001c71a0 con 0x7f8f000a2350 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 >> v1:172.21.15.189:6789/0 conn(0x7f8f0013ecd0 legacy=0x7f8f001d1e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 >> v1:172.21.15.76:6789/0 conn(0x7f8f000a27a0 legacy=0x7f8f001c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 monclient: found mon.c 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0013cdb0 con 0x7f8f000a2350 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 5 monclient: authenticate success, global_id 124319 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _renew_subs 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0013e540 con 0x7f8f000a2350 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _renew_subs 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0013cff0 con 0x7f8f000a2350 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8efc001c60 con 0x7f8f000a2350 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8efc004ce0 con 0x7f8f000a2350 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef67fc700 10 monclient: dump: 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.436 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.421+0000 7f8ef7fff700 4 set_mon_vals no callback set 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8f0801e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8f0801e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8f0801e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee8005100 con 0x7f8f000a2350 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8efc00f750 con 0x7f8f000a2350 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(61..61 src has 1..61) v4 ==== 4250+0+0 (unknown 28097275 0 0) 0x7f8efc0435a0 con 0x7f8f000a2350 2024-09-16T06:40:56.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.425+0000 7f8f055b9700 10 monclient: get_auth_request con 0x7f8ee4036310 auth_method 0 2024-09-16T06:40:56.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.437+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8efc010c50 con 0x7f8f000a2350 2024-09-16T06:40:56.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.437+0000 7f8ef67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:56.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.437+0000 7f8ef67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:56.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.617+0000 7f8f0801e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.617+0000 7f8f0801e700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.617+0000 7f8f0801e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.617+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f8ee80053b0 con 0x7f8f000a2350 2024-09-16T06:40:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.793+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v62) v1 ==== 80+0+0 (unknown 2248278314 0 0) 0x7f8efc012d10 con 0x7f8f000a2350 2024-09-16T06:40:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.793+0000 7f8ef67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.793+0000 7f8ef67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:40:56.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8f0801e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8f0801e700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8f0801e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:56.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f8ee8005900 con 0x7f8f000a2350 2024-09-16T06:40:56.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8ef67fc700 1 -- 172.21.15.189:0/1142473184 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v62) v1 ==== 103+0+0 (unknown 2648600787 0 0) 0x7f8efc014490 con 0x7f8f000a2350 2024-09-16T06:40:56.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8ef67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T06:40:56.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.909+0000 7f8ef67fc700 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T06:40:56.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8ee4036310 legacy=0x7f8ee40387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 10 monclient: shutdown 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 >> v1:172.21.15.76:6790/0 conn(0x7f8f000a2350 legacy=0x7f8f001c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 shutdown_connections 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 >> 172.21.15.189:0/1142473184 conn(0x7f8f000a7d70 msgr2=0x7f8f000a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 shutdown_connections 2024-09-16T06:40:56.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:56.913+0000 7f8f0801e700 1 -- 172.21.15.189:0/1142473184 wait complete. 2024-09-16T06:40:56.918 INFO:tasks.workunit.client.0.smithi189.stderr:rule bar does not exist 2024-09-16T06:40:56.927 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create pinning_pool 1 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 10 monclient: get_monmap_and_config 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 10 monclient: build_initial_monmap 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 10 monclient: monmap: 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:57.031213+0000 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:57.031213+0000 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:57.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:57.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:57.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 1 Processor -- start 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 1 -- start start 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.025+0000 7ff02a474700 10 monclient: init 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-c con 0x7ff02409e560 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-b con 0x7ff024132c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-a con 0x7ff024133090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02406ba80 con 0x7ff024133090 2024-09-16T06:40:57.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02406b860 con 0x7ff02409e560 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff023fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff02409e560 legacy=0x7ff024132570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02409e940 con 0x7ff024132c80 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient(hunting): authenticate will time out at 1009.544330s 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff028a11700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff024133090 0x7ff02413aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46342/0 (socket says 172.21.15.189:46342) 2024-09-16T06:40:57.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff028a11700 1 -- 172.21.15.189:0/1625012460 learned_addr learned my addr 172.21.15.189:0/1625012460 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464147013 0 0) 0x7ff02409e940 con 0x7ff024132c80 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): my global_id is 124322 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c0035d0 con 0x7ff024132c80 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483007123 0 0) 0x7ff02406ba80 con 0x7ff024133090 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient(hunting): my global_id is 104628 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff02409e940 con 0x7ff024133090 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654005120 0 0) 0x7ff00c0035d0 con 0x7ff024132c80 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 >> v1:172.21.15.189:6789/0 conn(0x7ff02409e560 legacy=0x7ff024132570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:57.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 >> v1:172.21.15.76:6789/0 conn(0x7ff024133090 legacy=0x7ff02413aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 monclient: found mon.noname-b 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02409e0b0 con 0x7ff024132c80 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: _finish_auth 0 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff014001610 con 0x7ff024132c80 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 5 monclient: authenticate success, global_id 124322 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:57.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: dump: 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:57.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff014001880 con 0x7ff024132c80 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 1 -- 172.21.15.189:0/1625012460 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff014001c10 con 0x7ff024132c80 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 4 set_mon_vals no callback set 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff022ffd700 10 monclient: dump: 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: shutdown 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- 172.21.15.189:0/1625012460 >> v1:172.21.15.76:6790/0 conn(0x7ff024132c80 legacy=0x7ff0241377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- 172.21.15.189:0/1625012460 shutdown_connections 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- 172.21.15.189:0/1625012460 >> 172.21.15.189:0/1625012460 conn(0x7ff0240a1100 msgr2=0x7ff0240a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- 172.21.15.189:0/1625012460 shutdown_connections 2024-09-16T06:40:57.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- 172.21.15.189:0/1625012460 wait complete. 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: build_initial_monmap 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: monmap: 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 Processor -- start 2024-09-16T06:40:57.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 1 -- start start 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.029+0000 7ff02a474700 10 monclient: init 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-c con 0x7ff02409e560 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-b con 0x7ff024132c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient(hunting): picked mon.noname-a con 0x7ff024133090 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0240a0650 con 0x7ff024133090 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02406b020 con 0x7ff024132c80 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff024130650 con 0x7ff02409e560 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient(hunting): authenticate will time out at 1009.548782s 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff0237fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff024132c80 legacy=0x7ff02412f1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:57.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff028a11700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff024133090 0x7ff02412f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46350/0 (socket says 172.21.15.189:46350) 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff023fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff02409e560 0x7ff02412eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48278/0 (socket says 172.21.15.189:48278) 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff028a11700 1 -- 172.21.15.189:0/3310960022 learned_addr learned my addr 172.21.15.189:0/3310960022 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873078500 0 0) 0x7ff024130650 con 0x7ff02409e560 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): my global_id is 124325 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0100035d0 con 0x7ff02409e560 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122205362 0 0) 0x7ff0240a0650 con 0x7ff024133090 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:57.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): my global_id is 104631 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024130650 con 0x7ff024133090 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469599466 0 0) 0x7ff0100035d0 con 0x7ff02409e560 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0240a0650 con 0x7ff02409e560 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff018004170 con 0x7ff02409e560 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058749860 0 0) 0x7ff024130650 con 0x7ff024133090 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0100035d0 con 0x7ff024133090 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff01c001c30 con 0x7ff024133090 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient(hunting): dump: 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:57.051 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4274351595 0 0) 0x7ff0240a0650 con 0x7ff02409e560 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 >> v1:172.21.15.189:6789/0 conn(0x7ff024132c80 legacy=0x7ff02412f1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 >> v1:172.21.15.76:6789/0 conn(0x7ff024133090 legacy=0x7ff02412f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:40:57.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 monclient: found mon.c 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02408b450 con 0x7ff02409e560 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: _finish_auth 0 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 5 monclient: authenticate success, global_id 124325 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _renew_subs 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff024130f60 con 0x7ff02409e560 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _renew_subs 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0241d3620 con 0x7ff02409e560 2024-09-16T06:40:57.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0180043e0 con 0x7ff02409e560 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff018004770 con 0x7ff02409e560 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff0227fc700 4 set_mon_vals no callback set 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 10 monclient: dump: 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:57.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.033+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff018004990 con 0x7ff02409e560 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff02a474700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff02a474700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff02a474700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff004005100 con 0x7ff02409e560 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(62..62 src has 1..62) v4 ==== 4195+0+0 (unknown 1847986175 0 0) 0x7ff018043300 con 0x7ff02409e560 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.037+0000 7ff0237fe700 10 monclient: get_auth_request con 0x7ff010036230 auth_method 0 2024-09-16T06:40:57.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.045+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff018014770 con 0x7ff02409e560 2024-09-16T06:40:57.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.045+0000 7ff020ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:57.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.045+0000 7ff020ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:57.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.221+0000 7ff02a474700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.221+0000 7ff02a474700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.221+0000 7ff02a474700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.221+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7ff004005930 con 0x7ff02409e560 2024-09-16T06:40:57.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.801+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v63) v1 ==== 127+0+0 (unknown 3136886715 0 0) 0x7ff018012be0 con 0x7ff02409e560 2024-09-16T06:40:57.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.801+0000 7ff020ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.801+0000 7ff020ff9700 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-16T06:40:57.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff02a474700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff02a474700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff02a474700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:57.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7ff0040053f0 con 0x7ff02409e560 2024-09-16T06:40:57.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff020ff9700 1 -- 172.21.15.189:0/3310960022 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v63) v1 ==== 134+0+0 (unknown 212549570 0 0) 0x7ff018014360 con 0x7ff02409e560 2024-09-16T06:40:57.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff020ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T06:40:57.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.913+0000 7ff020ff9700 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-16T06:40:57.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff010036230 legacy=0x7ff0100386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:57.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 10 monclient: shutdown 2024-09-16T06:40:57.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:57.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 >> v1:172.21.15.76:6790/0 conn(0x7ff02409e560 legacy=0x7ff02412eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:57.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 shutdown_connections 2024-09-16T06:40:57.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 >> 172.21.15.189:0/3310960022 conn(0x7ff0240a1100 msgr2=0x7ff024099990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:57.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 shutdown_connections 2024-09-16T06:40:57.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:57.917+0000 7ff02a474700 1 -- 172.21.15.189:0/3310960022 wait complete. 2024-09-16T06:40:57.923 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'pinning_pool' already exists 2024-09-16T06:40:57.932 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-16T06:40:57.932 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:57.932 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-16T06:40:58.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: get_monmap_and_config 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: build_initial_monmap 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: monmap: 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:58.041924+0000 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:58.041924+0000 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:58.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:58.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 1 Processor -- start 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 1 -- start start 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: init 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-a con 0x7f625c06a340 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:58.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-b con 0x7f625c0a19f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-c con 0x7f625c06ae30 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c09f8e0 con 0x7f625c06a340 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c09c960 con 0x7f625c06ae30 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c0990c0 con 0x7f625c0a19f0 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6262dee700 10 monclient(hunting): authenticate will time out at 1010.555093s 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f62625ed700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f625c06ae30 legacy=0x7f625c13dea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f62615eb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f625c0a19f0 0x7f625c13a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48284/0 (socket says 172.21.15.189:48284) 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f62615eb700 1 -- 172.21.15.189:0/1638016322 learned_addr learned my addr 172.21.15.189:0/1638016322 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.037+0000 7f6261dec700 1 --1- 172.21.15.189:0/1638016322 >> v1:172.21.15.76:6789/0 conn(0x7f625c06a340 0x7f625c06a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46366/0 (socket says 172.21.15.189:46366) 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754082194 0 0) 0x7f625c0990c0 con 0x7f625c0a19f0 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): my global_id is 124328 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62480035d0 con 0x7f625c0a19f0 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164291720 0 0) 0x7f625c09f8e0 con 0x7f625c06a340 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient(hunting): my global_id is 104634 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c0990c0 con 0x7f625c06a340 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368091192 0 0) 0x7f62480035d0 con 0x7f625c0a19f0 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 >> v1:172.21.15.189:6789/0 conn(0x7f625c06ae30 legacy=0x7f625c13dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 >> v1:172.21.15.76:6789/0 conn(0x7f625c06a340 legacy=0x7f625c06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: _finish_hunting 0 2024-09-16T06:40:58.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 monclient: found mon.noname-b 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c0a1570 con 0x7f625c0a19f0 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: _finish_auth 0 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: _check_auth_tickets 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 5 monclient: authenticate success, global_id 124328 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6258001540 con 0x7f625c0a19f0 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: dump: 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6258001820 con 0x7f625c0a19f0 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:58.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 1 -- 172.21.15.189:0/1638016322 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6258001bb0 con 0x7f625c0a19f0 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 4 set_mon_vals no callback set 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6260dea700 10 monclient: dump: 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: shutdown 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- 172.21.15.189:0/1638016322 >> v1:172.21.15.76:6790/0 conn(0x7f625c0a19f0 legacy=0x7f625c13a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- 172.21.15.189:0/1638016322 shutdown_connections 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- 172.21.15.189:0/1638016322 >> 172.21.15.189:0/1638016322 conn(0x7f625c0a20a0 msgr2=0x7f625c0a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- 172.21.15.189:0/1638016322 shutdown_connections 2024-09-16T06:40:58.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- 172.21.15.189:0/1638016322 wait complete. 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: build_initial_monmap 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: monmap: 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:58.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 Processor -- start 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- start start 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: init 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-b con 0x7f625c06a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-c con 0x7f625c06ae30 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient(hunting): picked mon.noname-a con 0x7f625c0a19f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c094230 con 0x7f625c0a19f0 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c09c5b0 con 0x7f625c06a340 2024-09-16T06:40:58.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c093c70 con 0x7f625c06ae30 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6262dee700 10 monclient(hunting): authenticate will time out at 1010.559721s 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.041+0000 7f6261dec700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f625c06a340 legacy=0x7f625c097a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62615eb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f625c06ae30 0x7f625c1ceda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48300/0 (socket says 172.21.15.189:48300) 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62625ed700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f625c0a19f0 0x7f625c1d24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46374/0 (socket says 172.21.15.189:46374) 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62615eb700 1 -- 172.21.15.189:0/358756010 learned_addr learned my addr 172.21.15.189:0/358756010 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548973029 0 0) 0x7f625c093c70 con 0x7f625c06ae30 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): my global_id is 124331 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62400035d0 con 0x7f625c06ae30 2024-09-16T06:40:58.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352221582 0 0) 0x7f625c094230 con 0x7f625c0a19f0 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): my global_id is 104637 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c093c70 con 0x7f625c0a19f0 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361057487 0 0) 0x7f62400035d0 con 0x7f625c06ae30 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f625c094230 con 0x7f625c06ae30 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62580015c0 con 0x7f625c06ae30 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): dump: 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2129822733 0 0) 0x7f625c093c70 con 0x7f625c0a19f0 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62400035d0 con 0x7f625c0a19f0 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6254004360 con 0x7f625c0a19f0 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient(hunting): dump: 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2097188157 0 0) 0x7f625c094230 con 0x7f625c06ae30 2024-09-16T06:40:58.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 >> v1:172.21.15.189:6789/0 conn(0x7f625c06a340 legacy=0x7f625c097a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 >> v1:172.21.15.76:6789/0 conn(0x7f625c0a19f0 legacy=0x7f625c1d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: _finish_hunting 0 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 monclient: found mon.c 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c093f50 con 0x7f625c06ae30 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: _finish_auth 0 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: _check_auth_tickets 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 5 monclient: authenticate success, global_id 124331 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 10 monclient: _renew_subs 2024-09-16T06:40:58.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c0924e0 con 0x7f625c06ae30 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 10 monclient: _renew_subs 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6262dee700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c098350 con 0x7f625c06ae30 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6258004bc0 con 0x7f625c06ae30 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6258004db0 con 0x7f625c06ae30 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f6253fff700 4 set_mon_vals no callback set 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 10 monclient: dump: 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.045+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6258004f90 con 0x7f625c06ae30 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(63..63 src has 1..63) v4 ==== 4546+0+0 (unknown 2725984662 0 0) 0x7f62580436f0 con 0x7f625c06ae30 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f6262dee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f6262dee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f6262dee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f6262dee700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6244005100 con 0x7f625c06ae30 2024-09-16T06:40:58.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.049+0000 7f6261dec700 10 monclient: get_auth_request con 0x7f62400362c0 auth_method 0 2024-09-16T06:40:58.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.057+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f62580067d0 con 0x7f625c06ae30 2024-09-16T06:40:58.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.057+0000 7f62527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.057+0000 7f62527fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:58.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.237+0000 7f6262dee700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T06:40:58.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.237+0000 7f6262dee700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T06:40:58.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.237+0000 7f6262dee700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:58.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.237+0000 7f6262dee700 1 -- 172.21.15.189:0/358756010 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7f6244005930 con 0x7f625c06ae30 2024-09-16T06:40:58.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.237+0000 7f62527fc700 1 -- 172.21.15.189:0/358756010 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v63) v1 ==== 132+0+0 (unknown 3553680343 0 0) 0x7f6258012eb0 con 0x7f625c06ae30 2024-09-16T06:40:58.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.241+0000 7f62527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T06:40:58.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.241+0000 7f62527fc700 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-16T06:40:58.245 INFO:tasks.workunit.client.0.smithi189.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-16T06:40:58.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:58.249 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-16T06:40:58.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: get_monmap_and_config 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: build_initial_monmap 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: monmap: 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:58.353712+0000 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:58.353712+0000 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:58.353 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 1 Processor -- start 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 1 -- start start 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: init 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-b con 0x7f63ac06a8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-c con 0x7f63ac0a6090 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-a con 0x7f63ac13bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac06acb0 con 0x7f63ac13bab0 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63a9d9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f63ac0a6090 legacy=0x7f63ac13a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:58.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac099d60 con 0x7f63ac0a6090 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac09a040 con 0x7f63ac06a8d0 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63b090d700 10 monclient(hunting): authenticate will time out at 1010.867254s 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63aad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f63ac13bab0 0x7f63ac13dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46378/0 (socket says 172.21.15.189:46378) 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63aad9d700 1 -- 172.21.15.189:0/4285355111 learned_addr learned my addr 172.21.15.189:0/4285355111 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:58.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.349+0000 7f63aa59c700 1 --1- 172.21.15.189:0/4285355111 >> v1:172.21.15.76:6790/0 conn(0x7f63ac06a8d0 0x7f63ac0a5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48302/0 (socket says 172.21.15.189:48302) 2024-09-16T06:40:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790593867 0 0) 0x7f63ac06acb0 con 0x7f63ac13bab0 2024-09-16T06:40:58.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): my global_id is 104640 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63940035d0 con 0x7f63ac13bab0 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389210404 0 0) 0x7f63ac09a040 con 0x7f63ac06a8d0 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient(hunting): my global_id is 124334 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac06acb0 con 0x7f63ac06a8d0 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293750936 0 0) 0x7f63940035d0 con 0x7f63ac13bab0 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 >> v1:172.21.15.76:6790/0 conn(0x7f63ac06a8d0 legacy=0x7f63ac0a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 >> v1:172.21.15.189:6789/0 conn(0x7f63ac0a6090 legacy=0x7f63ac13a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: _finish_hunting 0 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 monclient: found mon.noname-a 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac09a320 con 0x7f63ac13bab0 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: _finish_auth 0 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: _check_auth_tickets 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f63a4001990 con 0x7f63ac13bab0 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 5 monclient: authenticate success, global_id 104640 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:58.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: dump: 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:58.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a4001c70 con 0x7f63ac13bab0 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 4 set_mon_vals no callback set 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 1 -- 172.21.15.189:0/4285355111 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f63a4002000 con 0x7f63ac13bab0 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63a959a700 10 monclient: dump: 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 10 monclient: shutdown 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 1 -- 172.21.15.189:0/4285355111 >> v1:172.21.15.76:6789/0 conn(0x7f63ac13bab0 legacy=0x7f63ac13dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 1 -- 172.21.15.189:0/4285355111 shutdown_connections 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 1 -- 172.21.15.189:0/4285355111 >> 172.21.15.189:0/4285355111 conn(0x7f63ac0a7bf0 msgr2=0x7f63ac0a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 1 -- 172.21.15.189:0/4285355111 shutdown_connections 2024-09-16T06:40:58.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 1 -- 172.21.15.189:0/4285355111 wait complete. 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 10 monclient: build_initial_monmap 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.353+0000 7f63b090d700 10 monclient: monmap: 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:58.368 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 1 Processor -- start 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 1 -- start start 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient: init 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-c con 0x7f63ac06a8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-b con 0x7f63ac0a6090 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient(hunting): picked mon.noname-a con 0x7f63ac13bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac11ff20 con 0x7f63ac13bab0 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac06bd10 con 0x7f63ac0a6090 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63a9d9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f63ac0a6090 legacy=0x7f63ac11e530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:58.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac1207c0 con 0x7f63ac06a8d0 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63b090d700 10 monclient(hunting): authenticate will time out at 1010.874282s 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63aad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f63ac13bab0 0x7f63ac1d20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46394/0 (socket says 172.21.15.189:46394) 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63aad9d700 1 -- 172.21.15.189:0/2315095027 learned_addr learned my addr 172.21.15.189:0/2315095027 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f63aa59c700 1 --1- 172.21.15.189:0/2315095027 >> v1:172.21.15.76:6790/0 conn(0x7f63ac06a8d0 0x7f63ac11de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48318/0 (socket says 172.21.15.189:48318) 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518313880 0 0) 0x7f63ac11ff20 con 0x7f63ac13bab0 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): my global_id is 104643 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0035d0 con 0x7f63ac13bab0 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018295878 0 0) 0x7f63ac1207c0 con 0x7f63ac06a8d0 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _negotiate 2024-09-16T06:40:58.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): my global_id is 124337 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac11ff20 con 0x7f63ac06a8d0 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580572070 0 0) 0x7f638c0035d0 con 0x7f63ac13bab0 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63ac1207c0 con 0x7f63ac13bab0 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f63a4001950 con 0x7f63ac13bab0 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): dump: 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753980804 0 0) 0x7f63ac11ff20 con 0x7f63ac06a8d0 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f638c0035d0 con 0x7f63ac06a8d0 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f63a0001790 con 0x7f63ac06a8d0 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.357+0000 7f639affd700 10 monclient(hunting): dump: 2024-09-16T06:40:58.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763157506 0 0) 0x7f63ac1207c0 con 0x7f63ac13bab0 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 >> v1:172.21.15.76:6790/0 conn(0x7f63ac06a8d0 legacy=0x7f63ac11de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 >> v1:172.21.15.189:6789/0 conn(0x7f63ac0a6090 legacy=0x7f63ac11e530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: _finish_hunting 0 2024-09-16T06:40:58.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 monclient: found mon.a 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac11ec40 con 0x7f63ac13bab0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: _finish_auth 0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: _check_auth_tickets 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 5 monclient: authenticate success, global_id 104643 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _renew_subs 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac11f030 con 0x7f63ac13bab0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _renew_subs 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac0a0180 con 0x7f63ac13bab0 2024-09-16T06:40:58.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a40014c0 con 0x7f63ac13bab0 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f63a40051c0 con 0x7f63ac13bab0 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63a8d99700 4 set_mon_vals no callback set 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 10 monclient: dump: 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:58.375 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6390005100 con 0x7f63ac13bab0 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f63a400fc30 con 0x7f63ac13bab0 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(63..63 src has 1..63) v4 ==== 4546+0+0 (unknown 2725984662 0 0) 0x7f63a4043b10 con 0x7f63ac13bab0 2024-09-16T06:40:58.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.361+0000 7f63aa59c700 10 monclient: get_auth_request con 0x7f638c036340 auth_method 0 2024-09-16T06:40:58.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.373+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f63a4014b10 con 0x7f63ac13bab0 2024-09-16T06:40:58.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.373+0000 7f639affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:58.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.373+0000 7f639affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:58.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.545+0000 7f63b090d700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.545+0000 7f63b090d700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.545+0000 7f63b090d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.545+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63900053b0 con 0x7f63ac13bab0 2024-09-16T06:40:58.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.809+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v64) v1 ==== 172+0+0 (unknown 2486260894 0 0) 0x7f63a40131b0 con 0x7f63ac13bab0 2024-09-16T06:40:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.809+0000 7f639affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.809+0000 7f639affd700 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-16T06:40:58.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63900010a0 con 0x7f63ac13bab0 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f639affd700 1 -- 172.21.15.189:0/2315095027 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v64) v1 ==== 179+0+0 (unknown 2435180744 0 0) 0x7f63a4014930 con 0x7f63ac13bab0 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f639affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:40:58.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f639affd700 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-16T06:40:58.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 >> v1:172.21.15.189:6812/3098810674 conn(0x7f638c036340 legacy=0x7f638c038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 10 monclient: shutdown 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 >> v1:172.21.15.76:6789/0 conn(0x7f63ac13bab0 legacy=0x7f63ac1d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 shutdown_connections 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 >> 172.21.15.189:0/2315095027 conn(0x7f63ac0a7bf0 msgr2=0x7f63ac139910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 shutdown_connections 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:58.925+0000 7f63b090d700 1 -- 172.21.15.189:0/2315095027 wait complete. 2024-09-16T06:40:58.931 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'pinning_pool' does not exist 2024-09-16T06:40:58.940 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-16T06:40:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:40:58.941 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo osd 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: get_monmap_and_config 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: build_initial_monmap 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: monmap: 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:59.049083+0000 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:59.049083+0000 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:59.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:59.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:59.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 1 Processor -- start 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 1 -- start start 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: init 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-b con 0x7fa3d00974f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-a con 0x7fa3d0097fe0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-c con 0x7fa3d0140170 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d0099d40 con 0x7fa3d0097fe0 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d00999d0 con 0x7fa3d0140170 2024-09-16T06:40:59.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d00987a0 con 0x7fa3d00974f0 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient(hunting): authenticate will time out at 1011.561925s 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d59d9700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa3d0140170 legacy=0x7fa3d0142560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d49d7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa3d0097fe0 0x7fa3d013ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50882/0 (socket says 172.21.15.189:50882) 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d51d8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa3d00974f0 0x7fa3d00978d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48846/0 (socket says 172.21.15.189:48846) 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d49d7700 1 -- 172.21.15.189:0/2037373354 learned_addr learned my addr 172.21.15.189:0/2037373354 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858972723 0 0) 0x7fa3d0099d40 con 0x7fa3d0097fe0 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): my global_id is 104646 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b80035d0 con 0x7fa3d0097fe0 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190425386 0 0) 0x7fa3d00987a0 con 0x7fa3d00974f0 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient(hunting): my global_id is 124340 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d0099d40 con 0x7fa3d00974f0 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893771615 0 0) 0x7fa3b80035d0 con 0x7fa3d0097fe0 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 >> v1:172.21.15.76:6790/0 conn(0x7fa3d00974f0 legacy=0x7fa3d00978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 >> v1:172.21.15.189:6789/0 conn(0x7fa3d0140170 legacy=0x7fa3d0142560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: _finish_hunting 0 2024-09-16T06:40:59.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 monclient: found mon.noname-a 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00996f0 con 0x7fa3d0097fe0 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: _finish_auth 0 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: _check_auth_tickets 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 5 monclient: authenticate success, global_id 104646 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa3c0001990 con 0x7fa3d0097fe0 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:59.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: dump: 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.056 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c0001c70 con 0x7fa3d0097fe0 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3d743c700 4 set_mon_vals no callback set 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 1 -- 172.21.15.189:0/2037373354 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa3c0002000 con 0x7fa3d0097fe0 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.045+0000 7fa3cffff700 10 monclient: dump: 2024-09-16T06:40:59.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.058 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.059 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.059 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.059 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: shutdown 2024-09-16T06:40:59.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:59.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- 172.21.15.189:0/2037373354 >> v1:172.21.15.76:6789/0 conn(0x7fa3d0097fe0 legacy=0x7fa3d013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- 172.21.15.189:0/2037373354 shutdown_connections 2024-09-16T06:40:59.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- 172.21.15.189:0/2037373354 >> 172.21.15.189:0/2037373354 conn(0x7fa3d00a03e0 msgr2=0x7fa3d00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:59.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- 172.21.15.189:0/2037373354 shutdown_connections 2024-09-16T06:40:59.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- 172.21.15.189:0/2037373354 wait complete. 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: build_initial_monmap 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: monmap: 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:59.064 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 Processor -- start 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- start start 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: init 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:40:59.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-a con 0x7fa3d00974f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-b con 0x7fa3d0097fe0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient(hunting): picked mon.noname-c con 0x7fa3d0140170 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d013fd10 con 0x7fa3d00974f0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d00a2520 con 0x7fa3d0097fe0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d009a6e0 con 0x7fa3d0140170 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d743c700 10 monclient(hunting): authenticate will time out at 1011.566587s 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d49d7700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa3d0097fe0 legacy=0x7fa3d01d3740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d59d9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa3d0140170 0x7fa3d01d6e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48860/0 (socket says 172.21.15.189:48860) 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3d59d9700 1 -- 172.21.15.189:0/2773492817 learned_addr learned my addr 172.21.15.189:0/2773492817 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759255564 0 0) 0x7fa3d009a6e0 con 0x7fa3d0140170 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): my global_id is 124343 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3bc0035d0 con 0x7fa3d0140170 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428158428 0 0) 0x7fa3d013fd10 con 0x7fa3d00974f0 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): my global_id is 104649 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d009a6e0 con 0x7fa3d00974f0 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3854238810 0 0) 0x7fa3bc0035d0 con 0x7fa3d0140170 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3d013fd10 con 0x7fa3d0140170 2024-09-16T06:40:59.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa3c80040e0 con 0x7fa3d0140170 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): dump: 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.068 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.049+0000 7fa3cdffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2823830909 0 0) 0x7fa3d009a6e0 con 0x7fa3d00974f0 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3bc0035d0 con 0x7fa3d00974f0 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa3c40017d0 con 0x7fa3d00974f0 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient(hunting): dump: 2024-09-16T06:40:59.069 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.070 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.070 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.070 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.070 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.070 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3004129993 0 0) 0x7fa3d013fd10 con 0x7fa3d0140170 2024-09-16T06:40:59.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 >> v1:172.21.15.189:6789/0 conn(0x7fa3d0097fe0 legacy=0x7fa3d01d3740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 >> v1:172.21.15.76:6789/0 conn(0x7fa3d00974f0 legacy=0x7fa3d0138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 monclient: found mon.c 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d006dde0 con 0x7fa3d0140170 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: _finish_auth 0 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 5 monclient: authenticate success, global_id 124343 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _renew_subs 2024-09-16T06:40:59.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d006d500 con 0x7fa3d0140170 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _renew_subs 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d013f190 con 0x7fa3d0140170 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c8004350 con 0x7fa3d0140170 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:59.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa3c80046e0 con 0x7fa3d0140170 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cf7fe700 4 set_mon_vals no callback set 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 10 monclient: dump: 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.074 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa3c8004900 con 0x7fa3d0140170 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d743c700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b0005100 con 0x7fa3d0140170 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(64..64 src has 1..64) v4 ==== 4195+0+0 (unknown 3830037545 0 0) 0x7fa3c8043250 con 0x7fa3d0140170 2024-09-16T06:40:59.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.053+0000 7fa3d51d8700 10 monclient: get_auth_request con 0x7fa3bc0362c0 auth_method 0 2024-09-16T06:40:59.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.065+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa3c80146e0 con 0x7fa3d0140170 2024-09-16T06:40:59.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.065+0000 7fa3cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.065+0000 7fa3cdffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:59.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.237+0000 7fa3d743c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T06:40:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.237+0000 7fa3d743c700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T06:40:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.237+0000 7fa3d743c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.237+0000 7fa3d743c700 1 -- 172.21.15.189:0/2773492817 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7fa3b0005930 con 0x7fa3d0140170 2024-09-16T06:40:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.241+0000 7fa3cdffb700 1 -- 172.21.15.189:0/2773492817 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v64) v1 ==== 136+0+0 (unknown 3899507164 0 0) 0x7fa3c8012b50 con 0x7fa3d0140170 2024-09-16T06:40:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.241+0000 7fa3cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T06:40:59.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.241+0000 7fa3cdffb700 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-16T06:40:59.247 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-16T06:40:59.250 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:40:59.251 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo root 2024-09-16T06:40:59.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: get_monmap_and_config 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: build_initial_monmap 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: monmap: 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:40:59.360812+0000 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:40:59.360812+0000 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.360 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 1 Processor -- start 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 1 -- start start 2024-09-16T06:40:59.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: init 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-b con 0x7f5e3009f5e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-a con 0x7f5e300a00d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-c con 0x7f5e30138d00 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3009dc50 con 0x7f5e300a00d0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3009afd0 con 0x7f5e30138d00 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e34d46700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5e30138d00 legacy=0x7f5e3013b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3009c280 con 0x7f5e3009f5e0 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e367a9700 10 monclient(hunting): authenticate will time out at 1011.875144s 2024-09-16T06:40:59.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e277fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5e300a00d0 0x7f5e301375e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50902/0 (socket says 172.21.15.189:50902) 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e277fe700 1 -- 172.21.15.189:0/3423779293 learned_addr learned my addr 172.21.15.189:0/3423779293 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.357+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723960624 0 0) 0x7f5e3009dc50 con 0x7f5e300a00d0 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): my global_id is 104652 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c0035d0 con 0x7f5e300a00d0 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526971749 0 0) 0x7f5e3009c280 con 0x7f5e3009f5e0 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient(hunting): my global_id is 124346 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3009dc50 con 0x7f5e3009f5e0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065690135 0 0) 0x7f5e1c0035d0 con 0x7f5e300a00d0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 >> v1:172.21.15.76:6790/0 conn(0x7f5e3009f5e0 legacy=0x7f5e3009f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 >> v1:172.21.15.189:6789/0 conn(0x7f5e30138d00 legacy=0x7f5e3013b0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3009c560 con 0x7f5e300a00d0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:40:59.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 5 monclient: authenticate success, global_id 104652 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e18001610 con 0x7f5e300a00d0 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: dump: 2024-09-16T06:40:59.369 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e180018a0 con 0x7f5e300a00d0 2024-09-16T06:40:59.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 10 monclient: get_monmap_and_config success 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 1 -- 172.21.15.189:0/3423779293 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e18001c30 con 0x7f5e300a00d0 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 4 set_mon_vals no callback set 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e2f7fe700 10 monclient: dump: 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 10 monclient: shutdown 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 1 -- 172.21.15.189:0/3423779293 >> v1:172.21.15.76:6789/0 conn(0x7f5e300a00d0 legacy=0x7f5e301375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 1 -- 172.21.15.189:0/3423779293 shutdown_connections 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 1 -- 172.21.15.189:0/3423779293 >> 172.21.15.189:0/3423779293 conn(0x7f5e300a44e0 msgr2=0x7f5e300a06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 1 -- 172.21.15.189:0/3423779293 shutdown_connections 2024-09-16T06:40:59.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 1 -- 172.21.15.189:0/3423779293 wait complete. 2024-09-16T06:40:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 10 monclient: build_initial_monmap 2024-09-16T06:40:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.361+0000 7f5e367a9700 10 monclient: monmap: 2024-09-16T06:40:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:40:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:40:59.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 Processor -- start 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- start start 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: init 2024-09-16T06:40:59.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-c con 0x7f5e3009f5e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-b con 0x7f5e300a00d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient(hunting): picked mon.noname-a con 0x7f5e30138d00 addr v1:172.21.15.76:6789/0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3009b600 con 0x7f5e30138d00 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e30135a70 con 0x7f5e300a00d0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e30135d50 con 0x7f5e3009f5e0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e277fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5e300a00d0 legacy=0x7f5e301cc0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:40:59.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient(hunting): authenticate will time out at 1011.881948s 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e3009f5e0 0x7f5e301c70f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48884/0 (socket says 172.21.15.189:48884) 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2ffff700 1 -- 172.21.15.189:0/3269996583 learned_addr learned my addr 172.21.15.189:0/3269996583 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170637617 0 0) 0x7f5e30135d50 con 0x7f5e3009f5e0 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): my global_id is 124349 2024-09-16T06:40:59.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e100035d0 con 0x7f5e3009f5e0 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096848276 0 0) 0x7f5e3009b600 con 0x7f5e30138d00 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): my global_id is 104655 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e30135d50 con 0x7f5e30138d00 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2895344693 0 0) 0x7f5e100035d0 con 0x7f5e3009f5e0 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e3009b600 con 0x7f5e3009f5e0 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e20001970 con 0x7f5e3009f5e0 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 424842758 0 0) 0x7f5e30135d50 con 0x7f5e30138d00 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e100035d0 con 0x7f5e30138d00 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e280043e0 con 0x7f5e30138d00 2024-09-16T06:40:59.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient(hunting): dump: 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 79738029 0 0) 0x7f5e3009b600 con 0x7f5e3009f5e0 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 >> v1:172.21.15.189:6789/0 conn(0x7f5e300a00d0 legacy=0x7f5e301cc0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 >> v1:172.21.15.76:6789/0 conn(0x7f5e30138d00 legacy=0x7f5e301cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 monclient: found mon.c 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30070d40 con 0x7f5e3009f5e0 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e2d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 5 monclient: authenticate success, global_id 124349 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _renew_subs 2024-09-16T06:40:59.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3006b010 con 0x7f5e3009f5e0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _renew_subs 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.365+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e30133980 con 0x7f5e3009f5e0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e20001ce0 con 0x7f5e3009f5e0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e20003b10 con 0x7f5e3009f5e0 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2effd700 4 set_mon_vals no callback set 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 10 monclient: dump: 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:40:59.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e367a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e367a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e367a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e14005100 con 0x7f5e3009f5e0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5e20003d70 con 0x7f5e3009f5e0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(64..64 src has 1..64) v4 ==== 4195+0+0 (unknown 3830037545 0 0) 0x7f5e20043500 con 0x7f5e3009f5e0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.369+0000 7f5e277fe700 10 monclient: get_auth_request con 0x7f5e100362c0 auth_method 0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.381+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5e20014890 con 0x7f5e3009f5e0 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.381+0000 7f5e2d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:40:59.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.381+0000 7f5e2d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:40:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.557+0000 7f5e367a9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.557+0000 7f5e367a9700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.557+0000 7f5e367a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.557+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f5e14005930 con 0x7f5e3009f5e0 2024-09-16T06:40:59.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.821+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v65) v1 ==== 138+0+0 (unknown 208029034 0 0) 0x7f5e20012d00 con 0x7f5e3009f5e0 2024-09-16T06:40:59.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.821+0000 7f5e2d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.821+0000 7f5e2d7fa700 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-16T06:40:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.933+0000 7f5e367a9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.933+0000 7f5e367a9700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.933+0000 7f5e367a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:40:59.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.933+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f5e140053f0 con 0x7f5e3009f5e0 2024-09-16T06:40:59.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e2d7fa700 1 -- 172.21.15.189:0/3269996583 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v65) v1 ==== 126+0+0 (unknown 3202625816 0 0) 0x7f5e20014480 con 0x7f5e3009f5e0 2024-09-16T06:40:59.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e2d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T06:40:59.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e2d7fa700 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5e100362c0 legacy=0x7f5e10038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 10 monclient: shutdown 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 >> v1:172.21.15.76:6790/0 conn(0x7f5e3009f5e0 legacy=0x7f5e301c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 shutdown_connections 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 >> 172.21.15.189:0/3269996583 conn(0x7f5e300a44e0 msgr2=0x7f5e300a2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 shutdown_connections 2024-09-16T06:40:59.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:40:59.937+0000 7f5e367a9700 1 -- 172.21.15.189:0/3269996583 wait complete. 2024-09-16T06:40:59.944 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'foo' already exists 2024-09-16T06:40:59.954 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T06:41:00.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: get_monmap_and_config 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: build_initial_monmap 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: monmap: 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:00.054699+0000 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:00.054699+0000 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:00.054 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 1 Processor -- start 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 1 -- start start 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: init 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-c con 0x7f17f406b760 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-b con 0x7f17f4135b50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-a con 0x7f17f4135f30 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f406bb40 con 0x7f17f4135f30 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f406c310 con 0x7f17f406b760 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f406ac80 con 0x7f17f4135b50 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:00.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17fae22700 10 monclient(hunting): authenticate will time out at 1012.567730s 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.049+0000 7f17f8bbe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f17f406b760 legacy=0x7f17f4135440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f93bf700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f17f4135f30 0x7f17f413de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50924/0 (socket says 172.21.15.189:50924) 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f93bf700 1 -- 172.21.15.189:0/1813992855 learned_addr learned my addr 172.21.15.189:0/1813992855 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125061772 0 0) 0x7f17f406ac80 con 0x7f17f4135b50 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): my global_id is 124352 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e00035d0 con 0x7f17f4135b50 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181770377 0 0) 0x7f17f406bb40 con 0x7f17f4135f30 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:00.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient(hunting): my global_id is 104658 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f406ac80 con 0x7f17f4135f30 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1280956918 0 0) 0x7f17e00035d0 con 0x7f17f4135b50 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 >> v1:172.21.15.189:6789/0 conn(0x7f17f406b760 legacy=0x7f17f4135440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 >> v1:172.21.15.76:6789/0 conn(0x7f17f4135f30 legacy=0x7f17f413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 monclient: found mon.noname-b 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f406af60 con 0x7f17f4135b50 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:00.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 5 monclient: authenticate success, global_id 124352 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17dc001990 con 0x7f17f4135b50 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: dump: 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:00.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17dc001c70 con 0x7f17f4135b50 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 4 set_mon_vals no callback set 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 1 -- 172.21.15.189:0/1813992855 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17dc002000 con 0x7f17f4135b50 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17f37fe700 10 monclient: dump: 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.062 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 10 monclient: shutdown 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:00.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- 172.21.15.189:0/1813992855 >> v1:172.21.15.76:6790/0 conn(0x7f17f4135b50 legacy=0x7f17f413a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:00.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- 172.21.15.189:0/1813992855 shutdown_connections 2024-09-16T06:41:00.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- 172.21.15.189:0/1813992855 >> 172.21.15.189:0/1813992855 conn(0x7f17f40a4660 msgr2=0x7f17f40a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:00.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- 172.21.15.189:0/1813992855 shutdown_connections 2024-09-16T06:41:00.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- 172.21.15.189:0/1813992855 wait complete. 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 10 monclient: build_initial_monmap 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 10 monclient: monmap: 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:00.068 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 Processor -- start 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 1 -- start start 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.053+0000 7f17fae22700 10 monclient: init 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-a con 0x7f17f406b760 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-c con 0x7f17f4135b50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient(hunting): picked mon.noname-b con 0x7f17f4135f30 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:00.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f4139740 con 0x7f17f406b760 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f413a380 con 0x7f17f4135f30 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f413b7c0 con 0x7f17f4135b50 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient(hunting): authenticate will time out at 1012.572442s 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f93bf700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f17f4135f30 legacy=0x7f17f41d2670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f8bbe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f17f406b760 0x7f17f4137af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50934/0 (socket says 172.21.15.189:50934) 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f8bbe700 1 -- 172.21.15.189:0/2541169038 learned_addr learned my addr 172.21.15.189:0/2541169038 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655671621 0 0) 0x7f17f413b7c0 con 0x7f17f4135b50 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): my global_id is 124355 2024-09-16T06:41:00.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d40035d0 con 0x7f17f4135b50 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553740902 0 0) 0x7f17f4139740 con 0x7f17f406b760 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): my global_id is 104661 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f413b7c0 con 0x7f17f406b760 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566648112 0 0) 0x7f17d40035d0 con 0x7f17f4135b50 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17f4139740 con 0x7f17f4135b50 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17dc001950 con 0x7f17f4135b50 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): dump: 2024-09-16T06:41:00.071 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653118979 0 0) 0x7f17f413b7c0 con 0x7f17f406b760 2024-09-16T06:41:00.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17d40035d0 con 0x7f17f406b760 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17ec004180 con 0x7f17f406b760 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient(hunting): dump: 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:00.073 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2792445870 0 0) 0x7f17f4139740 con 0x7f17f4135b50 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 >> v1:172.21.15.189:6789/0 conn(0x7f17f4135f30 legacy=0x7f17f41d2670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 >> v1:172.21.15.76:6789/0 conn(0x7f17f406b760 legacy=0x7f17f4137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 monclient: found mon.c 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f4138910 con 0x7f17f4135b50 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 5 monclient: authenticate success, global_id 124355 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _renew_subs 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f413baa0 con 0x7f17f4135b50 2024-09-16T06:41:00.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _renew_subs 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f40a6910 con 0x7f17f4135b50 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17dc003d90 con 0x7f17f4135b50 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17dc004120 con 0x7f17f4135b50 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f2ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.057+0000 7f17f17fa700 10 monclient: dump: 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:00.075 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f17dc004300 con 0x7f17f4135b50 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17fae22700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17fae22700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17fae22700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d8005100 con 0x7f17f4135b50 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 4222+0+0 (unknown 3703189981 0 0) 0x7f17dc042840 con 0x7f17f4135b50 2024-09-16T06:41:00.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.061+0000 7f17f8bbe700 10 monclient: get_auth_request con 0x7f17d40362c0 auth_method 0 2024-09-16T06:41:00.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.069+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f17dc013ce0 con 0x7f17f4135b50 2024-09-16T06:41:00.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.069+0000 7f17f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:00.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.069+0000 7f17f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.249+0000 7f17fae22700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:41:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.249+0000 7f17fae22700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T06:41:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.249+0000 7f17fae22700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.249+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f17d8005930 con 0x7f17f4135b50 2024-09-16T06:41:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.825+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v66) v1 ==== 58+0+2 (unknown 2230295283 0 1113811849) 0x7f17dc012150 con 0x7f17f4135b50 2024-09-16T06:41:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.825+0000 7f17f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T06:41:00.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.825+0000 7f17f17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:00.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.949+0000 7f17fae22700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:41:00.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.949+0000 7f17fae22700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T06:41:00.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.949+0000 7f17fae22700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:00.952 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:00.949+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f17d8005420 con 0x7f17f4135b50 2024-09-16T06:41:01.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.837+0000 7f17f17fa700 1 -- 172.21.15.189:0/2541169038 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) v1 ==== 58+0+2 (unknown 832193543 0 1371665406) 0x7f17dc0138d0 con 0x7f17f4135b50 2024-09-16T06:41:01.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.837+0000 7f17f17fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T06:41:01.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.837+0000 7f17f17fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 >> v1:172.21.15.189:6812/3098810674 conn(0x7f17d40362c0 legacy=0x7f17d4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 10 monclient: shutdown 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 >> v1:172.21.15.76:6790/0 conn(0x7f17f4135b50 legacy=0x7f17f4138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 shutdown_connections 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 >> 172.21.15.189:0/2541169038 conn(0x7f17f40a4660 msgr2=0x7f17f409dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 shutdown_connections 2024-09-16T06:41:01.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.841+0000 7f17fae22700 1 -- 172.21.15.189:0/2541169038 wait complete. 2024-09-16T06:41:01.859 INFO:tasks.workunit.client.0.smithi189.stderr:+ o1=7 2024-09-16T06:41:01.859 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 10 monclient: get_monmap_and_config 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 10 monclient: build_initial_monmap 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 10 monclient: monmap: 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:01.967241+0000 2024-09-16T06:41:01.966 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:01.967241+0000 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:01.967 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 1 Processor -- start 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 1 -- start start 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.961+0000 7fa501b9f700 10 monclient: init 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc09ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:01.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc09f830 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc099b10 con 0x7fa4fc09ed40 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc098890 con 0x7fa4fc097860 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc098b70 con 0x7fa4fc09f830 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient(hunting): authenticate will time out at 1014.480106s 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4f2ffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa4fc097860 legacy=0x7fa4fc1386e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4fbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa4fc09f830 0x7fa4fc13be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48920/0 (socket says 172.21.15.189:48920) 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4fb7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa4fc09ed40 0x7fa4fc09f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50940/0 (socket says 172.21.15.189:50940) 2024-09-16T06:41:01.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4fbfff700 1 -- 172.21.15.189:0/2689560850 learned_addr learned my addr 172.21.15.189:0/2689560850 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247469080 0 0) 0x7fa4fc099b10 con 0x7fa4fc09ed40 2024-09-16T06:41:01.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): my global_id is 104664 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e80035d0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830828606 0 0) 0x7fa4fc098b70 con 0x7fa4fc09f830 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient(hunting): my global_id is 124358 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc099b10 con 0x7fa4fc09f830 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2767459255 0 0) 0x7fa4e80035d0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 >> v1:172.21.15.76:6790/0 conn(0x7fa4fc09f830 legacy=0x7fa4fc13be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 >> v1:172.21.15.189:6789/0 conn(0x7fa4fc097860 legacy=0x7fa4fc1386e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 monclient: found mon.noname-a 2024-09-16T06:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:01.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc0973a0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4ec001540 con 0x7fa4fc09ed40 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 5 monclient: authenticate success, global_id 104664 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: dump: 2024-09-16T06:41:01.974 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4ec001760 con 0x7fa4fc09ed40 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:01.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 1 -- 172.21.15.189:0/2689560850 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4ec001af0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 4 set_mon_vals no callback set 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa4faffd700 10 monclient: dump: 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:01.976 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: shutdown 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- 172.21.15.189:0/2689560850 >> v1:172.21.15.76:6789/0 conn(0x7fa4fc09ed40 legacy=0x7fa4fc09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- 172.21.15.189:0/2689560850 shutdown_connections 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- 172.21.15.189:0/2689560850 >> 172.21.15.189:0/2689560850 conn(0x7fa4fc09fe30 msgr2=0x7fa4fc0a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- 172.21.15.189:0/2689560850 shutdown_connections 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 1 -- 172.21.15.189:0/2689560850 wait complete. 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: build_initial_monmap 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.965+0000 7fa501b9f700 10 monclient: monmap: 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:01.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 Processor -- start 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- start start 2024-09-16T06:41:01.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: init 2024-09-16T06:41:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc09ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:01.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc09f830 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc12daf0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc09ab80 con 0x7fa4fc097860 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc12c260 con 0x7fa4fc09f830 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient(hunting): authenticate will time out at 1014.485079s 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4fb7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa4fc097860 legacy=0x7fa4fc133620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4fbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa4fc09f830 0x7fa4fc1ce000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48926/0 (socket says 172.21.15.189:48926) 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4fbfff700 1 -- 172.21.15.189:0/2873677100 learned_addr learned my addr 172.21.15.189:0/2873677100 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730469332 0 0) 0x7fa4fc12c260 con 0x7fa4fc09f830 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): my global_id is 124361 2024-09-16T06:41:01.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc0035d0 con 0x7fa4fc09f830 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859452443 0 0) 0x7fa4fc12daf0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): my global_id is 104667 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc12c260 con 0x7fa4fc09ed40 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57266457 0 0) 0x7fa4fc12c260 con 0x7fa4fc09ed40 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4fc12daf0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4e40040f0 con 0x7fa4fc09ed40 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:01.984 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136763224 0 0) 0x7fa4dc0035d0 con 0x7fa4fc09f830 2024-09-16T06:41:01.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4fc12c260 con 0x7fa4fc09f830 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4f4001cd0 con 0x7fa4fc09f830 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:01.986 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212019002 0 0) 0x7fa4fc12c260 con 0x7fa4fc09f830 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 >> v1:172.21.15.189:6789/0 conn(0x7fa4fc097860 legacy=0x7fa4fc133620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 >> v1:172.21.15.76:6789/0 conn(0x7fa4fc09ed40 legacy=0x7fa4fc1ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 monclient: found mon.c 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc12bd50 con 0x7fa4fc09f830 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 5 monclient: authenticate success, global_id 124361 2024-09-16T06:41:01.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _renew_subs 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc12d810 con 0x7fa4fc09f830 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _renew_subs 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc09a700 con 0x7fa4fc09f830 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f4004c10 con 0x7fa4fc09f830 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa4f4004f20 con 0x7fa4fc09f830 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4fa7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.969+0000 7fa4f8ff9700 10 monclient: dump: 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:01.988 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa501b9f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa4f400f990 con 0x7fa4fc09f830 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa501b9f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa501b9f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:01.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0005100 con 0x7fa4fc09f830 2024-09-16T06:41:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(67..67 src has 1..67) v4 ==== 4452+0+0 (unknown 2995005480 0 0) 0x7fa4f40438a0 con 0x7fa4fc09f830 2024-09-16T06:41:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.973+0000 7fa4fb7fe700 10 monclient: get_auth_request con 0x7fa4dc036270 auth_method 0 2024-09-16T06:41:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.985+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa4f4010e40 con 0x7fa4fc09f830 2024-09-16T06:41:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.985+0000 7fa4f8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:01.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:01.985+0000 7fa4f8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:02.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.165+0000 7fa501b9f700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:41:02.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.165+0000 7fa501b9f700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T06:41:02.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.165+0000 7fa501b9f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:02.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.165+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa4e0005930 con 0x7fa4fc09f830 2024-09-16T06:41:02.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.837+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v68) v1 ==== 58+0+2 (unknown 904382424 0 3065575683) 0x7fa4f4012ee0 con 0x7fa4fc09f830 2024-09-16T06:41:02.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.837+0000 7fa4f8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T06:41:02.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.837+0000 7fa4f8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.957+0000 7fa501b9f700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:41:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.957+0000 7fa501b9f700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T06:41:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.957+0000 7fa501b9f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:02.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:02.957+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa4e0005420 con 0x7fa4fc09f830 2024-09-16T06:41:03.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.745+0000 7fa4f8ff9700 1 -- 172.21.15.189:0/2873677100 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v69) v1 ==== 58+0+2 (unknown 2157058348 0 2769969524) 0x7fa4f4014660 con 0x7fa4fc09f830 2024-09-16T06:41:03.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.745+0000 7fa4f8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T06:41:03.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.745+0000 7fa4f8ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa4dc036270 legacy=0x7fa4dc038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 10 monclient: shutdown 2024-09-16T06:41:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 >> v1:172.21.15.76:6790/0 conn(0x7fa4fc09f830 legacy=0x7fa4fc1ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:03.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 shutdown_connections 2024-09-16T06:41:03.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 >> 172.21.15.189:0/2873677100 conn(0x7fa4fc09fe30 msgr2=0x7fa4fc06d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:03.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 shutdown_connections 2024-09-16T06:41:03.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.749+0000 7fa501b9f700 1 -- 172.21.15.189:0/2873677100 wait complete. 2024-09-16T06:41:03.770 INFO:tasks.workunit.client.0.smithi189.stderr:+ o2=9 2024-09-16T06:41:03.770 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T06:41:03.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.885+0000 7f4f2cbda700 10 monclient: get_monmap_and_config 2024-09-16T06:41:03.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.885+0000 7f4f2cbda700 10 monclient: build_initial_monmap 2024-09-16T06:41:03.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.885+0000 7f4f2cbda700 10 monclient: monmap: 2024-09-16T06:41:03.890 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:03.891632+0000 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:03.891632+0000 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:03.891 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 Processor -- start 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- start start 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient: init 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-a con 0x7f4f280a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-c con 0x7f4f2809aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-b con 0x7f4f280a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2809a040 con 0x7f4f280a2350 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280a33d0 con 0x7f4f2809aa20 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f28099d60 con 0x7f4f280a3ca0 2024-09-16T06:41:03.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:03.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient(hunting): authenticate will time out at 1016.405014s 2024-09-16T06:41:03.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f25d9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4f2809aa20 legacy=0x7f4f2813a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:03.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4f280a2350 0x7f4f280a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50958/0 (socket says 172.21.15.189:50958) 2024-09-16T06:41:03.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f26d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4f280a3ca0 0x7f4f2813dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48936/0 (socket says 172.21.15.189:48936) 2024-09-16T06:41:03.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2659c700 1 -- 172.21.15.189:0/3312484359 learned_addr learned my addr 172.21.15.189:0/3312484359 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:03.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816247744 0 0) 0x7f4f28099d60 con 0x7f4f280a3ca0 2024-09-16T06:41:03.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:03.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:03.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:03.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): my global_id is 124364 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f140035d0 con 0x7f4f280a3ca0 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427265114 0 0) 0x7f4f2809a040 con 0x7f4f280a2350 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient(hunting): my global_id is 104670 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f28099d60 con 0x7f4f280a2350 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788534900 0 0) 0x7f4f28099d60 con 0x7f4f280a2350 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 >> v1:172.21.15.76:6790/0 conn(0x7f4f280a3ca0 legacy=0x7f4f2813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 >> v1:172.21.15.189:6789/0 conn(0x7f4f2809aa20 legacy=0x7f4f2813a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 monclient: found mon.noname-a 2024-09-16T06:41:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280a3920 con 0x7f4f280a2350 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: _finish_auth 0 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f180015f0 con 0x7f4f280a2350 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 5 monclient: authenticate success, global_id 104670 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: dump: 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:03.898 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f18001910 con 0x7f4f280a2350 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:03.899 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 1 -- 172.21.15.189:0/3312484359 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f18001ce0 con 0x7f4f280a2350 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 4 set_mon_vals no callback set 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2559a700 10 monclient: dump: 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.900 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 10 monclient: shutdown 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- 172.21.15.189:0/3312484359 >> v1:172.21.15.76:6789/0 conn(0x7f4f280a2350 legacy=0x7f4f280a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- 172.21.15.189:0/3312484359 shutdown_connections 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- 172.21.15.189:0/3312484359 >> 172.21.15.189:0/3312484359 conn(0x7f4f280a7d70 msgr2=0x7f4f280a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- 172.21.15.189:0/3312484359 shutdown_connections 2024-09-16T06:41:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.889+0000 7f4f2cbda700 1 -- 172.21.15.189:0/3312484359 wait complete. 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: build_initial_monmap 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: monmap: 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 Processor -- start 2024-09-16T06:41:03.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- start start 2024-09-16T06:41:03.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: init 2024-09-16T06:41:03.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-c con 0x7f4f2809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-a con 0x7f4f280a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient(hunting): picked mon.noname-b con 0x7f4f280a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f28127600 con 0x7f4f280a2350 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280a6cc0 con 0x7f4f280a3ca0 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280a0350 con 0x7f4f2809aa20 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:03.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient(hunting): authenticate will time out at 1016.409758s 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f26d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4f280a3ca0 legacy=0x7f4f281d6660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f25d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4f280a2350 0x7f4f2809fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50960/0 (socket says 172.21.15.189:50960) 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f25d9b700 1 -- 172.21.15.189:0/1652979596 learned_addr learned my addr 172.21.15.189:0/1652979596 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820399728 0 0) 0x7f4f28127600 con 0x7f4f280a2350 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): my global_id is 104673 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f080035d0 con 0x7f4f280a2350 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846841710 0 0) 0x7f4f280a0350 con 0x7f4f2809aa20 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:03.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): my global_id is 124367 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f28127600 con 0x7f4f2809aa20 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662847744 0 0) 0x7f4f080035d0 con 0x7f4f280a2350 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f280a0350 con 0x7f4f280a2350 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f10004200 con 0x7f4f280a2350 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): dump: 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.909 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561164879 0 0) 0x7f4f28127600 con 0x7f4f2809aa20 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f080035d0 con 0x7f4f2809aa20 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f18001c20 con 0x7f4f2809aa20 2024-09-16T06:41:03.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient(hunting): dump: 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 668167924 0 0) 0x7f4f280a0350 con 0x7f4f280a2350 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 >> v1:172.21.15.76:6790/0 conn(0x7f4f2809aa20 legacy=0x7f4f2809f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 >> v1:172.21.15.189:6789/0 conn(0x7f4f280a3ca0 legacy=0x7f4f281d6660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 monclient: found mon.a 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f281271b0 con 0x7f4f280a2350 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: _finish_auth 0 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 5 monclient: authenticate success, global_id 104673 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _renew_subs 2024-09-16T06:41:03.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f280a70f0 con 0x7f4f280a2350 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _renew_subs 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f281288e0 con 0x7f4f280a2350 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f10004420 con 0x7f4f280a2350 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f10004730 con 0x7f4f280a2350 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f24d99700 4 set_mon_vals no callback set 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.893+0000 7f4f1effd700 10 monclient: dump: 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:03.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4f10004910 con 0x7f4f280a2350 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f2cbda700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f2cbda700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f2cbda700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:03.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c005100 con 0x7f4f280a2350 2024-09-16T06:41:03.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 4682+0+0 (unknown 2560176607 0 0) 0x7f4f10043660 con 0x7f4f280a2350 2024-09-16T06:41:03.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.897+0000 7f4f2659c700 10 monclient: get_auth_request con 0x7f4f080362f0 auth_method 0 2024-09-16T06:41:03.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.909+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4f10005160 con 0x7f4f280a2350 2024-09-16T06:41:03.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.909+0000 7f4f1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:03.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:03.909+0000 7f4f1effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:04.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.093+0000 7f4f2cbda700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.093+0000 7f4f2cbda700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.093+0000 7f4f2cbda700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:04.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.093+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f4f0c005930 con 0x7f4f280a2350 2024-09-16T06:41:04.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.857+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v70) v1 ==== 203+0+0 (unknown 2104601758 0 0) 0x7f4f10012bc0 con 0x7f4f280a2350 2024-09-16T06:41:04.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.857+0000 7f4f1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.857+0000 7f4f1effd700 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-16T06:41:04.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f2cbda700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f2cbda700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f2cbda700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:04.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f4f0c0053b0 con 0x7f4f280a2350 2024-09-16T06:41:04.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f1effd700 1 -- 172.21.15.189:0/1652979596 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 1589083800 0 0) 0x7f4f10014340 con 0x7f4f280a2350 2024-09-16T06:41:04.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f1effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:04.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.973+0000 7f4f1effd700 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T06:41:04.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4f080362f0 legacy=0x7f4f080387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:04.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 10 monclient: shutdown 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 >> v1:172.21.15.76:6789/0 conn(0x7f4f280a2350 legacy=0x7f4f2809fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 shutdown_connections 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 >> 172.21.15.189:0/1652979596 conn(0x7f4f280a7d70 msgr2=0x7f4f281bb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 shutdown_connections 2024-09-16T06:41:04.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:04.977+0000 7f4f2cbda700 1 -- 172.21.15.189:0/1652979596 wait complete. 2024-09-16T06:41:04.981 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T06:41:04.990 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T06:41:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: get_monmap_and_config 2024-09-16T06:41:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: monmap: 2024-09-16T06:41:05.100 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:05.101319+0000 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:05.101319+0000 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:05.101 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 1 Processor -- start 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 1 -- start start 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: init 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:05.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-b con 0x7f9da009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-a con 0x7f9da0099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-c con 0x7f9da01396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da009f520 con 0x7f9da0099700 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da009bc10 con 0x7f9da01396a0 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da009b810 con 0x7f9da009f140 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da71a9700 10 monclient(hunting): authenticate will time out at 1017.614383s 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da5746700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9da01396a0 legacy=0x7f9da013ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9da0099700 0x7f9da009b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50966/0 (socket says 172.21.15.189:50966) 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9da4f45700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9da009f140 0x7f9da009a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48958/0 (socket says 172.21.15.189:48958) 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9ffff700 1 -- 172.21.15.189:0/403849424 learned_addr learned my addr 172.21.15.189:0/403849424 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:05.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729963785 0 0) 0x7f9da009b810 con 0x7f9da009f140 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): my global_id is 124370 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d880035d0 con 0x7f9da009f140 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160447867 0 0) 0x7f9da009f520 con 0x7f9da0099700 2024-09-16T06:41:05.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 10 monclient(hunting): my global_id is 104676 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da009b810 con 0x7f9da0099700 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 754199495 0 0) 0x7f9da009b810 con 0x7f9da0099700 2024-09-16T06:41:05.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 >> v1:172.21.15.76:6790/0 conn(0x7f9da009f140 legacy=0x7f9da009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.097+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 >> v1:172.21.15.189:6789/0 conn(0x7f9da01396a0 legacy=0x7f9da013ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da009ceb0 con 0x7f9da0099700 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 5 monclient: authenticate success, global_id 104676 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d900018f0 con 0x7f9da0099700 2024-09-16T06:41:05.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:05.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: dump: 2024-09-16T06:41:05.110 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.110 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.110 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.111 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.112 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:05.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d90001c10 con 0x7f9da0099700 2024-09-16T06:41:05.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 4 set_mon_vals no callback set 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 1 -- 172.21.15.189:0/403849424 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d90001fe0 con 0x7f9da0099700 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9f7fe700 10 monclient: dump: 2024-09-16T06:41:05.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: shutdown 2024-09-16T06:41:05.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:05.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- 172.21.15.189:0/403849424 >> v1:172.21.15.76:6789/0 conn(0x7f9da0099700 legacy=0x7f9da009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- 172.21.15.189:0/403849424 shutdown_connections 2024-09-16T06:41:05.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- 172.21.15.189:0/403849424 >> 172.21.15.189:0/403849424 conn(0x7f9da00a03e0 msgr2=0x7f9da00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:05.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- 172.21.15.189:0/403849424 shutdown_connections 2024-09-16T06:41:05.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- 172.21.15.189:0/403849424 wait complete. 2024-09-16T06:41:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: monmap: 2024-09-16T06:41:05.124 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.125 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 Processor -- start 2024-09-16T06:41:05.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- start start 2024-09-16T06:41:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: init 2024-09-16T06:41:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-a con 0x7f9da0099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-b con 0x7f9da009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient(hunting): picked mon.noname-c con 0x7f9da01396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00a2e60 con 0x7f9da0099700 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da009a6e0 con 0x7f9da009f140 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da0098fe0 con 0x7f9da01396a0 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9da009f140 legacy=0x7f9da01cf280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da71a9700 10 monclient(hunting): authenticate will time out at 1017.618469s 2024-09-16T06:41:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da5746700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9da01396a0 0x7f9da01d29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48974/0 (socket says 172.21.15.189:48974) 2024-09-16T06:41:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da4f45700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9da0099700 0x7f9da01ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50976/0 (socket says 172.21.15.189:50976) 2024-09-16T06:41:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9da5746700 1 -- 172.21.15.189:0/3776075904 learned_addr learned my addr 172.21.15.189:0/3776075904 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:05.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728241511 0 0) 0x7f9da0098fe0 con 0x7f9da01396a0 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): my global_id is 124373 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c0035d0 con 0x7f9da01396a0 2024-09-16T06:41:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414777632 0 0) 0x7f9da00a2e60 con 0x7f9da0099700 2024-09-16T06:41:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 10 monclient(hunting): my global_id is 104679 2024-09-16T06:41:05.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da0098fe0 con 0x7f9da0099700 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172061192 0 0) 0x7f9d8c0035d0 con 0x7f9da01396a0 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9da00a2e60 con 0x7f9da01396a0 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3832178928 0 0) 0x7f9da0098fe0 con 0x7f9da0099700 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.101+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c0035d0 con 0x7f9da0099700 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d980040e0 con 0x7f9da01396a0 2024-09-16T06:41:05.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): dump: 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.133 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d94001a10 con 0x7f9da0099700 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient(hunting): dump: 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.135 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918441717 0 0) 0x7f9da00a2e60 con 0x7f9da01396a0 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 >> v1:172.21.15.189:6789/0 conn(0x7f9da009f140 legacy=0x7f9da01cf280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 >> v1:172.21.15.76:6789/0 conn(0x7f9da0099700 legacy=0x7f9da01ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 monclient: found mon.c 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da0098d00 con 0x7f9da01396a0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 5 monclient: authenticate success, global_id 124373 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _renew_subs 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0098340 con 0x7f9da01396a0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _renew_subs 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da0099510 con 0x7f9da01396a0 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d98004340 con 0x7f9da01396a0 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d980046d0 con 0x7f9da01396a0 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9e7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 10 monclient: dump: 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.138 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d98004830 con 0x7f9da01396a0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d80005100 con 0x7f9da01396a0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 4964+0+0 (unknown 1381275288 0 0) 0x7f9d980435a0 con 0x7f9da01396a0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.105+0000 7f9da4f45700 10 monclient: get_auth_request con 0x7f9d8c0362b0 auth_method 0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.117+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d98014630 con 0x7f9da01396a0 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.117+0000 7f9d9d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.117+0000 7f9d9d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:05.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.289+0000 7f9da71a9700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.289+0000 7f9da71a9700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.289+0000 7f9da71a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.289+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9d800053b0 con 0x7f9da01396a0 2024-09-16T06:41:05.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.293+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 1589083800 0 0) 0x7f9d98018830 con 0x7f9da01396a0 2024-09-16T06:41:05.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.293+0000 7f9d9d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.293+0000 7f9d9d7fa700 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T06:41:05.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9da71a9700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9da71a9700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9da71a9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9d80005af0 con 0x7f9da01396a0 2024-09-16T06:41:05.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9d9d7fa700 1 -- 172.21.15.189:0/3776075904 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v70) v1 ==== 201+0+0 (unknown 1589083800 0 0) 0x7f9d9801cc20 con 0x7f9da01396a0 2024-09-16T06:41:05.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9d9d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T06:41:05.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.389+0000 7f9d9d7fa700 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T06:41:05.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9d8c0362b0 legacy=0x7f9d8c038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 10 monclient: shutdown 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 >> v1:172.21.15.76:6790/0 conn(0x7f9da01396a0 legacy=0x7f9da01d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 shutdown_connections 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 >> 172.21.15.189:0/3776075904 conn(0x7f9da00a03e0 msgr2=0x7f9da013b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 shutdown_connections 2024-09-16T06:41:05.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.393+0000 7f9da71a9700 1 -- 172.21.15.189:0/3776075904 wait complete. 2024-09-16T06:41:05.398 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T06:41:05.405 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T06:41:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 10 monclient: get_monmap_and_config 2024-09-16T06:41:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 10 monclient: monmap: 2024-09-16T06:41:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:05.518 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:05.519237+0000 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:05.519237+0000 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:05.519 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 1 Processor -- start 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 1 -- start start 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.513+0000 7fdda64c8700 10 monclient: init 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-a con 0x7fdda0099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-b con 0x7fdda0099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-c con 0x7fdda009a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda009c460 con 0x7fdda0099a60 2024-09-16T06:41:05.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda009c1e0 con 0x7fdda009a250 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda009bd60 con 0x7fdda0099e40 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient(hunting): authenticate will time out at 1018.032668s 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda4a65700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fdda009a250 legacy=0x7fdda013a090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdda0099e40 0x7fdda0136960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48982/0 (socket says 172.21.15.189:48982) 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9f7fe700 1 -- 172.21.15.189:0/2375964027 learned_addr learned my addr 172.21.15.189:0/2375964027 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdda0099a60 0x7fdda0098070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50990/0 (socket says 172.21.15.189:50990) 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409259398 0 0) 0x7fdda009bd60 con 0x7fdda0099e40 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): my global_id is 124376 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd880034f0 con 0x7fdda0099e40 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726519100 0 0) 0x7fdda009c460 con 0x7fdda0099a60 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient(hunting): my global_id is 104682 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda009bd60 con 0x7fdda0099a60 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895824363 0 0) 0x7fdd880034f0 con 0x7fdda0099e40 2024-09-16T06:41:05.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 >> v1:172.21.15.189:6789/0 conn(0x7fdda009a250 legacy=0x7fdda013a090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 >> v1:172.21.15.76:6789/0 conn(0x7fdda0099a60 legacy=0x7fdda0098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 monclient: found mon.noname-b 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda009d490 con 0x7fdda0099e40 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd94001610 con 0x7fdda0099e40 2024-09-16T06:41:05.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 5 monclient: authenticate success, global_id 124376 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: dump: 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.530 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd94001830 con 0x7fdda0099e40 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 4 set_mon_vals no callback set 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 1 -- 172.21.15.189:0/2375964027 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd94001c00 con 0x7fdda0099e40 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:05.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdd9effd700 10 monclient: dump: 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.532 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 10 monclient: shutdown 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- 172.21.15.189:0/2375964027 >> v1:172.21.15.76:6790/0 conn(0x7fdda0099e40 legacy=0x7fdda0136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- 172.21.15.189:0/2375964027 shutdown_connections 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- 172.21.15.189:0/2375964027 >> 172.21.15.189:0/2375964027 conn(0x7fdda00a03e0 msgr2=0x7fdda00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- 172.21.15.189:0/2375964027 shutdown_connections 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.517+0000 7fdda64c8700 1 -- 172.21.15.189:0/2375964027 wait complete. 2024-09-16T06:41:05.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: monmap: 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:05.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 Processor -- start 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 -- start start 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: init 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-a con 0x7fdda0099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-c con 0x7fdda0099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient(hunting): picked mon.noname-b con 0x7fdda009a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda0094cb0 con 0x7fdda0099a60 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda009a6e0 con 0x7fdda009a250 2024-09-16T06:41:05.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda00949d0 con 0x7fdda0099e40 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient(hunting): authenticate will time out at 1018.038037s 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda4a65700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fdda009a250 legacy=0x7fdda01d24c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdda0099e40 0x7fdda01ced90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48996/0 (socket says 172.21.15.189:48996) 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdda0099a60 0x7fdda00967a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50996/0 (socket says 172.21.15.189:50996) 2024-09-16T06:41:05.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9f7fe700 1 -- 172.21.15.189:0/3412545667 learned_addr learned my addr 172.21.15.189:0/3412545667 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864583535 0 0) 0x7fdda0094cb0 con 0x7fdda0099a60 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): my global_id is 104685 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd8c0035d0 con 0x7fdda0099a60 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401503714 0 0) 0x7fdda00949d0 con 0x7fdda0099e40 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): my global_id is 124379 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda0094cb0 con 0x7fdda0099e40 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481373518 0 0) 0x7fdd8c0035d0 con 0x7fdda0099a60 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda00949d0 con 0x7fdda0099a60 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd90001880 con 0x7fdda0099a60 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:05.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.542 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210652707 0 0) 0x7fdda0094cb0 con 0x7fdda0099e40 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd8c0035d0 con 0x7fdda0099e40 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd94001bc0 con 0x7fdda0099e40 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4251540900 0 0) 0x7fdda00949d0 con 0x7fdda0099a60 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 >> v1:172.21.15.76:6790/0 conn(0x7fdda0099e40 legacy=0x7fdda01ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 >> v1:172.21.15.189:6789/0 conn(0x7fdda009a250 legacy=0x7fdda01d24c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 monclient: found mon.a 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0096eb0 con 0x7fdda0099a60 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdd9cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 5 monclient: authenticate success, global_id 104685 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: _renew_subs 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.521+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda0093c90 con 0x7fdda0099a60 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 10 monclient: _renew_subs 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda00930c0 con 0x7fdda0099a60 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd90004890 con 0x7fdda0099a60 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd90004ba0 con 0x7fdda0099a60 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 10 monclient: dump: 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9e7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda009e990 con 0x7fdda0099a60 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdd9000f690 con 0x7fdda0099a60 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 4964+0+0 (unknown 1381275288 0 0) 0x7fdd900437e0 con 0x7fdda0099a60 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.525+0000 7fdd9f7fe700 10 monclient: get_auth_request con 0x7fdd8c036280 auth_method 0 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.533+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdd900145e0 con 0x7fdda0099a60 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.533+0000 7fdd9cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.533+0000 7fdd9cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:05.623 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:41:05.623 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:41:05.623 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:41:05.623 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:41:05.626 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:41:05.626 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:41:05.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:05.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.709+0000 7fdda64c8700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.709+0000 7fdda64c8700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.709+0000 7fdda64c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.709+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fdda00929d0 con 0x7fdda0099a60 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: get_monmap_and_config 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: monmap: 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:05.744600+0000 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:05.744600+0000 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:05.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.744 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:05.744 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:05.744 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 1 Processor -- start 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 1 -- start start 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: init 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-b con 0x7f6e4409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-a con 0x7f6e44098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-c con 0x7f6e44099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4409d560 con 0x7f6e44098be0 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4409f140 con 0x7f6e44099030 2024-09-16T06:41:05.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4409d790 con 0x7f6e4409ed60 2024-09-16T06:41:05.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4b2fd700 10 monclient(hunting): authenticate will time out at 1012.877868s 2024-09-16T06:41:05.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4a2fb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6e4409ed60 0x7f6e440984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50070/0 (socket says 172.21.15.76:50070) 2024-09-16T06:41:05.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.740+0000 7f6e4a2fb700 1 -- 172.21.15.76:0/836009172 learned_addr learned my addr 172.21.15.76:0/836009172 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745150567 0 0) 0x7f6e4409d790 con 0x7f6e4409ed60 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient(hunting): my global_id is 124382 2024-09-16T06:41:05.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e300035d0 con 0x7f6e4409ed60 2024-09-16T06:41:05.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2630767270 0 0) 0x7f6e300035d0 con 0x7f6e4409ed60 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 >> v1:172.21.15.189:6789/0 conn(0x7f6e44099030 legacy=0x7f6e4413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 >> v1:172.21.15.76:6789/0 conn(0x7f6e44098be0 legacy=0x7f6e441374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 monclient: found mon.noname-b 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4409f620 con 0x7f6e4409ed60 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e3c001990 con 0x7f6e4409ed60 2024-09-16T06:41:05.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: dump: 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 5 monclient: authenticate success, global_id 124382 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e3c001c40 con 0x7f6e4409ed60 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 1 -- 172.21.15.76:0/836009172 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e3c002010 con 0x7f6e4409ed60 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 4 set_mon_vals no callback set 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e492f9700 10 monclient: dump: 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.768 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 10 monclient: shutdown 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/836009172 >> v1:172.21.15.76:6790/0 conn(0x7f6e4409ed60 legacy=0x7f6e440984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/836009172 shutdown_connections 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/836009172 >> 172.21.15.76:0/836009172 conn(0x7f6e440a0030 msgr2=0x7f6e440a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:05.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/836009172 shutdown_connections 2024-09-16T06:41:05.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.760+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/836009172 wait complete. 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient: build_initial_monmap 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient: monmap: 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:05.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 1 Processor -- start 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 1 -- start start 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient: init 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-c con 0x7f6e44098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-a con 0x7f6e44099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:05.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient(hunting): picked mon.noname-b con 0x7f6e4409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e44092230 con 0x7f6e44099030 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4408f0d0 con 0x7f6e4409ed60 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4409a540 con 0x7f6e44098be0 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e4b2fd700 10 monclient(hunting): authenticate will time out at 1012.903203s 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e49afa700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6e44099030 0x7f6e4408e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53194/0 (socket says 172.21.15.76:53194) 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.764+0000 7f6e49afa700 1 -- 172.21.15.76:0/3125457177 learned_addr learned my addr 172.21.15.76:0/3125457177 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572328613 0 0) 0x7f6e44092230 con 0x7f6e44099030 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): my global_id is 104688 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e280035d0 con 0x7f6e44099030 2024-09-16T06:41:05.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112830528 0 0) 0x7f6e280035d0 con 0x7f6e44099030 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e44092230 con 0x7f6e44099030 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e34001810 con 0x7f6e44099030 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): dump: 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3971986581 0 0) 0x7f6e44092230 con 0x7f6e44099030 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 >> v1:172.21.15.76:6790/0 conn(0x7f6e44098be0 legacy=0x7f6e4408e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 >> v1:172.21.15.189:6789/0 conn(0x7f6e4409ed60 legacy=0x7f6e441cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 monclient: found mon.a 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e440a28d0 con 0x7f6e44099030 2024-09-16T06:41:05.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 5 monclient: authenticate success, global_id 104688 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _renew_subs 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e440a2380 con 0x7f6e44099030 2024-09-16T06:41:05.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _renew_subs 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e44091f50 con 0x7f6e44099030 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e340036c0 con 0x7f6e44099030 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e340039d0 con 0x7f6e44099030 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e48af8700 4 set_mon_vals no callback set 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:05.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 10 monclient: dump: 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6e34003bb0 con 0x7f6e44099030 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 4964+0+0 (unknown 1381275288 0 0) 0x7f6e340436e0 con 0x7f6e44099030 2024-09-16T06:41:05.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.768+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e44091990 con 0x7f6e44099030 2024-09-16T06:41:05.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.772+0000 7f6e4a2fb700 10 monclient: get_auth_request con 0x7f6e2803a580 auth_method 0 2024-09-16T06:41:05.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.780+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6e340146b0 con 0x7f6e44099030 2024-09-16T06:41:05.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.780+0000 7f6e3affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:05.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:05.780+0000 7f6e3affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:05.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.865+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v71) v1 ==== 203+0+0 (unknown 136103579 0 0) 0x7fdd90012c80 con 0x7fdda0099a60 2024-09-16T06:41:05.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.865+0000 7fdd9cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.865+0000 7fdd9cff9700 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-16T06:41:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fdda00008d0 con 0x7fdda0099a60 2024-09-16T06:41:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdd9cff9700 1 -- 172.21.15.189:0/3412545667 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (unknown 3095293837 0 0) 0x7fdd90014400 con 0x7fdda0099a60 2024-09-16T06:41:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdd9cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdd9cff9700 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T06:41:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdd8c036280 legacy=0x7fdd8c038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 10 monclient: shutdown 2024-09-16T06:41:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.973+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 >> v1:172.21.15.76:6789/0 conn(0x7fdda0099a60 legacy=0x7fdda00967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.977+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 shutdown_connections 2024-09-16T06:41:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.977+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 >> 172.21.15.189:0/3412545667 conn(0x7fdda00a03e0 msgr2=0x7fdda0097f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.977+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 shutdown_connections 2024-09-16T06:41:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:05.977+0000 7fdda64c8700 1 -- 172.21.15.189:0/3412545667 wait complete. 2024-09-16T06:41:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T06:41:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T06:41:06.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:06.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e4409b0f0 con 0x7f6e44099030 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e3affd700 1 -- 172.21.15.76:0/3125457177 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 637748108) 0x7f6e34014480 con 0x7f6e44099030 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e3affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e3affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:06.019 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:06.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6e2803a580 legacy=0x7f6e2803ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 10 monclient: shutdown 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 >> v1:172.21.15.76:6789/0 conn(0x7f6e44099030 legacy=0x7f6e4408e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 shutdown_connections 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 >> 172.21.15.76:0/3125457177 conn(0x7f6e440a0030 msgr2=0x7f6e440a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 shutdown_connections 2024-09-16T06:41:06.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.016+0000 7f6e4b2fd700 1 -- 172.21.15.76:0/3125457177 wait complete. 2024-09-16T06:41:06.033 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:41:06.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:41:06.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:41:06.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: monmap: 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:06.101872+0000 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:06.101872+0000 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.101 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 1 Processor -- start 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 1 -- start start 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: init 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-c con 0x7f62580a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-a con 0x7f625809aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-b con 0x7f62580a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625809a040 con 0x7f625809aa20 2024-09-16T06:41:06.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580a33d0 con 0x7f62580a2350 2024-09-16T06:41:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6258099d60 con 0x7f62580a3ca0 2024-09-16T06:41:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f625dc37700 10 monclient(hunting): authenticate will time out at 1018.615628s 2024-09-16T06:41:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f62577fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f62580a2350 0x7f62580a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45806/0 (socket says 172.21.15.189:45806) 2024-09-16T06:41:06.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.097+0000 7f62577fe700 1 -- 172.21.15.189:0/1466922615 learned_addr learned my addr 172.21.15.189:0/1466922615 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249866806 0 0) 0x7f62580a33d0 con 0x7f62580a2350 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): my global_id is 134098 2024-09-16T06:41:06.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62400035d0 con 0x7f62580a2350 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474179494 0 0) 0x7f6258099d60 con 0x7f62580a3ca0 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): my global_id is 124385 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62580a33d0 con 0x7f62580a3ca0 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303572251 0 0) 0x7f625809a040 con 0x7f625809aa20 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient(hunting): my global_id is 104691 2024-09-16T06:41:06.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258099d60 con 0x7f625809aa20 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2669579758 0 0) 0x7f62400035d0 con 0x7f62580a2350 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 >> v1:172.21.15.76:6790/0 conn(0x7f62580a3ca0 legacy=0x7f625813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 >> v1:172.21.15.76:6789/0 conn(0x7f625809aa20 legacy=0x7f625813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580a3920 con 0x7f62580a2350 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 5 monclient: authenticate success, global_id 134098 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6248001610 con 0x7f62580a2350 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: dump: 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.110 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6248001830 con 0x7f62580a2350 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 4 set_mon_vals no callback set 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 1 -- 172.21.15.189:0/1466922615 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6248001bc0 con 0x7f62580a2350 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f62567fc700 10 monclient: dump: 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.111 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 10 monclient: shutdown 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 1 -- 172.21.15.189:0/1466922615 >> v1:172.21.15.189:6789/0 conn(0x7f62580a2350 legacy=0x7f62580a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 1 -- 172.21.15.189:0/1466922615 shutdown_connections 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 1 -- 172.21.15.189:0/1466922615 >> 172.21.15.189:0/1466922615 conn(0x7f62580a7d70 msgr2=0x7f62580a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 1 -- 172.21.15.189:0/1466922615 shutdown_connections 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 1 -- 172.21.15.189:0/1466922615 wait complete. 2024-09-16T06:41:06.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.101+0000 7f625dc37700 10 monclient: monmap: 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 Processor -- start 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- start start 2024-09-16T06:41:06.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: init 2024-09-16T06:41:06.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:06.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-a con 0x7f625809aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-c con 0x7f62580a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient(hunting): picked mon.noname-b con 0x7f62580a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625806bed0 con 0x7f625809aa20 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625809b730 con 0x7f62580a3ca0 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6258136120 con 0x7f62580a2350 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f6257fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f62580a3ca0 0x7f62581d2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45820/0 (socket says 172.21.15.189:45820) 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient(hunting): authenticate will time out at 1018.621528s 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f62577fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f625809aa20 0x7f62581354b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:51026/0 (socket says 172.21.15.189:51026) 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f6257fff700 1 -- 172.21.15.189:0/3505216841 learned_addr learned my addr 172.21.15.189:0/3505216841 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312197064 0 0) 0x7f625809b730 con 0x7f62580a3ca0 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): my global_id is 134101 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62440035d0 con 0x7f62580a3ca0 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422986090 0 0) 0x7f625806bed0 con 0x7f625809aa20 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): my global_id is 104694 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625809b730 con 0x7f625809aa20 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218193668 0 0) 0x7f6258136120 con 0x7f62580a2350 2024-09-16T06:41:06.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): my global_id is 124388 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625806bed0 con 0x7f62580a2350 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3236305960 0 0) 0x7f62440035d0 con 0x7f62580a3ca0 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6258136120 con 0x7f62580a3ca0 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6250001c70 con 0x7f62580a3ca0 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): dump: 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.121 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 75864759 0 0) 0x7f625806bed0 con 0x7f62580a2350 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62440035d0 con 0x7f62580a2350 2024-09-16T06:41:06.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f624c001830 con 0x7f62580a2350 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): dump: 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2737465931 0 0) 0x7f625809b730 con 0x7f625809aa20 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f625806bed0 con 0x7f625809aa20 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6248001510 con 0x7f625809aa20 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient(hunting): dump: 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.124 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2610767525 0 0) 0x7f6258136120 con 0x7f62580a3ca0 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 >> v1:172.21.15.76:6790/0 conn(0x7f62580a2350 legacy=0x7f62581cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 >> v1:172.21.15.76:6789/0 conn(0x7f625809aa20 legacy=0x7f62581354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 monclient: found mon.b 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258122b90 con 0x7f62580a3ca0 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 5 monclient: authenticate success, global_id 134101 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _renew_subs 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258122e70 con 0x7f62580a3ca0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _renew_subs 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6258135bc0 con 0x7f62580a3ca0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6250003a40 con 0x7f62580a3ca0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6250003d50 con 0x7f62580a3ca0 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f62557fa700 4 set_mon_vals no callback set 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.105+0000 7f623ffff700 10 monclient: dump: 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6250003eb0 con 0x7f62580a3ca0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f625dc37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f625dc37700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f625dc37700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6238005100 con 0x7f62580a3ca0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7f6250043be0 con 0x7f62580a3ca0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.109+0000 7f62577fe700 10 monclient: get_auth_request con 0x7f6244036510 auth_method 0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.121+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6250004700 con 0x7f62580a3ca0 2024-09-16T06:41:06.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.121+0000 7f623ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.121+0000 7f623ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.140+0000 7fb052089700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.140+0000 7fb052089700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.140+0000 7fb052089700 10 monclient: monmap: 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:06.147767+0000 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:06.147767+0000 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.146 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 1 Processor -- start 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 1 -- start start 2024-09-16T06:41:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient: init 2024-09-16T06:41:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient(hunting): picked mon.noname-b con 0x7fb04c0a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient(hunting): picked mon.noname-a con 0x7fb04c0a11a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient(hunting): picked mon.noname-c con 0x7fb04c071d80 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c0a4c20 con 0x7fb04c0a11a0 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c09fc40 con 0x7fb04c071d80 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c09c120 con 0x7fb04c0a4840 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient(hunting): authenticate will time out at 1013.282687s 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb050886700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb04c0a11a0 0x7fb04c09ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53196/0 (socket says 172.21.15.76:53196) 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb051087700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb04c0a4840 0x7fb04c09b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50076/0 (socket says 172.21.15.76:50076) 2024-09-16T06:41:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb050886700 1 -- 172.21.15.76:0/217547715 learned_addr learned my addr 172.21.15.76:0/217547715 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052551664 0 0) 0x7fb04c0a4c20 con 0x7fb04c0a11a0 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): my global_id is 104697 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380035d0 con 0x7fb04c0a11a0 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493785216 0 0) 0x7fb04c09c120 con 0x7fb04c0a4840 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): my global_id is 124391 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c0a4c20 con 0x7fb04c0a4840 2024-09-16T06:41:06.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144706209 0 0) 0x7fb04c09fc40 con 0x7fb04c071d80 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient(hunting): my global_id is 134104 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c09c120 con 0x7fb04c071d80 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878640216 0 0) 0x7fb0380035d0 con 0x7fb04c0a11a0 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 >> v1:172.21.15.76:6790/0 conn(0x7fb04c0a4840 legacy=0x7fb04c09b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 >> v1:172.21.15.189:6789/0 conn(0x7fb04c071d80 legacy=0x7fb04c13a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 monclient: found mon.noname-a 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c09d550 con 0x7fb04c0a11a0 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb03c001540 con 0x7fb04c0a11a0 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 5 monclient: authenticate success, global_id 104697 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: dump: 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.154 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c001820 con 0x7fb04c0a11a0 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 1 -- 172.21.15.76:0/217547715 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb03c001bb0 con 0x7fb04c0a11a0 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb052089700 4 set_mon_vals no callback set 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.144+0000 7fb043fff700 10 monclient: dump: 2024-09-16T06:41:06.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: shutdown 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/217547715 >> v1:172.21.15.76:6789/0 conn(0x7fb04c0a11a0 legacy=0x7fb04c09ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/217547715 shutdown_connections 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/217547715 >> 172.21.15.76:0/217547715 conn(0x7fb04c0a4430 msgr2=0x7fb04c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/217547715 shutdown_connections 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/217547715 wait complete. 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: monmap: 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.157 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.158 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 Processor -- start 2024-09-16T06:41:06.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- start start 2024-09-16T06:41:06.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: init 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient(hunting): picked mon.noname-a con 0x7fb04c071d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient(hunting): picked mon.noname-b con 0x7fb04c0a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient(hunting): picked mon.noname-c con 0x7fb04c06da10 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c09b0f0 con 0x7fb04c071d80 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c125ad0 con 0x7fb04c0a4840 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c0a2380 con 0x7fb04c06da10 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient(hunting): authenticate will time out at 1013.286101s 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb051888700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb04c06da10 0x7fb04c1d1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50082/0 (socket says 172.21.15.76:50082) 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb051888700 1 -- 172.21.15.76:0/1714137673 learned_addr learned my addr 172.21.15.76:0/1714137673 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb051087700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb04c071d80 0x7fb04c06cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53208/0 (socket says 172.21.15.76:53208) 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783408607 0 0) 0x7fb04c0a2380 con 0x7fb04c06da10 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): my global_id is 124394 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0300035d0 con 0x7fb04c06da10 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402605333 0 0) 0x7fb04c09b0f0 con 0x7fb04c071d80 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): my global_id is 104700 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c0a2380 con 0x7fb04c071d80 2024-09-16T06:41:06.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266344550 0 0) 0x7fb04c125ad0 con 0x7fb04c0a4840 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): my global_id is 134107 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c09b0f0 con 0x7fb04c0a4840 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 371721700 0 0) 0x7fb0300035d0 con 0x7fb04c06da10 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb04c125ad0 con 0x7fb04c06da10 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103001360 0 0) 0x7fb04c0a2380 con 0x7fb04c071d80 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0300035d0 con 0x7fb04c071d80 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596019289 0 0) 0x7fb04c09b0f0 con 0x7fb04c0a4840 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb04c0a2380 con 0x7fb04c0a4840 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb044001c90 con 0x7fb04c06da10 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb048001830 con 0x7fb04c071d80 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb03c001500 con 0x7fb04c0a4840 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 62737733 0 0) 0x7fb04c0a2380 con 0x7fb04c0a4840 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6790/0 conn(0x7fb04c06da10 legacy=0x7fb04c1d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6789/0 conn(0x7fb04c071d80 legacy=0x7fb04c06cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 monclient: found mon.b 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c126090 con 0x7fb04c0a4840 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 5 monclient: authenticate success, global_id 134107 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _renew_subs 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c1257f0 con 0x7fb04c0a4840 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _renew_subs 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c13b9e0 con 0x7fb04c0a4840 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c004b40 con 0x7fb04c0a4840 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb03c004e50 con 0x7fb04c0a4840 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb041ffb700 10 monclient: dump: 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.148+0000 7fb042ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb03c00f8c0 con 0x7fb04c0a4840 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb052089700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb02c006590 con 0x7fb02c002830 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7fb03c043980 con 0x7fb04c0a4840 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb051888700 10 monclient: get_auth_request con 0x7fb0300366b0 auth_method 0 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878152403 0 0) 0x7fb02c006590 con 0x7fb02c002830 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient(hunting): my global_id is 104703 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c0a2380 con 0x7fb02c002830 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 593315308 0 0) 0x7fb04c0a2380 con 0x7fb02c002830 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb02c006590 con 0x7fb02c002830 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb048004890 con 0x7fb02c002830 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient: dump: 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1769000334 0 0) 0x7fb02c006590 con 0x7fb02c002830 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb02c00a190 con 0x7fb02c002830 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb048006ad0 con 0x7fb02c002830 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6789/0 conn(0x7fb02c002830 legacy=0x7fb02c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.152+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6789/0 conn(0x7fb02c002830 legacy=0x7fb02c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb052089700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:06.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb02c009e30 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397069478 0 0) 0x7fb02c009e30 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient(hunting): my global_id is 104706 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb02c006590 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1868070699 0 0) 0x7fb02c006590 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb02c009e30 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb0480051a0 con 0x7fb02c009150 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient: dump: 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2368996227 0 0) 0x7fb02c009e30 con 0x7fb02c009150 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb02c00c340 con 0x7fb02c009150 2024-09-16T06:41:06.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1802900380) 0x7fb0480064c0 con 0x7fb02c009150 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6789/0 conn(0x7fb02c009150 legacy=0x7fb02c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.164+0000 7fb041ffb700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.76:6789/0 conn(0x7fb02c009150 legacy=0x7fb02c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:06.176 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.177 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.178 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.179 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.181 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.182 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.184 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:06.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.186 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:06.187 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:06.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb0300366b0 legacy=0x7fb030038b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 10 monclient: shutdown 2024-09-16T06:41:06.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 >> v1:172.21.15.189:6789/0 conn(0x7fb04c0a4840 legacy=0x7fb04c06d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb051888700 1 -- 172.21.15.76:0/1714137673 reap_dead start 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb051888700 1 -- 172.21.15.76:0/1714137673 reap_dead start 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 shutdown_connections 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 >> 172.21.15.76:0/1714137673 conn(0x7fb04c0a4430 msgr2=0x7fb04c09f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 shutdown_connections 2024-09-16T06:41:06.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.168+0000 7fb052089700 1 -- 172.21.15.76:0/1714137673 wait complete. 2024-09-16T06:41:06.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:06.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:41:06.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:41:06.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f625dc37700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f625dc37700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f625dc37700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f6238005930 con 0x7f62580a3ca0 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (unknown 3095293837 0 0) 0x7f6250012f90 con 0x7f62580a3ca0 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f623ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.293+0000 7f623ffff700 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T06:41:06.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: monmap: 2024-09-16T06:41:06.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:06.308238+0000 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:06.308238+0000 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 Processor -- start 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- start start 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: init 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-c con 0x7fa02401f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-b con 0x7fa024025410 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-a con 0x7fa024025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02401f8a0 con 0x7fa024025860 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240233f0 con 0x7fa02401f4c0 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa024020050 con 0x7fa024025410 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient(hunting): authenticate will time out at 1013.441555s 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa024025410 0x7fa0240bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50084/0 (socket says 172.21.15.76:50084) 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02b7fe700 1 -- 172.21.15.76:0/3515279796 learned_addr learned my addr 172.21.15.76:0/3515279796 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513943168 0 0) 0x7fa024020050 con 0x7fa024025410 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): my global_id is 124397 2024-09-16T06:41:06.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa01c0035d0 con 0x7fa024025410 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233920664 0 0) 0x7fa02401f8a0 con 0x7fa024025860 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): my global_id is 104709 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024020050 con 0x7fa024025860 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057196593 0 0) 0x7fa0240233f0 con 0x7fa02401f4c0 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient(hunting): my global_id is 134110 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02401f8a0 con 0x7fa02401f4c0 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2531950959 0 0) 0x7fa024020050 con 0x7fa024025860 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 >> v1:172.21.15.76:6790/0 conn(0x7fa024025410 legacy=0x7fa0240bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 >> v1:172.21.15.189:6789/0 conn(0x7fa02401f4c0 legacy=0x7fa024024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 monclient: found mon.noname-a 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa024021a90 con 0x7fa024025860 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa02c04ec20 con 0x7fa024025860 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: dump: 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.326 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 5 monclient: authenticate success, global_id 104709 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa02c04ef50 con 0x7fa024025860 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 1 -- 172.21.15.76:0/3515279796 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa02c06a310 con 0x7fa024025860 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 4 set_mon_vals no callback set 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa02affd700 10 monclient: dump: 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 10 monclient: shutdown 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- 172.21.15.76:0/3515279796 >> v1:172.21.15.76:6789/0 conn(0x7fa024025860 legacy=0x7fa0240c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- 172.21.15.76:0/3515279796 shutdown_connections 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- 172.21.15.76:0/3515279796 >> 172.21.15.76:0/3515279796 conn(0x7fa024005b40 msgr2=0x7fa024005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- 172.21.15.76:0/3515279796 shutdown_connections 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.304+0000 7fa0313c2700 1 -- 172.21.15.76:0/3515279796 wait complete. 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: monmap: 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 Processor -- start 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- start start 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: init 2024-09-16T06:41:06.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-c con 0x7fa02401f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-b con 0x7fa024025410 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient(hunting): picked mon.noname-a con 0x7fa024025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240204a0 con 0x7fa024025860 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0241481f0 con 0x7fa024025410 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02401bc50 con 0x7fa02401f4c0 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient(hunting): authenticate will time out at 1013.445891s 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa02bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa02401f4c0 0x7fa024147ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50094/0 (socket says 172.21.15.76:50094) 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa030bc1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa024025860 0x7fa024156540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53246/0 (socket says 172.21.15.76:53246) 2024-09-16T06:41:06.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa02bfff700 1 -- 172.21.15.76:0/3426212912 learned_addr learned my addr 172.21.15.76:0/3426212912 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025397170 0 0) 0x7fa0240204a0 con 0x7fa024025860 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): my global_id is 104712 2024-09-16T06:41:06.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0100035d0 con 0x7fa024025860 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416949217 0 0) 0x7fa02401bc50 con 0x7fa02401f4c0 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): my global_id is 124400 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0240204a0 con 0x7fa02401f4c0 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231988114 0 0) 0x7fa0241481f0 con 0x7fa024025410 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): my global_id is 134113 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02401bc50 con 0x7fa024025410 2024-09-16T06:41:06.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2999898795 0 0) 0x7fa02401bc50 con 0x7fa024025410 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0241481f0 con 0x7fa024025410 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193986385 0 0) 0x7fa0240204a0 con 0x7fa02401f4c0 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa02401bc50 con 0x7fa02401f4c0 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa0180018d0 con 0x7fa024025410 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.334 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa020001930 con 0x7fa02401f4c0 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.335 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 915536198 0 0) 0x7fa0100035d0 con 0x7fa024025860 2024-09-16T06:41:06.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0240204a0 con 0x7fa024025860 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa02c04ec20 con 0x7fa024025860 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.337 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250989778 0 0) 0x7fa02401bc50 con 0x7fa02401f4c0 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.189:6789/0 conn(0x7fa024025410 legacy=0x7fa024152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6789/0 conn(0x7fa024025860 legacy=0x7fa024156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 monclient: found mon.c 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02412c9b0 con 0x7fa02401f4c0 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 5 monclient: authenticate success, global_id 124400 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _renew_subs 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa024028d50 con 0x7fa02401f4c0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _renew_subs 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02412d100 con 0x7fa02401f4c0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa020001a90 con 0x7fa02401f4c0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa020003970 con 0x7fa02401f4c0 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa028ff9700 10 monclient: dump: 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.341 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.341 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.341 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.308+0000 7fa02a7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:06.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa0313c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa020003b90 con 0x7fa02401f4c0 2024-09-16T06:41:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa00c006590 con 0x7fa00c002830 2024-09-16T06:41:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7fa020043820 con 0x7fa02401f4c0 2024-09-16T06:41:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa030bc1700 10 monclient: get_auth_request con 0x7fa010036590 auth_method 0 2024-09-16T06:41:06.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482608494 0 0) 0x7fa00c006590 con 0x7fa00c002830 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient(hunting): my global_id is 124403 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02401bc50 con 0x7fa00c002830 2024-09-16T06:41:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1996337415 0 0) 0x7fa02401bc50 con 0x7fa00c002830 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa00c006590 con 0x7fa00c002830 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa018004c20 con 0x7fa00c002830 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient: dump: 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 620869060 0 0) 0x7fa00c006590 con 0x7fa00c002830 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa00c00a190 con 0x7fa00c002830 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa018005100 con 0x7fa00c002830 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6790/0 conn(0x7fa00c002830 legacy=0x7fa00c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.312+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6790/0 conn(0x7fa00c002830 legacy=0x7fa00c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:06.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa00c009e50 con 0x7fa00c009170 2024-09-16T06:41:06.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029833089 0 0) 0x7fa00c009e50 con 0x7fa00c009170 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient(hunting): my global_id is 124406 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa00c006590 con 0x7fa00c009170 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3929868303 0 0) 0x7fa00c006590 con 0x7fa00c009170 2024-09-16T06:41:06.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa00c009e50 con 0x7fa00c009170 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa018005680 con 0x7fa00c009170 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient: dump: 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.351 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 76648896 0 0) 0x7fa00c009e50 con 0x7fa00c009170 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa00c00c340 con 0x7fa00c009170 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1931435752) 0x7fa018005970 con 0x7fa00c009170 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6790/0 conn(0x7fa00c009170 legacy=0x7fa00c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa028ff9700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6790/0 conn(0x7fa00c009170 legacy=0x7fa00c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:06.352 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.355 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:06.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.357 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.358 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.359 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:06.360 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:06.361 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:06.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.364 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:06.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:06.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.367 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:06.367 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa010036590 legacy=0x7fa010038a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 10 monclient: shutdown 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 >> v1:172.21.15.76:6790/0 conn(0x7fa02401f4c0 legacy=0x7fa024147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa030bc1700 1 -- 172.21.15.76:0/3426212912 reap_dead start 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa030bc1700 1 -- 172.21.15.76:0/3426212912 reap_dead start 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 shutdown_connections 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 >> 172.21.15.76:0/3426212912 conn(0x7fa024005b40 msgr2=0x7fa02401a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 shutdown_connections 2024-09-16T06:41:06.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.324+0000 7fa0313c2700 1 -- 172.21.15.76:0/3426212912 wait complete. 2024-09-16T06:41:06.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:06.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:41:06.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:41:06.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.393+0000 7f625dc37700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.393+0000 7f625dc37700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.393+0000 7f625dc37700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:06.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.393+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f62380053b0 con 0x7f62580a3ca0 2024-09-16T06:41:06.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.393+0000 7f623ffff700 1 -- 172.21.15.189:0/3505216841 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v71) v1 ==== 201+0+0 (unknown 3095293837 0 0) 0x7f6250014710 con 0x7f62580a3ca0 2024-09-16T06:41:06.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f623ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T06:41:06.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f623ffff700 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6244036510 legacy=0x7f62440389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 10 monclient: shutdown 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 >> v1:172.21.15.189:6789/0 conn(0x7f62580a3ca0 legacy=0x7f62581d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 shutdown_connections 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 >> 172.21.15.189:0/3505216841 conn(0x7f62580a7d70 msgr2=0x7f625809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 shutdown_connections 2024-09-16T06:41:06.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.397+0000 7f625dc37700 1 -- 172.21.15.189:0/3505216841 wait complete. 2024-09-16T06:41:06.403 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T06:41:06.411 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: monmap: 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.495 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:06.496888+0000 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:06.496888+0000 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.496 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 1 Processor -- start 2024-09-16T06:41:06.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 1 -- start start 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: init 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-a con 0x7f4c040964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-b con 0x7f4c0409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-c con 0x7f4c04137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c04098560 con 0x7f4c040964f0 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c04099ec0 con 0x7f4c04137e70 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0409b5f0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 10 monclient(hunting): authenticate will time out at 1013.630637s 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0a506700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4c040964f0 0x7f4c040968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53248/0 (socket says 172.21.15.76:53248) 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09d05700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4c0409b1c0 0x7f4c04136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50116/0 (socket says 172.21.15.76:50116) 2024-09-16T06:41:06.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0a506700 1 -- 172.21.15.76:0/432901153 learned_addr learned my addr 172.21.15.76:0/432901153 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245880332 0 0) 0x7f4c0409b5f0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): my global_id is 124409 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf00035d0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325166076 0 0) 0x7f4c04098560 con 0x7f4c040964f0 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): my global_id is 104715 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0409b5f0 con 0x7f4c040964f0 2024-09-16T06:41:06.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170871217 0 0) 0x7f4c04099ec0 con 0x7f4c04137e70 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient(hunting): my global_id is 134116 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04098560 con 0x7f4c04137e70 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176336877 0 0) 0x7f4bf00035d0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 >> v1:172.21.15.189:6789/0 conn(0x7f4c04137e70 legacy=0x7f4c0413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 >> v1:172.21.15.76:6789/0 conn(0x7f4c040964f0 legacy=0x7f4c040968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 monclient: found mon.noname-b 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04099be0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4bf4001610 con 0x7f4c0409b1c0 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 5 monclient: authenticate success, global_id 124409 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: dump: 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c0b508700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bf40018f0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.492+0000 7f4c09504700 1 -- 172.21.15.76:0/432901153 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4bf4001c80 con 0x7f4c0409b1c0 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 4 set_mon_vals no callback set 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c09504700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c09504700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c09504700 10 monclient: dump: 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 10 monclient: shutdown 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- 172.21.15.76:0/432901153 >> v1:172.21.15.76:6790/0 conn(0x7f4c0409b1c0 legacy=0x7f4c04136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- 172.21.15.76:0/432901153 shutdown_connections 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- 172.21.15.76:0/432901153 >> 172.21.15.76:0/432901153 conn(0x7f4c040a0030 msgr2=0x7f4c040a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- 172.21.15.76:0/432901153 shutdown_connections 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- 172.21.15.76:0/432901153 wait complete. 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 10 monclient: monmap: 2024-09-16T06:41:06.507 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.508 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 Processor -- start 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 1 -- start start 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.496+0000 7f4c0b508700 10 monclient: init 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-a con 0x7f4c040964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-c con 0x7f4c0409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient(hunting): picked mon.noname-b con 0x7f4c04137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0408d110 con 0x7f4c040964f0 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0a506700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4c040964f0 0x7f4c0408b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53258/0 (socket says 172.21.15.76:53258) 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0a506700 1 -- 172.21.15.76:0/3559528119 learned_addr learned my addr 172.21.15.76:0/3559528119 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0408f190 con 0x7f4c04137e70 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0408c3e0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient(hunting): authenticate will time out at 1013.637298s 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237285280 0 0) 0x7f4c0408d110 con 0x7f4c040964f0 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): my global_id is 104718 2024-09-16T06:41:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be80035d0 con 0x7f4c040964f0 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060797808 0 0) 0x7f4c0408c3e0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): my global_id is 124412 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0408d110 con 0x7f4c0409b1c0 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593175185 0 0) 0x7f4c0408f190 con 0x7f4c04137e70 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): my global_id is 134119 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0408c3e0 con 0x7f4c04137e70 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226640523 0 0) 0x7f4be80035d0 con 0x7f4c040964f0 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c0408f190 con 0x7f4c040964f0 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c00001870 con 0x7f4c040964f0 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:06.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): dump: 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958954818 0 0) 0x7f4c0408c3e0 con 0x7f4c04137e70 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be80035d0 con 0x7f4c04137e70 2024-09-16T06:41:06.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136822823 0 0) 0x7f4c0408d110 con 0x7f4c0409b1c0 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c0408c3e0 con 0x7f4c0409b1c0 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4bfc001bb0 con 0x7f4c04137e70 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient(hunting): dump: 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.519 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2110305990 0 0) 0x7f4c0408f190 con 0x7f4c040964f0 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.76:6790/0 conn(0x7f4c0409b1c0 legacy=0x7f4c0408bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6789/0 conn(0x7f4c04137e70 legacy=0x7f4c041ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 monclient: found mon.a 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0408dce0 con 0x7f4c040964f0 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 5 monclient: authenticate success, global_id 104718 2024-09-16T06:41:06.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _renew_subs 2024-09-16T06:41:06.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c04097e50 con 0x7f4c040964f0 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _renew_subs 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0408f470 con 0x7f4c040964f0 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c00004890 con 0x7f4c040964f0 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c00001d80 con 0x7f4c040964f0 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4c08d03700 4 set_mon_vals no callback set 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.500+0000 7f4bfaffd700 10 monclient: dump: 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.525 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4c0000f530 con 0x7f4c040964f0 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4c0b508700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4bec006590 con 0x7f4bec002830 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7f4c00043760 con 0x7f4c040964f0 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4c0ad07700 10 monclient: get_auth_request con 0x7f4be8036440 auth_method 0 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818153390 0 0) 0x7f4bec006590 con 0x7f4bec002830 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient(hunting): my global_id is 134122 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0408f190 con 0x7f4bec002830 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1096697017 0 0) 0x7f4c0408f190 con 0x7f4bec002830 2024-09-16T06:41:06.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bec006590 con 0x7f4bec002830 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4bf4004c20 con 0x7f4bec002830 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient: dump: 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.527 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3843101584 0 0) 0x7f4bec006590 con 0x7f4bec002830 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bec00a190 con 0x7f4bec002830 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4bf4005030 con 0x7f4bec002830 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6789/0 conn(0x7f4bec002830 legacy=0x7f4bec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.504+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6789/0 conn(0x7f4bec002830 legacy=0x7f4bec004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4bec009e30 con 0x7f4bec009150 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170342663 0 0) 0x7f4bec009e30 con 0x7f4bec009150 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient(hunting): my global_id is 134128 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec006590 con 0x7f4bec009150 2024-09-16T06:41:06.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3962402464 0 0) 0x7f4bec006590 con 0x7f4bec009150 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bec009e30 con 0x7f4bec009150 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4bf40055b0 con 0x7f4bec009150 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient: dump: 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.533 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2810600435 0 0) 0x7f4bec009e30 con 0x7f4bec009150 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4bec00c340 con 0x7f4bec009150 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 85755580) 0x7f4bf4005900 con 0x7f4bec009150 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6789/0 conn(0x7f4bec009150 legacy=0x7f4bec0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4bfaffd700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6789/0 conn(0x7f4bec009150 legacy=0x7f4bec0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:06.535 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.536 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.537 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.538 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.538 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:06.540 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:06.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.543 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:06.544 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:06.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:06.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:06.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:06.548 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:06.548 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:06.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4be8036440 legacy=0x7f4be8038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 10 monclient: shutdown 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 >> v1:172.21.15.76:6789/0 conn(0x7f4c040964f0 legacy=0x7f4c0408b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0ad07700 1 -- 172.21.15.76:0/3559528119 reap_dead start 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0ad07700 1 -- 172.21.15.76:0/3559528119 reap_dead start 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 shutdown_connections 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 >> 172.21.15.76:0/3559528119 conn(0x7f4c040a0030 msgr2=0x7f4c0409bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 shutdown_connections 2024-09-16T06:41:06.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.516+0000 7f4c0b508700 1 -- 172.21.15.76:0/3559528119 wait complete. 2024-09-16T06:41:06.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:06.552 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:41:06.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.509+0000 7f53cf382700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.509+0000 7f53cf382700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.509+0000 7f53cf382700 10 monclient: monmap: 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:06.516108+0000 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:06.516108+0000 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 Processor -- start 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- start start 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient: init 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-c con 0x7f53c809eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-a con 0x7f53c809f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-b con 0x7f53c8139a50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c809ab80 con 0x7f53c809f690 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c809a690 con 0x7f53c809eba0 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c8072230 con 0x7f53c8139a50 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient(hunting): authenticate will time out at 1019.028978s 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53ce380700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53c809eba0 0x7f53c809ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45834/0 (socket says 172.21.15.189:45834) 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53ce380700 1 -- 172.21.15.189:0/365825858 learned_addr learned my addr 172.21.15.189:0/365825858 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556475442 0 0) 0x7f53c809ab80 con 0x7f53c809f690 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): my global_id is 104721 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b40035d0 con 0x7f53c809f690 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993730980 0 0) 0x7f53c809a690 con 0x7f53c809eba0 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): my global_id is 134125 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c809ab80 con 0x7f53c809eba0 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893731281 0 0) 0x7f53c8072230 con 0x7f53c8139a50 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient(hunting): my global_id is 124415 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c809a690 con 0x7f53c8139a50 2024-09-16T06:41:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3413604299 0 0) 0x7f53b40035d0 con 0x7f53c809f690 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 >> v1:172.21.15.76:6790/0 conn(0x7f53c8139a50 legacy=0x7f53c813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 >> v1:172.21.15.189:6789/0 conn(0x7f53c809eba0 legacy=0x7f53c809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 monclient: found mon.noname-a 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c809be90 con 0x7f53c809f690 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 5 monclient: authenticate success, global_id 104721 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53b8001610 con 0x7f53c809f690 2024-09-16T06:41:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: dump: 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b80018f0 con 0x7f53c809f690 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 1 -- 172.21.15.189:0/365825858 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53b8001c80 con 0x7f53c809f690 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 4 set_mon_vals no callback set 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cd37e700 10 monclient: dump: 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 10 monclient: shutdown 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- 172.21.15.189:0/365825858 >> v1:172.21.15.76:6789/0 conn(0x7f53c809f690 legacy=0x7f53c8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- 172.21.15.189:0/365825858 shutdown_connections 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- 172.21.15.189:0/365825858 >> 172.21.15.189:0/365825858 conn(0x7f53c809fe70 msgr2=0x7f53c80a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- 172.21.15.189:0/365825858 shutdown_connections 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.513+0000 7f53cf382700 1 -- 172.21.15.189:0/365825858 wait complete. 2024-09-16T06:41:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient: monmap: 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 1 Processor -- start 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 1 -- start start 2024-09-16T06:41:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient: init 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-b con 0x7f53c809eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-c con 0x7f53c809f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient(hunting): picked mon.noname-a con 0x7f53c8139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c8136140 con 0x7f53c8139a50 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c806b8c0 con 0x7f53c809eba0 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c80714f0 con 0x7f53c809f690 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53cf382700 10 monclient(hunting): authenticate will time out at 1019.035412s 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53ce380700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53c809eba0 0x7f53c8135320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45844/0 (socket says 172.21.15.189:45844) 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53ce380700 1 -- 172.21.15.189:0/1600543739 learned_addr learned my addr 172.21.15.189:0/1600543739 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778990345 0 0) 0x7f53c806b8c0 con 0x7f53c809eba0 2024-09-16T06:41:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.517+0000 7f53beffd700 10 monclient(hunting): my global_id is 134131 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ac0035d0 con 0x7f53c809eba0 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311387165 0 0) 0x7f53c8136140 con 0x7f53c8139a50 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): my global_id is 104724 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c806b8c0 con 0x7f53c8139a50 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048852220 0 0) 0x7f53c80714f0 con 0x7f53c809f690 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): my global_id is 124418 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c8136140 con 0x7f53c809f690 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598497910 0 0) 0x7f53c806b8c0 con 0x7f53c8139a50 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c80714f0 con 0x7f53c8139a50 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53c0001c30 con 0x7f53c8139a50 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): dump: 2024-09-16T06:41:06.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086957774 0 0) 0x7f53ac0035d0 con 0x7f53c809eba0 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c806b8c0 con 0x7f53c809eba0 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53c4001950 con 0x7f53c809eba0 2024-09-16T06:41:06.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): dump: 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.571 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2487720756 0 0) 0x7f53c8136140 con 0x7f53c809f690 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53ac0035d0 con 0x7f53c809f690 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53b8001c00 con 0x7f53c809f690 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient(hunting): dump: 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2335580883 0 0) 0x7f53c80714f0 con 0x7f53c8139a50 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 >> v1:172.21.15.76:6790/0 conn(0x7f53c809f690 legacy=0x7f53c8135a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 >> v1:172.21.15.189:6789/0 conn(0x7f53c809eba0 legacy=0x7f53c8135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 monclient: found mon.a 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c808f730 con 0x7f53c8139a50 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 5 monclient: authenticate success, global_id 104724 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _renew_subs 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c808fa10 con 0x7f53c8139a50 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _renew_subs 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c808cbc0 con 0x7f53c8139a50 2024-09-16T06:41:06.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c0003a40 con 0x7f53c8139a50 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53c0003d50 con 0x7f53c8139a50 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 10 monclient: dump: 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53ccb7d700 4 set_mon_vals no callback set 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f53c000e7c0 con 0x7f53c8139a50 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7f53c00428c0 con 0x7f53c8139a50 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53ce380700 10 monclient: get_auth_request con 0x7f53ac0364e0 auth_method 0 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.521+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0005100 con 0x7f53c8139a50 2024-09-16T06:41:06.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.529+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f53c0013940 con 0x7f53c8139a50 2024-09-16T06:41:06.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.529+0000 7f53beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.529+0000 7f53beffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: get_monmap_and_config 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: monmap: 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:06.677595+0000 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:06.677595+0000 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.677 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:06.677 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:06.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 1 Processor -- start 2024-09-16T06:41:06.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 1 -- start start 2024-09-16T06:41:06.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: init 2024-09-16T06:41:06.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient(hunting): picked mon.noname-b con 0x7f278c1342c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient(hunting): picked mon.noname-a con 0x7f278c06cfb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient(hunting): picked mon.noname-c con 0x7f278c06d410 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0a31f0 con 0x7f278c06cfb0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c06c4f0 con 0x7f278c06d410 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c06bbe0 con 0x7f278c1342c0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f2791464700 10 monclient(hunting): authenticate will time out at 1013.810993s 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f278c06cfb0 0x7f278c13a280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53264/0 (socket says 172.21.15.76:53264) 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278b7fe700 1 -- 172.21.15.76:0/4261026376 learned_addr learned my addr 172.21.15.76:0/4261026376 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703764489 0 0) 0x7f278c0a31f0 con 0x7f278c06cfb0 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): my global_id is 104727 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27780035d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279087484 0 0) 0x7f278c06bbe0 con 0x7f278c1342c0 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): my global_id is 124421 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c0a31f0 con 0x7f278c1342c0 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281812395 0 0) 0x7f278c06c4f0 con 0x7f278c06d410 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 10 monclient(hunting): my global_id is 134134 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c06bbe0 con 0x7f278c06d410 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.672+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360097053 0 0) 0x7f278c0a31f0 con 0x7f278c1342c0 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 >> v1:172.21.15.189:6789/0 conn(0x7f278c06d410 legacy=0x7f278c13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 >> v1:172.21.15.76:6789/0 conn(0x7f278c06cfb0 legacy=0x7f278c13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 monclient: found mon.noname-b 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c09ace0 con 0x7f278c1342c0 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 5 monclient: authenticate success, global_id 124421 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f27740015f0 con 0x7f278c1342c0 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: dump: 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2774001910 con 0x7f278c1342c0 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 4 set_mon_vals no callback set 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 1 -- 172.21.15.76:0/4261026376 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2774001ca0 con 0x7f278c1342c0 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278affd700 10 monclient: dump: 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.685 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: shutdown 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- 172.21.15.76:0/4261026376 >> v1:172.21.15.76:6790/0 conn(0x7f278c1342c0 legacy=0x7f278c1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- 172.21.15.76:0/4261026376 shutdown_connections 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- 172.21.15.76:0/4261026376 >> 172.21.15.76:0/4261026376 conn(0x7f278c0a41c0 msgr2=0x7f278c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- 172.21.15.76:0/4261026376 shutdown_connections 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- 172.21.15.76:0/4261026376 wait complete. 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: build_initial_monmap 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: monmap: 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 Processor -- start 2024-09-16T06:41:06.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- start start 2024-09-16T06:41:06.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: init 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient(hunting): picked mon.noname-a con 0x7f278c06cfb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient(hunting): picked mon.noname-c con 0x7f278c06d410 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient(hunting): picked mon.noname-b con 0x7f278c1342c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c1324d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c12f510 con 0x7f278c1342c0 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c09e8b0 con 0x7f278c06d410 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2791464700 10 monclient(hunting): authenticate will time out at 1013.815300s 2024-09-16T06:41:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f278c06cfb0 0x7f278c12e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53280/0 (socket says 172.21.15.76:53280) 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f278bfff700 1 -- 172.21.15.76:0/971089601 learned_addr learned my addr 172.21.15.76:0/971089601 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2790c63700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6789/0 conn(0x7f278c1342c0 legacy=0x7f278c1d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2790c63700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6789/0 conn(0x7f278c1342c0 legacy=0x7f278c1d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2790c63700 1 --1- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6789/0 conn(0x7f278c1342c0 0x7f278c1d2520 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2790c63700 1 --1- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6789/0 conn(0x7f278c1342c0 0x7f278c1d2520 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194921883 0 0) 0x7f278c1324d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): my global_id is 104730 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276c0035d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839320298 0 0) 0x7f278c09e8b0 con 0x7f278c06d410 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 10 monclient(hunting): my global_id is 124424 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.676+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c1324d0 con 0x7f278c06d410 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3363746988 0 0) 0x7f278c1324d0 con 0x7f278c06d410 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f278c09e8b0 con 0x7f278c06d410 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f277c001c50 con 0x7f278c06d410 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.694 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707256000 0 0) 0x7f276c0035d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f278c1324d0 con 0x7f278c06cfb0 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2774001c20 con 0x7f278c06cfb0 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3295620286 0 0) 0x7f278c09e8b0 con 0x7f278c06d410 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6789/0 conn(0x7f278c1342c0 legacy=0x7f278c1d2520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:06.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.76:6789/0 conn(0x7f278c06cfb0 legacy=0x7f278c12e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 monclient: found mon.c 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0a5e30 con 0x7f278c06d410 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 5 monclient: authenticate success, global_id 124424 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _renew_subs 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c0a58e0 con 0x7f278c06d410 2024-09-16T06:41:06.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _renew_subs 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c131c30 con 0x7f278c06d410 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f277c004c10 con 0x7f278c06d410 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f277c004fa0 con 0x7f278c06d410 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f278a7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 10 monclient: dump: 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:06.698 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770005100 con 0x7f278c06d410 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f277c005180 con 0x7f278c06d410 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 5208+0+0 (unknown 3875812706 0 0) 0x7f277c043b20 con 0x7f278c06d410 2024-09-16T06:41:06.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.680+0000 7f278bfff700 10 monclient: get_auth_request con 0x7f276c0362c0 auth_method 0 2024-09-16T06:41:06.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.688+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f277c014970 con 0x7f278c06d410 2024-09-16T06:41:06.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.688+0000 7f2788ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:06.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.688+0000 7f2788ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.685+0000 7f53cf382700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.685+0000 7f53cf382700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.685+0000 7f53cf382700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.689+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f53b0005930 con 0x7f53c8139a50 2024-09-16T06:41:06.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.877+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v72) v1 ==== 138+0+0 (unknown 3564594782 0 0) 0x7f53c0011db0 con 0x7f53c8139a50 2024-09-16T06:41:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.877+0000 7f53beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.877+0000 7f53beffd700 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-16T06:41:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.920+0000 7f2791464700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:41:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.920+0000 7f2791464700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:41:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.920+0000 7f2791464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.920+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f2770005bc0 con 0x7f278c06d410 2024-09-16T06:41:06.929 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.929 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=2380876981,config=1097430493,health=1886805112,kv=790649379,logm=2924705768}) 2024-09-16T06:41:06.929 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=13,logm=54} crc {auth=2880990158,config=1097430493,health=2157093694,kv=2121020328,logm=3485415066}) 2024-09-16T06:41:06.929 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.929 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=2380876981,config=1097430493,health=1886805112,kv=790649379,logm=2924705768}) 2024-09-16T06:41:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.924+0000 7f2788ff9700 1 -- 172.21.15.76:0/971089601 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f277c013010 con 0x7f278c06d410 2024-09-16T06:41:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.924+0000 7f2788ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:41:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.924+0000 7f2788ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:06.930 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.924+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=13,logm=53} crc {auth=1209923135,config=1097430493,health=2157093694,kv=1001081567,logm=2549507136}) 2024-09-16T06:41:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.189:6812/3098810674 conn(0x7f276c0362c0 legacy=0x7f276c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 10 monclient: shutdown 2024-09-16T06:41:06.931 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 >> v1:172.21.15.76:6790/0 conn(0x7f278c06d410 legacy=0x7f278c12ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.932 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457431472}) 2024-09-16T06:41:06.932 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1413622306}) 2024-09-16T06:41:06.932 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.932 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457431472}) 2024-09-16T06:41:06.932 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4773662}) 2024-09-16T06:41:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 shutdown_connections 2024-09-16T06:41:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 >> 172.21.15.76:0/971089601 conn(0x7f278c0a41c0 msgr2=0x7f278c0a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 shutdown_connections 2024-09-16T06:41:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:06.928+0000 7f2791464700 1 -- 172.21.15.76:0/971089601 wait complete. 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933168420}) 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001001432}) 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933168420}) 2024-09-16T06:41:06.934 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.928+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593107186}) 2024-09-16T06:41:06.935 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.936 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102372547}) 2024-09-16T06:41:06.936 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779823193}) 2024-09-16T06:41:06.936 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.936 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102372547}) 2024-09-16T06:41:06.936 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242051092}) 2024-09-16T06:41:06.937 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.937 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294407794}) 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376160846}) 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294407794}) 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.932+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270845734}) 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.938 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506254756}) 2024-09-16T06:41:06.939 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018666746}) 2024-09-16T06:41:06.939 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.939 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506254756}) 2024-09-16T06:41:06.939 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601985240}) 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441880859}) 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3542196171}) 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441880859}) 2024-09-16T06:41:06.940 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115678688}) 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102860413}) 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538121258}) 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102860413}) 2024-09-16T06:41:06.941 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.936+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207214205}) 2024-09-16T06:41:06.942 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.942 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408700086}) 2024-09-16T06:41:06.942 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4014843741}) 2024-09-16T06:41:06.942 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.942 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408700086}) 2024-09-16T06:41:06.943 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3401060048}) 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=2457451620,mdsmap=1531151339,mgr=2759932744,mgr_command_descs=794974419,mgr_metadata=1347778423}) 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=2192555059,mdsmap=1531151339,mgr=2925341485}) 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=2457451620,mdsmap=1531151339,mgr=2759932744,mgr_command_descs=794974419,mgr_metadata=1347778423}) 2024-09-16T06:41:06.944 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=82,mdsmap=3,mgr=7,mgrstat=8} crc {logm=655267867,mdsmap=1531151339,mgr=4125395841,mgrstat=502146274}) 2024-09-16T06:41:06.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=6,osd_snap=1,osdmap=60} crc {mgrstat=1319999600,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_snap=3823785925,osdmap=1678906309}) 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {mgrstat=3514858817,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2516224954,osd_snap=2033388227,osdmap=416668932}) 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=6,osd_snap=1,osdmap=60} crc {mgrstat=1319999600,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_snap=3823785925,osdmap=1678906309}) 2024-09-16T06:41:06.946 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.940+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {mgrstat=2756796095,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=2516224954,osd_snap=2033388227,osdmap=3528432150}) 2024-09-16T06:41:06.948 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.948 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=2184058307}) 2024-09-16T06:41:06.949 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=3692483989}) 2024-09-16T06:41:06.949 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:06.949 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=2184058307}) 2024-09-16T06:41:06.949 INFO:tasks.ceph.mon.a.smithi076.stderr:2024-09-16T06:41:06.944+0000 7fd0de7cd700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=2564696147}) 2024-09-16T06:41:06.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.977+0000 7f53cf382700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.977+0000 7f53cf382700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.977+0000 7f53cf382700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.977+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f53b00053e0 con 0x7f53c8139a50 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53beffd700 1 -- 172.21.15.189:0/1600543739 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (unknown 1073408522 0 0) 0x7f53c0013530 con 0x7f53c8139a50 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53beffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:06.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53beffd700 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 >> v1:172.21.15.189:6812/3098810674 conn(0x7f53ac0364e0 legacy=0x7f53ac0389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 10 monclient: shutdown 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 >> v1:172.21.15.76:6789/0 conn(0x7f53c8139a50 legacy=0x7f53c81d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 shutdown_connections 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 >> 172.21.15.189:0/1600543739 conn(0x7f53c809fe70 msgr2=0x7f53c81383d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 shutdown_connections 2024-09-16T06:41:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:06.981+0000 7f53cf382700 1 -- 172.21.15.189:0/1600543739 wait complete. 2024-09-16T06:41:06.987 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'bar' already exists 2024-09-16T06:41:06.995 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T06:41:07.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: build_initial_monmap 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: monmap: 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:07.097755+0000 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:07.097755+0000 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:07.097 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:07.098 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:07.098 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 1 Processor -- start 2024-09-16T06:41:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 1 -- start start 2024-09-16T06:41:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: init 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f231006a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f23100a19f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f231006ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231009f8e0 con 0x7f231006ae30 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231009c960 con 0x7f23100a19f0 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100990c0 con 0x7f231006a340 2024-09-16T06:41:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230f59e700 10 monclient(hunting): authenticate will time out at 1019.610910s 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f23100a19f0 0x7f231013a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45850/0 (socket says 172.21.15.189:45850) 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230dd9b700 1 -- 172.21.15.189:0/1420009655 learned_addr learned my addr 172.21.15.189:0/1420009655 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167383001 0 0) 0x7f231009c960 con 0x7f23100a19f0 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): my global_id is 134137 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80035d0 con 0x7f23100a19f0 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713976035 0 0) 0x7f23100990c0 con 0x7f231006a340 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): my global_id is 124427 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231009c960 con 0x7f231006a340 2024-09-16T06:41:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032875195 0 0) 0x7f231009f8e0 con 0x7f231006ae30 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient(hunting): my global_id is 104733 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23100990c0 con 0x7f231006ae30 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410265028 0 0) 0x7f22f80035d0 con 0x7f23100a19f0 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 >> v1:172.21.15.76:6790/0 conn(0x7f231006a340 legacy=0x7f231006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 >> v1:172.21.15.76:6789/0 conn(0x7f231006ae30 legacy=0x7f231013dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 monclient: found mon.noname-c 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.093+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23100a1570 con 0x7f23100a19f0 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: _finish_auth 0 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 5 monclient: authenticate success, global_id 134137 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2300001610 con 0x7f23100a19f0 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: dump: 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.112 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23000018f0 con 0x7f23100a19f0 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 4 set_mon_vals no callback set 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 1 -- 172.21.15.189:0/1420009655 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2300001c80 con 0x7f23100a19f0 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230d59a700 10 monclient: dump: 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: shutdown 2024-09-16T06:41:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- 172.21.15.189:0/1420009655 >> v1:172.21.15.189:6789/0 conn(0x7f23100a19f0 legacy=0x7f231013a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- 172.21.15.189:0/1420009655 shutdown_connections 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- 172.21.15.189:0/1420009655 >> 172.21.15.189:0/1420009655 conn(0x7f23100a20a0 msgr2=0x7f23100a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- 172.21.15.189:0/1420009655 shutdown_connections 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- 172.21.15.189:0/1420009655 wait complete. 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: build_initial_monmap 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: monmap: 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 Processor -- start 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- start start 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: init 2024-09-16T06:41:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f231006a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f231006ae30 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f2310098000 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231009de80 con 0x7f2310098000 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231009c5b0 con 0x7f231006ae30 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100947d0 con 0x7f231006a340 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230f59e700 10 monclient(hunting): authenticate will time out at 1019.614306s 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f231006ae30 0x7f2310091020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45852/0 (socket says 172.21.15.189:45852) 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f230dd9b700 1 -- 172.21.15.189:0/661121021 learned_addr learned my addr 172.21.15.189:0/661121021 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039920929 0 0) 0x7f231009c5b0 con 0x7f231006ae30 2024-09-16T06:41:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): my global_id is 134140 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f00035d0 con 0x7f231006ae30 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419535189 0 0) 0x7f23100990c0 con 0x7f2310098000 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): my global_id is 104736 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231009c5b0 con 0x7f2310098000 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537136921 0 0) 0x7f23100947d0 con 0x7f231006a340 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): my global_id is 124430 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23100990c0 con 0x7f231006a340 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3528800144 0 0) 0x7f22f00035d0 con 0x7f231006ae30 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23100947d0 con 0x7f231006ae30 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23000015d0 con 0x7f231006ae30 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): dump: 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1184072259 0 0) 0x7f23100947d0 con 0x7f231006ae30 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 >> v1:172.21.15.76:6790/0 conn(0x7f231006a340 legacy=0x7f2310090910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 >> v1:172.21.15.76:6789/0 conn(0x7f2310098000 legacy=0x7f23101d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 monclient: found mon.b 2024-09-16T06:41:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.097+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310091730 con 0x7f231006ae30 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 5 monclient: authenticate success, global_id 134140 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _renew_subs 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310092470 con 0x7f231006ae30 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _renew_subs 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23100944f0 con 0x7f231006ae30 2024-09-16T06:41:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2300001190 con 0x7f231006ae30 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2300004e40 con 0x7f231006ae30 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 10 monclient: dump: 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230cd99700 4 set_mon_vals no callback set 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f4005100 con 0x7f231006ae30 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f230000f8b0 con 0x7f231006ae30 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 5235+0+0 (unknown 2528383054 0 0) 0x7f2300043890 con 0x7f231006ae30 2024-09-16T06:41:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.101+0000 7f230e59c700 10 monclient: get_auth_request con 0x7f22f0036090 auth_method 0 2024-09-16T06:41:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.113+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2300014780 con 0x7f231006ae30 2024-09-16T06:41:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.113+0000 7f22feffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.113+0000 7f22feffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:07.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f230f59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f230f59e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f230f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f22f4005930 con 0x7f231006ae30 2024-09-16T06:41:07.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (unknown 1073408522 0 0) 0x7f23000145a0 con 0x7f231006ae30 2024-09-16T06:41:07.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f22feffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.285+0000 7f22feffd700 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-16T06:41:07.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.389+0000 7f230f59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.389+0000 7f230f59e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.389+0000 7f230f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.389+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f22f4005440 con 0x7f231006ae30 2024-09-16T06:41:07.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.393+0000 7f22feffd700 1 -- 172.21.15.189:0/661121021 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v72) v1 ==== 126+0+0 (unknown 1073408522 0 0) 0x7f23000145a0 con 0x7f231006ae30 2024-09-16T06:41:07.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.393+0000 7f22feffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T06:41:07.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.393+0000 7f22feffd700 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 >> v1:172.21.15.189:6812/3098810674 conn(0x7f22f0036090 legacy=0x7f22f0038550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 10 monclient: shutdown 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 >> v1:172.21.15.189:6789/0 conn(0x7f231006ae30 legacy=0x7f2310091020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 shutdown_connections 2024-09-16T06:41:07.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 >> 172.21.15.189:0/661121021 conn(0x7f23100a20a0 msgr2=0x7f231009ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:07.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 shutdown_connections 2024-09-16T06:41:07.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.397+0000 7f230f59e700 1 -- 172.21.15.189:0/661121021 wait complete. 2024-09-16T06:41:07.402 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'bar' already exists 2024-09-16T06:41:07.415 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: get_monmap_and_config 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: monmap: 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:07.522178+0000 2024-09-16T06:41:07.521 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:07.522178+0000 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:07.522 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 1 Processor -- start 2024-09-16T06:41:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 1 -- start start 2024-09-16T06:41:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: init 2024-09-16T06:41:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:07.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-c con 0x7f53581352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5358135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-a con 0x7f53580a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535809b050 con 0x7f53580a2030 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535806ab20 con 0x7f53581352c0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535806a840 con 0x7f5358135db0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5357fff700 10 monclient(hunting): authenticate will time out at 1020.035708s 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.517+0000 7f5356ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53581352c0 0x7f53581356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45856/0 (socket says 172.21.15.189:45856) 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5356ffd700 1 -- 172.21.15.189:0/1141442030 learned_addr learned my addr 172.21.15.189:0/1141442030 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972412498 0 0) 0x7f535806ab20 con 0x7f53581352c0 2024-09-16T06:41:07.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): my global_id is 134143 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53440035d0 con 0x7f53581352c0 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244879485 0 0) 0x7f535806a840 con 0x7f5358135db0 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): my global_id is 124433 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535806ab20 con 0x7f5358135db0 2024-09-16T06:41:07.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316630258 0 0) 0x7f535809b050 con 0x7f53580a2030 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient(hunting): my global_id is 104739 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535806a840 con 0x7f53580a2030 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784371034 0 0) 0x7f53440035d0 con 0x7f53581352c0 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 >> v1:172.21.15.76:6790/0 conn(0x7f5358135db0 legacy=0x7f535813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 >> v1:172.21.15.76:6789/0 conn(0x7f53580a2030 legacy=0x7f535813df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:07.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535806c1b0 con 0x7f53581352c0 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5348001610 con 0x7f53581352c0 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 5 monclient: authenticate success, global_id 134143 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: dump: 2024-09-16T06:41:07.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53480018f0 con 0x7f53581352c0 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 1 -- 172.21.15.189:0/1141442030 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5348001c80 con 0x7f53581352c0 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f53567fc700 10 monclient: dump: 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 4 set_mon_vals no callback set 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 10 monclient: shutdown 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 1 -- 172.21.15.189:0/1141442030 >> v1:172.21.15.189:6789/0 conn(0x7f53581352c0 legacy=0x7f53581356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 1 -- 172.21.15.189:0/1141442030 shutdown_connections 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 1 -- 172.21.15.189:0/1141442030 >> 172.21.15.189:0/1141442030 conn(0x7f53580a4550 msgr2=0x7f53580a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 1 -- 172.21.15.189:0/1141442030 shutdown_connections 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 1 -- 172.21.15.189:0/1141442030 wait complete. 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.521+0000 7f5357fff700 10 monclient: monmap: 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:07.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 Processor -- start 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- start start 2024-09-16T06:41:07.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: init 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-a con 0x7f53580a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-b con 0x7f53581352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5358135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535809e820 con 0x7f53580a2030 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5358130800 con 0x7f53581352c0 2024-09-16T06:41:07.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580a6400 con 0x7f5358135db0 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient(hunting): authenticate will time out at 1020.042205s 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53581352c0 0x7f53581cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45860/0 (socket says 172.21.15.189:45860) 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534ffff700 1 -- 172.21.15.189:0/717027910 learned_addr learned my addr 172.21.15.189:0/717027910 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398539808 0 0) 0x7f5358130800 con 0x7f53581352c0 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): my global_id is 134146 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53380035d0 con 0x7f53581352c0 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744396355 0 0) 0x7f535809e820 con 0x7f53580a2030 2024-09-16T06:41:07.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): my global_id is 104742 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358130800 con 0x7f53580a2030 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337519957 0 0) 0x7f53580a6400 con 0x7f5358135db0 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): my global_id is 124436 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535809e820 con 0x7f5358135db0 2024-09-16T06:41:07.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3337968466 0 0) 0x7f53380035d0 con 0x7f53581352c0 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53580a6400 con 0x7f53581352c0 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53400018b0 con 0x7f53581352c0 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638334352 0 0) 0x7f53580a6400 con 0x7f53581352c0 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 >> v1:172.21.15.76:6790/0 conn(0x7f5358135db0 legacy=0x7f53581d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 >> v1:172.21.15.76:6789/0 conn(0x7f53580a2030 legacy=0x7f53581322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 monclient: found mon.b 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:07.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535812e9f0 con 0x7f53581352c0 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 5 monclient: authenticate success, global_id 134146 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _renew_subs 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5358130520 con 0x7f53581352c0 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _renew_subs 2024-09-16T06:41:07.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5358132a00 con 0x7f53581352c0 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5340001ca0 con 0x7f53581352c0 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5340003b10 con 0x7f53581352c0 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f5355ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.525+0000 7f534f7fe700 10 monclient: dump: 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:07.548 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5340003d30 con 0x7f53581352c0 2024-09-16T06:41:07.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f5357fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f5357fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f5357fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c005100 con 0x7f53581352c0 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 5235+0+0 (unknown 2528383054 0 0) 0x7f5340043790 con 0x7f53581352c0 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.529+0000 7f5356ffd700 10 monclient: get_auth_request con 0x7f53380362c0 auth_method 0 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.533+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f53400145c0 con 0x7f53581352c0 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.533+0000 7f534f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:07.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.533+0000 7f534f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.709+0000 7f5357fff700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:07.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.709+0000 7f5357fff700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:07.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.709+0000 7f5357fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:07.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.709+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f533c005930 con 0x7f53581352c0 2024-09-16T06:41:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.893+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 168+0+0 (unknown 485579508 0 0) 0x7f5340012c60 con 0x7f53581352c0 2024-09-16T06:41:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.893+0000 7f534f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:07.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:07.893+0000 7f534f7fe700 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:07.947 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:41:07.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.005+0000 7f5357fff700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.005+0000 7f5357fff700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.005+0000 7f5357fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.005+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f533c0010a0 con 0x7f53581352c0 2024-09-16T06:41:08.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.009+0000 7f534f7fe700 1 -- 172.21.15.189:0/717027910 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 4050003456 0 0) 0x7f53400143e0 con 0x7f53581352c0 2024-09-16T06:41:08.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.009+0000 7f534f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.009+0000 7f534f7fe700 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 >> v1:172.21.15.189:6812/3098810674 conn(0x7f53380362c0 legacy=0x7f5338038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 10 monclient: shutdown 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 >> v1:172.21.15.189:6789/0 conn(0x7f53581352c0 legacy=0x7f53581cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 shutdown_connections 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 >> 172.21.15.189:0/717027910 conn(0x7f53580a4550 msgr2=0x7f53580a24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 shutdown_connections 2024-09-16T06:41:08.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.013+0000 7f5357fff700 1 -- 172.21.15.189:0/717027910 wait complete. 2024-09-16T06:41:08.017 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:08.027 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T06:41:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: monmap: 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:08.066444+0000 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:08.066444+0000 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.066 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 1 Processor -- start 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 1 -- start start 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: init 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-b con 0x7fd374097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-a con 0x7fd374099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-c con 0x7fd374139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374098990 con 0x7fd374099090 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd374097f40 con 0x7fd374139500 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37409d610 con 0x7fd374097b60 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd379b0c700 10 monclient(hunting): authenticate will time out at 1015.199795s 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.060+0000 7fd378b0a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd374097b60 0x7fd37409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50162/0 (socket says 172.21.15.76:50162) 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd373fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd374099090 0x7fd37409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53288/0 (socket says 172.21.15.76:53288) 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd378b0a700 1 -- 172.21.15.76:0/1218385725 learned_addr learned my addr 172.21.15.76:0/1218385725 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591898202 0 0) 0x7fd374098990 con 0x7fd374099090 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): my global_id is 104745 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3640035d0 con 0x7fd374099090 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086792532 0 0) 0x7fd37409d610 con 0x7fd374097b60 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): my global_id is 124439 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd374098990 con 0x7fd374097b60 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859153381 0 0) 0x7fd374097f40 con 0x7fd374139500 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient(hunting): my global_id is 134149 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37409d610 con 0x7fd374139500 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093693678 0 0) 0x7fd3640035d0 con 0x7fd374099090 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 >> v1:172.21.15.76:6790/0 conn(0x7fd374097b60 legacy=0x7fd37409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 >> v1:172.21.15.189:6789/0 conn(0x7fd374139500 legacy=0x7fd37413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 monclient: found mon.noname-a 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740986b0 con 0x7fd374099090 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 5 monclient: authenticate success, global_id 104745 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd368001610 con 0x7fd374099090 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: dump: 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3680018f0 con 0x7fd374099090 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 1 -- 172.21.15.76:0/1218385725 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd368001c80 con 0x7fd374099090 2024-09-16T06:41:08.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 4 set_mon_vals no callback set 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd3737fe700 10 monclient: dump: 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.075 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 10 monclient: shutdown 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- 172.21.15.76:0/1218385725 >> v1:172.21.15.76:6789/0 conn(0x7fd374099090 legacy=0x7fd37409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- 172.21.15.76:0/1218385725 shutdown_connections 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- 172.21.15.76:0/1218385725 >> 172.21.15.76:0/1218385725 conn(0x7fd3740a0030 msgr2=0x7fd3740a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- 172.21.15.76:0/1218385725 shutdown_connections 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- 172.21.15.76:0/1218385725 wait complete. 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 10 monclient: monmap: 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 Processor -- start 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 1 -- start start 2024-09-16T06:41:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.064+0000 7fd379b0c700 10 monclient: init 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-b con 0x7fd374097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-a con 0x7fd374099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient(hunting): picked mon.noname-c con 0x7fd374139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740a2c80 con 0x7fd374099090 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740712d0 con 0x7fd374097b60 2024-09-16T06:41:08.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37409a540 con 0x7fd374139500 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient(hunting): authenticate will time out at 1015.204633s 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd373fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd374099090 0x7fd374137760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53302/0 (socket says 172.21.15.76:53302) 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd373fff700 1 -- 172.21.15.76:0/755767348 learned_addr learned my addr 172.21.15.76:0/755767348 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711866327 0 0) 0x7fd3740986b0 con 0x7fd374099090 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): my global_id is 104748 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3580035d0 con 0x7fd374099090 2024-09-16T06:41:08.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260039833 0 0) 0x7fd37409a540 con 0x7fd374139500 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): my global_id is 124442 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3740986b0 con 0x7fd374139500 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439202874 0 0) 0x7fd3740712d0 con 0x7fd374097b60 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): my global_id is 134152 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37409a540 con 0x7fd374097b60 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3565949641 0 0) 0x7fd3580035d0 con 0x7fd374099090 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3740712d0 con 0x7fd374099090 2024-09-16T06:41:08.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd3680014b0 con 0x7fd374099090 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): dump: 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494087347 0 0) 0x7fd37409a540 con 0x7fd374097b60 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3580035d0 con 0x7fd374097b60 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd3600018b0 con 0x7fd374097b60 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): dump: 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.085 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692545110 0 0) 0x7fd3740986b0 con 0x7fd374139500 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd37409a540 con 0x7fd374139500 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd36c001bb0 con 0x7fd374139500 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient(hunting): dump: 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1229356482 0 0) 0x7fd3740712d0 con 0x7fd374099090 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 >> v1:172.21.15.76:6790/0 conn(0x7fd374139500 legacy=0x7fd3741cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 >> v1:172.21.15.189:6789/0 conn(0x7fd374097b60 legacy=0x7fd374137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 monclient: found mon.a 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37406ccd0 con 0x7fd374099090 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 5 monclient: authenticate success, global_id 104748 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _renew_subs 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37406cfb0 con 0x7fd374099090 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _renew_subs 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37406be90 con 0x7fd374099090 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3680011b0 con 0x7fd374099090 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd368004ce0 con 0x7fd374099090 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd372ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 10 monclient: dump: 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd354005100 con 0x7fd374099090 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.068+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd36800f750 con 0x7fd374099090 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.072+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7fd3680438f0 con 0x7fd374099090 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.072+0000 7fd378b0a700 10 monclient: get_auth_request con 0x7fd358036510 auth_method 0 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.076+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd368010ca0 con 0x7fd374099090 2024-09-16T06:41:08.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.076+0000 7fd3717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.076+0000 7fd3717fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: monmap: 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:08.137891+0000 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:08.137891+0000 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.137 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 1 Processor -- start 2024-09-16T06:41:08.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 1 -- start start 2024-09-16T06:41:08.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: init 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-b con 0x7fdc180a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-c con 0x7fdc1809aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-a con 0x7fdc180a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1809a040 con 0x7fdc180a3ca0 2024-09-16T06:41:08.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180a33d0 con 0x7fdc1809aa20 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc18099d60 con 0x7fdc180a2350 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc1d37e700 10 monclient(hunting): authenticate will time out at 1020.651660s 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc0ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc1809aa20 0x7fdc1813a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45864/0 (socket says 172.21.15.189:45864) 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.133+0000 7fdc0ffff700 1 -- 172.21.15.189:0/3732762752 learned_addr learned my addr 172.21.15.189:0/3732762752 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452467583 0 0) 0x7fdc180a33d0 con 0x7fdc1809aa20 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): my global_id is 134155 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf80035d0 con 0x7fdc1809aa20 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047009540 0 0) 0x7fdc1809a040 con 0x7fdc180a3ca0 2024-09-16T06:41:08.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): my global_id is 104751 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc180a33d0 con 0x7fdc180a3ca0 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216636728 0 0) 0x7fdc18099d60 con 0x7fdc180a2350 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient(hunting): my global_id is 124445 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1809a040 con 0x7fdc180a2350 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3542969469 0 0) 0x7fdbf80035d0 con 0x7fdc1809aa20 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 >> v1:172.21.15.76:6790/0 conn(0x7fdc180a2350 legacy=0x7fdc180a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 >> v1:172.21.15.76:6789/0 conn(0x7fdc180a3ca0 legacy=0x7fdc1813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180a3920 con 0x7fdc1809aa20 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 5 monclient: authenticate success, global_id 134155 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc00001610 con 0x7fdc1809aa20 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: dump: 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc000018f0 con 0x7fdc1809aa20 2024-09-16T06:41:08.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 4 set_mon_vals no callback set 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 1 -- 172.21.15.189:0/3732762752 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc00001c80 con 0x7fdc1809aa20 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc167fc700 10 monclient: dump: 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.147 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: shutdown 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3732762752 >> v1:172.21.15.189:6789/0 conn(0x7fdc1809aa20 legacy=0x7fdc1813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3732762752 shutdown_connections 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3732762752 >> 172.21.15.189:0/3732762752 conn(0x7fdc180a7d70 msgr2=0x7fdc180a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3732762752 shutdown_connections 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3732762752 wait complete. 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: monmap: 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:08.148 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 Processor -- start 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- start start 2024-09-16T06:41:08.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: init 2024-09-16T06:41:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-c con 0x7fdc1809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-b con 0x7fdc180a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient(hunting): picked mon.noname-a con 0x7fdc180a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1809b770 con 0x7fdc180a3ca0 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc181361b0 con 0x7fdc180a2350 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1806bed0 con 0x7fdc1809aa20 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.137+0000 7fdc1d37e700 10 monclient(hunting): authenticate will time out at 1020.655899s 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc180a2350 0x7fdc181cef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45878/0 (socket says 172.21.15.189:45878) 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0ffff700 1 -- 172.21.15.189:0/3949962921 learned_addr learned my addr 172.21.15.189:0/3949962921 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578048579 0 0) 0x7fdc181361b0 con 0x7fdc180a2350 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): my global_id is 134158 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbfc0035d0 con 0x7fdc180a2350 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15603148 0 0) 0x7fdc1806bed0 con 0x7fdc1809aa20 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): my global_id is 124448 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc181361b0 con 0x7fdc1809aa20 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316582068 0 0) 0x7fdc1809b770 con 0x7fdc180a3ca0 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): my global_id is 104754 2024-09-16T06:41:08.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc1806bed0 con 0x7fdc180a3ca0 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3528014059 0 0) 0x7fdbfc0035d0 con 0x7fdc180a2350 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc1809b770 con 0x7fdc180a2350 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc000015d0 con 0x7fdc180a2350 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.157 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36550991 0 0) 0x7fdc181361b0 con 0x7fdc1809aa20 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbfc0035d0 con 0x7fdc1809aa20 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc080018b0 con 0x7fdc1809aa20 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.158 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084037563 0 0) 0x7fdc1806bed0 con 0x7fdc180a3ca0 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc181361b0 con 0x7fdc180a3ca0 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3303222196 0 0) 0x7fdc1809b770 con 0x7fdc180a2350 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 >> v1:172.21.15.76:6790/0 conn(0x7fdc1809aa20 legacy=0x7fdc18135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 >> v1:172.21.15.76:6789/0 conn(0x7fdc180a3ca0 legacy=0x7fdc181d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 monclient: found mon.b 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18122e40 con 0x7fdc180a2350 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 5 monclient: authenticate success, global_id 134158 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _renew_subs 2024-09-16T06:41:08.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc18123120 con 0x7fdc180a2350 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _renew_subs 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc18135e70 con 0x7fdc180a2350 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc00004d90 con 0x7fdc180a2350 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc000050a0 con 0x7fdc180a2350 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc15ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 10 monclient: dump: 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.161 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdc00005200 con 0x7fdc180a2350 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0005100 con 0x7fdc180a2350 2024-09-16T06:41:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7fdc00043b40 con 0x7fdc180a2350 2024-09-16T06:41:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.141+0000 7fdc16ffd700 10 monclient: get_auth_request con 0x7fdbfc0364a0 auth_method 0 2024-09-16T06:41:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.149+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdc00014b00 con 0x7fdc180a2350 2024-09-16T06:41:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.149+0000 7fdc0f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.149+0000 7fdc0f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd379b0c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd379b0c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd379b0c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd354005930 con 0x7fd374099090 2024-09-16T06:41:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd3717fa700 1 -- 172.21.15.76:0/755767348 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1376619267) 0x7fd3680144a0 con 0x7fd374099090 2024-09-16T06:41:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd3717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:08.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.312+0000 7fd3717fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:08.317 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:08.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd358036510 legacy=0x7fd3580389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 10 monclient: shutdown 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 >> v1:172.21.15.76:6789/0 conn(0x7fd374099090 legacy=0x7fd374137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 shutdown_connections 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 >> 172.21.15.76:0/755767348 conn(0x7fd3740a0030 msgr2=0x7fd374132690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 shutdown_connections 2024-09-16T06:41:08.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.316+0000 7fd379b0c700 1 -- 172.21.15.76:0/755767348 wait complete. 2024-09-16T06:41:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc1d37e700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc1d37e700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc1d37e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fdbf0005930 con 0x7fdc180a2350 2024-09-16T06:41:08.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 4050003456 0 0) 0x7fdc00012f70 con 0x7fdc180a2350 2024-09-16T06:41:08.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc0f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.325+0000 7fdc0f7fe700 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:08.331 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:41:08.331 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:41:08.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:41:08.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:41:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.437+0000 7fdc1d37e700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.437+0000 7fdc1d37e700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.437+0000 7fdc1d37e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.437+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fdbf00053b0 con 0x7fdc180a2350 2024-09-16T06:41:08.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc0f7fe700 1 -- 172.21.15.189:0/3949962921 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v73) v1 ==== 177+0+0 (unknown 4050003456 0 0) 0x7fdc000148d0 con 0x7fdc180a2350 2024-09-16T06:41:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc0f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T06:41:08.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc0f7fe700 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:08.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 10 monclient: monmap: 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:08.446664+0000 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:08.446664+0000 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 1 Processor -- start 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 1 -- start start 2024-09-16T06:41:08.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.440+0000 7f7bb7212700 10 monclient: init 2024-09-16T06:41:08.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdbfc0364a0 legacy=0x7fdbfc038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 10 monclient: shutdown 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 >> v1:172.21.15.189:6789/0 conn(0x7fdc180a2350 legacy=0x7fdc181cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 shutdown_connections 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 >> 172.21.15.189:0/3949962921 conn(0x7fdc180a7d70 msgr2=0x7fdc181bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 shutdown_connections 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.441+0000 7fdc1d37e700 1 -- 172.21.15.189:0/3949962921 wait complete. 2024-09-16T06:41:08.449 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-a con 0x7f7bb006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb006c280 con 0x7f7bb006cf50 2024-09-16T06:41:08.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb006bfa0 con 0x7f7bb0132330 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb009ace0 con 0x7f7bb006d480 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): authenticate will time out at 1015.580195s 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb5a0f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7bb006d480 0x7f7bb013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50178/0 (socket says 172.21.15.76:50178) 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb6a11700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7bb006cf50 0x7f7bb013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53318/0 (socket says 172.21.15.76:53318) 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb5a0f700 1 -- 172.21.15.76:0/1580784089 learned_addr learned my addr 172.21.15.76:0/1580784089 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb6210700 1 --1- 172.21.15.76:0/1580784089 >> v1:172.21.15.189:6789/0 conn(0x7f7bb0132330 0x7f7bb0132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:59564/0 (socket says 172.21.15.76:59564) 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257479513 0 0) 0x7f7bb006c280 con 0x7f7bb006cf50 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): my global_id is 104757 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c0035d0 con 0x7f7bb006cf50 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358142095 0 0) 0x7f7bb009ace0 con 0x7f7bb006d480 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): my global_id is 124451 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb006c280 con 0x7f7bb006d480 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512642943 0 0) 0x7f7bb006bfa0 con 0x7f7bb0132330 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient(hunting): my global_id is 134161 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb009ace0 con 0x7f7bb0132330 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268682887 0 0) 0x7f7b9c0035d0 con 0x7f7bb006cf50 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 >> v1:172.21.15.76:6790/0 conn(0x7f7bb006d480 legacy=0x7f7bb013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 >> v1:172.21.15.189:6789/0 conn(0x7f7bb0132330 legacy=0x7f7bb0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 monclient: found mon.noname-a 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb009ae70 con 0x7f7bb006cf50 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 5 monclient: authenticate success, global_id 104757 2024-09-16T06:41:08.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ba8001990 con 0x7f7bb006cf50 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: dump: 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba8001c70 con 0x7f7bb006cf50 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 4 set_mon_vals no callback set 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 1 -- 172.21.15.76:0/1580784089 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ba8001e60 con 0x7f7bb006cf50 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb520e700 10 monclient: dump: 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.461 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: shutdown 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- 172.21.15.76:0/1580784089 >> v1:172.21.15.76:6789/0 conn(0x7f7bb006cf50 legacy=0x7f7bb013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- 172.21.15.76:0/1580784089 shutdown_connections 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- 172.21.15.76:0/1580784089 >> 172.21.15.76:0/1580784089 conn(0x7f7bb00a41c0 msgr2=0x7f7bb00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- 172.21.15.76:0/1580784089 shutdown_connections 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- 172.21.15.76:0/1580784089 wait complete. 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: monmap: 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 Processor -- start 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- start start 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: init 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): picked mon.noname-a con 0x7f7bb0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb006de30 con 0x7f7bb0132330 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb006ea70 con 0x7f7bb006d480 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00a11a0 con 0x7f7bb006cf50 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb7212700 10 monclient(hunting): authenticate will time out at 1015.583861s 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb6a11700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7bb0132330 0x7f7bb01d6490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53328/0 (socket says 172.21.15.76:53328) 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.444+0000 7f7bb6a11700 1 -- 172.21.15.76:0/670598970 learned_addr learned my addr 172.21.15.76:0/670598970 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978438943 0 0) 0x7f7bb006de30 con 0x7f7bb0132330 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): my global_id is 104760 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b940035d0 con 0x7f7bb0132330 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459829539 0 0) 0x7f7bb00a11a0 con 0x7f7bb006cf50 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): my global_id is 124454 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb006de30 con 0x7f7bb006cf50 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201278119 0 0) 0x7f7bb006ea70 con 0x7f7bb006d480 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): my global_id is 134164 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb00a11a0 con 0x7f7bb006d480 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529725184 0 0) 0x7f7bb006de30 con 0x7f7bb006cf50 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb006ea70 con 0x7f7bb006cf50 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7bac001870 con 0x7f7bb006cf50 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.466 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184056813 0 0) 0x7f7b940035d0 con 0x7f7bb0132330 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb006de30 con 0x7f7bb0132330 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ba80013a0 con 0x7f7bb0132330 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.467 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 269535891 0 0) 0x7f7bb006ea70 con 0x7f7bb006cf50 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.189:6789/0 conn(0x7f7bb006d480 legacy=0x7f7bb00a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6789/0 conn(0x7f7bb0132330 legacy=0x7f7bb01d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 monclient: found mon.c 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0070200 con 0x7f7bb006cf50 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 5 monclient: authenticate success, global_id 124454 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 10 monclient: _renew_subs 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb006fc40 con 0x7f7bb006cf50 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 10 monclient: _renew_subs 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb006e780 con 0x7f7bb006cf50 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bac004890 con 0x7f7bb006cf50 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7bac004c20 con 0x7f7bb006cf50 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb4a0d700 4 set_mon_vals no callback set 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: dump: 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.470 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b98006590 con 0x7f7b98002830 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7bac004e00 con 0x7f7bb006cf50 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f7bac043820 con 0x7f7bb006cf50 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836169426 0 0) 0x7f7b98006590 con 0x7f7b98002830 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient(hunting): my global_id is 104763 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb006ea70 con 0x7f7b98002830 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7bb6a11700 10 monclient: get_auth_request con 0x7f7b94036470 auth_method 0 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1468638285 0 0) 0x7f7bb006ea70 con 0x7f7b98002830 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b98006590 con 0x7f7b98002830 2024-09-16T06:41:08.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ba4001740 con 0x7f7b98002830 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: dump: 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 669935825 0 0) 0x7f7b98006590 con 0x7f7b98002830 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b9800a190 con 0x7f7b98002830 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7ba4001d00 con 0x7f7b98002830 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6789/0 conn(0x7f7b98002830 legacy=0x7f7b98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.448+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6789/0 conn(0x7f7b98002830 legacy=0x7f7b98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7b98009e30 con 0x7f7b98009150 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656759915 0 0) 0x7f7b98009e30 con 0x7f7b98009150 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient(hunting): my global_id is 104766 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98006590 con 0x7f7b98009150 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 258568040 0 0) 0x7f7b98006590 con 0x7f7b98009150 2024-09-16T06:41:08.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b98009e30 con 0x7f7b98009150 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ba4004c20 con 0x7f7b98009150 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient: dump: 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.476 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4028989252 0 0) 0x7f7b98009e30 con 0x7f7b98009150 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b9800c340 con 0x7f7b98009150 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 4266460851) 0x7f7ba4004e90 con 0x7f7b98009150 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6789/0 conn(0x7f7b98009150 legacy=0x7f7b980095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7ba2ffd700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6789/0 conn(0x7f7b98009150 legacy=0x7f7b980095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:08.477 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.478 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:08.479 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.480 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:08.481 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.482 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.483 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.484 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.486 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.487 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.488 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.489 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:08.489 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7b94036470 legacy=0x7f7b94038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 10 monclient: shutdown 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 >> v1:172.21.15.76:6790/0 conn(0x7f7bb006cf50 legacy=0x7f7bb00a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb6a11700 1 -- 172.21.15.76:0/670598970 reap_dead start 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 shutdown_connections 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 >> 172.21.15.76:0/670598970 conn(0x7f7bb00a41c0 msgr2=0x7f7bb013e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 shutdown_connections 2024-09-16T06:41:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.460+0000 7f7bb7212700 1 -- 172.21.15.76:0/670598970 wait complete. 2024-09-16T06:41:08.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:08.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:41:08.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:41:08.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 10 monclient: monmap: 2024-09-16T06:41:08.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:08.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:08.563296+0000 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:08.563296+0000 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 1 Processor -- start 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 1 -- start start 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.557+0000 7f2380cfe700 10 monclient: init 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:08.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-c con 0x7f237c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-b con 0x7f237c09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-a con 0x7f237c138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c09cca0 con 0x7f237c138da0 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c09f750 con 0x7f237c09bb00 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c097620 con 0x7f237c09c5f0 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient(hunting): authenticate will time out at 1021.076344s 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f237c09bb00 0x7f237c09bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52550/0 (socket says 172.21.15.189:52550) 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237a59c700 1 -- 172.21.15.189:0/2544044071 learned_addr learned my addr 172.21.15.189:0/2544044071 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452784348 0 0) 0x7f237c09f750 con 0x7f237c09bb00 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): my global_id is 134167 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23640035d0 con 0x7f237c09bb00 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449551179 0 0) 0x7f237c09cca0 con 0x7f237c138da0 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): my global_id is 104769 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c09f750 con 0x7f237c138da0 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209528576 0 0) 0x7f237c097620 con 0x7f237c09c5f0 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient(hunting): my global_id is 124457 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c09cca0 con 0x7f237c09c5f0 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332671776 0 0) 0x7f23640035d0 con 0x7f237c09bb00 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 >> v1:172.21.15.76:6790/0 conn(0x7f237c09c5f0 legacy=0x7f237c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 >> v1:172.21.15.76:6789/0 conn(0x7f237c138da0 legacy=0x7f237c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 monclient: found mon.noname-c 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c097900 con 0x7f237c09bb00 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 5 monclient: authenticate success, global_id 134167 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2370001610 con 0x7f237c09bb00 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: dump: 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23700018f0 con 0x7f237c09bb00 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 4 set_mon_vals no callback set 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 1 -- 172.21.15.189:0/2544044071 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2370001c80 con 0x7f237c09bb00 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f237959a700 10 monclient: dump: 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: shutdown 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- 172.21.15.189:0/2544044071 >> v1:172.21.15.189:6789/0 conn(0x7f237c09bb00 legacy=0x7f237c09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- 172.21.15.189:0/2544044071 shutdown_connections 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- 172.21.15.189:0/2544044071 >> 172.21.15.189:0/2544044071 conn(0x7f237c0a03e0 msgr2=0x7f237c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- 172.21.15.189:0/2544044071 shutdown_connections 2024-09-16T06:41:08.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 -- 172.21.15.189:0/2544044071 wait complete. 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 10 monclient: monmap: 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:08.576 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.561+0000 7f2380cfe700 1 Processor -- start 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- start start 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: init 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-b con 0x7f237c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-a con 0x7f237c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient(hunting): picked mon.noname-c con 0x7f237c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c091d30 con 0x7f237c09c5f0 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c1cad70 con 0x7f237c09bb00 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c092010 con 0x7f237c138da0 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient(hunting): authenticate will time out at 1021.080781s 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f237a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f237c09bb00 0x7f237c092430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52566/0 (socket says 172.21.15.189:52566) 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f237a59c700 1 -- 172.21.15.189:0/3081871705 learned_addr learned my addr 172.21.15.189:0/3081871705 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095543948 0 0) 0x7f237c1cad70 con 0x7f237c09bb00 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): my global_id is 134170 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f235c0035d0 con 0x7f237c09bb00 2024-09-16T06:41:08.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017561915 0 0) 0x7f237c092010 con 0x7f237c138da0 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): my global_id is 124460 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c1cad70 con 0x7f237c138da0 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152276097 0 0) 0x7f237c091d30 con 0x7f237c09c5f0 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): my global_id is 104772 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c092010 con 0x7f237c09c5f0 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2449490789 0 0) 0x7f235c0035d0 con 0x7f237c09bb00 2024-09-16T06:41:08.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237c091d30 con 0x7f237c09bb00 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23700015d0 con 0x7f237c09bb00 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): dump: 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209130636 0 0) 0x7f237c1cad70 con 0x7f237c138da0 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f235c0035d0 con 0x7f237c138da0 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2374001c30 con 0x7f237c138da0 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): dump: 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927539266 0 0) 0x7f237c092010 con 0x7f237c09c5f0 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237c1cad70 con 0x7f237c09c5f0 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f236c0017e0 con 0x7f237c09c5f0 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient(hunting): dump: 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508921493 0 0) 0x7f237c091d30 con 0x7f237c09bb00 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 >> v1:172.21.15.76:6790/0 conn(0x7f237c138da0 legacy=0x7f237c1ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 >> v1:172.21.15.76:6789/0 conn(0x7f237c09c5f0 legacy=0x7f237c1c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 monclient: found mon.b 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c09ac60 con 0x7f237c09bb00 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 5 monclient: authenticate success, global_id 134170 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _renew_subs 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c08c3b0 con 0x7f237c09bb00 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _renew_subs 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c1d3690 con 0x7f237c09bb00 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2370004ca0 con 0x7f237c09bb00 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236bfff700 4 set_mon_vals no callback set 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2370004fb0 con 0x7f237c09bb00 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f236affd700 10 monclient: dump: 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.585 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.565+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2360005100 con 0x7f237c09bb00 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.569+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2370005190 con 0x7f237c09bb00 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.569+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f2370043a40 con 0x7f237c09bb00 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.569+0000 7f2379d9b700 10 monclient: get_auth_request con 0x7f235c036560 auth_method 0 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.577+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2370014b60 con 0x7f237c09bb00 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.577+0000 7f236affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.577+0000 7f236affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.604+0000 7f91a3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.604+0000 7f91a3fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.604+0000 7f91a3fff700 10 monclient: monmap: 2024-09-16T06:41:08.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:08.611114+0000 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:08.611114+0000 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 1 Processor -- start 2024-09-16T06:41:08.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 1 -- start start 2024-09-16T06:41:08.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient: init 2024-09-16T06:41:08.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f91a40a4840 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f91a4071d80 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f91a4138610 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a40a11a0 con 0x7f91a40a4840 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a40a4c20 con 0x7f91a4071d80 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a40a06e0 con 0x7f91a4138610 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f91a40a4840 0x7f91a409b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53354/0 (socket says 172.21.15.76:53354) 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient(hunting): authenticate will time out at 1015.746048s 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a2ffd700 1 -- 172.21.15.76:0/1182056905 learned_addr learned my addr 172.21.15.76:0/1182056905 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561449059 0 0) 0x7f91a40a11a0 con 0x7f91a40a4840 2024-09-16T06:41:08.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): my global_id is 104775 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91900035d0 con 0x7f91a40a4840 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779577484 0 0) 0x7f91a40a06e0 con 0x7f91a4138610 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): my global_id is 124463 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a40a11a0 con 0x7f91a4138610 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9240436 0 0) 0x7f91a40a4c20 con 0x7f91a4071d80 2024-09-16T06:41:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient(hunting): my global_id is 134173 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a40a06e0 con 0x7f91a4071d80 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379902524 0 0) 0x7f91900035d0 con 0x7f91a40a4840 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 >> v1:172.21.15.76:6790/0 conn(0x7f91a4138610 legacy=0x7f91a413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 >> v1:172.21.15.189:6789/0 conn(0x7f91a4071d80 legacy=0x7f91a409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 monclient: found mon.noname-a 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a409c120 con 0x7f91a40a4840 2024-09-16T06:41:08.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 5 monclient: authenticate success, global_id 104775 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9194001610 con 0x7f91a40a4840 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: dump: 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91940018f0 con 0x7f91a40a4840 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 1 -- 172.21.15.76:0/1182056905 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9194001cc0 con 0x7f91a40a4840 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a3fff700 4 set_mon_vals no callback set 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.608+0000 7f91a27fc700 10 monclient: dump: 2024-09-16T06:41:08.619 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: shutdown 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- 172.21.15.76:0/1182056905 >> v1:172.21.15.76:6789/0 conn(0x7f91a40a4840 legacy=0x7f91a409b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- 172.21.15.76:0/1182056905 shutdown_connections 2024-09-16T06:41:08.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- 172.21.15.76:0/1182056905 >> 172.21.15.76:0/1182056905 conn(0x7f91a40a4430 msgr2=0x7f91a40a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- 172.21.15.76:0/1182056905 shutdown_connections 2024-09-16T06:41:08.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- 172.21.15.76:0/1182056905 wait complete. 2024-09-16T06:41:08.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: monmap: 2024-09-16T06:41:08.626 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.626 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 Processor -- start 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- start start 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: init 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:08.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f91a4071d80 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f91a40a4840 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f91a4138610 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a40a3270 con 0x7f91a40a4840 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a4122dc0 con 0x7f91a4138610 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a4071440 con 0x7f91a4071d80 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f91a3fff700 10 monclient(hunting): authenticate will time out at 1015.751056s 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f91a40a4840 0x7f91a41cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53366/0 (socket says 172.21.15.76:53366) 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919bfff700 1 -- 172.21.15.76:0/3034730662 learned_addr learned my addr 172.21.15.76:0/3034730662 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781358791 0 0) 0x7f91a40a3270 con 0x7f91a40a4840 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): my global_id is 104778 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91840035d0 con 0x7f91a40a4840 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926763452 0 0) 0x7f91a4071440 con 0x7f91a4071d80 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): my global_id is 124466 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a40a3270 con 0x7f91a4071d80 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093916424 0 0) 0x7f91a4122dc0 con 0x7f91a4138610 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 10 monclient(hunting): my global_id is 134176 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4071440 con 0x7f91a4138610 2024-09-16T06:41:08.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126589996 0 0) 0x7f91840035d0 con 0x7f91a40a4840 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.612+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a4122dc0 con 0x7f91a40a4840 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f918c001890 con 0x7f91a40a4840 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.630 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2291328431 0 0) 0x7f91a40a3270 con 0x7f91a4071d80 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91840035d0 con 0x7f91a4071d80 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9194001c00 con 0x7f91a4071d80 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.631 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513428674 0 0) 0x7f91a4071440 con 0x7f91a4138610 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a40a3270 con 0x7f91a4138610 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f919c001bb0 con 0x7f91a4138610 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769600935 0 0) 0x7f91a4122dc0 con 0x7f91a40a4840 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6790/0 conn(0x7f91a4071d80 legacy=0x7f91a4136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.189:6789/0 conn(0x7f91a4138610 legacy=0x7f91a41cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 monclient: found mon.a 2024-09-16T06:41:08.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a409b0f0 con 0x7f91a40a4840 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 5 monclient: authenticate success, global_id 104778 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 10 monclient: _renew_subs 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a41213b0 con 0x7f91a40a4840 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 10 monclient: _renew_subs 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a4122230 con 0x7f91a40a4840 2024-09-16T06:41:08.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f918c0036c0 con 0x7f91a40a4840 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f918c003a50 con 0x7f91a40a4840 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a1ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: dump: 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.635 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9188006590 con 0x7f9188002830 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746644389 0 0) 0x7f9188006590 con 0x7f9188002830 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient(hunting): my global_id is 124469 2024-09-16T06:41:08.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a4122dc0 con 0x7f9188002830 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f918c003c30 con 0x7f91a40a4840 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f918c043910 con 0x7f91a40a4840 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 193329170 0 0) 0x7f91a4122dc0 con 0x7f9188002830 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9188006590 con 0x7f9188002830 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9194001d60 con 0x7f9188002830 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 10 monclient: dump: 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f91a37fe700 10 monclient: get_auth_request con 0x7f9184036510 auth_method 0 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 500514682 0 0) 0x7f9188006590 con 0x7f9188002830 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.616+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f918800a190 con 0x7f9188002830 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.620+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9194004ef0 con 0x7f9188002830 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.620+0000 7f919b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.620+0000 7f919b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.620+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6790/0 conn(0x7f9188002830 legacy=0x7f9188004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.620+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6790/0 conn(0x7f9188002830 legacy=0x7f9188004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f91a3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9188009e50 con 0x7f9188009170 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670971556 0 0) 0x7f9188009e50 con 0x7f9188009170 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient(hunting): my global_id is 124472 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9188006590 con 0x7f9188009170 2024-09-16T06:41:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 464756574 0 0) 0x7f9188006590 con 0x7f9188009170 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9188009e50 con 0x7f9188009170 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91940054f0 con 0x7f9188009170 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient: dump: 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2227528921 0 0) 0x7f9188009e50 con 0x7f9188009170 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f918800c340 con 0x7f9188009170 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2085097733) 0x7f9194006990 con 0x7f9188009170 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6790/0 conn(0x7f9188009170 legacy=0x7f9188009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.628+0000 7f919b7fe700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6790/0 conn(0x7f9188009170 legacy=0x7f9188009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:08.642 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.643 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.644 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:08.645 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.646 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:08.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.648 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:08.649 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:08.650 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.652 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:08.653 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9184036510 legacy=0x7f91840389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 10 monclient: shutdown 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 >> v1:172.21.15.76:6789/0 conn(0x7f91a40a4840 legacy=0x7f91a41cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a37fe700 1 -- 172.21.15.76:0/3034730662 reap_dead start 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a37fe700 1 -- 172.21.15.76:0/3034730662 reap_dead start 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 shutdown_connections 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 >> 172.21.15.76:0/3034730662 conn(0x7f91a40a4430 msgr2=0x7f91a40a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 shutdown_connections 2024-09-16T06:41:08.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.632+0000 7f91a3fff700 1 -- 172.21.15.76:0/3034730662 wait complete. 2024-09-16T06:41:08.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:08.656 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:41:08.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:41:08.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.753+0000 7f2380cfe700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:08.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.753+0000 7f2380cfe700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:08.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.753+0000 7f2380cfe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:08.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:08.753+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f2360005930 con 0x7f237c09bb00 2024-09-16T06:41:08.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.772+0000 7fe81560d700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.772+0000 7fe81560d700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.772+0000 7fe81560d700 10 monclient: monmap: 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:08.779300+0000 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:08.779300+0000 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.778 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.779 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.772+0000 7fe81560d700 1 Processor -- start 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- start start 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient: init 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-b con 0x7fe810097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-a con 0x7fe810099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-c con 0x7fe810139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe810098990 con 0x7fe810099090 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe810097f40 con 0x7fe810139500 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81009d610 con 0x7fe810097b60 2024-09-16T06:41:08.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient(hunting): authenticate will time out at 1015.912534s 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe807fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe810099090 0x7fe81009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53370/0 (socket says 172.21.15.76:53370) 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe807fff700 1 -- 172.21.15.76:0/166473849 learned_addr learned my addr 172.21.15.76:0/166473849 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166041998 0 0) 0x7fe810098990 con 0x7fe810099090 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): my global_id is 104781 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc0035d0 con 0x7fe810099090 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443465528 0 0) 0x7fe81009d610 con 0x7fe810097b60 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient(hunting): my global_id is 124475 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe810098990 con 0x7fe810097b60 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 895575554 0 0) 0x7fe7fc0035d0 con 0x7fe810099090 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 >> v1:172.21.15.76:6790/0 conn(0x7fe810097b60 legacy=0x7fe81009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 >> v1:172.21.15.189:6789/0 conn(0x7fe810139500 legacy=0x7fe81013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8100986b0 con 0x7fe810099090 2024-09-16T06:41:08.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe800001610 con 0x7fe810099090 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: dump: 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 5 monclient: authenticate success, global_id 104781 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8000018f0 con 0x7fe810099090 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 1 -- 172.21.15.76:0/166473849 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe800001c80 con 0x7fe810099090 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe80f7fe700 10 monclient: dump: 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 4 set_mon_vals no callback set 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 10 monclient: shutdown 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- 172.21.15.76:0/166473849 >> v1:172.21.15.76:6789/0 conn(0x7fe810099090 legacy=0x7fe81009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- 172.21.15.76:0/166473849 shutdown_connections 2024-09-16T06:41:08.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- 172.21.15.76:0/166473849 >> 172.21.15.76:0/166473849 conn(0x7fe8100a0030 msgr2=0x7fe8100a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- 172.21.15.76:0/166473849 shutdown_connections 2024-09-16T06:41:08.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.776+0000 7fe81560d700 1 -- 172.21.15.76:0/166473849 wait complete. 2024-09-16T06:41:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: monmap: 2024-09-16T06:41:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 Processor -- start 2024-09-16T06:41:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- start start 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: init 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-c con 0x7fe810097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-a con 0x7fe810099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient(hunting): picked mon.noname-b con 0x7fe810139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81009abb0 con 0x7fe810099090 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe810137750 con 0x7fe810139500 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81009a970 con 0x7fe810097b60 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient(hunting): authenticate will time out at 1015.918512s 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe810097b60 0x7fe8101ca250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50262/0 (socket says 172.21.15.76:50262) 2024-09-16T06:41:08.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80ffff700 1 -- 172.21.15.76:0/1716873818 learned_addr learned my addr 172.21.15.76:0/1716873818 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe807fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe810099090 0x7fe8101cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53380/0 (socket says 172.21.15.76:53380) 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882208198 0 0) 0x7fe81009a970 con 0x7fe810097b60 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): my global_id is 124478 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f00035d0 con 0x7fe810097b60 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660989013 0 0) 0x7fe81009abb0 con 0x7fe810099090 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): my global_id is 104784 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81009a970 con 0x7fe810099090 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651355234 0 0) 0x7fe810137750 con 0x7fe810139500 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): my global_id is 134182 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81009abb0 con 0x7fe810139500 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855819842 0 0) 0x7fe7f00035d0 con 0x7fe810097b60 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe810137750 con 0x7fe810097b60 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7f8001910 con 0x7fe810097b60 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): dump: 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14156648 0 0) 0x7fe81009a970 con 0x7fe810099090 2024-09-16T06:41:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f00035d0 con 0x7fe810099090 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe800001c00 con 0x7fe810099090 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient(hunting): dump: 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1472804069 0 0) 0x7fe810137750 con 0x7fe810097b60 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6789/0 conn(0x7fe810139500 legacy=0x7fe8101d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.76:6789/0 conn(0x7fe810099090 legacy=0x7fe8101cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 monclient: found mon.c 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe810136d50 con 0x7fe810097b60 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 5 monclient: authenticate success, global_id 124478 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _renew_subs 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81013c2e0 con 0x7fe810097b60 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _renew_subs 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81009c720 con 0x7fe810097b60 2024-09-16T06:41:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f8001ca0 con 0x7fe810097b60 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7f8003b10 con 0x7fe810097b60 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80d7fa700 10 monclient: dump: 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.780+0000 7fe80effd700 4 set_mon_vals no callback set 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe81560d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7f4006590 con 0x7fe7f4002830 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe7f8003d30 con 0x7fe810097b60 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7fe7f8043870 con 0x7fe810097b60 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe814e0c700 10 monclient: get_auth_request con 0x7fe7f0036430 auth_method 0 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392572577 0 0) 0x7fe7f4006590 con 0x7fe7f4002830 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient(hunting): my global_id is 134185 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe810137750 con 0x7fe7f4002830 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1874240033 0 0) 0x7fe810137750 con 0x7fe7f4002830 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f4006590 con 0x7fe7f4002830 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe800004b90 con 0x7fe7f4002830 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient: dump: 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.803 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2691523575 0 0) 0x7fe7f4006590 con 0x7fe7f4002830 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7f400a190 con 0x7fe7f4002830 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe800005030 con 0x7fe7f4002830 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6789/0 conn(0x7fe7f4002830 legacy=0x7fe7f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.784+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6789/0 conn(0x7fe7f4002830 legacy=0x7fe7f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe81560d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7f4009e50 con 0x7fe7f4009170 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695336206 0 0) 0x7fe7f4009e50 con 0x7fe7f4009170 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient(hunting): my global_id is 134188 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f4006590 con 0x7fe7f4009170 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3624504825 0 0) 0x7fe7f4006590 con 0x7fe7f4009170 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f4009e50 con 0x7fe7f4009170 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe800005640 con 0x7fe7f4009170 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient: dump: 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3840733352 0 0) 0x7fe7f4009e50 con 0x7fe7f4009170 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe7f400c340 con 0x7fe7f4009170 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 233978756) 0x7fe800006960 con 0x7fe7f4009170 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6789/0 conn(0x7fe7f4009170 legacy=0x7fe7f4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.800+0000 7fe80d7fa700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6789/0 conn(0x7fe7f4009170 legacy=0x7fe7f4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 34, 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:08.809 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.810 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.811 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:08.812 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:08.813 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:08.815 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:08.816 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:08.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:08.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:08.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:08.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:08.820 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:08.820 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe7f0036430 legacy=0x7fe7f00388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 10 monclient: shutdown 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 >> v1:172.21.15.76:6790/0 conn(0x7fe810097b60 legacy=0x7fe8101ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe814e0c700 1 -- 172.21.15.76:0/1716873818 reap_dead start 2024-09-16T06:41:08.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 shutdown_connections 2024-09-16T06:41:08.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 >> 172.21.15.76:0/1716873818 conn(0x7fe8100a0030 msgr2=0x7fe8101326b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:08.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 shutdown_connections 2024-09-16T06:41:08.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.804+0000 7fe81560d700 1 -- 172.21.15.76:0/1716873818 wait complete. 2024-09-16T06:41:08.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:08.823 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:41:08.823 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:41:08.823 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:41:08.823 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:41:08.823 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:41:08.823 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:41:08.868 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:41:08.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:41:08.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: get_monmap_and_config 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: build_initial_monmap 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: monmap: 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:08.988826+0000 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:08.988826+0000 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:08.988 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 1 Processor -- start 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 1 -- start start 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: init 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-b con 0x7f58800973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-a con 0x7f5880097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-c con 0x7f58801400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5880099350 con 0x7f5880097eb0 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588009f690 con 0x7f58801400d0 2024-09-16T06:41:08.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5880099bf0 con 0x7f58800973c0 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient(hunting): authenticate will time out at 1016.122213s 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5885310700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5880097eb0 legacy=0x7f588013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5885b11700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f58800973c0 0x7f58800977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50290/0 (socket says 172.21.15.76:50290) 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5885b11700 1 -- 172.21.15.76:0/1243287572 learned_addr learned my addr 172.21.15.76:0/1243287572 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539058132 0 0) 0x7f5880099bf0 con 0x7f58800973c0 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): my global_id is 124481 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586c0035d0 con 0x7f58800973c0 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050238521 0 0) 0x7f588009f690 con 0x7f58801400d0 2024-09-16T06:41:08.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _negotiate 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient(hunting): my global_id is 134191 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880099bf0 con 0x7f58801400d0 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282578311 0 0) 0x7f586c0035d0 con 0x7f58800973c0 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 >> v1:172.21.15.189:6789/0 conn(0x7f58801400d0 legacy=0x7f58801424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 >> v1:172.21.15.76:6789/0 conn(0x7f5880097eb0 legacy=0x7f588013e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: _finish_hunting 0 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 monclient: found mon.noname-b 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:08.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880098360 con 0x7f58800973c0 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: _finish_auth 0 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: _check_auth_tickets 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 5 monclient: authenticate success, global_id 124481 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5870001610 con 0x7f58800973c0 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: dump: 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.000 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58700018f0 con 0x7f58800973c0 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5886b13700 4 set_mon_vals no callback set 2024-09-16T06:41:09.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 1 -- 172.21.15.76:0/1243287572 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5870001cc0 con 0x7f58800973c0 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.984+0000 7f5884b0f700 10 monclient: dump: 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:09.002 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: shutdown 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- 172.21.15.76:0/1243287572 >> v1:172.21.15.76:6790/0 conn(0x7f58800973c0 legacy=0x7f58800977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- 172.21.15.76:0/1243287572 shutdown_connections 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- 172.21.15.76:0/1243287572 >> 172.21.15.76:0/1243287572 conn(0x7f58800a0030 msgr2=0x7f58800a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- 172.21.15.76:0/1243287572 shutdown_connections 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- 172.21.15.76:0/1243287572 wait complete. 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: build_initial_monmap 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: monmap: 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:09.003 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 Processor -- start 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- start start 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: init 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:09.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-c con 0x7f58800973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-b con 0x7f5880097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient(hunting): picked mon.noname-a con 0x7f58801400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5880072660 con 0x7f58801400d0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588006c7a0 con 0x7f5880097eb0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588009e950 con 0x7f58800973c0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient(hunting): authenticate will time out at 1016.126137s 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886312700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f58801400d0 legacy=0x7f58801d62a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5885b11700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f58800973c0 0x7f588013c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50296/0 (socket says 172.21.15.76:50296) 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5885b11700 1 -- 172.21.15.76:0/4007392075 learned_addr learned my addr 172.21.15.76:0/4007392075 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088727074 0 0) 0x7f588009e950 con 0x7f58800973c0 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:09.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): my global_id is 124484 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58640035d0 con 0x7f58800973c0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510418946 0 0) 0x7f588006c7a0 con 0x7f5880097eb0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): my global_id is 134194 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f588009e950 con 0x7f5880097eb0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357092710 0 0) 0x7f58640035d0 con 0x7f58800973c0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f588006c7a0 con 0x7f58800973c0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5870001c00 con 0x7f58800973c0 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): dump: 2024-09-16T06:41:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2216808400 0 0) 0x7f588009e950 con 0x7f5880097eb0 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58640035d0 con 0x7f5880097eb0 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f587c0044f0 con 0x7f5880097eb0 2024-09-16T06:41:09.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient(hunting): dump: 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:09.008 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505717381 0 0) 0x7f588006c7a0 con 0x7f58800973c0 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 >> v1:172.21.15.189:6789/0 conn(0x7f5880097eb0 legacy=0x7f588013ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 >> v1:172.21.15.76:6789/0 conn(0x7f58801400d0 legacy=0x7f58801d62a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 monclient: found mon.c 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588009a540 con 0x7f58800973c0 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f58767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 5 monclient: authenticate success, global_id 124484 2024-09-16T06:41:09.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: _renew_subs 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.988+0000 7f5886b13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588013ea50 con 0x7f58800973c0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 10 monclient: _renew_subs 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f588013e100 con 0x7f58800973c0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5870004ca0 con 0x7f58800973c0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5870004fb0 con 0x7f58800973c0 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58777fe700 4 set_mon_vals no callback set 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 10 monclient: dump: 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5870005190 con 0x7f58800973c0 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:09.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868005100 con 0x7f58800973c0 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f5870043b70 con 0x7f58800973c0 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:08.992+0000 7f5885310700 10 monclient: get_auth_request con 0x7f5864036280 auth_method 0 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.000+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5870014b30 con 0x7f58800973c0 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.000+0000 7f58767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:09.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.000+0000 7f58767fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.228+0000 7f5886b13700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.228+0000 7f5886b13700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:09.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.228+0000 7f5886b13700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.228+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5868005930 con 0x7f58800973c0 2024-09-16T06:41:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.232+0000 7f58767fc700 1 -- 172.21.15.76:0/4007392075 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3942437948) 0x7f5870012fa0 con 0x7f58800973c0 2024-09-16T06:41:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.232+0000 7f58767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:09.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.232+0000 7f58767fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:09.235 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5864036280 legacy=0x7f5864038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 10 monclient: shutdown 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 >> v1:172.21.15.76:6790/0 conn(0x7f58800973c0 legacy=0x7f588013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 shutdown_connections 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 >> 172.21.15.76:0/4007392075 conn(0x7f58800a0030 msgr2=0x7f5880096060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 shutdown_connections 2024-09-16T06:41:09.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:09.236+0000 7f5886b13700 1 -- 172.21.15.76:0/4007392075 wait complete. 2024-09-16T06:41:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:11.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:11.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:11.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:11.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:11.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:11.565+0000 7f2369ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:12.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:12.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: get_monmap_and_config 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: build_initial_monmap 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: monmap: 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:12.376264+0000 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:12.376264+0000 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:12.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:12.376 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:12.376 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 Processor -- start 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 -- start start 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: init 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-b con 0x7f1014132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-a con 0x7f101406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-c con 0x7f101406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101406c280 con 0x7f101406d480 2024-09-16T06:41:12.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101406bfa0 con 0x7f101406cf50 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101409ace0 con 0x7f1014132330 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient(hunting): authenticate will time out at 1019.509695s 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f10188a7700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f101406d480 legacy=0x7f101413a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f10190a8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1014132330 0x7f1014132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50302/0 (socket says 172.21.15.76:50302) 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f10190a8700 1 -- 172.21.15.76:0/2888991852 learned_addr learned my addr 172.21.15.76:0/2888991852 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986740840 0 0) 0x7f101409ace0 con 0x7f1014132330 2024-09-16T06:41:12.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): my global_id is 124487 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10000035d0 con 0x7f1014132330 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035170511 0 0) 0x7f101406bfa0 con 0x7f101406cf50 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient(hunting): my global_id is 134197 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101409ace0 con 0x7f101406cf50 2024-09-16T06:41:12.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952773275 0 0) 0x7f10000035d0 con 0x7f1014132330 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 >> v1:172.21.15.189:6789/0 conn(0x7f101406cf50 legacy=0x7f101413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 >> v1:172.21.15.76:6789/0 conn(0x7f101406d480 legacy=0x7f101413a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 monclient: found mon.noname-b 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101409ae70 con 0x7f1014132330 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: _finish_auth 0 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1004001610 con 0x7f1014132330 2024-09-16T06:41:12.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: dump: 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:12.384 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 5 monclient: authenticate success, global_id 124487 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1004001830 con 0x7f1014132330 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 1 -- 172.21.15.76:0/2888991852 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1004001a20 con 0x7f1014132330 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 4 set_mon_vals no callback set 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f100bfff700 10 monclient: dump: 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:12.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 10 monclient: shutdown 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.372+0000 7f101a0aa700 1 -- 172.21.15.76:0/2888991852 >> v1:172.21.15.76:6790/0 conn(0x7f1014132330 legacy=0x7f1014132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- 172.21.15.76:0/2888991852 shutdown_connections 2024-09-16T06:41:12.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- 172.21.15.76:0/2888991852 >> 172.21.15.76:0/2888991852 conn(0x7f10140a41c0 msgr2=0x7f10140a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- 172.21.15.76:0/2888991852 shutdown_connections 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- 172.21.15.76:0/2888991852 wait complete. 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient: build_initial_monmap 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient: monmap: 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:12.387 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 Processor -- start 2024-09-16T06:41:12.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- start start 2024-09-16T06:41:12.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient: init 2024-09-16T06:41:12.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:12.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:12.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-c con 0x7f101406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-a con 0x7f101406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient(hunting): picked mon.noname-b con 0x7f1014132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1014131630 con 0x7f101406d480 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1014131100 con 0x7f1014132330 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101409e8b0 con 0x7f101406cf50 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f101a0aa700 10 monclient(hunting): authenticate will time out at 1019.514776s 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f10188a7700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f101406d480 legacy=0x7f10141cece0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f10190a8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f101406cf50 0x7f101413e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50310/0 (socket says 172.21.15.76:50310) 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f10190a8700 1 -- 172.21.15.76:0/974251798 learned_addr learned my addr 172.21.15.76:0/974251798 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925914570 0 0) 0x7f101409e8b0 con 0x7f101406cf50 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:12.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): my global_id is 124490 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff80035d0 con 0x7f101406cf50 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745316593 0 0) 0x7f1014131100 con 0x7f1014132330 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): my global_id is 134200 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101409e8b0 con 0x7f1014132330 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982012222 0 0) 0x7f0ff80035d0 con 0x7f101406cf50 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1014131100 con 0x7f101406cf50 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1004004890 con 0x7f101406cf50 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:12.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.376+0000 7f1009ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:12.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 600079592 0 0) 0x7f101409e8b0 con 0x7f1014132330 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff80035d0 con 0x7f1014132330 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f100c001c30 con 0x7f1014132330 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:12.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2442773115 0 0) 0x7f1014131100 con 0x7f101406cf50 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 >> v1:172.21.15.189:6789/0 conn(0x7f1014132330 legacy=0x7f10141d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 >> v1:172.21.15.76:6789/0 conn(0x7f101406d480 legacy=0x7f10141cece0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 monclient: found mon.c 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014132f70 con 0x7f101406cf50 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:12.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 5 monclient: authenticate success, global_id 124490 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _renew_subs 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101412fc20 con 0x7f101406cf50 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _renew_subs 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f101413ec50 con 0x7f101406cf50 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1004004ca0 con 0x7f101406cf50 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1004004fb0 con 0x7f101406cf50 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:12.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 10 monclient: dump: 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f100b7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1004005190 con 0x7f101406cf50 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f1004043bb0 con 0x7f101406cf50 2024-09-16T06:41:12.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff4005100 con 0x7f101406cf50 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.380+0000 7f10188a7700 10 monclient: get_auth_request con 0x7f0ff80362f0 auth_method 0 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.388+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1004014b70 con 0x7f101406cf50 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.388+0000 7f1009ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:12.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.388+0000 7f1009ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:12.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:12.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:12.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:12.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:12.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:12.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:12.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:12.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:12.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:12.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ff4005930 con 0x7f101406cf50 2024-09-16T06:41:12.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f1009ffb700 1 -- 172.21.15.76:0/974251798 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3121617437) 0x7f1004012fe0 con 0x7f101406cf50 2024-09-16T06:41:12.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f1009ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:12.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f1009ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:12.628 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ff80362f0 legacy=0x7f0ff80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 10 monclient: shutdown 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 >> v1:172.21.15.76:6790/0 conn(0x7f101406cf50 legacy=0x7f101413e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 shutdown_connections 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 >> 172.21.15.76:0/974251798 conn(0x7f10140a41c0 msgr2=0x7f101409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 shutdown_connections 2024-09-16T06:41:12.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:12.624+0000 7f101a0aa700 1 -- 172.21.15.76:0/974251798 wait complete. 2024-09-16T06:41:13.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:13.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:13.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:13.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:13.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:13.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:14.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:14.569+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:14.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:14.569+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:14.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:14.569+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:15.569+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:15.569+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:15.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:15.569+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:15.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:15.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.756+0000 7f0acb37c700 10 monclient: get_monmap_and_config 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.756+0000 7f0acb37c700 10 monclient: build_initial_monmap 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.756+0000 7f0acb37c700 10 monclient: monmap: 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:15.763390+0000 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:15.763390+0000 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:15.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:15.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 1 Processor -- start 2024-09-16T06:41:15.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 1 -- start start 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient: init 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-b con 0x7f0ac4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-c con 0x7f0ac4099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-a con 0x7f0ac4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac4098990 con 0x7f0ac4139500 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac4097f40 con 0x7f0ac4099090 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac409d610 con 0x7f0ac4097b60 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acb37c700 10 monclient(hunting): authenticate will time out at 1022.896870s 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0acab7b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0ac413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:15.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0aca37a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ac4097b60 0x7f0ac409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50026/0 (socket says 172.21.15.76:50026) 2024-09-16T06:41:15.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.760+0000 7f0aca37a700 1 -- 172.21.15.76:0/2638943758 learned_addr learned my addr 172.21.15.76:0/2638943758 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:15.964+0000 7f0acab7b700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0ac413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:16.368+0000 7f0acab7b700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0ac413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:16.569+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:16.569+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:16.569+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:17.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:17.172+0000 7f0acab7b700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0ac413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:17.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:17.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:17.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:17.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:17.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:17.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:18.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:18.569+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:18.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:18.569+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:18.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:18.569+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): tick 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 monclient(hunting): continuing hunt 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6790/0 conn(0x7f0ac4097b60 legacy=0x7f0ac409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.189:6789/0 conn(0x7f0ac4099090 legacy=0x7f0ac409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0ac413b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f0aa8003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f0ac4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:18.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f0ac4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:18.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac40986b0 con 0x7f0ac4139500 2024-09-16T06:41:18.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac4098990 con 0x7f0aa8003740 2024-09-16T06:41:18.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0ac9b79700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:18.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa8010420 con 0x7f0ac4097b60 2024-09-16T06:41:18.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.764+0000 7f0abbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:18.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:18.968+0000 7f0ac9b79700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:19.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:19.372+0000 7f0ac9b79700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:19.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:19.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:19.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:19.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:19.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:19.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:20.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:20.176+0000 7f0ac9b79700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:20.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:20.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:20.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:20.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:20.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:21.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:21.565+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:21.565+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:21.565+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:21.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:21.565+0000 7f2369ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): tick 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 monclient(hunting): continuing hunt 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6790/0 conn(0x7f0ac4097b60 legacy=0x7f0aa800cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.189:6789/0 conn(0x7f0aa8003740 legacy=0x7f0aa8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0ac4139500 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:21.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f0aa800f4a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f0aa8009b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f0aa8003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa8011630 con 0x7f0aa800f4a0 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac40986b0 con 0x7f0aa8003740 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa8009f70 con 0x7f0aa8009b70 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0abbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:21.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.760+0000 7f0aca37a700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0aa800f4a0 legacy=0x7f0aa800f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:21.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:21.964+0000 7f0aca37a700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0aa800f4a0 legacy=0x7f0aa800f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.368+0000 7f0aca37a700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0aa800f4a0 legacy=0x7f0aa800f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:22.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:22.569+0000 7f2369ffb700 10 monclient: tick 2024-09-16T06:41:22.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:22.569+0000 7f2369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:22.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:22.569+0000 7f2369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939290018 0 0) 0x7f0ac40986b0 con 0x7f0aa8003740 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 10 monclient(hunting): _negotiate 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 10 monclient(hunting): my global_id is 134203 2024-09-16T06:41:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.936+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab00035d0 con 0x7f0aa8003740 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911747240 0 0) 0x7f0ab00035d0 con 0x7f0aa8003740 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6790/0 conn(0x7f0aa8009b70 legacy=0x7f0aa8009440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.76:6789/0 conn(0x7f0aa800f4a0 legacy=0x7f0aa800f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: _finish_hunting 0 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 monclient: found mon.noname-c 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa800cc60 con 0x7f0aa8003740 2024-09-16T06:41:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: _finish_auth 0 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: _check_auth_tickets 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0abc004580 con 0x7f0aa8003740 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 5 monclient: authenticate success, global_id 134203 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: dump: 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0abc0047a0 con 0x7f0aa8003740 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 1 -- 172.21.15.76:0/2638943758 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0abc004bb0 con 0x7f0aa8003740 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0ac9378700 10 monclient: dump: 2024-09-16T06:41:22.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 4 set_mon_vals no callback set 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 10 monclient: shutdown 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:22.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 1 -- 172.21.15.76:0/2638943758 >> v1:172.21.15.189:6789/0 conn(0x7f0aa8003740 legacy=0x7f0aa8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:22.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 1 -- 172.21.15.76:0/2638943758 shutdown_connections 2024-09-16T06:41:22.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 1 -- 172.21.15.76:0/2638943758 >> 172.21.15.76:0/2638943758 conn(0x7f0ac40a0030 msgr2=0x7f0ac40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:22.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 1 -- 172.21.15.76:0/2638943758 shutdown_connections 2024-09-16T06:41:22.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.940+0000 7f0acb37c700 1 -- 172.21.15.76:0/2638943758 wait complete. 2024-09-16T06:41:22.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient: build_initial_monmap 2024-09-16T06:41:22.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient: monmap: 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:22.951 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 1 Processor -- start 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 1 -- start start 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient: init 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-a con 0x7f0aa8003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-c con 0x7f0aa8009b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient(hunting): picked mon.noname-b con 0x7f0aa800f4a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac4092910 con 0x7f0aa8003740 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac40718e0 con 0x7f0aa800f4a0 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac40a2c10 con 0x7f0aa8009b70 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:22.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0aca37a700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0aa8003740 legacy=0x7f0ac4096370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:22.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0acb37c700 10 monclient(hunting): authenticate will time out at 1030.083044s 2024-09-16T06:41:22.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0ac9b79700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0aa8009b70 0x7f0ac41cc6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50060/0 (socket says 172.21.15.76:50060) 2024-09-16T06:41:22.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.944+0000 7f0ac9b79700 1 -- 172.21.15.76:0/1224524315 learned_addr learned my addr 172.21.15.76:0/1224524315 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494192076 0 0) 0x7f0ac40a2c10 con 0x7f0aa8009b70 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 10 monclient(hunting): my global_id is 124493 2024-09-16T06:41:22.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa800f880 con 0x7f0aa8009b70 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251508215 0 0) 0x7f0aa800f880 con 0x7f0aa8009b70 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.956+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ac40a2c10 con 0x7f0aa8009b70 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab40017a0 con 0x7f0aa8009b70 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient(hunting): dump: 2024-09-16T06:41:22.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:22.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:22.964 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.964 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3242368041 0 0) 0x7f0ac40a2c10 con 0x7f0aa8009b70 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 >> v1:172.21.15.189:6789/0 conn(0x7f0aa800f4a0 legacy=0x7f0ac41cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 >> v1:172.21.15.76:6789/0 conn(0x7f0aa8003740 legacy=0x7f0ac4096370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 monclient: found mon.c 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:22.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac4096a80 con 0x7f0aa8009b70 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 5 monclient: authenticate success, global_id 124493 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _renew_subs 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4092bf0 con 0x7f0aa8009b70 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _renew_subs 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ab40036a0 con 0x7f0aa8009b70 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac4090890 con 0x7f0aa8009b70 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:22.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab4003a30 con 0x7f0aa8009b70 2024-09-16T06:41:22.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:22.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0ac8b77700 4 set_mon_vals no callback set 2024-09-16T06:41:22.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 10 monclient: dump: 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:22.968 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ab4003b90 con 0x7f0aa8009b70 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac40926a0 con 0x7f0aa8009b70 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 5361+0+0 (unknown 1559103531 0 0) 0x7f0ab4043800 con 0x7f0aa8009b70 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.960+0000 7f0aca37a700 10 monclient: get_auth_request con 0x7f0aa80377c0 auth_method 0 2024-09-16T06:41:22.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.964+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ac40926a0 con 0x7f0aa8009b70 2024-09-16T06:41:22.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.964+0000 7f0abaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:22.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:22.964+0000 7f0abaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:23.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.013+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 168+0+0 (unknown 1499378534 0 0) 0x7f2370012fd0 con 0x7f237c09bb00 2024-09-16T06:41:23.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.013+0000 7f236affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.013+0000 7f236affd700 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.125+0000 7f2380cfe700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.125+0000 7f2380cfe700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.125+0000 7f2380cfe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.125+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f23600053b0 con 0x7f237c09bb00 2024-09-16T06:41:23.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f236affd700 1 -- 172.21.15.189:0/3081871705 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (unknown 3708369363 0 0) 0x7f2370014750 con 0x7f237c09bb00 2024-09-16T06:41:23.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f236affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f236affd700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 >> v1:172.21.15.189:6812/3098810674 conn(0x7f235c036560 legacy=0x7f235c038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 10 monclient: shutdown 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 >> v1:172.21.15.189:6789/0 conn(0x7f237c09bb00 legacy=0x7f237c092430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 shutdown_connections 2024-09-16T06:41:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 >> 172.21.15.189:0/3081871705 conn(0x7f237c0a03e0 msgr2=0x7f237c1384b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 shutdown_connections 2024-09-16T06:41:23.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.129+0000 7f2380cfe700 1 -- 172.21.15.189:0/3081871705 wait complete. 2024-09-16T06:41:23.136 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.149 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T06:41:23.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0acb37c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0acb37c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0acb37c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ac40672a0 con 0x7f0aa8009b70 2024-09-16T06:41:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0abaffd700 1 -- 172.21.15.76:0/1224524315 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 790424303) 0x7f0ab40146a0 con 0x7f0aa8009b70 2024-09-16T06:41:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0abaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.203+0000 7f0abaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:23.209 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":36,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:23.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0aa80377c0 legacy=0x7f0aa8039c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 10 monclient: shutdown 2024-09-16T06:41:23.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 >> v1:172.21.15.76:6790/0 conn(0x7f0aa8009b70 legacy=0x7f0ac41cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 shutdown_connections 2024-09-16T06:41:23.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 >> 172.21.15.76:0/1224524315 conn(0x7f0ac40a0030 msgr2=0x7f0ac40a1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 shutdown_connections 2024-09-16T06:41:23.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.207+0000 7f0acb37c700 1 -- 172.21.15.76:0/1224524315 wait complete. 2024-09-16T06:41:23.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:41:23.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:41:23.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 10 monclient: get_monmap_and_config 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 10 monclient: monmap: 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:23.259306+0000 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:23.259306+0000 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.258 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:23.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:23.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 1 Processor -- start 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 1 -- start start 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.253+0000 7f2c753f2700 10 monclient: init 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-b con 0x7f2c700a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-a con 0x7f2c7009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-c con 0x7f2c700a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7009a040 con 0x7f2c7009aa20 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c700a33d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c70099d60 con 0x7f2c700a2350 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): authenticate will time out at 1035.772399s 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6e7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2c7009aa20 legacy=0x7f2c7013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2c700a3ca0 0x7f2c7013dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51288/0 (socket says 172.21.15.189:51288) 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6f7fe700 1 -- 172.21.15.189:0/2622939242 learned_addr learned my addr 172.21.15.189:0/2622939242 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183904655 0 0) 0x7f2c700a33d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): my global_id is 134206 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5c0035d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819496659 0 0) 0x7f2c70099d60 con 0x7f2c700a2350 2024-09-16T06:41:23.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient(hunting): my global_id is 124496 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c700a33d0 con 0x7f2c700a2350 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176173048 0 0) 0x7f2c5c0035d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 >> v1:172.21.15.76:6790/0 conn(0x7f2c700a2350 legacy=0x7f2c700a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 >> v1:172.21.15.76:6789/0 conn(0x7f2c7009aa20 legacy=0x7f2c7013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 monclient: found mon.noname-c 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700a3920 con 0x7f2c700a3ca0 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 5 monclient: authenticate success, global_id 134206 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c68001610 con 0x7f2c700a3ca0 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: dump: 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c680018f0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 4 set_mon_vals no callback set 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 1 -- 172.21.15.189:0/2622939242 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c68001c80 con 0x7f2c700a3ca0 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c6dffb700 10 monclient: dump: 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: shutdown 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- 172.21.15.189:0/2622939242 >> v1:172.21.15.189:6789/0 conn(0x7f2c700a3ca0 legacy=0x7f2c7013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- 172.21.15.189:0/2622939242 shutdown_connections 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- 172.21.15.189:0/2622939242 >> 172.21.15.189:0/2622939242 conn(0x7f2c700a7d70 msgr2=0x7f2c700a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- 172.21.15.189:0/2622939242 shutdown_connections 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- 172.21.15.189:0/2622939242 wait complete. 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: monmap: 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 Processor -- start 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 1 -- start start 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: init 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-a con 0x7f2c7009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.257+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-c con 0x7f2c700a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient(hunting): picked mon.noname-b con 0x7f2c700a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c701220b0 con 0x7f2c7009aa20 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7011fd70 con 0x7f2c700a3ca0 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7009b140 con 0x7f2c700a2350 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient(hunting): authenticate will time out at 1035.776065s 2024-09-16T06:41:23.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2c7009aa20 legacy=0x7f2c701c9c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2c700a2350 0x7f2c701346f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:43534/0 (socket says 172.21.15.189:43534) 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2c700a3ca0 0x7f2c701d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51292/0 (socket says 172.21.15.189:51292) 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6e7fc700 1 -- 172.21.15.189:0/1536151741 learned_addr learned my addr 172.21.15.189:0/1536151741 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719620397 0 0) 0x7f2c7011fd70 con 0x7f2c700a3ca0 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): my global_id is 134209 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c500035d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998011644 0 0) 0x7f2c7009b140 con 0x7f2c700a2350 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): my global_id is 124499 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7011fd70 con 0x7f2c700a2350 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805626971 0 0) 0x7f2c500035d0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c7009b140 con 0x7f2c700a3ca0 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c68001430 con 0x7f2c700a3ca0 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): dump: 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338076221 0 0) 0x7f2c7011fd70 con 0x7f2c700a2350 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c500035d0 con 0x7f2c700a2350 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c58004230 con 0x7f2c700a2350 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient(hunting): dump: 2024-09-16T06:41:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1514710337 0 0) 0x7f2c7009b140 con 0x7f2c700a3ca0 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 >> v1:172.21.15.76:6790/0 conn(0x7f2c700a2350 legacy=0x7f2c701346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 >> v1:172.21.15.76:6789/0 conn(0x7f2c7009aa20 legacy=0x7f2c701c9c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 monclient: found mon.b 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7006bed0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 5 monclient: authenticate success, global_id 134209 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _renew_subs 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c70121dd0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _renew_subs 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c70134e00 con 0x7f2c700a3ca0 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c68004ca0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c68004fb0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6cff9700 4 set_mon_vals no callback set 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 10 monclient: dump: 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2c6800fa20 con 0x7f2c700a3ca0 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7f2c68043990 con 0x7f2c700a3ca0 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c54005100 con 0x7f2c700a3ca0 2024-09-16T06:41:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.261+0000 7f2c6effd700 10 monclient: get_auth_request con 0x7f2c500362c0 auth_method 0 2024-09-16T06:41:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.269+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2c68014660 con 0x7f2c700a3ca0 2024-09-16T06:41:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.269+0000 7f2c677fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.269+0000 7f2c677fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:23.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: get_monmap_and_config 2024-09-16T06:41:23.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: monmap: 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:23.336256+0000 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:23.336256+0000 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:23.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 Processor -- start 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- start start 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: init 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-b con 0x7f4780097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-c con 0x7f4780099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-a con 0x7f4780139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4780098990 con 0x7f4780139500 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4780097f40 con 0x7f4780099090 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478009d610 con 0x7f4780097b60 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient(hunting): authenticate will time out at 1030.469440s 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f4784ef3700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4780139500 legacy=0x7f478013b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4780097b60 0x7f478009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50074/0 (socket says 172.21.15.76:50074) 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477ffff700 1 -- 172.21.15.76:0/3909848141 learned_addr learned my addr 172.21.15.76:0/3909848141 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146121355 0 0) 0x7f478009d610 con 0x7f4780097b60 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): my global_id is 124502 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47680035d0 con 0x7f4780097b60 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374159442 0 0) 0x7f4780097f40 con 0x7f4780099090 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient(hunting): my global_id is 134212 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478009d610 con 0x7f4780099090 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554040993 0 0) 0x7f47680035d0 con 0x7f4780097b60 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 >> v1:172.21.15.189:6789/0 conn(0x7f4780099090 legacy=0x7f478009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 >> v1:172.21.15.76:6789/0 conn(0x7f4780139500 legacy=0x7f478013b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:23.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 monclient: found mon.noname-b 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47800986b0 con 0x7f4780097b60 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 5 monclient: authenticate success, global_id 124502 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4774001610 con 0x7f4780097b60 2024-09-16T06:41:23.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: dump: 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47740018f0 con 0x7f4780097b60 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 4 set_mon_vals no callback set 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 1 -- 172.21.15.76:0/3909848141 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4774001c80 con 0x7f4780097b60 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f477effd700 10 monclient: dump: 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: shutdown 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- 172.21.15.76:0/3909848141 >> v1:172.21.15.76:6790/0 conn(0x7f4780097b60 legacy=0x7f478009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- 172.21.15.76:0/3909848141 shutdown_connections 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- 172.21.15.76:0/3909848141 >> 172.21.15.76:0/3909848141 conn(0x7f47800a0030 msgr2=0x7f47800a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- 172.21.15.76:0/3909848141 shutdown_connections 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 1 -- 172.21.15.76:0/3909848141 wait complete. 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.331+0000 7f47856f4700 10 monclient: monmap: 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:23.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 Processor -- start 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- start start 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: init 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-a con 0x7f4780097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-c con 0x7f4780099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient(hunting): picked mon.noname-b con 0x7f4780139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478009b550 con 0x7f4780097b60 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47800a2c10 con 0x7f4780139500 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478009c470 con 0x7f4780099090 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4780097b60 legacy=0x7f4780070510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4780099090 0x7f4780070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50086/0 (socket says 172.21.15.76:50086) 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient(hunting): authenticate will time out at 1030.473615s 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477f7fe700 1 -- 172.21.15.76:0/2840979465 learned_addr learned my addr 172.21.15.76:0/2840979465 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641939796 0 0) 0x7f478009c470 con 0x7f4780099090 2024-09-16T06:41:23.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): my global_id is 124505 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f476c0035d0 con 0x7f4780099090 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587633965 0 0) 0x7f47800a2c10 con 0x7f4780139500 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): my global_id is 134215 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f478009c470 con 0x7f4780139500 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4089313171 0 0) 0x7f476c0035d0 con 0x7f4780099090 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47800a2c10 con 0x7f4780099090 2024-09-16T06:41:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4778001c90 con 0x7f4780099090 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586581855 0 0) 0x7f478009c470 con 0x7f4780139500 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f476c0035d0 con 0x7f4780139500 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4770004060 con 0x7f4780139500 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.359 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3485030915 0 0) 0x7f47800a2c10 con 0x7f4780099090 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.189:6789/0 conn(0x7f4780139500 legacy=0x7f47801d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6789/0 conn(0x7f4780097b60 legacy=0x7f4780070510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 monclient: found mon.c 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478013c200 con 0x7f4780099090 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f477cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 5 monclient: authenticate success, global_id 124505 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _renew_subs 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478009a540 con 0x7f4780099090 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _renew_subs 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4780136b90 con 0x7f4780099090 2024-09-16T06:41:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.335+0000 7f47856f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4780139280 con 0x7f4780071330 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4778002020 con 0x7f4780099090 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4778003e90 con 0x7f4780099090 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: dump: 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477e7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4778004070 con 0x7f4780099090 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7f4778043b60 con 0x7f4780099090 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f4784ef3700 10 monclient: get_auth_request con 0x7f476c0362c0 auth_method 0 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628758972 0 0) 0x7f4780139280 con 0x7f4780071330 2024-09-16T06:41:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient(hunting): my global_id is 124508 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47800a2c10 con 0x7f4780071330 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2199340557 0 0) 0x7f47800a2c10 con 0x7f4780071330 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4780139280 con 0x7f4780071330 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4774001620 con 0x7f4780071330 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: dump: 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1284888465 0 0) 0x7f4780139280 con 0x7f4780071330 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f478009fb80 con 0x7f4780071330 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f47740049f0 con 0x7f4780071330 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6790/0 conn(0x7f4780071330 legacy=0x7f4780133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.339+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6790/0 conn(0x7f4780071330 legacy=0x7f4780133960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f47856f4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f47800672a0 con 0x7f478002c6b0 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358345190 0 0) 0x7f47800672a0 con 0x7f478002c6b0 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient(hunting): my global_id is 124511 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4780139280 con 0x7f478002c6b0 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1413815013 0 0) 0x7f4780139280 con 0x7f478002c6b0 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47800672a0 con 0x7f478002c6b0 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4774004fb0 con 0x7f478002c6b0 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 10 monclient: dump: 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.347+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 867527157 0 0) 0x7f47800672a0 con 0x7f478002c6b0 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4780099500 con 0x7f478002c6b0 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 3517255522) 0x7f47740072e0 con 0x7f478002c6b0 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6790/0 conn(0x7f478002c6b0 legacy=0x7f4780134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f477cff9700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6790/0 conn(0x7f478002c6b0 legacy=0x7f4780134070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 36, 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:23.370 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:23.371 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.372 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.374 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:23.376 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:23.377 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:23.378 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.379 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.380 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:23.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:23.384 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:23.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.386 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.389 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.390 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.391 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.393 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.394 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.396 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:41:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:23.399 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:23.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.189:6812/3098810674 conn(0x7f476c0362c0 legacy=0x7f476c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 10 monclient: shutdown 2024-09-16T06:41:23.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 >> v1:172.21.15.76:6790/0 conn(0x7f4780099090 legacy=0x7f4780070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f4784ef3700 1 -- 172.21.15.76:0/2840979465 reap_dead start 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f4784ef3700 1 -- 172.21.15.76:0/2840979465 reap_dead start 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 shutdown_connections 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 >> 172.21.15.76:0/2840979465 conn(0x7f47800a0030 msgr2=0x7f47800a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 shutdown_connections 2024-09-16T06:41:23.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.351+0000 7f47856f4700 1 -- 172.21.15.76:0/2840979465 wait complete. 2024-09-16T06:41:23.401 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:41:23.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:41:23.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:41:23.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.441+0000 7f2c753f2700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.441+0000 7f2c753f2700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.441+0000 7f2c753f2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.441+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f2c54005930 con 0x7f2c700a3ca0 2024-09-16T06:41:23.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.445+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (unknown 3708369363 0 0) 0x7f2c68012d00 con 0x7f2c700a3ca0 2024-09-16T06:41:23.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.445+0000 7f2c677fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.445+0000 7f2c677fe700 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: get_monmap_and_config 2024-09-16T06:41:23.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: monmap: 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:23.522349+0000 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:23.522349+0000 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.533 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 1 Processor -- start 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 1 -- start start 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: init 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-a con 0x7f08ac09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-b con 0x7f08ac09d520 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-c con 0x7f08ac09fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac09b520 con 0x7f08ac09f230 2024-09-16T06:41:23.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac071f30 con 0x7f08ac09fd20 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac09b0e0 con 0x7f08ac09d520 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b447d700 10 monclient(hunting): authenticate will time out at 1030.655680s 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b2219700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f08ac09f230 legacy=0x7f08ac09f610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b1a18700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f08ac09d520 0x7f08ac137440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50108/0 (socket says 172.21.15.76:50108) 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.515+0000 7f08b1a18700 1 -- 172.21.15.76:0/1008671563 learned_addr learned my addr 172.21.15.76:0/1008671563 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261705444 0 0) 0x7f08ac09b0e0 con 0x7f08ac09d520 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): my global_id is 124514 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08980035d0 con 0x7f08ac09d520 2024-09-16T06:41:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290189990 0 0) 0x7f08ac071f30 con 0x7f08ac09fd20 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient(hunting): my global_id is 134218 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac09b0e0 con 0x7f08ac09fd20 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3415706303 0 0) 0x7f08980035d0 con 0x7f08ac09d520 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 >> v1:172.21.15.189:6789/0 conn(0x7f08ac09fd20 legacy=0x7f08ac13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 >> v1:172.21.15.76:6789/0 conn(0x7f08ac09f230 legacy=0x7f08ac09f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 monclient: found mon.noname-b 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac09c7d0 con 0x7f08ac09d520 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08a8001990 con 0x7f08ac09d520 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 5 monclient: authenticate success, global_id 124514 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: dump: 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08a8001c00 con 0x7f08ac09d520 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:23.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 1 -- 172.21.15.76:0/1008671563 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08a8001f90 con 0x7f08ac09d520 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 4 set_mon_vals no callback set 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b1217700 10 monclient: dump: 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.539 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 10 monclient: shutdown 2024-09-16T06:41:23.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- 172.21.15.76:0/1008671563 >> v1:172.21.15.76:6790/0 conn(0x7f08ac09d520 legacy=0x7f08ac137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- 172.21.15.76:0/1008671563 shutdown_connections 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- 172.21.15.76:0/1008671563 >> 172.21.15.76:0/1008671563 conn(0x7f08ac0a3e50 msgr2=0x7f08ac0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- 172.21.15.76:0/1008671563 shutdown_connections 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- 172.21.15.76:0/1008671563 wait complete. 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 10 monclient: monmap: 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:23.541 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:23.542 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 Processor -- start 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 1 -- start start 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.519+0000 7f08b447d700 10 monclient: init 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-c con 0x7f08ac09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-a con 0x7f08ac09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient(hunting): picked mon.noname-b con 0x7f08ac13ae30 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac13a370 con 0x7f08ac09fd20 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac072450 con 0x7f08ac13ae30 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac135120 con 0x7f08ac09f230 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient(hunting): authenticate will time out at 1030.660155s 2024-09-16T06:41:23.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b1a18700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f08ac09fd20 legacy=0x7f08ac09a170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b2219700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f08ac09f230 0x7f08ac099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50118/0 (socket says 172.21.15.76:50118) 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b2219700 1 -- 172.21.15.76:0/177740333 learned_addr learned my addr 172.21.15.76:0/177740333 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584092654 0 0) 0x7f08ac135120 con 0x7f08ac09f230 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): my global_id is 124517 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08900035d0 con 0x7f08ac09f230 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054416673 0 0) 0x7f08ac072450 con 0x7f08ac13ae30 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): my global_id is 134221 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac135120 con 0x7f08ac13ae30 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4099109863 0 0) 0x7f08900035d0 con 0x7f08ac09f230 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08ac072450 con 0x7f08ac09f230 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f089c004200 con 0x7f08ac09f230 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.545 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086420892 0 0) 0x7f08ac072450 con 0x7f08ac09f230 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6789/0 conn(0x7f08ac13ae30 legacy=0x7f08ac1d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.76:6789/0 conn(0x7f08ac09fd20 legacy=0x7f08ac09a170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 monclient: found mon.c 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac09a880 con 0x7f08ac09f230 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 5 monclient: authenticate success, global_id 124517 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _renew_subs 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac134e40 con 0x7f08ac09f230 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _renew_subs 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac1326e0 con 0x7f08ac09f230 2024-09-16T06:41:23.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f089c0044e0 con 0x7f08ac09f230 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f089c004870 con 0x7f08ac09f230 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b0a16700 4 set_mon_vals no callback set 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: dump: 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.548 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0894006590 con 0x7f0894002830 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857853885 0 0) 0x7f0894006590 con 0x7f0894002830 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient(hunting): my global_id is 134224 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac072450 con 0x7f0894002830 2024-09-16T06:41:23.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f089c004a50 con 0x7f08ac09f230 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7f089c0439a0 con 0x7f08ac09f230 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4066293537 0 0) 0x7f08ac072450 con 0x7f0894002830 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0894006590 con 0x7f0894002830 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08a80013f0 con 0x7f0894002830 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08a2ffd700 10 monclient: dump: 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.550 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.523+0000 7f08b2a1a700 10 monclient: get_auth_request con 0x7f0890036210 auth_method 0 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1456072394 0 0) 0x7f0894006590 con 0x7f0894002830 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f089400a190 con 0x7f0894002830 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f08a8007480 con 0x7f0894002830 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6789/0 conn(0x7f0894002830 legacy=0x7f0894004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.527+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6789/0 conn(0x7f0894002830 legacy=0x7f0894004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:23.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08b447d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0894009e30 con 0x7f0894009150 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862379180 0 0) 0x7f0894009e30 con 0x7f0894009150 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient(hunting): my global_id is 134227 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0894006590 con 0x7f0894009150 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2089694017 0 0) 0x7f0894006590 con 0x7f0894009150 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0894009e30 con 0x7f0894009150 2024-09-16T06:41:23.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08a8005410 con 0x7f0894009150 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 10 monclient: dump: 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.554 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 646686722 0 0) 0x7f0894009e30 con 0x7f0894009150 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.535+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f089400c340 con 0x7f0894009150 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 1849782475) 0x7f08a80075f0 con 0x7f0894009150 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08a2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6789/0 conn(0x7f0894009150 legacy=0x7f08940095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08a2ffd700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6789/0 conn(0x7f0894009150 legacy=0x7f08940095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:23.555 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 36, 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.558 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:23.559 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:23.560 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:23.561 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.562 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.564 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.565 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.566 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:23.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:23.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:23.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:23.568 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:23.568 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:23.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0890036210 legacy=0x7f08900386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 10 monclient: shutdown 2024-09-16T06:41:23.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 >> v1:172.21.15.76:6790/0 conn(0x7f08ac09f230 legacy=0x7f08ac099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b2a1a700 1 -- 172.21.15.76:0/177740333 reap_dead start 2024-09-16T06:41:23.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b2a1a700 1 -- 172.21.15.76:0/177740333 reap_dead start 2024-09-16T06:41:23.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 shutdown_connections 2024-09-16T06:41:23.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 >> 172.21.15.76:0/177740333 conn(0x7f08ac0a3e50 msgr2=0x7f08ac0a1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 shutdown_connections 2024-09-16T06:41:23.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:23.539+0000 7f08b447d700 1 -- 172.21.15.76:0/177740333 wait complete. 2024-09-16T06:41:23.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:41:23.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c753f2700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c753f2700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c753f2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f2c54005bc0 con 0x7f2c700a3ca0 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c677fe700 1 -- 172.21.15.189:0/1536151741 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v74) v1 ==== 177+0+0 (unknown 3708369363 0 0) 0x7f2c68014480 con 0x7f2c700a3ca0 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c677fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.549+0000 7f2c677fe700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2c500362c0 legacy=0x7f2c50038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 10 monclient: shutdown 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 >> v1:172.21.15.189:6789/0 conn(0x7f2c700a3ca0 legacy=0x7f2c701d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 shutdown_connections 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 >> 172.21.15.189:0/1536151741 conn(0x7f2c700a7d70 msgr2=0x7f2c700a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 shutdown_connections 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.553+0000 7f2c753f2700 1 -- 172.21.15.189:0/1536151741 wait complete. 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c osd.7 2024-09-16T06:41:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: get_monmap_and_config 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: monmap: 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:23.669237+0000 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:23.669237+0000 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:23.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 1 Processor -- start 2024-09-16T06:41:23.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 1 -- start start 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: init 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c809b7a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c809c290 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c8138a40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c8097b10 con 0x7fa9c809c290 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c8097550 con 0x7fa9c809b7a0 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c809f990 con 0x7fa9c8138a40 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient(hunting): authenticate will time out at 1036.181990s 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ccf8c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa9c809c290 legacy=0x7fa9c8137320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9cd78d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9c809b7a0 0x7fa9c809bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51302/0 (socket says 172.21.15.189:51302) 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9cd78d700 1 -- 172.21.15.189:0/1617975614 learned_addr learned my addr 172.21.15.189:0/1617975614 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754649178 0 0) 0x7fa9c8097550 con 0x7fa9c809b7a0 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient(hunting): my global_id is 134230 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b00035d0 con 0x7fa9c809b7a0 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698347329 0 0) 0x7fa9b00035d0 con 0x7fa9c809b7a0 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 >> v1:172.21.15.76:6790/0 conn(0x7fa9c8138a40 legacy=0x7fa9c813ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 >> v1:172.21.15.76:6789/0 conn(0x7fa9c809c290 legacy=0x7fa9c8137320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 monclient: found mon.noname-c 2024-09-16T06:41:23.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8097830 con 0x7fa9c809b7a0 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9b8001610 con 0x7fa9c809b7a0 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: dump: 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.675 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 5 monclient: authenticate success, global_id 134230 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b80018c0 con 0x7fa9c809b7a0 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 1 -- 172.21.15.189:0/1617975614 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9b8001c50 con 0x7fa9c809b7a0 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9c7fff700 10 monclient: dump: 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.665+0000 7fa9ce78f700 4 set_mon_vals no callback set 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 10 monclient: shutdown 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- 172.21.15.189:0/1617975614 >> v1:172.21.15.189:6789/0 conn(0x7fa9c809b7a0 legacy=0x7fa9c809bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- 172.21.15.189:0/1617975614 shutdown_connections 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- 172.21.15.189:0/1617975614 >> 172.21.15.189:0/1617975614 conn(0x7fa9c809fe70 msgr2=0x7fa9c80a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- 172.21.15.189:0/1617975614 shutdown_connections 2024-09-16T06:41:23.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- 172.21.15.189:0/1617975614 wait complete. 2024-09-16T06:41:23.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 10 monclient: build_initial_monmap 2024-09-16T06:41:23.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 10 monclient: monmap: 2024-09-16T06:41:23.682 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:23.682 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 Processor -- start 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 1 -- start start 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.669+0000 7fa9ce78f700 10 monclient: init 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:23.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c809b7a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c809c290 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c8138a40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c80714d0 con 0x7fa9c809b7a0 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9cd78d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa9c809b7a0 legacy=0x7fa9c81c6e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c808f060 con 0x7fa9c8138a40 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c808ed10 con 0x7fa9c809c290 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient(hunting): authenticate will time out at 1036.188613s 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9cdf8e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9c8138a40 0x7fa9c81cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51318/0 (socket says 172.21.15.189:51318) 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9cdf8e700 1 -- 172.21.15.189:0/2459006607 learned_addr learned my addr 172.21.15.189:0/2459006607 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553057069 0 0) 0x7fa9c808f060 con 0x7fa9c8138a40 2024-09-16T06:41:23.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): my global_id is 134233 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b40035d0 con 0x7fa9c8138a40 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796409124 0 0) 0x7fa9c808ed10 con 0x7fa9c809c290 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): my global_id is 124523 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c808f060 con 0x7fa9c809c290 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444975869 0 0) 0x7fa9b40035d0 con 0x7fa9c8138a40 2024-09-16T06:41:23.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c808ed10 con 0x7fa9c8138a40 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9c0001c30 con 0x7fa9c8138a40 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.686 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3872926246 0 0) 0x7fa9c808ed10 con 0x7fa9c8138a40 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 >> v1:172.21.15.76:6790/0 conn(0x7fa9c809c290 legacy=0x7fa9c81cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 >> v1:172.21.15.76:6789/0 conn(0x7fa9c809b7a0 legacy=0x7fa9c81c6e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 monclient: found mon.b 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c809a380 con 0x7fa9c8138a40 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:23.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 5 monclient: authenticate success, global_id 134233 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _renew_subs 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c808e470 con 0x7fa9c8138a40 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _renew_subs 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c808c9c0 con 0x7fa9c8138a40 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c0001fc0 con 0x7fa9c8138a40 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:23.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9c0003e90 con 0x7fa9c8138a40 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c77fe700 4 set_mon_vals no callback set 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 10 monclient: dump: 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:23.689 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa9c0003ff0 con 0x7fa9c8138a40 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a8005100 con 0x7fa9c8138a40 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7fa9c0043bb0 con 0x7fa9c8138a40 2024-09-16T06:41:23.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.673+0000 7fa9cd78d700 10 monclient: get_auth_request con 0x7fa9b4035fa0 auth_method 0 2024-09-16T06:41:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.681+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa9c0013030 con 0x7fa9c8138a40 2024-09-16T06:41:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.681+0000 7fa9c5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:23.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.681+0000 7fa9c5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:23.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9ce78f700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:23.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9ce78f700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:23.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9ce78f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa9a8005bc0 con 0x7fa9c8138a40 2024-09-16T06:41:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fa9c00154f0 con 0x7fa9c8138a40 2024-09-16T06:41:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9c5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:23.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.857+0000 7fa9c5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:23.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9ce78f700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:23.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9ce78f700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:23.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9ce78f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:23.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa9a8005680 con 0x7fa9c8138a40 2024-09-16T06:41:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9c5ffb700 1 -- 172.21.15.189:0/2459006607 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fa9c0014b80 con 0x7fa9c8138a40 2024-09-16T06:41:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9c5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.960+0000 7fa9c5ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa9b4035fa0 legacy=0x7fa9b4038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 10 monclient: shutdown 2024-09-16T06:41:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 >> v1:172.21.15.189:6789/0 conn(0x7fa9c8138a40 legacy=0x7fa9c81cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 shutdown_connections 2024-09-16T06:41:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 >> 172.21.15.189:0/2459006607 conn(0x7fa9c809fe70 msgr2=0x7fa9c81381e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 shutdown_connections 2024-09-16T06:41:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:23.964+0000 7fa9ce78f700 1 -- 172.21.15.189:0/2459006607 wait complete. 2024-09-16T06:41:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T06:41:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: get_monmap_and_config 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: monmap: 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:24.078052+0000 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:24.078052+0000 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.077 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.078 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.078 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:24.078 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:24.078 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 1 Processor -- start 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 1 -- start start 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: init 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:24.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-a con 0x7fe31409ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-b con 0x7fe314097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-c con 0x7fe31409f830 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe314099b10 con 0x7fe31409ed40 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe314098890 con 0x7fe31409f830 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe314098b70 con 0x7fe314097860 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient(hunting): authenticate will time out at 1036.590617s 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe313fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe31409ed40 legacy=0x7fe31409f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe318988700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe31409f830 0x7fe31413be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51334/0 (socket says 172.21.15.189:51334) 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe318988700 1 -- 172.21.15.189:0/1838396717 learned_addr learned my addr 172.21.15.189:0/1838396717 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949525575 0 0) 0x7fe314098890 con 0x7fe31409f830 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): my global_id is 134236 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc0035d0 con 0x7fe31409f830 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846809877 0 0) 0x7fe314098b70 con 0x7fe314097860 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient(hunting): my global_id is 124526 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314098890 con 0x7fe314097860 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751594613 0 0) 0x7fe2fc0035d0 con 0x7fe31409f830 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 >> v1:172.21.15.76:6790/0 conn(0x7fe314097860 legacy=0x7fe3141386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 >> v1:172.21.15.76:6789/0 conn(0x7fe31409ed40 legacy=0x7fe31409f120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:24.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 monclient: found mon.noname-c 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140973a0 con 0x7fe31409f830 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 5 monclient: authenticate success, global_id 134236 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe30c001990 con 0x7fe31409f830 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: dump: 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.089 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe30c001c70 con 0x7fe31409f830 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe31a3eb700 4 set_mon_vals no callback set 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 1 -- 172.21.15.189:0/1838396717 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe30c002000 con 0x7fe31409f830 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.072+0000 7fe312ffd700 10 monclient: dump: 2024-09-16T06:41:24.092 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.092 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.092 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.092 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.093 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: shutdown 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/1838396717 >> v1:172.21.15.189:6789/0 conn(0x7fe31409f830 legacy=0x7fe31413be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/1838396717 shutdown_connections 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/1838396717 >> 172.21.15.189:0/1838396717 conn(0x7fe31409fe30 msgr2=0x7fe3140a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:24.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/1838396717 shutdown_connections 2024-09-16T06:41:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/1838396717 wait complete. 2024-09-16T06:41:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: monmap: 2024-09-16T06:41:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.095 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:24.096 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:24.096 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:24.096 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.096 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:24.097 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.097 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:24.097 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:24.097 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 Processor -- start 2024-09-16T06:41:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- start start 2024-09-16T06:41:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: init 2024-09-16T06:41:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:24.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-c con 0x7fe314097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-a con 0x7fe31409ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient(hunting): picked mon.noname-b con 0x7fe31409f830 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140716b0 con 0x7fe31409ed40 2024-09-16T06:41:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31409a520 con 0x7fe31409f830 2024-09-16T06:41:24.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3141301c0 con 0x7fe314097860 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient(hunting): authenticate will time out at 1036.594458s 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe3137fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe31409ed40 legacy=0x7fe3141cc940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe318988700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe31409f830 0x7fe3141d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51348/0 (socket says 172.21.15.189:51348) 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe318988700 1 -- 172.21.15.189:0/771843795 learned_addr learned my addr 172.21.15.189:0/771843795 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961118028 0 0) 0x7fe31409a520 con 0x7fe31409f830 2024-09-16T06:41:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): my global_id is 134239 2024-09-16T06:41:24.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3000035d0 con 0x7fe31409f830 2024-09-16T06:41:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230844182 0 0) 0x7fe3141301c0 con 0x7fe314097860 2024-09-16T06:41:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): my global_id is 124529 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe31409a520 con 0x7fe314097860 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596494539 0 0) 0x7fe3000035d0 con 0x7fe31409f830 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3141301c0 con 0x7fe31409f830 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe30c001950 con 0x7fe31409f830 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.103 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451440249 0 0) 0x7fe31409a520 con 0x7fe314097860 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3000035d0 con 0x7fe314097860 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe3080041a0 con 0x7fe314097860 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994348729 0 0) 0x7fe3141301c0 con 0x7fe31409f830 2024-09-16T06:41:24.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 >> v1:172.21.15.76:6790/0 conn(0x7fe314097860 legacy=0x7fe314133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 >> v1:172.21.15.76:6789/0 conn(0x7fe31409ed40 legacy=0x7fe3141cc940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 monclient: found mon.b 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31412f920 con 0x7fe31409f830 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe310ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 5 monclient: authenticate success, global_id 134239 2024-09-16T06:41:24.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _renew_subs 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31412de30 con 0x7fe31409f830 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _renew_subs 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.076+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe314134000 con 0x7fe31409f830 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe30c005020 con 0x7fe31409f830 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe30c005330 con 0x7fe31409f830 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe3127fc700 4 set_mon_vals no callback set 2024-09-16T06:41:24.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 10 monclient: dump: 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe31a3eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe31a3eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe31a3eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f4005100 con 0x7fe31409f830 2024-09-16T06:41:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe30c005510 con 0x7fe31409f830 2024-09-16T06:41:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7fe30c043e60 con 0x7fe31409f830 2024-09-16T06:41:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.080+0000 7fe313fff700 10 monclient: get_auth_request con 0x7fe3000362a0 auth_method 0 2024-09-16T06:41:24.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.088+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe30c014e70 con 0x7fe31409f830 2024-09-16T06:41:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.088+0000 7fe310ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.088+0000 7fe310ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:24.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe31a3eb700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:24.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe31a3eb700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe31a3eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe2f4005bc0 con 0x7fe31409f830 2024-09-16T06:41:24.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fe30c0132e0 con 0x7fe31409f830 2024-09-16T06:41:24.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe310ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.260+0000 7fe310ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:24.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe31a3eb700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:24.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe31a3eb700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe31a3eb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe2f4005680 con 0x7fe31409f830 2024-09-16T06:41:24.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe310ff9700 1 -- 172.21.15.189:0/771843795 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fe30c014a60 con 0x7fe31409f830 2024-09-16T06:41:24.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe310ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.356+0000 7fe310ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:24.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe3000362a0 legacy=0x7fe300038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 10 monclient: shutdown 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 >> v1:172.21.15.189:6789/0 conn(0x7fe31409f830 legacy=0x7fe3141d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 shutdown_connections 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 >> 172.21.15.189:0/771843795 conn(0x7fe31409fe30 msgr2=0x7fe31406f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 shutdown_connections 2024-09-16T06:41:24.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.360+0000 7fe31a3eb700 1 -- 172.21.15.189:0/771843795 wait complete. 2024-09-16T06:41:24.380 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:24.380 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T06:41:24.380 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c osd.9 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.472+0000 7fb4f5834700 10 monclient: get_monmap_and_config 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.472+0000 7fb4f5834700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.472+0000 7fb4f5834700 10 monclient: monmap: 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:24.479982+0000 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:24.479982+0000 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.480 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:24.480 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:24.480 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 Processor -- start 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- start start 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: init 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-b con 0x7fb4f009ed40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-c con 0x7fb4f0097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-a con 0x7fb4f009f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f0099b10 con 0x7fb4f009f830 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f0098890 con 0x7fb4f0097860 2024-09-16T06:41:24.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f0098b70 con 0x7fb4f009ed40 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient(hunting): authenticate will time out at 1036.992989s 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4ee7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb4f0097860 0x7fb4f01386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51360/0 (socket says 172.21.15.189:51360) 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4ee7fc700 1 -- 172.21.15.189:0/3623662716 learned_addr learned my addr 172.21.15.189:0/3623662716 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4ef7fe700 1 -- 172.21.15.189:0/3623662716 >> v1:172.21.15.76:6789/0 conn(0x7fb4f009f830 legacy=0x7fb4f013be10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211770639 0 0) 0x7fb4f0098890 con 0x7fb4f0097860 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient(hunting): my global_id is 134242 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d80035d0 con 0x7fb4f0097860 2024-09-16T06:41:24.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179107293 0 0) 0x7fb4d80035d0 con 0x7fb4f0097860 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 >> v1:172.21.15.76:6790/0 conn(0x7fb4f009ed40 legacy=0x7fb4f009f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 >> v1:172.21.15.76:6789/0 conn(0x7fb4f009f830 legacy=0x7fb4f013be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 monclient: found mon.noname-c 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f00973a0 con 0x7fb4f0097860 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e0001610 con 0x7fb4f0097860 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 5 monclient: authenticate success, global_id 134242 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: dump: 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.487 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e0001930 con 0x7fb4f0097860 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 1 -- 172.21.15.189:0/3623662716 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e0001cc0 con 0x7fb4f0097860 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 4 set_mon_vals no callback set 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4edffb700 10 monclient: dump: 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.488 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: shutdown 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- 172.21.15.189:0/3623662716 >> v1:172.21.15.189:6789/0 conn(0x7fb4f0097860 legacy=0x7fb4f01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- 172.21.15.189:0/3623662716 shutdown_connections 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- 172.21.15.189:0/3623662716 >> 172.21.15.189:0/3623662716 conn(0x7fb4f009fe30 msgr2=0x7fb4f00a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- 172.21.15.189:0/3623662716 shutdown_connections 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 1 -- 172.21.15.189:0/3623662716 wait complete. 2024-09-16T06:41:24.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.476+0000 7fb4f5834700 10 monclient: monmap: 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 Processor -- start 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- start start 2024-09-16T06:41:24.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: init 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-c con 0x7fb4f0097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-b con 0x7fb4f009ed40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient(hunting): picked mon.noname-a con 0x7fb4f009f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f012d920 con 0x7fb4f009f830 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f009a520 con 0x7fb4f009ed40 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f009c340 con 0x7fb4f0097860 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient(hunting): authenticate will time out at 1036.997378s 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4ef7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb4f009f830 legacy=0x7fb4f01cd9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4ee7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb4f009ed40 0x7fb4f0133710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51368/0 (socket says 172.21.15.189:51368) 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4ee7fc700 1 -- 172.21.15.189:0/2108926274 learned_addr learned my addr 172.21.15.189:0/2108926274 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908923976 0 0) 0x7fb4f009a520 con 0x7fb4f009ed40 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): my global_id is 134245 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc0035d0 con 0x7fb4f009ed40 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336788682 0 0) 0x7fb4f009c340 con 0x7fb4f0097860 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): my global_id is 124535 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f009a520 con 0x7fb4f0097860 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 253826273 0 0) 0x7fb4f009a520 con 0x7fb4f0097860 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4f009c340 con 0x7fb4f0097860 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e4001870 con 0x7fb4f0097860 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): dump: 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.497 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203889555 0 0) 0x7fb4dc0035d0 con 0x7fb4f009ed40 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4f009a520 con 0x7fb4f009ed40 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e00015d0 con 0x7fb4f009ed40 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient(hunting): dump: 2024-09-16T06:41:24.498 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1045507507 0 0) 0x7fb4f009c340 con 0x7fb4f0097860 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 >> v1:172.21.15.189:6789/0 conn(0x7fb4f009ed40 legacy=0x7fb4f0133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 >> v1:172.21.15.76:6789/0 conn(0x7fb4f009f830 legacy=0x7fb4f01cd9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:24.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 monclient: found mon.c 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f012c330 con 0x7fb4f0097860 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 5 monclient: authenticate success, global_id 124535 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _renew_subs 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f009a850 con 0x7fb4f0097860 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _renew_subs 2024-09-16T06:41:24.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f0133e20 con 0x7fb4f0097860 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e40036c0 con 0x7fb4f0097860 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e4003a50 con 0x7fb4f0097860 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4ed7fa700 4 set_mon_vals no callback set 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.480+0000 7fb4d7fff700 10 monclient: dump: 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.501 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4f5834700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb4e400e540 con 0x7fb4f0097860 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4f5834700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4f5834700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0005100 con 0x7fb4f0097860 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7fb4e4042790 con 0x7fb4f0097860 2024-09-16T06:41:24.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.484+0000 7fb4ee7fc700 10 monclient: get_auth_request con 0x7fb4dc0362c0 auth_method 0 2024-09-16T06:41:24.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.496+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb4e400f9f0 con 0x7fb4f0097860 2024-09-16T06:41:24.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.496+0000 7fb4d7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.496+0000 7fb4d7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:24.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4f5834700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:24.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4f5834700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4f5834700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb4d0005bc0 con 0x7fb4f0097860 2024-09-16T06:41:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fb4e4011a90 con 0x7fb4f0097860 2024-09-16T06:41:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4d7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.652+0000 7fb4d7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.744+0000 7fb4f5834700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.744+0000 7fb4f5834700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.744+0000 7fb4f5834700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.744+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb4d0005680 con 0x7fb4f0097860 2024-09-16T06:41:24.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4d7fff700 1 -- 172.21.15.189:0/2108926274 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fb4e4013210 con 0x7fb4f0097860 2024-09-16T06:41:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4d7fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:24.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4d7fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb4dc0362c0 legacy=0x7fb4dc038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 10 monclient: shutdown 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 >> v1:172.21.15.76:6790/0 conn(0x7fb4f0097860 legacy=0x7fb4f01c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 shutdown_connections 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 >> 172.21.15.189:0/2108926274 conn(0x7fb4f009fe30 msgr2=0x7fb4f006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:24.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 shutdown_connections 2024-09-16T06:41:24.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.748+0000 7fb4f5834700 1 -- 172.21.15.189:0/2108926274 wait complete. 2024-09-16T06:41:24.764 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:24.764 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host2 2024-09-16T06:41:24.765 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 2 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: get_monmap_and_config 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: monmap: 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:24.866927+0000 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:24.866927+0000 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:24.867 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:24.867 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 1 Processor -- start 2024-09-16T06:41:24.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 1 -- start start 2024-09-16T06:41:24.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: init 2024-09-16T06:41:24.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc5080993a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc50809c840 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc508139550 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508099780 con 0x7fc508139550 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50809bb80 con 0x7fc5080993a0 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50809be60 con 0x7fc50809c840 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.860+0000 7fc50759e700 10 monclient(hunting): authenticate will time out at 1037.379873s 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc506d9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc508139550 legacy=0x7fc50813b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5080993a0 0x7fc50809eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51372/0 (socket says 172.21.15.189:51372) 2024-09-16T06:41:24.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50659c700 1 -- 172.21.15.189:0/2426811041 learned_addr learned my addr 172.21.15.189:0/2426811041 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686397716 0 0) 0x7fc50809bb80 con 0x7fc5080993a0 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): my global_id is 134248 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f00035d0 con 0x7fc5080993a0 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462636816 0 0) 0x7fc50809be60 con 0x7fc50809c840 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient(hunting): my global_id is 124538 2024-09-16T06:41:24.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50809bb80 con 0x7fc50809c840 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051853976 0 0) 0x7fc4f00035d0 con 0x7fc5080993a0 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 >> v1:172.21.15.76:6790/0 conn(0x7fc50809c840 legacy=0x7fc50809f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 >> v1:172.21.15.76:6789/0 conn(0x7fc508139550 legacy=0x7fc50813b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 monclient: found mon.noname-c 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50809f9c0 con 0x7fc5080993a0 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 5 monclient: authenticate success, global_id 134248 2024-09-16T06:41:24.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4fc001610 con 0x7fc5080993a0 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: dump: 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.873 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4fc0018f0 con 0x7fc5080993a0 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 1 -- 172.21.15.189:0/2426811041 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc4fc001c80 con 0x7fc5080993a0 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 4 set_mon_vals no callback set 2024-09-16T06:41:24.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50559a700 10 monclient: dump: 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: shutdown 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:24.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- 172.21.15.189:0/2426811041 >> v1:172.21.15.189:6789/0 conn(0x7fc5080993a0 legacy=0x7fc50809eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- 172.21.15.189:0/2426811041 shutdown_connections 2024-09-16T06:41:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- 172.21.15.189:0/2426811041 >> 172.21.15.189:0/2426811041 conn(0x7fc50809fe70 msgr2=0x7fc5080a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- 172.21.15.189:0/2426811041 shutdown_connections 2024-09-16T06:41:24.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- 172.21.15.189:0/2426811041 wait complete. 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: build_initial_monmap 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: monmap: 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:24.880 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 Processor -- start 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 1 -- start start 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: init 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc5080993a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.864+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc50809c840 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc508139550 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50813c290 con 0x7fc50809c840 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50809a380 con 0x7fc508139550 2024-09-16T06:41:24.881 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508071440 con 0x7fc5080993a0 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient(hunting): authenticate will time out at 1037.384081s 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc506d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc508139550 0x7fc5081d07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51386/0 (socket says 172.21.15.189:51386) 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc505d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc50809c840 legacy=0x7fc5081cd0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc506d9d700 1 -- 172.21.15.189:0/2640568334 learned_addr learned my addr 172.21.15.189:0/2640568334 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508641862 0 0) 0x7fc50809a380 con 0x7fc508139550 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): my global_id is 134251 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e80035d0 con 0x7fc508139550 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749881654 0 0) 0x7fc4e80035d0 con 0x7fc508139550 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc50809a380 con 0x7fc508139550 2024-09-16T06:41:24.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622935330 0 0) 0x7fc508071440 con 0x7fc5080993a0 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): my global_id is 124541 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e80035d0 con 0x7fc5080993a0 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5000044c0 con 0x7fc508139550 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.883 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 367269096 0 0) 0x7fc50809a380 con 0x7fc508139550 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 >> v1:172.21.15.76:6790/0 conn(0x7fc5080993a0 legacy=0x7fc5081c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 >> v1:172.21.15.76:6789/0 conn(0x7fc50809c840 legacy=0x7fc5081cd0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 monclient: found mon.b 2024-09-16T06:41:24.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080a2ad0 con 0x7fc508139550 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 5 monclient: authenticate success, global_id 134251 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _renew_subs 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc508071020 con 0x7fc508139550 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _renew_subs 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc508070430 con 0x7fc508139550 2024-09-16T06:41:24.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc500004760 con 0x7fc508139550 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc504d99700 4 set_mon_vals no callback set 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc500004af0 con 0x7fc508139550 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 10 monclient: dump: 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:24.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec005100 con 0x7fc508139550 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc500004cd0 con 0x7fc508139550 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7fc500043b20 con 0x7fc508139550 2024-09-16T06:41:24.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.868+0000 7fc50659c700 10 monclient: get_auth_request con 0x7fc4e8036080 auth_method 0 2024-09-16T06:41:24.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.880+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc500005520 con 0x7fc508139550 2024-09-16T06:41:24.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.880+0000 7fc4f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:24.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:24.880+0000 7fc4f6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.036+0000 7fc50759e700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.036+0000 7fc50759e700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.036+0000 7fc50759e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.036+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc4ec005bc0 con 0x7fc508139550 2024-09-16T06:41:25.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.040+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fc500012f60 con 0x7fc508139550 2024-09-16T06:41:25.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.040+0000 7fc4f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:25.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.040+0000 7fc4f6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc50759e700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc50759e700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc50759e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc4ec005930 con 0x7fc508139550 2024-09-16T06:41:25.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc4f6ffd700 1 -- 172.21.15.189:0/2640568334 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v74) v1 ==== 56+0+1474 (unknown 1366924582 0 3234054088) 0x7fc5000146e0 con 0x7fc508139550 2024-09-16T06:41:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc4f6ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:25.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.132+0000 7fc4f6ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:25.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc4e8036080 legacy=0x7fc4e8038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 10 monclient: shutdown 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 >> v1:172.21.15.189:6789/0 conn(0x7fc508139550 legacy=0x7fc5081d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 shutdown_connections 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 >> 172.21.15.189:0/2640568334 conn(0x7fc50809fe70 msgr2=0x7fc50813ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 shutdown_connections 2024-09-16T06:41:25.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.136+0000 7fc50759e700 1 -- 172.21.15.189:0/2640568334 wait complete. 2024-09-16T06:41:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-16T06:41:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:25.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host1 foo 2024-09-16T06:41:25.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: get_monmap_and_config 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: build_initial_monmap 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: monmap: 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:25.245716+0000 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:25.245716+0000 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:25.245 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 1 Processor -- start 2024-09-16T06:41:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 1 -- start start 2024-09-16T06:41:25.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: init 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-a con 0x7f9d58132dc0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-b con 0x7f9d5809f330 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-c con 0x7f9d581338b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5809e5e0 con 0x7f9d58132dc0 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5809e8c0 con 0x7f9d581338b0 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5809dd80 con 0x7f9d5809f330 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.240+0000 7f9d5c880700 10 monclient(hunting): authenticate will time out at 1037.759956s 2024-09-16T06:41:25.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d57fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d581338b0 0x7f9d5813b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51390/0 (socket says 172.21.15.189:51390) 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d57fff700 1 -- 172.21.15.189:0/3692148167 learned_addr learned my addr 172.21.15.189:0/3692148167 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d577fe700 1 -- 172.21.15.189:0/3692148167 >> v1:172.21.15.76:6789/0 conn(0x7f9d58132dc0 legacy=0x7f9d580a1750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320279696 0 0) 0x7f9d5809e8c0 con 0x7f9d581338b0 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): my global_id is 134254 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d400035d0 con 0x7f9d581338b0 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396991230 0 0) 0x7f9d5809dd80 con 0x7f9d5809f330 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient(hunting): my global_id is 124544 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5809e8c0 con 0x7f9d5809f330 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139865886 0 0) 0x7f9d400035d0 con 0x7f9d581338b0 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 >> v1:172.21.15.76:6790/0 conn(0x7f9d5809f330 legacy=0x7f9d581331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 >> v1:172.21.15.76:6789/0 conn(0x7f9d58132dc0 legacy=0x7f9d580a1750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5813fa60 con 0x7f9d581338b0 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:25.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d50001990 con 0x7f9d581338b0 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: dump: 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.254 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 5 monclient: authenticate success, global_id 134254 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d50001c70 con 0x7f9d581338b0 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 1 -- 172.21.15.189:0/3692148167 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d50002000 con 0x7f9d581338b0 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d567fc700 10 monclient: dump: 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.255 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 4 set_mon_vals no callback set 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 10 monclient: shutdown 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:25.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- 172.21.15.189:0/3692148167 >> v1:172.21.15.189:6789/0 conn(0x7f9d581338b0 legacy=0x7f9d5813b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- 172.21.15.189:0/3692148167 shutdown_connections 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- 172.21.15.189:0/3692148167 >> 172.21.15.189:0/3692148167 conn(0x7f9d580a2350 msgr2=0x7f9d580a2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- 172.21.15.189:0/3692148167 shutdown_connections 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- 172.21.15.189:0/3692148167 wait complete. 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 10 monclient: build_initial_monmap 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 10 monclient: monmap: 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:25.257 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:25.258 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:25.258 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 Processor -- start 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 1 -- start start 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.244+0000 7f9d5c880700 10 monclient: init 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-a con 0x7f9d5809f330 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-c con 0x7f9d58132dc0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient(hunting): picked mon.noname-b con 0x7f9d581338b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5809ab90 con 0x7f9d5809f330 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5812bd00 con 0x7f9d581338b0 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5809c620 con 0x7f9d58132dc0 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:25.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient(hunting): authenticate will time out at 1037.764869s 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d577fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9d5809f330 legacy=0x7f9d5812aee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d57fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d581338b0 0x7f9d581cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51392/0 (socket says 172.21.15.189:51392) 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d57fff700 1 -- 172.21.15.189:0/4260084688 learned_addr learned my addr 172.21.15.189:0/4260084688 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561380136 0 0) 0x7f9d5812bd00 con 0x7f9d581338b0 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): my global_id is 134257 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d440035d0 con 0x7f9d581338b0 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228420223 0 0) 0x7f9d5809c620 con 0x7f9d58132dc0 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): my global_id is 124547 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5812bd00 con 0x7f9d58132dc0 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262938249 0 0) 0x7f9d440035d0 con 0x7f9d581338b0 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d5809c620 con 0x7f9d581338b0 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d50001950 con 0x7f9d581338b0 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): dump: 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.264 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849385855 0 0) 0x7f9d5812bd00 con 0x7f9d58132dc0 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d440035d0 con 0x7f9d58132dc0 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d4c0018b0 con 0x7f9d58132dc0 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient(hunting): dump: 2024-09-16T06:41:25.265 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3373821395 0 0) 0x7f9d5809c620 con 0x7f9d581338b0 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 >> v1:172.21.15.76:6790/0 conn(0x7f9d58132dc0 legacy=0x7f9d5812b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 >> v1:172.21.15.76:6789/0 conn(0x7f9d5809f330 legacy=0x7f9d5812aee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:25.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 monclient: found mon.b 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5809a030 con 0x7f9d581338b0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: _finish_auth 0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 5 monclient: authenticate success, global_id 134257 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _renew_subs 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5809ba20 con 0x7f9d581338b0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _renew_subs 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5812ca50 con 0x7f9d581338b0 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d50005020 con 0x7f9d581338b0 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d50005330 con 0x7f9d581338b0 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d55ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 10 monclient: dump: 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.268 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d5000fda0 con 0x7f9d581338b0 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d5c880700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d38005100 con 0x7f9d581338b0 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7f9d50043dd0 con 0x7f9d581338b0 2024-09-16T06:41:25.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.248+0000 7f9d577fe700 10 monclient: get_auth_request con 0x7f9d440362d0 auth_method 0 2024-09-16T06:41:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.256+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d50014d80 con 0x7f9d581338b0 2024-09-16T06:41:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.256+0000 7f9d3ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.256+0000 7f9d3ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d5c880700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d5c880700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d5c880700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d5c880700 1 -- 172.21.15.189:0/4260084688 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f9d38005930 con 0x7f9d581338b0 2024-09-16T06:41:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d3ffff700 1 -- 172.21.15.189:0/4260084688 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v74) v1 ==== 120+0+0 (unknown 2594394665 0 0) 0x7f9d500131f0 con 0x7f9d581338b0 2024-09-16T06:41:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d3ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:25.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.420+0000 7f9d3ffff700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T06:41:25.427 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T06:41:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:25.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: get_monmap_and_config 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: build_initial_monmap 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: monmap: 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:25.531052+0000 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:25.531052+0000 2024-09-16T06:41:25.530 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:25.531 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:25.531 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 1 Processor -- start 2024-09-16T06:41:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 1 -- start start 2024-09-16T06:41:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: init 2024-09-16T06:41:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:25.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.524+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-a con 0x7f367409f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-b con 0x7f3674099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-c con 0x7f36741396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367409f520 con 0x7f367409f140 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367409bc10 con 0x7f36741396a0 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367409b810 con 0x7f3674099700 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient(hunting): authenticate will time out at 1038.044124s 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3673fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f367409f140 legacy=0x7f367409a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3678866700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f36741396a0 0x7f367413ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51394/0 (socket says 172.21.15.189:51394) 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3678866700 1 -- 172.21.15.189:0/246832622 learned_addr learned my addr 172.21.15.189:0/246832622 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:25.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418571891 0 0) 0x7f367409bc10 con 0x7f36741396a0 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): my global_id is 134260 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36600035d0 con 0x7f36741396a0 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614236775 0 0) 0x7f367409b810 con 0x7f3674099700 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient(hunting): my global_id is 124550 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367409bc10 con 0x7f3674099700 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086849344 0 0) 0x7f36600035d0 con 0x7f36741396a0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 >> v1:172.21.15.76:6790/0 conn(0x7f3674099700 legacy=0x7f367409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 >> v1:172.21.15.76:6789/0 conn(0x7f367409f140 legacy=0x7f367409a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 monclient: found mon.noname-c 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367409ceb0 con 0x7f36741396a0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f366c001610 con 0x7f36741396a0 2024-09-16T06:41:25.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: dump: 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 5 monclient: authenticate success, global_id 134260 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f366c0018f0 con 0x7f36741396a0 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 1 -- 172.21.15.189:0/246832622 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f366c001c80 con 0x7f36741396a0 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 4 set_mon_vals no callback set 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f3672ffd700 10 monclient: dump: 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.539 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient: shutdown 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- 172.21.15.189:0/246832622 >> v1:172.21.15.189:6789/0 conn(0x7f36741396a0 legacy=0x7f367413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- 172.21.15.189:0/246832622 shutdown_connections 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- 172.21.15.189:0/246832622 >> 172.21.15.189:0/246832622 conn(0x7f36740a03e0 msgr2=0x7f36740a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- 172.21.15.189:0/246832622 shutdown_connections 2024-09-16T06:41:25.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 1 -- 172.21.15.189:0/246832622 wait complete. 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient: build_initial_monmap 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.528+0000 7f367a2c9700 10 monclient: monmap: 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:25.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 Processor -- start 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- start start 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: init 2024-09-16T06:41:25.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-a con 0x7f3674099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-c con 0x7f367409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient(hunting): picked mon.noname-b con 0x7f36741396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36740907f0 con 0x7f3674099700 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367409a6e0 con 0x7f36741396a0 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3673fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3674099700 legacy=0x7f3674096110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36740911f0 con 0x7f367409f140 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient(hunting): authenticate will time out at 1038.049365s 2024-09-16T06:41:25.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3678866700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f36741396a0 0x7f36741cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51406/0 (socket says 172.21.15.189:51406) 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3678866700 1 -- 172.21.15.189:0/2258477764 learned_addr learned my addr 172.21.15.189:0/2258477764 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529417175 0 0) 0x7f367409a6e0 con 0x7f36741396a0 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): my global_id is 134263 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36540035d0 con 0x7f36741396a0 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202205841 0 0) 0x7f36740911f0 con 0x7f367409f140 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): my global_id is 124553 2024-09-16T06:41:25.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367409a6e0 con 0x7f367409f140 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925270135 0 0) 0x7f36540035d0 con 0x7f36741396a0 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36740911f0 con 0x7f36741396a0 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f366c0015d0 con 0x7f36741396a0 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.550 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576190004 0 0) 0x7f367409a6e0 con 0x7f367409f140 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36540035d0 con 0x7f367409f140 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f365c0018b0 con 0x7f367409f140 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.551 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1879201555 0 0) 0x7f36740911f0 con 0x7f36741396a0 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 >> v1:172.21.15.76:6790/0 conn(0x7f367409f140 legacy=0x7f36741cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 >> v1:172.21.15.76:6789/0 conn(0x7f3674099700 legacy=0x7f3674096110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 monclient: found mon.b 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:25.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36741394e0 con 0x7f36741396a0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 5 monclient: authenticate success, global_id 134263 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _renew_subs 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3674092bc0 con 0x7f36741396a0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _renew_subs 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674071480 con 0x7f36741396a0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f366c004ca0 con 0x7f36741396a0 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:25.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f366c005030 con 0x7f36741396a0 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f36727fc700 4 set_mon_vals no callback set 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.532+0000 7f3670ff9700 10 monclient: dump: 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:25.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f367a2c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f367a2c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f367a2c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3658005100 con 0x7f36741396a0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f366c005210 con 0x7f36741396a0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 5385+0+0 (unknown 4290543350 0 0) 0x7f366c043a60 con 0x7f36741396a0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.536+0000 7f3673fff700 10 monclient: get_auth_request con 0x7f36540362c0 auth_method 0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.544+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f366c014920 con 0x7f36741396a0 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.544+0000 7f3670ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:25.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.544+0000 7f3670ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:25.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.716+0000 7f367a2c9700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.716+0000 7f367a2c9700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.716+0000 7f367a2c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:25.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:25.716+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f3658005930 con 0x7f36741396a0 2024-09-16T06:41:26.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.032+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v75) v1 ==== 147+0+0 (unknown 167261303 0 0) 0x7f366c012fc0 con 0x7f36741396a0 2024-09-16T06:41:26.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.032+0000 7f3670ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.032+0000 7f3670ff9700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T06:41:26.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f367a2c9700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f367a2c9700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f367a2c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f36580053b0 con 0x7f36741396a0 2024-09-16T06:41:26.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f3670ff9700 1 -- 172.21.15.189:0/2258477764 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (unknown 670102375 0 0) 0x7f366c014740 con 0x7f36741396a0 2024-09-16T06:41:26.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f3670ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.148+0000 7f3670ff9700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T06:41:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 >> v1:172.21.15.189:6812/3098810674 conn(0x7f36540362c0 legacy=0x7f3654038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 10 monclient: shutdown 2024-09-16T06:41:26.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 >> v1:172.21.15.189:6789/0 conn(0x7f36741396a0 legacy=0x7f36741cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 shutdown_connections 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 >> 172.21.15.189:0/2258477764 conn(0x7f36740a03e0 msgr2=0x7f36740a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 shutdown_connections 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.152+0000 7f367a2c9700 1 -- 172.21.15.189:0/2258477764 wait complete. 2024-09-16T06:41:26.158 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T06:41:26.167 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T06:41:26.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: get_monmap_and_config 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: build_initial_monmap 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: monmap: 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:26.277659+0000 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:26.277659+0000 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:26.277 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:26.278 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:26.278 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 1 Processor -- start 2024-09-16T06:41:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 1 -- start start 2024-09-16T06:41:26.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: init 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-c con 0x7f00cc09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-b con 0x7f00cc09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-a con 0x7f00cc138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc09cca0 con 0x7f00cc138da0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc09f750 con 0x7f00cc09bb00 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc097620 con 0x7f00cc09c5f0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient(hunting): authenticate will time out at 1038.790634s 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00cad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00cc138da0 legacy=0x7f00cc13b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:26.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00ca59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00cc09bb00 0x7f00cc09bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51420/0 (socket says 172.21.15.189:51420) 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00ca59c700 1 -- 172.21.15.189:0/1327568310 learned_addr learned my addr 172.21.15.189:0/1327568310 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678090197 0 0) 0x7f00cc09f750 con 0x7f00cc09bb00 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient(hunting): my global_id is 134266 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b40035d0 con 0x7f00cc09bb00 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191008263 0 0) 0x7f00b40035d0 con 0x7f00cc09bb00 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 >> v1:172.21.15.76:6790/0 conn(0x7f00cc09c5f0 legacy=0x7f00cc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 >> v1:172.21.15.76:6789/0 conn(0x7f00cc138da0 legacy=0x7f00cc13b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 monclient: found mon.noname-c 2024-09-16T06:41:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:26.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc097900 con 0x7f00cc09bb00 2024-09-16T06:41:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: _finish_auth 0 2024-09-16T06:41:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:26.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 5 monclient: authenticate success, global_id 134266 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00c0001610 con 0x7f00cc09bb00 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: dump: 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.289 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00c00018b0 con 0x7f00cc09bb00 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00d0f55700 4 set_mon_vals no callback set 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 1 -- 172.21.15.189:0/1327568310 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00c0001c40 con 0x7f00cc09bb00 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:26.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.272+0000 7f00c959a700 10 monclient: dump: 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: shutdown 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:26.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1327568310 >> v1:172.21.15.189:6789/0 conn(0x7f00cc09bb00 legacy=0x7f00cc09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1327568310 shutdown_connections 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1327568310 >> 172.21.15.189:0/1327568310 conn(0x7f00cc0a03e0 msgr2=0x7f00cc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1327568310 shutdown_connections 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1327568310 wait complete. 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: build_initial_monmap 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: monmap: 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:26.293 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 Processor -- start 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- start start 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: init 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:26.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-c con 0x7f00cc09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-b con 0x7f00cc09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient(hunting): picked mon.noname-a con 0x7f00cc138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc094480 con 0x7f00cc138da0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc094760 con 0x7f00cc09c5f0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc09a6e0 con 0x7f00cc09bb00 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient(hunting): authenticate will time out at 1038.793958s 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00cad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00cc138da0 legacy=0x7f00cc1d2930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00c9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00cc09c5f0 0x7f00cc1cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51424/0 (socket says 172.21.15.189:51424) 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00c9d9b700 1 -- 172.21.15.189:0/1794091903 learned_addr learned my addr 172.21.15.189:0/1794091903 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:26.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97358483 0 0) 0x7f00cc094760 con 0x7f00cc09c5f0 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): my global_id is 134269 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac0035d0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023203596 0 0) 0x7f00cc09a6e0 con 0x7f00cc09bb00 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): my global_id is 124559 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00cc094760 con 0x7f00cc09bb00 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307650965 0 0) 0x7f00ac0035d0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00cc09a6e0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00bc0019b0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): dump: 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2598264101 0 0) 0x7f00cc09a6e0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 >> v1:172.21.15.76:6790/0 conn(0x7f00cc09bb00 legacy=0x7f00cc1ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 >> v1:172.21.15.76:6789/0 conn(0x7f00cc138da0 legacy=0x7f00cc1d2930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 monclient: found mon.b 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc0a2d00 con 0x7f00cc09c5f0 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 5 monclient: authenticate success, global_id 134269 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _renew_subs 2024-09-16T06:41:26.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc099380 con 0x7f00cc09c5f0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _renew_subs 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc094a40 con 0x7f00cc09c5f0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00bc003970 con 0x7f00cc09c5f0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00bc001d80 con 0x7f00cc09c5f0 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00c8d99700 4 set_mon_vals no callback set 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:26.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.276+0000 7f00baffd700 10 monclient: dump: 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f00bc003d80 con 0x7f00cc09c5f0 2024-09-16T06:41:26.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00d0f55700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00d0f55700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00d0f55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 5377+0+0 (unknown 2959285018 0 0) 0x7f00bc043b10 con 0x7f00cc09c5f0 2024-09-16T06:41:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc06f250 con 0x7f00cc09c5f0 2024-09-16T06:41:26.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.280+0000 7f00ca59c700 10 monclient: get_auth_request con 0x7f00ac0361b0 auth_method 0 2024-09-16T06:41:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.288+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f00bc014770 con 0x7f00cc09c5f0 2024-09-16T06:41:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.288+0000 7f00baffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.288+0000 7f00baffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:26.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00d0f55700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00d0f55700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00d0f55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f00cc092730 con 0x7f00cc09c5f0 2024-09-16T06:41:26.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (unknown 670102375 0 0) 0x7f00bc012e10 con 0x7f00cc09c5f0 2024-09-16T06:41:26.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00baffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.456+0000 7f00baffd700 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-16T06:41:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.556+0000 7f00d0f55700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.556+0000 7f00d0f55700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.556+0000 7f00d0f55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.556+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f00cc0008d0 con 0x7f00cc09c5f0 2024-09-16T06:41:26.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00baffd700 1 -- 172.21.15.189:0/1794091903 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v75) v1 ==== 141+0+0 (unknown 670102375 0 0) 0x7f00bc014590 con 0x7f00cc09c5f0 2024-09-16T06:41:26.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00baffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T06:41:26.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00baffd700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T06:41:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00ac0361b0 legacy=0x7f00ac038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00d0f55700 10 monclient: shutdown 2024-09-16T06:41:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00d0f55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:26.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 >> v1:172.21.15.189:6789/0 conn(0x7f00cc09c5f0 legacy=0x7f00cc1cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.560+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 shutdown_connections 2024-09-16T06:41:26.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.564+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 >> 172.21.15.189:0/1794091903 conn(0x7f00cc0a03e0 msgr2=0x7f00cc071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:26.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.564+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 shutdown_connections 2024-09-16T06:41:26.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.564+0000 7f00d0f55700 1 -- 172.21.15.189:0/1794091903 wait complete. 2024-09-16T06:41:26.569 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T06:41:26.578 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:26.578 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T06:41:26.578 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T06:41:26.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.664+0000 7f26379e6700 10 monclient: get_monmap_and_config 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.664+0000 7f26379e6700 10 monclient: build_initial_monmap 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.664+0000 7f26379e6700 10 monclient: monmap: 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:26.671742+0000 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:26.671742+0000 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:26.671 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 Processor -- start 2024-09-16T06:41:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- start start 2024-09-16T06:41:26.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient: init 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-c con 0x7f26300a28c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-b con 0x7f26300a3c10 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-a con 0x7f263013bc20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26300a33b0 con 0x7f263013bc20 2024-09-16T06:41:26.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263009a8c0 con 0x7f26300a28c0 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2630099f90 con 0x7f26300a3c10 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient(hunting): authenticate will time out at 1039.186158s 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2635782700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f26300a28c0 0x7f26300a2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51438/0 (socket says 172.21.15.189:51438) 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2635782700 1 -- 172.21.15.189:0/3478205952 learned_addr learned my addr 172.21.15.189:0/3478205952 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2635f83700 1 -- 172.21.15.189:0/3478205952 >> v1:172.21.15.76:6789/0 conn(0x7f263013bc20 legacy=0x7f263013e010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783841447 0 0) 0x7f263009a8c0 con 0x7f26300a28c0 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient(hunting): my global_id is 134272 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c0035d0 con 0x7f26300a28c0 2024-09-16T06:41:26.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127181986 0 0) 0x7f261c0035d0 con 0x7f26300a28c0 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 >> v1:172.21.15.76:6790/0 conn(0x7f26300a3c10 legacy=0x7f263013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 >> v1:172.21.15.76:6789/0 conn(0x7f263013bc20 legacy=0x7f263013e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 monclient: found mon.noname-c 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263009a270 con 0x7f26300a28c0 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 5 monclient: authenticate success, global_id 134272 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:26.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f262c001610 con 0x7f26300a28c0 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: dump: 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:26.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f262c001930 con 0x7f26300a28c0 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 4 set_mon_vals no callback set 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 1 -- 172.21.15.189:0/3478205952 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f262c001d00 con 0x7f26300a28c0 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f2627fff700 10 monclient: dump: 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 10 monclient: shutdown 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- 172.21.15.189:0/3478205952 >> v1:172.21.15.189:6789/0 conn(0x7f26300a28c0 legacy=0x7f26300a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- 172.21.15.189:0/3478205952 shutdown_connections 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- 172.21.15.189:0/3478205952 >> 172.21.15.189:0/3478205952 conn(0x7f26300a7e70 msgr2=0x7f26300a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- 172.21.15.189:0/3478205952 shutdown_connections 2024-09-16T06:41:26.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.668+0000 7f26379e6700 1 -- 172.21.15.189:0/3478205952 wait complete. 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: build_initial_monmap 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: monmap: 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:26.684 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 Processor -- start 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 -- start start 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: init 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-c con 0x7f26300a28c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-a con 0x7f26300a3c10 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient(hunting): picked mon.noname-b con 0x7f263013bc20 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263006bdb0 con 0x7f26300a3c10 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263011f890 con 0x7f263013bc20 2024-09-16T06:41:26.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2634f81700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f26300a3c10 legacy=0x7f26301cf130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2630120850 con 0x7f26300a28c0 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient(hunting): authenticate will time out at 1039.190608s 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2635f83700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f263013bc20 0x7f26301d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51442/0 (socket says 172.21.15.189:51442) 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2635f83700 1 -- 172.21.15.189:0/1047994241 learned_addr learned my addr 172.21.15.189:0/1047994241 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292664221 0 0) 0x7f2628003e70 con 0x7f263013bc20 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): my global_id is 134275 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26140035d0 con 0x7f263013bc20 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289874340 0 0) 0x7f2630120850 con 0x7f26300a28c0 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:26.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): my global_id is 124565 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628003e70 con 0x7f26300a28c0 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 8273076 0 0) 0x7f26140035d0 con 0x7f263013bc20 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2630120850 con 0x7f263013bc20 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26280044e0 con 0x7f263013bc20 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.687 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3542069815 0 0) 0x7f2630120850 con 0x7f263013bc20 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 >> v1:172.21.15.76:6790/0 conn(0x7f26300a28c0 legacy=0x7f26301353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 >> v1:172.21.15.76:6789/0 conn(0x7f26300a3c10 legacy=0x7f26301cf130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 monclient: found mon.b 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263011f610 con 0x7f263013bc20 2024-09-16T06:41:26.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f2625ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 5 monclient: authenticate success, global_id 134275 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: _renew_subs 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.672+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2630120d80 con 0x7f263013bc20 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 10 monclient: _renew_subs 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f263013b9f0 con 0x7f263013bc20 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26280047c0 con 0x7f263013bc20 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26277fe700 4 set_mon_vals no callback set 2024-09-16T06:41:26.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2628004b50 con 0x7f263013bc20 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 10 monclient: dump: 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:26.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2610005100 con 0x7f263013bc20 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2628004d30 con 0x7f263013bc20 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 5377+0+0 (unknown 2959285018 0 0) 0x7f2628043c70 con 0x7f263013bc20 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.676+0000 7f2635782700 10 monclient: get_auth_request con 0x7f2614036180 auth_method 0 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.684+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2628005530 con 0x7f263013bc20 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.684+0000 7f2625ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:26.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.684+0000 7f2625ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:26.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f26379e6700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:26.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f26379e6700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:26.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f26379e6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f2610005bc0 con 0x7f263013bc20 2024-09-16T06:41:26.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (unknown 3476242654 0 3778217129) 0x7f2628012f70 con 0x7f263013bc20 2024-09-16T06:41:26.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f2625ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:26.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.840+0000 7f2625ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:26.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:26.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:26.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:26.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f2610005680 con 0x7f263013bc20 2024-09-16T06:41:26.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f2625ffb700 1 -- 172.21.15.189:0/1047994241 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) v1 ==== 56+0+1340 (unknown 3476242654 0 3778217129) 0x7f26280146f0 con 0x7f263013bc20 2024-09-16T06:41:26.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f2625ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:26.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f2625ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2614036180 legacy=0x7f2614038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 10 monclient: shutdown 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 >> v1:172.21.15.189:6789/0 conn(0x7f263013bc20 legacy=0x7f26301d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 shutdown_connections 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 >> 172.21.15.189:0/1047994241 conn(0x7f26300a7e70 msgr2=0x7f26300a5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:26.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 shutdown_connections 2024-09-16T06:41:26.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:26.932+0000 7f26379e6700 1 -- 172.21.15.189:0/1047994241 wait complete. 2024-09-16T06:41:26.949 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T06:41:26.949 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:26.949 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: get_monmap_and_config 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: monmap: 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:27.054831+0000 2024-09-16T06:41:27.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:27.054831+0000 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:27.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 1 Processor -- start 2024-09-16T06:41:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 1 -- start start 2024-09-16T06:41:27.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: init 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-c con 0x7fb89c09f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-b con 0x7fb89c099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-a con 0x7fb89c1396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c09f520 con 0x7fb89c1396a0 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c09bc10 con 0x7fb89c09f140 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c09b810 con 0x7fb89c099700 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a46fe700 10 monclient(hunting): authenticate will time out at 1039.567747s 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a249a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb89c09f140 0x7fb89c09a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51452/0 (socket says 172.21.15.189:51452) 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a249a700 1 -- 172.21.15.189:0/435421423 learned_addr learned my addr 172.21.15.189:0/435421423 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.048+0000 7fb8a2c9b700 1 -- 172.21.15.189:0/435421423 >> v1:172.21.15.76:6789/0 conn(0x7fb89c1396a0 legacy=0x7fb89c13ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335425543 0 0) 0x7fb89c09bc10 con 0x7fb89c09f140 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): my global_id is 134278 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8880035d0 con 0x7fb89c09f140 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667867800 0 0) 0x7fb89c09b810 con 0x7fb89c099700 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient(hunting): my global_id is 124568 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb89c09bc10 con 0x7fb89c099700 2024-09-16T06:41:27.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977783373 0 0) 0x7fb8880035d0 con 0x7fb89c09f140 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 >> v1:172.21.15.76:6790/0 conn(0x7fb89c099700 legacy=0x7fb89c09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 >> v1:172.21.15.76:6789/0 conn(0x7fb89c1396a0 legacy=0x7fb89c13ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 monclient: found mon.noname-c 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c09ceb0 con 0x7fb89c09f140 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb894001610 con 0x7fb89c09f140 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 5 monclient: authenticate success, global_id 134278 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:27.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: dump: 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:27.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8940018f0 con 0x7fb89c09f140 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 4 set_mon_vals no callback set 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 1 -- 172.21.15.189:0/435421423 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb894001c80 con 0x7fb89c09f140 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a1498700 10 monclient: dump: 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 10 monclient: shutdown 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- 172.21.15.189:0/435421423 >> v1:172.21.15.189:6789/0 conn(0x7fb89c09f140 legacy=0x7fb89c09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- 172.21.15.189:0/435421423 shutdown_connections 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- 172.21.15.189:0/435421423 >> 172.21.15.189:0/435421423 conn(0x7fb89c0a03e0 msgr2=0x7fb89c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- 172.21.15.189:0/435421423 shutdown_connections 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- 172.21.15.189:0/435421423 wait complete. 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 10 monclient: monmap: 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.064 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 Processor -- start 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 1 -- start start 2024-09-16T06:41:27.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.052+0000 7fb8a46fe700 10 monclient: init 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-a con 0x7fb89c099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-c con 0x7fb89c09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient(hunting): picked mon.noname-b con 0x7fb89c1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c090740 con 0x7fb89c099700 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c09a6e0 con 0x7fb89c1396a0 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c0a2dd0 con 0x7fb89c09f140 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient(hunting): authenticate will time out at 1039.572149s 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a249a700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb89c099700 legacy=0x7fb89c096060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a2c9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb89c1396a0 0x7fb89c1cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51458/0 (socket says 172.21.15.189:51458) 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a2c9b700 1 -- 172.21.15.189:0/2229217612 learned_addr learned my addr 172.21.15.189:0/2229217612 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339543404 0 0) 0x7fb89c09a6e0 con 0x7fb89c1396a0 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): my global_id is 134281 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87c0035d0 con 0x7fb89c1396a0 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635179709 0 0) 0x7fb87c0035d0 con 0x7fb89c1396a0 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb89c09a6e0 con 0x7fb89c1396a0 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb88c004170 con 0x7fb89c1396a0 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:27.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): dump: 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2722256153 0 0) 0x7fb89c09a6e0 con 0x7fb89c1396a0 2024-09-16T06:41:27.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 >> v1:172.21.15.76:6790/0 conn(0x7fb89c09f140 legacy=0x7fb89c1cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 >> v1:172.21.15.76:6789/0 conn(0x7fb89c099700 legacy=0x7fb89c096060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 monclient: found mon.b 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c091140 con 0x7fb89c1396a0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 5 monclient: authenticate success, global_id 134281 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _renew_subs 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89c092830 con 0x7fb89c1396a0 2024-09-16T06:41:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _renew_subs 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c0714a0 con 0x7fb89c1396a0 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb88c004490 con 0x7fb89c1396a0 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb88c004820 con 0x7fb89c1396a0 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a0c97700 4 set_mon_vals no callback set 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 10 monclient: dump: 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb88c004a20 con 0x7fb89c1396a0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a46fe700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb880005100 con 0x7fb89c1396a0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 5377+0+0 (unknown 2959285018 0 0) 0x7fb88c043990 con 0x7fb89c1396a0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.056+0000 7fb8a249a700 10 monclient: get_auth_request con 0x7fb87c0360a0 auth_method 0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.060+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb88c014820 con 0x7fb89c1396a0 2024-09-16T06:41:27.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.060+0000 7fb89affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.060+0000 7fb89affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:27.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb8a46fe700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:27.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb8a46fe700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:27.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb8a46fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb8a46fe700 1 -- 172.21.15.189:0/2229217612 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fb8800053b0 con 0x7fb89c1396a0 2024-09-16T06:41:27.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb89affd700 1 -- 172.21.15.189:0/2229217612 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v75) v1 ==== 99+0+0 (unknown 943881153 0 0) 0x7fb88c012c90 con 0x7fb89c1396a0 2024-09-16T06:41:27.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb89affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:27.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.236+0000 7fb89affd700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T06:41:27.244 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T06:41:27.247 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:27.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T06:41:27.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:27.248 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T06:41:27.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: get_monmap_and_config 2024-09-16T06:41:27.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: monmap: 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:27.354485+0000 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:27.354485+0000 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:27.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 1 Processor -- start 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 1 -- start start 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: init 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-a con 0x7fe0f809bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-c con 0x7fe0f809c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-b con 0x7fe0f8138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f809cca0 con 0x7fe0f809bb00 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f809f750 con 0x7fe0f809c5f0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f8097620 con 0x7fe0f8138da0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.348+0000 7fe0ff396700 10 monclient(hunting): authenticate will time out at 1039.867902s 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0fd132700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe0f809bb00 legacy=0x7fe0f809bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0fc931700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe0f809c5f0 0x7fe0f8137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51474/0 (socket says 172.21.15.189:51474) 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0fc931700 1 -- 172.21.15.189:0/333661858 learned_addr learned my addr 172.21.15.189:0/333661858 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265982464 0 0) 0x7fe0f809f750 con 0x7fe0f809c5f0 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): my global_id is 134284 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e40035d0 con 0x7fe0f809c5f0 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843159634 0 0) 0x7fe0f8097620 con 0x7fe0f8138da0 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient(hunting): my global_id is 124571 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f809f750 con 0x7fe0f8138da0 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1318240667 0 0) 0x7fe0e40035d0 con 0x7fe0f809c5f0 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 >> v1:172.21.15.76:6790/0 conn(0x7fe0f8138da0 legacy=0x7fe0f813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 >> v1:172.21.15.76:6789/0 conn(0x7fe0f809bb00 legacy=0x7fe0f809bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 monclient: found mon.noname-c 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8097900 con 0x7fe0f809c5f0 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 5 monclient: authenticate success, global_id 134284 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe0e8001610 con 0x7fe0f809c5f0 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:27.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: dump: 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e80018f0 con 0x7fe0f809c5f0 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 1 -- 172.21.15.189:0/333661858 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe0e8001c80 con 0x7fe0f809c5f0 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 4 set_mon_vals no callback set 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0effff700 10 monclient: dump: 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 10 monclient: shutdown 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- 172.21.15.189:0/333661858 >> v1:172.21.15.189:6789/0 conn(0x7fe0f809c5f0 legacy=0x7fe0f8137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- 172.21.15.189:0/333661858 shutdown_connections 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- 172.21.15.189:0/333661858 >> 172.21.15.189:0/333661858 conn(0x7fe0f80a03e0 msgr2=0x7fe0f80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:27.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- 172.21.15.189:0/333661858 shutdown_connections 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- 172.21.15.189:0/333661858 wait complete. 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 10 monclient: monmap: 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:27.365 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 Processor -- start 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 1 -- start start 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.352+0000 7fe0ff396700 10 monclient: init 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-b con 0x7fe0f809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-c con 0x7fe0f809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient(hunting): picked mon.noname-a con 0x7fe0f8138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f812a7d0 con 0x7fe0f8138da0 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f80993f0 con 0x7fe0f809bb00 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f80a2c90 con 0x7fe0f809c5f0 2024-09-16T06:41:27.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient(hunting): authenticate will time out at 1039.872261s 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0fd132700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe0f809bb00 0x7fe0f81ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51486/0 (socket says 172.21.15.189:51486) 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0fd132700 1 -- 172.21.15.189:0/348386722 learned_addr learned my addr 172.21.15.189:0/348386722 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0fd933700 1 -- 172.21.15.189:0/348386722 >> v1:172.21.15.76:6789/0 conn(0x7fe0f8138da0 legacy=0x7fe0f81d69e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130355701 0 0) 0x7fe0f80993f0 con 0x7fe0f809bb00 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): my global_id is 134287 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0dc0035d0 con 0x7fe0f809bb00 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619801753 0 0) 0x7fe0dc0035d0 con 0x7fe0f809bb00 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f80993f0 con 0x7fe0f809bb00 2024-09-16T06:41:27.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe0f4004280 con 0x7fe0f809bb00 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): dump: 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083226169 0 0) 0x7fe0f80a2c90 con 0x7fe0f809c5f0 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient(hunting): my global_id is 124574 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0dc0035d0 con 0x7fe0f809c5f0 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 193265016 0 0) 0x7fe0f80993f0 con 0x7fe0f809bb00 2024-09-16T06:41:27.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 >> v1:172.21.15.76:6790/0 conn(0x7fe0f809c5f0 legacy=0x7fe0f81d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 >> v1:172.21.15.76:6789/0 conn(0x7fe0f8138da0 legacy=0x7fe0f81d69e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 monclient: found mon.b 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f809a6e0 con 0x7fe0f809bb00 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 5 monclient: authenticate success, global_id 134287 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _renew_subs 2024-09-16T06:41:27.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f812a4f0 con 0x7fe0f809bb00 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _renew_subs 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8129040 con 0x7fe0f809bb00 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0f40045a0 con 0x7fe0f809bb00 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe0f4004930 con 0x7fe0f809bb00 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 10 monclient: dump: 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ef7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe0f4004b10 con 0x7fe0f809bb00 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0ff396700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8005100 con 0x7fe0f809bb00 2024-09-16T06:41:27.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 50..75) v4 ==== 5377+0+0 (unknown 3364556671 0 0) 0x7fe0f40439c0 con 0x7fe0f809bb00 2024-09-16T06:41:27.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.356+0000 7fe0fc931700 10 monclient: get_auth_request con 0x7fe0dc036220 auth_method 0 2024-09-16T06:41:27.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.364+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe0f4014990 con 0x7fe0f809bb00 2024-09-16T06:41:27.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.364+0000 7fe0edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.364+0000 7fe0edffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:27.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.536+0000 7fe0ff396700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:27.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.536+0000 7fe0ff396700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:27.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.536+0000 7fe0ff396700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.536+0000 7fe0ff396700 1 -- 172.21.15.189:0/348386722 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fe0d80053b0 con 0x7fe0f809bb00 2024-09-16T06:41:27.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.540+0000 7fe0edffb700 1 -- 172.21.15.189:0/348386722 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v75) v1 ==== 99+0+0 (unknown 2573910478 0 0) 0x7fe0f4012e00 con 0x7fe0f809bb00 2024-09-16T06:41:27.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.540+0000 7fe0edffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:27.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.540+0000 7fe0edffb700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T06:41:27.545 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T06:41:27.549 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:27.549 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host1 bar 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: get_monmap_and_config 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: monmap: 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:27.654938+0000 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:27.654938+0000 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.654 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.655 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.655 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:27.655 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:27.655 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 1 Processor -- start 2024-09-16T06:41:27.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 1 -- start start 2024-09-16T06:41:27.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: init 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-a con 0x7faa5c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-c con 0x7faa5c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-b con 0x7faa5c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c09cca0 con 0x7faa5c09bb00 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.648+0000 7faa62edf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c09f750 con 0x7faa5c09c5f0 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c097620 con 0x7faa5c138da0 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient(hunting): authenticate will time out at 1040.168029s 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa5c09c5f0 0x7faa5c137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51500/0 (socket says 172.21.15.189:51500) 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5bfff700 1 -- 172.21.15.189:0/3315636047 learned_addr learned my addr 172.21.15.189:0/3315636047 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa60c7b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7faa5c09bb00 legacy=0x7faa5c09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648925726 0 0) 0x7faa5c09f750 con 0x7faa5c09c5f0 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient(hunting): my global_id is 134290 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa440035d0 con 0x7faa5c09c5f0 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753088535 0 0) 0x7faa440035d0 con 0x7faa5c09c5f0 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 >> v1:172.21.15.76:6790/0 conn(0x7faa5c138da0 legacy=0x7faa5c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 >> v1:172.21.15.76:6789/0 conn(0x7faa5c09bb00 legacy=0x7faa5c09bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:41:27.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c097900 con 0x7faa5c09c5f0 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 5 monclient: authenticate success, global_id 134290 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c001610 con 0x7faa5c09c5f0 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: dump: 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa4c001930 con 0x7faa5c09c5f0 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 1 -- 172.21.15.189:0/3315636047 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c001d00 con 0x7faa5c09c5f0 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 4 set_mon_vals no callback set 2024-09-16T06:41:27.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa5b7fe700 10 monclient: dump: 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient: shutdown 2024-09-16T06:41:27.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- 172.21.15.189:0/3315636047 >> v1:172.21.15.189:6789/0 conn(0x7faa5c09c5f0 legacy=0x7faa5c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- 172.21.15.189:0/3315636047 shutdown_connections 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- 172.21.15.189:0/3315636047 >> 172.21.15.189:0/3315636047 conn(0x7faa5c0a03e0 msgr2=0x7faa5c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- 172.21.15.189:0/3315636047 shutdown_connections 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- 172.21.15.189:0/3315636047 wait complete. 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient: build_initial_monmap 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient: monmap: 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:27.664 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 Processor -- start 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 1 -- start start 2024-09-16T06:41:27.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.652+0000 7faa62edf700 10 monclient: init 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-a con 0x7faa5c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-b con 0x7faa5c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient(hunting): picked mon.noname-c con 0x7faa5c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c1337c0 con 0x7faa5c09bb00 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c131970 con 0x7faa5c09c5f0 2024-09-16T06:41:27.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa60c7b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7faa5c09bb00 legacy=0x7faa5c096640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa5bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faa5c09c5f0 0x7faa5c1cca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51516/0 (socket says 172.21.15.189:51516) 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c099260 con 0x7faa5c138da0 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa5bfff700 1 -- 172.21.15.189:0/1090143171 learned_addr learned my addr 172.21.15.189:0/1090143171 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient(hunting): authenticate will time out at 1040.172685s 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384573693 0 0) 0x7faa5c131970 con 0x7faa5c09c5f0 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): my global_id is 134293 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480035d0 con 0x7faa5c09c5f0 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578461029 0 0) 0x7faa480035d0 con 0x7faa5c09c5f0 2024-09-16T06:41:27.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa5c131970 con 0x7faa5c09c5f0 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470368678 0 0) 0x7faa5c099260 con 0x7faa5c138da0 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): my global_id is 124580 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480035d0 con 0x7faa5c138da0 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c001610 con 0x7faa5c09c5f0 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): dump: 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.670 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1130537276 0 0) 0x7faa5c131970 con 0x7faa5c09c5f0 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 >> v1:172.21.15.76:6790/0 conn(0x7faa5c138da0 legacy=0x7faa5c1d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 >> v1:172.21.15.76:6789/0 conn(0x7faa5c09bb00 legacy=0x7faa5c096640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:27.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 monclient: found mon.b 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c13c240 con 0x7faa5c09c5f0 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:27.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 5 monclient: authenticate success, global_id 134293 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _renew_subs 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c1334e0 con 0x7faa5c09c5f0 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _renew_subs 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c096d50 con 0x7faa5c09c5f0 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa4c004d60 con 0x7faa5c09c5f0 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa4c005070 con 0x7faa5c09c5f0 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 10 monclient: dump: 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa5a7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faa4c005250 con 0x7faa5c09c5f0 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c005100 con 0x7faa5c09c5f0 2024-09-16T06:41:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.656+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(75..75 src has 50..75) v4 ==== 5377+0+0 (unknown 3364556671 0 0) 0x7faa4c043c20 con 0x7faa5c09c5f0 2024-09-16T06:41:27.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.660+0000 7faa60c7b700 10 monclient: get_auth_request con 0x7faa48036180 auth_method 0 2024-09-16T06:41:27.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.664+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faa4c014c30 con 0x7faa5c09c5f0 2024-09-16T06:41:27.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.668+0000 7faa597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:27.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.668+0000 7faa597fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:27.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.828+0000 7faa62edf700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:27.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.828+0000 7faa62edf700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:27.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.828+0000 7faa62edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:27.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:27.828+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7faa3c005930 con 0x7faa5c09c5f0 2024-09-16T06:41:28.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.084+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v76) v1 ==== 147+0+0 (unknown 1165639375 0 0) 0x7faa4c0130a0 con 0x7faa5c09c5f0 2024-09-16T06:41:28.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.084+0000 7faa597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:28.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.084+0000 7faa597fa700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T06:41:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.188+0000 7faa62edf700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.188+0000 7faa62edf700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.188+0000 7faa62edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.188+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7faa3c0053b0 con 0x7faa5c09c5f0 2024-09-16T06:41:28.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa597fa700 1 -- 172.21.15.189:0/1090143171 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v76) v1 ==== 141+0+0 (unknown 3498277629 0 0) 0x7faa4c014a00 con 0x7faa5c09c5f0 2024-09-16T06:41:28.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa597fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T06:41:28.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa597fa700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T06:41:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 >> v1:172.21.15.189:6812/3098810674 conn(0x7faa48036180 legacy=0x7faa48038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 10 monclient: shutdown 2024-09-16T06:41:28.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:28.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 >> v1:172.21.15.189:6789/0 conn(0x7faa5c09c5f0 legacy=0x7faa5c1cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 shutdown_connections 2024-09-16T06:41:28.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 >> 172.21.15.189:0/1090143171 conn(0x7faa5c0a03e0 msgr2=0x7faa5c0a22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:28.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 shutdown_connections 2024-09-16T06:41:28.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.192+0000 7faa62edf700 1 -- 172.21.15.189:0/1090143171 wait complete. 2024-09-16T06:41:28.200 INFO:tasks.workunit.client.0.smithi189.stderr:item -11 does not appear in that position 2024-09-16T06:41:28.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:28.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T06:41:28.209 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T06:41:28.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: get_monmap_and_config 2024-09-16T06:41:28.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: build_initial_monmap 2024-09-16T06:41:28.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: monmap: 2024-09-16T06:41:28.309 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:28.310002+0000 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:28.310002+0000 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 1 Processor -- start 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 1 -- start start 2024-09-16T06:41:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: init 2024-09-16T06:41:28.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-a con 0x7f5df009ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-b con 0x7f5df0097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-c con 0x7f5df009f830 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df0099b10 con 0x7f5df009ed40 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df0098890 con 0x7f5df009f830 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df0098b70 con 0x7f5df0097860 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:28.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 10 monclient(hunting): authenticate will time out at 1040.822670s 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5deed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5df009f830 0x7f5df013be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51530/0 (socket says 172.21.15.189:51530) 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5deed9d700 1 -- 172.21.15.189:0/1727298343 learned_addr learned my addr 172.21.15.189:0/1727298343 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5dee59c700 1 -- 172.21.15.189:0/1727298343 >> v1:172.21.15.76:6789/0 conn(0x7f5df009ed40 legacy=0x7f5df009f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809318582 0 0) 0x7f5df0098890 con 0x7f5df009f830 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): my global_id is 134296 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ddc0035d0 con 0x7f5df009f830 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283523343 0 0) 0x7f5df0098b70 con 0x7f5df0097860 2024-09-16T06:41:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient(hunting): my global_id is 124583 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df0098890 con 0x7f5df0097860 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632965533 0 0) 0x7f5ddc0035d0 con 0x7f5df009f830 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 >> v1:172.21.15.76:6790/0 conn(0x7f5df0097860 legacy=0x7f5df01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 >> v1:172.21.15.76:6789/0 conn(0x7f5df009ed40 legacy=0x7f5df009f120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: _finish_hunting 0 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 monclient: found mon.noname-c 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df00973a0 con 0x7f5df009f830 2024-09-16T06:41:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: _finish_auth 0 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: _check_auth_tickets 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 5 monclient: authenticate success, global_id 134296 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5df4c1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5de0001610 con 0x7f5df009f830 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.304+0000 7f5ded59a700 10 monclient: dump: 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de00018f0 con 0x7f5df009f830 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 4 set_mon_vals no callback set 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 1 -- 172.21.15.189:0/1727298343 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5de0001c80 con 0x7f5df009f830 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5ded59a700 10 monclient: dump: 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: shutdown 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1727298343 >> v1:172.21.15.189:6789/0 conn(0x7f5df009f830 legacy=0x7f5df013be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1727298343 shutdown_connections 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1727298343 >> 172.21.15.189:0/1727298343 conn(0x7f5df009fe30 msgr2=0x7f5df00a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1727298343 shutdown_connections 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1727298343 wait complete. 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: build_initial_monmap 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: monmap: 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 Processor -- start 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- start start 2024-09-16T06:41:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: init 2024-09-16T06:41:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-c con 0x7f5df0097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-b con 0x7f5df009ed40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient(hunting): picked mon.noname-a con 0x7f5df009f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df0130310 con 0x7f5df009f830 2024-09-16T06:41:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df01342b0 con 0x7f5df009ed40 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df01312b0 con 0x7f5df0097860 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5df4c1e700 10 monclient(hunting): authenticate will time out at 1040.827655s 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5deed9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5df009f830 legacy=0x7f5df01d25d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5dedd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5df009ed40 0x7f5df01ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51536/0 (socket says 172.21.15.189:51536) 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.308+0000 7f5dedd9b700 1 -- 172.21.15.189:0/1693131498 learned_addr learned my addr 172.21.15.189:0/1693131498 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487807151 0 0) 0x7f5df01342b0 con 0x7f5df009ed40 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): my global_id is 134299 2024-09-16T06:41:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd00035d0 con 0x7f5df009ed40 2024-09-16T06:41:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342714052 0 0) 0x7f5df01312b0 con 0x7f5df0097860 2024-09-16T06:41:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): my global_id is 124586 2024-09-16T06:41:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df01342b0 con 0x7f5df0097860 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3876611636 0 0) 0x7f5dd00035d0 con 0x7f5df009ed40 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5df01312b0 con 0x7f5df009ed40 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5dd8001970 con 0x7f5df009ed40 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288206188 0 0) 0x7f5df01342b0 con 0x7f5df0097860 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dd00035d0 con 0x7f5df0097860 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5de8004520 con 0x7f5df0097860 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:28.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.336 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 482912254 0 0) 0x7f5df01312b0 con 0x7f5df009ed40 2024-09-16T06:41:28.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 >> v1:172.21.15.76:6790/0 conn(0x7f5df0097860 legacy=0x7f5df0133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 >> v1:172.21.15.76:6789/0 conn(0x7f5df009f830 legacy=0x7f5df01d25d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 monclient: found mon.b 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df0131dd0 con 0x7f5df009ed40 2024-09-16T06:41:28.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 5 monclient: authenticate success, global_id 134299 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _renew_subs 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df01320b0 con 0x7f5df009ed40 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _renew_subs 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df01ca600 con 0x7f5df009ed40 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dd80036c0 con 0x7f5df009ed40 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:28.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de7fff700 4 set_mon_vals no callback set 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5dd80039d0 con 0x7f5df009ed40 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 10 monclient: dump: 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5dd8003bb0 con 0x7f5df009ed40 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(76..76 src has 50..76) v4 ==== 5337+0+0 (unknown 996980743 0 0) 0x7f5dd80438a0 con 0x7f5df009ed40 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5dee59c700 10 monclient: get_auth_request con 0x7f5dd0036310 auth_method 0 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.312+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd4005100 con 0x7f5df009ed40 2024-09-16T06:41:28.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.320+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5dd80043d0 con 0x7f5df009ed40 2024-09-16T06:41:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.320+0000 7f5de6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.320+0000 7f5de6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:28.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.488+0000 7f5df4c1e700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:28.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.488+0000 7f5df4c1e700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:28.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.488+0000 7f5df4c1e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.488+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5dd4005bc0 con 0x7f5df009ed40 2024-09-16T06:41:28.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.492+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1340 (unknown 452184613 0 3315320304) 0x7f5dd8012c50 con 0x7f5df009ed40 2024-09-16T06:41:28.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.492+0000 7f5de6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:28.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.492+0000 7f5de6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5dd4005680 con 0x7f5df009ed40 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5de6ffd700 1 -- 172.21.15.189:0/1693131498 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1340 (unknown 452184613 0 3315320304) 0x7f5dd80143d0 con 0x7f5df009ed40 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5de6ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:28.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5de6ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5dd0036310 legacy=0x7f5dd00387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 10 monclient: shutdown 2024-09-16T06:41:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:28.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 >> v1:172.21.15.189:6789/0 conn(0x7f5df009ed40 legacy=0x7f5df01ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.588+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 shutdown_connections 2024-09-16T06:41:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.592+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 >> 172.21.15.189:0/1693131498 conn(0x7f5df009fe30 msgr2=0x7f5df006d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.592+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 shutdown_connections 2024-09-16T06:41:28.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.592+0000 7f5df4c1e700 1 -- 172.21.15.189:0/1693131498 wait complete. 2024-09-16T06:41:28.605 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.7 host1 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 10 monclient: get_monmap_and_config 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 10 monclient: build_initial_monmap 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 10 monclient: monmap: 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:28.715232+0000 2024-09-16T06:41:28.714 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:28.715232+0000 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:28.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 1 Processor -- start 2024-09-16T06:41:28.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 1 -- start start 2024-09-16T06:41:28.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.708+0000 7f2bdf801700 10 monclient: init 2024-09-16T06:41:28.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:28.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-a con 0x7f2bd8099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-c con 0x7f2bd8099e40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-b con 0x7f2bd809a250 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd809c460 con 0x7f2bd8099a60 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd809c1e0 con 0x7f2bd8099e40 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd809bd60 con 0x7f2bd809a250 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): authenticate will time out at 1041.228377s 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdcd9c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2bd8099e40 0x7f2bd8136960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53534/0 (socket says 172.21.15.189:53534) 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdcd9c700 1 -- 172.21.15.189:0/3146942084 learned_addr learned my addr 172.21.15.189:0/3146942084 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdd59d700 1 -- 172.21.15.189:0/3146942084 >> v1:172.21.15.76:6789/0 conn(0x7f2bd8099a60 legacy=0x7f2bd8098070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280172840 0 0) 0x7f2bd809c1e0 con 0x7f2bd8099e40 2024-09-16T06:41:28.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient(hunting): my global_id is 134302 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc40035d0 con 0x7f2bd8099e40 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4233342161 0 0) 0x7f2bc40035d0 con 0x7f2bd8099e40 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 >> v1:172.21.15.76:6790/0 conn(0x7f2bd809a250 legacy=0x7f2bd813a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 >> v1:172.21.15.76:6789/0 conn(0x7f2bd8099a60 legacy=0x7f2bd8098070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 monclient: found mon.noname-c 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd809d490 con 0x7f2bd8099e40 2024-09-16T06:41:28.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: _finish_auth 0 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 5 monclient: authenticate success, global_id 134302 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2bd4001540 con 0x7f2bd8099e40 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: dump: 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.726 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bd4001860 con 0x7f2bd8099e40 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 4 set_mon_vals no callback set 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 1 -- 172.21.15.189:0/3146942084 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2bd4001bf0 con 0x7f2bd8099e40 2024-09-16T06:41:28.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:28.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bcbfff700 10 monclient: dump: 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: shutdown 2024-09-16T06:41:28.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- 172.21.15.189:0/3146942084 >> v1:172.21.15.189:6789/0 conn(0x7f2bd8099e40 legacy=0x7f2bd8136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- 172.21.15.189:0/3146942084 shutdown_connections 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- 172.21.15.189:0/3146942084 >> 172.21.15.189:0/3146942084 conn(0x7f2bd80a03e0 msgr2=0x7f2bd80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- 172.21.15.189:0/3146942084 shutdown_connections 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- 172.21.15.189:0/3146942084 wait complete. 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: build_initial_monmap 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: monmap: 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:28.730 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 Processor -- start 2024-09-16T06:41:28.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- start start 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: init 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-a con 0x7f2bd8099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-c con 0x7f2bd8099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): picked mon.noname-b con 0x7f2bd809a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd8094ae0 con 0x7f2bd8099a60 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd809a6e0 con 0x7f2bd809a250 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd80938a0 con 0x7f2bd8099e40 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdf801700 10 monclient(hunting): authenticate will time out at 1041.231747s 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bdd59d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2bd8099a60 legacy=0x7f2bd81c9870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bddd9e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2bd809a250 0x7f2bd81d1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53544/0 (socket says 172.21.15.189:53544) 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.712+0000 7f2bddd9e700 1 -- 172.21.15.189:0/3814569598 learned_addr learned my addr 172.21.15.189:0/3814569598 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846503472 0 0) 0x7f2bd809a6e0 con 0x7f2bd809a250 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): my global_id is 134305 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc0035d0 con 0x7f2bd809a250 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601660190 0 0) 0x7f2bd80938a0 con 0x7f2bd8099e40 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:28.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): my global_id is 124592 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd809a6e0 con 0x7f2bd8099e40 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4168544944 0 0) 0x7f2bbc0035d0 con 0x7f2bd809a250 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bd80938a0 con 0x7f2bd809a250 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2bd0001c30 con 0x7f2bd809a250 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:28.734 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455380274 0 0) 0x7f2bd809a6e0 con 0x7f2bd8099e40 2024-09-16T06:41:28.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bbc0035d0 con 0x7f2bd8099e40 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2bd4001500 con 0x7f2bd8099e40 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.736 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3319930908 0 0) 0x7f2bd80938a0 con 0x7f2bd809a250 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 >> v1:172.21.15.76:6790/0 conn(0x7f2bd8099e40 legacy=0x7f2bd80965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 >> v1:172.21.15.76:6789/0 conn(0x7f2bd8099a60 legacy=0x7f2bd81c9870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 monclient: found mon.b 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd80950a0 con 0x7f2bd809a250 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:28.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 5 monclient: authenticate success, global_id 134305 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _renew_subs 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd8094800 con 0x7f2bd809a250 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _renew_subs 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd8096ce0 con 0x7f2bd809a250 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bd0001fc0 con 0x7f2bd809a250 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2bd0003e90 con 0x7f2bd809a250 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 10 monclient: dump: 2024-09-16T06:41:28.738 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bcb7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2bd0003ff0 con 0x7f2bd809a250 2024-09-16T06:41:28.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(76..76 src has 50..76) v4 ==== 5337+0+0 (unknown 996980743 0 0) 0x7f2bd0043c30 con 0x7f2bd809a250 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd8094c90 con 0x7f2bd809a250 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.716+0000 7f2bdd59d700 10 monclient: get_auth_request con 0x7f2bbc0362f0 auth_method 0 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.724+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2bd8094c90 con 0x7f2bd809a250 2024-09-16T06:41:28.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.724+0000 7f2bc9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:28.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.724+0000 7f2bc9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:28.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.896+0000 7f2bdf801700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:28.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.896+0000 7f2bdf801700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:28.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.896+0000 7f2bdf801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:28.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:28.896+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f2bd8066480 con 0x7f2bd809a250 2024-09-16T06:41:29.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.092+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v77) v1 ==== 143+0+0 (unknown 1469705424 0 0) 0x7f2bd0014c00 con 0x7f2bd809a250 2024-09-16T06:41:29.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.092+0000 7f2bc9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:29.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.092+0000 7f2bc9ffb700 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T06:41:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.204+0000 7f2bdf801700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.204+0000 7f2bdf801700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.204+0000 7f2bdf801700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.204+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f2bd809fc40 con 0x7f2bd809a250 2024-09-16T06:41:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.204+0000 7f2bc9ffb700 1 -- 172.21.15.189:0/3814569598 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v77) v1 ==== 145+0+0 (unknown 747315572 0 0) 0x7f2bd0013070 con 0x7f2bd809a250 2024-09-16T06:41:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.208+0000 7f2bc9ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T06:41:29.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.208+0000 7f2bc9ffb700 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T06:41:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2bbc0362f0 legacy=0x7f2bbc0387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 10 monclient: shutdown 2024-09-16T06:41:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 >> v1:172.21.15.189:6789/0 conn(0x7f2bd809a250 legacy=0x7f2bd81d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:29.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 shutdown_connections 2024-09-16T06:41:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 >> 172.21.15.189:0/3814569598 conn(0x7f2bd80a03e0 msgr2=0x7f2bd8097f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 shutdown_connections 2024-09-16T06:41:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.212+0000 7f2bdf801700 1 -- 172.21.15.189:0/3814569598 wait complete. 2024-09-16T06:41:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T06:41:29.229 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host1 2024-09-16T06:41:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.324+0000 7f0a24f23700 10 monclient: get_monmap_and_config 2024-09-16T06:41:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.324+0000 7f0a24f23700 10 monclient: build_initial_monmap 2024-09-16T06:41:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.324+0000 7f0a24f23700 10 monclient: monmap: 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:29.331367+0000 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:29.331367+0000 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:29.331 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 Processor -- start 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- start start 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient: init 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-a con 0x7f0a2009eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-b con 0x7f0a2009beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-c con 0x7f0a2009c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a20099ed0 con 0x7f0a2009eba0 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2009d2e0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2009ef80 con 0x7f0a2009beb0 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient(hunting): authenticate will time out at 1041.844788s 2024-09-16T06:41:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0a2009eba0 legacy=0x7f0a2009b7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0a2009c2e0 0x7f0a2013ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53556/0 (socket says 172.21.15.189:53556) 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1ffff700 1 -- 172.21.15.189:0/2805141547 learned_addr learned my addr 172.21.15.189:0/2805141547 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248639428 0 0) 0x7f0a2009d2e0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): my global_id is 134308 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c0035d0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917746335 0 0) 0x7f0a2009ef80 con 0x7f0a2009beb0 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:29.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient(hunting): my global_id is 124595 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2009d2e0 con 0x7f0a2009beb0 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340111133 0 0) 0x7f0a0c0035d0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 >> v1:172.21.15.76:6790/0 conn(0x7f0a2009beb0 legacy=0x7f0a20137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 >> v1:172.21.15.76:6789/0 conn(0x7f0a2009eba0 legacy=0x7f0a2009b7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 monclient: found mon.noname-c 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2009cc70 con 0x7f0a2009c2e0 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: _finish_auth 0 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 5 monclient: authenticate success, global_id 134308 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a18001990 con 0x7f0a2009c2e0 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: dump: 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a18001bb0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 1 -- 172.21.15.189:0/2805141547 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a18001ec0 con 0x7f0a2009c2e0 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 4 set_mon_vals no callback set 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a1effd700 10 monclient: dump: 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 10 monclient: shutdown 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- 172.21.15.189:0/2805141547 >> v1:172.21.15.189:6789/0 conn(0x7f0a2009c2e0 legacy=0x7f0a2013ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- 172.21.15.189:0/2805141547 shutdown_connections 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- 172.21.15.189:0/2805141547 >> 172.21.15.189:0/2805141547 conn(0x7f0a2009fe70 msgr2=0x7f0a200a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- 172.21.15.189:0/2805141547 shutdown_connections 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.328+0000 7f0a24f23700 1 -- 172.21.15.189:0/2805141547 wait complete. 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: build_initial_monmap 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: monmap: 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:29.343 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 Processor -- start 2024-09-16T06:41:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- start start 2024-09-16T06:41:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: init 2024-09-16T06:41:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-c con 0x7f0a2009beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-a con 0x7f0a2009c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient(hunting): picked mon.noname-b con 0x7f0a2009eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200a1090 con 0x7f0a2009c2e0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2009a380 con 0x7f0a2009eba0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200a15e0 con 0x7f0a2009beb0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient(hunting): authenticate will time out at 1041.849829s 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a16ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0a2009c2e0 legacy=0x7f0a201ce330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0a2009eba0 0x7f0a201d1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53558/0 (socket says 172.21.15.189:53558) 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1ffff700 1 -- 172.21.15.189:0/1030264236 learned_addr learned my addr 172.21.15.189:0/1030264236 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169368293 0 0) 0x7f0a2009a380 con 0x7f0a2009eba0 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): my global_id is 134311 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000035d0 con 0x7f0a2009eba0 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793484829 0 0) 0x7f0a200a15e0 con 0x7f0a2009beb0 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): my global_id is 124598 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2009a380 con 0x7f0a2009beb0 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283989644 0 0) 0x7f0a000035d0 con 0x7f0a2009eba0 2024-09-16T06:41:29.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a200a15e0 con 0x7f0a2009eba0 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a18001890 con 0x7f0a2009eba0 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:29.350 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469717668 0 0) 0x7f0a2009a380 con 0x7f0a2009beb0 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a000035d0 con 0x7f0a2009beb0 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a100041a0 con 0x7f0a2009beb0 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 585731931 0 0) 0x7f0a200a15e0 con 0x7f0a2009eba0 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 >> v1:172.21.15.76:6790/0 conn(0x7f0a2009beb0 legacy=0x7f0a200969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 >> v1:172.21.15.76:6789/0 conn(0x7f0a2009c2e0 legacy=0x7f0a201ce330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 monclient: found mon.b 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20095ff0 con 0x7f0a2009eba0 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:29.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 5 monclient: authenticate success, global_id 134311 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _renew_subs 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a200962d0 con 0x7f0a2009eba0 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _renew_subs 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201d3180 con 0x7f0a2009eba0 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a18004ec0 con 0x7f0a2009eba0 2024-09-16T06:41:29.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a180051d0 con 0x7f0a2009eba0 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.332+0000 7f0a1e7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1cff9700 10 monclient: dump: 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:29.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0a18005330 con 0x7f0a2009eba0 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a24f23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a24f23700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a24f23700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(77..77 src has 50..77) v4 ==== 5316+0+0 (unknown 4160890713 0 0) 0x7f0a18043d40 con 0x7f0a2009eba0 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a200008d0 con 0x7f0a2009eba0 2024-09-16T06:41:29.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.336+0000 7f0a1f7fe700 10 monclient: get_auth_request con 0x7f0a000362f0 auth_method 0 2024-09-16T06:41:29.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.344+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0a18014c20 con 0x7f0a2009eba0 2024-09-16T06:41:29.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.344+0000 7f0a1cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:29.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.344+0000 7f0a1cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:29.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.512+0000 7f0a24f23700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:29.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.512+0000 7f0a24f23700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:29.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.512+0000 7f0a24f23700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:29.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:29.512+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f0a2013d950 con 0x7f0a2009eba0 2024-09-16T06:41:30.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.100+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v78) v1 ==== 124+0+0 (unknown 1302964151 0 0) 0x7f0a18014810 con 0x7f0a2009eba0 2024-09-16T06:41:30.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.100+0000 7f0a1cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:30.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.100+0000 7f0a1cff9700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T06:41:30.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a24f23700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:30.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a24f23700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:30.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a24f23700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f0a201d3310 con 0x7f0a2009eba0 2024-09-16T06:41:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a1cff9700 1 -- 172.21.15.189:0/1030264236 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v78) v1 ==== 124+0+0 (unknown 3586971478 0 0) 0x7f0a180149f0 con 0x7f0a2009eba0 2024-09-16T06:41:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a1cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T06:41:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.208+0000 7f0a1cff9700 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0a000362f0 legacy=0x7f0a000387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 10 monclient: shutdown 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 >> v1:172.21.15.189:6789/0 conn(0x7f0a2009eba0 legacy=0x7f0a201d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 shutdown_connections 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 >> 172.21.15.189:0/1030264236 conn(0x7f0a2009fe70 msgr2=0x7f0a2009a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:30.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 shutdown_connections 2024-09-16T06:41:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.212+0000 7f0a24f23700 1 -- 172.21.15.189:0/1030264236 wait complete. 2024-09-16T06:41:30.218 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host1' does not appear in the crush map 2024-09-16T06:41:30.228 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:30.228 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c host1 2024-09-16T06:41:30.228 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T06:41:30.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: get_monmap_and_config 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: build_initial_monmap 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: monmap: 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:30.322586+0000 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:30.322586+0000 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:30.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:30.323 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:30.323 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 1 Processor -- start 2024-09-16T06:41:30.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 1 -- start start 2024-09-16T06:41:30.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: init 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-c con 0x7f419c0984b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-b con 0x7f419c072140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-a con 0x7f419c1384d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c09b2a0 con 0x7f419c1384d0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c099940 con 0x7f419c0984b0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c099d10 con 0x7f419c072140 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a35c3700 10 monclient(hunting): authenticate will time out at 1042.835714s 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.316+0000 7f41a1b60700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f419c1384d0 legacy=0x7f419c13a8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a135f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f419c0984b0 0x7f419c098890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53572/0 (socket says 172.21.15.189:53572) 2024-09-16T06:41:30.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a135f700 1 -- 172.21.15.189:0/1761096227 learned_addr learned my addr 172.21.15.189:0/1761096227 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272519272 0 0) 0x7f419c099940 con 0x7f419c0984b0 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): my global_id is 134314 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41880035d0 con 0x7f419c0984b0 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860741562 0 0) 0x7f419c099d10 con 0x7f419c072140 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient(hunting): my global_id is 124601 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419c099940 con 0x7f419c072140 2024-09-16T06:41:30.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236153659 0 0) 0x7f41880035d0 con 0x7f419c0984b0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 >> v1:172.21.15.76:6790/0 conn(0x7f419c072140 legacy=0x7f419c098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 >> v1:172.21.15.76:6789/0 conn(0x7f419c1384d0 legacy=0x7f419c13a8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 monclient: found mon.noname-c 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0996b0 con 0x7f419c0984b0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4198001610 con 0x7f419c0984b0 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 5 monclient: authenticate success, global_id 134314 2024-09-16T06:41:30.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: dump: 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41980018f0 con 0x7f419c0984b0 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 4 set_mon_vals no callback set 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 1 -- 172.21.15.189:0/1761096227 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4198001c80 con 0x7f419c0984b0 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f4193fff700 10 monclient: dump: 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.330 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 10 monclient: shutdown 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- 172.21.15.189:0/1761096227 >> v1:172.21.15.189:6789/0 conn(0x7f419c0984b0 legacy=0x7f419c098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- 172.21.15.189:0/1761096227 shutdown_connections 2024-09-16T06:41:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- 172.21.15.189:0/1761096227 >> 172.21.15.189:0/1761096227 conn(0x7f419c09fe30 msgr2=0x7f419c0a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- 172.21.15.189:0/1761096227 shutdown_connections 2024-09-16T06:41:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- 172.21.15.189:0/1761096227 wait complete. 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 10 monclient: build_initial_monmap 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 10 monclient: monmap: 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:30.335 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 Processor -- start 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 1 -- start start 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.320+0000 7f41a35c3700 10 monclient: init 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-c con 0x7f419c072140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-b con 0x7f419c0984b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient(hunting): picked mon.noname-a con 0x7f419c1384d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c090de0 con 0x7f419c1384d0 2024-09-16T06:41:30.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c09a520 con 0x7f419c0984b0 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c0a2b90 con 0x7f419c072140 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient(hunting): authenticate will time out at 1042.840437s 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a0b5e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f419c0984b0 0x7f419c1ca670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53582/0 (socket says 172.21.15.189:53582) 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a0b5e700 1 -- 172.21.15.189:0/3714191338 learned_addr learned my addr 172.21.15.189:0/3714191338 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a1b60700 1 -- 172.21.15.189:0/3714191338 >> v1:172.21.15.76:6789/0 conn(0x7f419c1384d0 legacy=0x7f419c1cdda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567879969 0 0) 0x7f419c09a520 con 0x7f419c0984b0 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): my global_id is 134317 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41800035d0 con 0x7f419c0984b0 2024-09-16T06:41:30.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417183562 0 0) 0x7f419c0a2b90 con 0x7f419c072140 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): my global_id is 124604 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419c09a520 con 0x7f419c072140 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894107448 0 0) 0x7f41800035d0 con 0x7f419c0984b0 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f419c0a2b90 con 0x7f419c0984b0 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f418c001910 con 0x7f419c0984b0 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1327329391 0 0) 0x7f419c0a2b90 con 0x7f419c0984b0 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 >> v1:172.21.15.76:6790/0 conn(0x7f419c072140 legacy=0x7f419c096580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 >> v1:172.21.15.76:6789/0 conn(0x7f419c1384d0 legacy=0x7f419c1cdda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 monclient: found mon.b 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c09e5d0 con 0x7f419c0984b0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 5 monclient: authenticate success, global_id 134317 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _renew_subs 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c096c90 con 0x7f419c0984b0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _renew_subs 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c08ea10 con 0x7f419c0984b0 2024-09-16T06:41:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f418c0036c0 con 0x7f419c0984b0 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f418c0039d0 con 0x7f419c0984b0 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41937fe700 4 set_mon_vals no callback set 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 10 monclient: dump: 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c005100 con 0x7f419c0984b0 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f418c003bb0 con 0x7f419c0984b0 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7f418c043850 con 0x7f419c0984b0 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.324+0000 7f41a135f700 10 monclient: get_auth_request con 0x7f41800361d0 auth_method 0 2024-09-16T06:41:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.332+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f418c014830 con 0x7f419c0984b0 2024-09-16T06:41:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.332+0000 7f4191ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.332+0000 7f4191ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f41a35c3700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f41a35c3700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f41a35c3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f417c005bc0 con 0x7f419c0984b0 2024-09-16T06:41:30.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (unknown 596248356 0 149626083) 0x7f418c01ce20 con 0x7f419c0984b0 2024-09-16T06:41:30.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f4191ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:30.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.500+0000 7f4191ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.612 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f417c005650 con 0x7f419c0984b0 2024-09-16T06:41:30.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f4191ffb700 1 -- 172.21.15.189:0/3714191338 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1273 (unknown 596248356 0 149626083) 0x7f418c012ca0 con 0x7f419c0984b0 2024-09-16T06:41:30.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f4191ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:30.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f4191ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 >> v1:172.21.15.189:6812/3098810674 conn(0x7f41800361d0 legacy=0x7f4180038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 10 monclient: shutdown 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 >> v1:172.21.15.189:6789/0 conn(0x7f419c0984b0 legacy=0x7f419c1ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 shutdown_connections 2024-09-16T06:41:30.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.608+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 >> 172.21.15.189:0/3714191338 conn(0x7f419c09fe30 msgr2=0x7f419c09c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:30.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.612+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 shutdown_connections 2024-09-16T06:41:30.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.612+0000 7f41a35c3700 1 -- 172.21.15.189:0/3714191338 wait complete. 2024-09-16T06:41:30.626 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd tree-from host1 2024-09-16T06:41:30.626 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:30.626 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from host1 2024-09-16T06:41:30.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: get_monmap_and_config 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: build_initial_monmap 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: monmap: 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:30.729478+0000 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:30.729478+0000 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:30.729 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:30.730 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:30.730 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 Processor -- start 2024-09-16T06:41:30.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 -- start start 2024-09-16T06:41:30.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: init 2024-09-16T06:41:30.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:30.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-c con 0x7efcf009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-a con 0x7efcf0099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-b con 0x7efcf01396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf009f520 con 0x7efcf0099700 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf009bc10 con 0x7efcf009f140 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf009b810 con 0x7efcf01396a0 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient(hunting): authenticate will time out at 1043.242194s 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efceed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcf009f140 0x7efcf009a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53598/0 (socket says 172.21.15.189:53598) 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efceed9d700 1 -- 172.21.15.189:0/2341354588 learned_addr learned my addr 172.21.15.189:0/2341354588 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcee59c700 1 -- 172.21.15.189:0/2341354588 >> v1:172.21.15.76:6789/0 conn(0x7efcf0099700 legacy=0x7efcf009b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486625466 0 0) 0x7efcf009bc10 con 0x7efcf009f140 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): my global_id is 134320 2024-09-16T06:41:30.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd00035d0 con 0x7efcf009f140 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809652301 0 0) 0x7efcf009b810 con 0x7efcf01396a0 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient(hunting): my global_id is 124607 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf009bc10 con 0x7efcf01396a0 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298181541 0 0) 0x7efcd00035d0 con 0x7efcf009f140 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 >> v1:172.21.15.76:6790/0 conn(0x7efcf01396a0 legacy=0x7efcf013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 >> v1:172.21.15.76:6789/0 conn(0x7efcf0099700 legacy=0x7efcf009b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 monclient: found mon.noname-c 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf009ceb0 con 0x7efcf009f140 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: _finish_auth 0 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 5 monclient: authenticate success, global_id 134320 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efce0001610 con 0x7efcf009f140 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: dump: 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.736 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efce00018f0 con 0x7efcf009f140 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:30.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 1 -- 172.21.15.189:0/2341354588 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efce0001c80 con 0x7efcf009f140 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 4 set_mon_vals no callback set 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcedd9b700 10 monclient: dump: 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.738 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 10 monclient: shutdown 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.724+0000 7efcf51a2700 1 -- 172.21.15.189:0/2341354588 >> v1:172.21.15.189:6789/0 conn(0x7efcf009f140 legacy=0x7efcf009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/2341354588 shutdown_connections 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/2341354588 >> 172.21.15.189:0/2341354588 conn(0x7efcf00a03e0 msgr2=0x7efcf00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/2341354588 shutdown_connections 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/2341354588 wait complete. 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: build_initial_monmap 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: monmap: 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:30.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 Processor -- start 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- start start 2024-09-16T06:41:30.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: init 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-c con 0x7efcf0099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-a con 0x7efcf009f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient(hunting): picked mon.noname-b con 0x7efcf01396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf00a2f10 con 0x7efcf009f140 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf0090210 con 0x7efcf01396a0 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf0071440 con 0x7efcf0099700 2024-09-16T06:41:30.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient(hunting): authenticate will time out at 1043.245612s 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcee59c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7efcf009f140 legacy=0x7efcf01cc850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcef59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcf01396a0 0x7efcf01cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53614/0 (socket says 172.21.15.189:53614) 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcef59e700 1 -- 172.21.15.189:0/3230845008 learned_addr learned my addr 172.21.15.189:0/3230845008 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942704916 0 0) 0x7efcf0090210 con 0x7efcf01396a0 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): my global_id is 134323 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd40035d0 con 0x7efcf01396a0 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926863508 0 0) 0x7efcf0071440 con 0x7efcf0099700 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:30.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): my global_id is 124610 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf0090210 con 0x7efcf0099700 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559548140 0 0) 0x7efcd40035d0 con 0x7efcf01396a0 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcf0071440 con 0x7efcf01396a0 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efce8001c70 con 0x7efcf01396a0 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): dump: 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94516052 0 0) 0x7efcf0090210 con 0x7efcf0099700 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcd40035d0 con 0x7efcf0099700 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efce00015d0 con 0x7efcf0099700 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient(hunting): dump: 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 287398397 0 0) 0x7efcf0071440 con 0x7efcf01396a0 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 >> v1:172.21.15.76:6790/0 conn(0x7efcf0099700 legacy=0x7efcf0096470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 >> v1:172.21.15.76:6789/0 conn(0x7efcf009f140 legacy=0x7efcf01cc850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 monclient: found mon.b 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:30.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf0096b80 con 0x7efcf01396a0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 5 monclient: authenticate success, global_id 134323 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _renew_subs 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf0092cf0 con 0x7efcf01396a0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _renew_subs 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efcf51a2700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf0092a10 con 0x7efcf01396a0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efce8002000 con 0x7efcf01396a0 2024-09-16T06:41:30.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efce8003e90 con 0x7efcf01396a0 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efce77fe700 10 monclient: dump: 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:30.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.728+0000 7efced59a700 4 set_mon_vals no callback set 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efce8004070 con 0x7efcf01396a0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7efce8043ab0 con 0x7efcf01396a0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efcf51a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efcf51a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efcf51a2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efcf51a2700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0090990 con 0x7efcf01396a0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.732+0000 7efceed9d700 10 monclient: get_auth_request con 0x7efcd40362c0 auth_method 0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.744+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efce8004890 con 0x7efcf01396a0 2024-09-16T06:41:30.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.744+0000 7efce77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:30.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.744+0000 7efce77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:30.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efcf51a2700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T06:41:30.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efcf51a2700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T06:41:30.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efcf51a2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:30.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efcf51a2700 1 -- 172.21.15.189:0/3230845008 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7efcf00924b0 con 0x7efcf01396a0 2024-09-16T06:41:30.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efce77fe700 1 -- 172.21.15.189:0/3230845008 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v78) v1 ==== 109+0+0 (unknown 3653902130 0 0) 0x7efcf00924b0 con 0x7efcf01396a0 2024-09-16T06:41:30.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efce77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T06:41:30.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:30.912+0000 7efce77fe700 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-16T06:41:30.921 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-16T06:41:30.923 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:30.923 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from host2 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: get_monmap_and_config 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: monmap: 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:31.025052+0000 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:31.025052+0000 2024-09-16T06:41:31.025 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 1 Processor -- start 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 1 -- start start 2024-09-16T06:41:31.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: init 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc0707a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc070b80 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc13b620 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc09f110 con 0x7fbfdc070b80 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0a0670 con 0x7fbfdc0707a0 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc09f380 con 0x7fbfdc13b620 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient(hunting): authenticate will time out at 1043.538317s 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfdb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0707a0 0x7fbfdc09ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53630/0 (socket says 172.21.15.189:53630) 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfdaffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fbfdc070b80 legacy=0x7fbfdc070fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfdb7fe700 1 -- 172.21.15.189:0/436814617 learned_addr learned my addr 172.21.15.189:0/436814617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694433857 0 0) 0x7fbfdc0a0670 con 0x7fbfdc0707a0 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): my global_id is 134326 2024-09-16T06:41:31.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc80035d0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429808378 0 0) 0x7fbfdc09f380 con 0x7fbfdc13b620 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient(hunting): my global_id is 124613 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc0a0670 con 0x7fbfdc13b620 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862581554 0 0) 0x7fbfc80035d0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 >> v1:172.21.15.76:6790/0 conn(0x7fbfdc13b620 legacy=0x7fbfdc13da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 >> v1:172.21.15.76:6789/0 conn(0x7fbfdc070b80 legacy=0x7fbfdc070fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc09f660 con 0x7fbfdc0707a0 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbfcc001990 con 0x7fbfdc0707a0 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 5 monclient: authenticate success, global_id 134326 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: dump: 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfcc001c70 con 0x7fbfdc0707a0 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfe1aa1700 4 set_mon_vals no callback set 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 1 -- 172.21.15.189:0/436814617 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbfcc002000 con 0x7fbfdc0707a0 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.020+0000 7fbfda7fc700 10 monclient: dump: 2024-09-16T06:41:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: shutdown 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/436814617 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0707a0 legacy=0x7fbfdc09ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/436814617 shutdown_connections 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/436814617 >> 172.21.15.189:0/436814617 conn(0x7fbfdc0a44a0 msgr2=0x7fbfdc0a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/436814617 shutdown_connections 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/436814617 wait complete. 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: monmap: 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 Processor -- start 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- start start 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: init 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-b con 0x7fbfdc0707a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-a con 0x7fbfdc070b80 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient(hunting): picked mon.noname-c con 0x7fbfdc13b620 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc13eb50 con 0x7fbfdc070b80 2024-09-16T06:41:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc132800 con 0x7fbfdc0707a0 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc13a420 con 0x7fbfdc13b620 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient(hunting): authenticate will time out at 1043.542596s 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfdaffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fbfdc070b80 legacy=0x7fbfdc13c5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfdb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0707a0 0x7fbfdc13bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53636/0 (socket says 172.21.15.189:53636) 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfdb7fe700 1 -- 172.21.15.189:0/3076640550 learned_addr learned my addr 172.21.15.189:0/3076640550 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707581124 0 0) 0x7fbfdc132800 con 0x7fbfdc0707a0 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): my global_id is 134329 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfbc0035d0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185979542 0 0) 0x7fbfdc13a420 con 0x7fbfdc13b620 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): my global_id is 124616 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc132800 con 0x7fbfdc13b620 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238305709 0 0) 0x7fbfbc0035d0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfdc13a420 con 0x7fbfdc0707a0 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbfcc001950 con 0x7fbfdc0707a0 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): dump: 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3110809669 0 0) 0x7fbfdc13a420 con 0x7fbfdc0707a0 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 >> v1:172.21.15.76:6790/0 conn(0x7fbfdc13b620 legacy=0x7fbfdc1d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 >> v1:172.21.15.76:6789/0 conn(0x7fbfdc070b80 legacy=0x7fbfdc13c5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 monclient: found mon.b 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc137540 con 0x7fbfdc0707a0 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfd3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 5 monclient: authenticate success, global_id 134329 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _renew_subs 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc13e350 con 0x7fbfdc0707a0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _renew_subs 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.024+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc13cd00 con 0x7fbfdc0707a0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfcc005020 con 0x7fbfdc0707a0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbfcc0053b0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 10 monclient: dump: 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd97fa700 4 set_mon_vals no callback set 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfe1aa1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbfcc006580 con 0x7fbfdc0707a0 2024-09-16T06:41:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7fbfcc043d50 con 0x7fbfdc0707a0 2024-09-16T06:41:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfe1aa1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfe1aa1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc13cfb0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.028+0000 7fbfdaffd700 10 monclient: get_auth_request con 0x7fbfbc036200 auth_method 0 2024-09-16T06:41:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.036+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbfcc006bb0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.036+0000 7fbfd3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.036+0000 7fbfd3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:31.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.204+0000 7fbfe1aa1700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.204+0000 7fbfe1aa1700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.204+0000 7fbfe1aa1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.204+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7fbfdc067990 con 0x7fbfdc0707a0 2024-09-16T06:41:31.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.208+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (unknown 3535260130 0 2845383371) 0x7fbfcc0133a0 con 0x7fbfdc0707a0 2024-09-16T06:41:31.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.208+0000 7fbfd3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.208+0000 7fbfd3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:31.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7fbfdc0a4b20 con 0x7fbfdc0707a0 2024-09-16T06:41:31.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfd3fff700 1 -- 172.21.15.189:0/3076640550 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v78) v1 ==== 80+0+177 (unknown 3535260130 0 2845383371) 0x7fbfcc014b20 con 0x7fbfdc0707a0 2024-09-16T06:41:31.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfd3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T06:41:31.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfd3fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:31.313 INFO:tasks.workunit.client.0.smithi189.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T06:41:31.313 INFO:tasks.workunit.client.0.smithi189.stdout:-16 1.00000 host host2 2024-09-16T06:41:31.313 INFO:tasks.workunit.client.0.smithi189.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbfbc036200 legacy=0x7fbfbc0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 10 monclient: shutdown 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 >> v1:172.21.15.189:6789/0 conn(0x7fbfdc0707a0 legacy=0x7fbfdc13bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 shutdown_connections 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 >> 172.21.15.189:0/3076640550 conn(0x7fbfdc0a44a0 msgr2=0x7fbfdc0a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 shutdown_connections 2024-09-16T06:41:31.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.308+0000 7fbfe1aa1700 1 -- 172.21.15.189:0/3076640550 wait complete. 2024-09-16T06:41:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-16T06:41:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:31.326 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree-from osd.9 2024-09-16T06:41:31.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: get_monmap_and_config 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: monmap: 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:31.433763+0000 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:31.433763+0000 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:31.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:31.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 1 Processor -- start 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 1 -- start start 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: init 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-c con 0x7f2ab009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-b con 0x7f2ab009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-a con 0x7f2ab0138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab009cca0 con 0x7f2ab0138da0 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab009f750 con 0x7f2ab009bb00 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab0097620 con 0x7f2ab009c5f0 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab85a7700 10 monclient(hunting): authenticate will time out at 1043.947021s 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab6b44700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2ab0138da0 legacy=0x7f2ab013b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab6343700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2ab009bb00 0x7f2ab009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53648/0 (socket says 172.21.15.189:53648) 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab6343700 1 -- 172.21.15.189:0/498899510 learned_addr learned my addr 172.21.15.189:0/498899510 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455125610 0 0) 0x7f2ab009f750 con 0x7f2ab009bb00 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): my global_id is 134332 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a9c0035d0 con 0x7f2ab009bb00 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093232623 0 0) 0x7f2ab0097620 con 0x7f2ab009c5f0 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 10 monclient(hunting): my global_id is 124619 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.428+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab009f750 con 0x7f2ab009c5f0 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712489084 0 0) 0x7f2a9c0035d0 con 0x7f2ab009bb00 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 >> v1:172.21.15.76:6790/0 conn(0x7f2ab009c5f0 legacy=0x7f2ab0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 >> v1:172.21.15.76:6789/0 conn(0x7f2ab0138da0 legacy=0x7f2ab013b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 monclient: found mon.noname-c 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0097900 con 0x7f2ab009bb00 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa4001610 con 0x7f2ab009bb00 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: dump: 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.439 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 5 monclient: authenticate success, global_id 134332 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aa40018f0 con 0x7f2ab009bb00 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 1 -- 172.21.15.189:0/498899510 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa4001c80 con 0x7f2ab009bb00 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 4 set_mon_vals no callback set 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab5341700 10 monclient: dump: 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: shutdown 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- 172.21.15.189:0/498899510 >> v1:172.21.15.189:6789/0 conn(0x7f2ab009bb00 legacy=0x7f2ab009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- 172.21.15.189:0/498899510 shutdown_connections 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- 172.21.15.189:0/498899510 >> 172.21.15.189:0/498899510 conn(0x7f2ab00a03e0 msgr2=0x7f2ab00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- 172.21.15.189:0/498899510 shutdown_connections 2024-09-16T06:41:31.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- 172.21.15.189:0/498899510 wait complete. 2024-09-16T06:41:31.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: monmap: 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 Processor -- start 2024-09-16T06:41:31.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- start start 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: init 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-a con 0x7f2ab009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-c con 0x7f2ab009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient(hunting): picked mon.noname-b con 0x7f2ab0138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab00a12f0 con 0x7f2ab009bb00 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab00a2e00 con 0x7f2ab0138da0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab009a6e0 con 0x7f2ab009c5f0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab85a7700 10 monclient(hunting): authenticate will time out at 1043.950413s 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab6343700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2ab009bb00 legacy=0x7f2ab01ca1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab6b44700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2ab0138da0 0x7f2ab01d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53656/0 (socket says 172.21.15.189:53656) 2024-09-16T06:41:31.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2ab6b44700 1 -- 172.21.15.189:0/3141472802 learned_addr learned my addr 172.21.15.189:0/3141472802 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629341707 0 0) 0x7f2ab00a2e00 con 0x7f2ab0138da0 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): my global_id is 134335 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a940035d0 con 0x7f2ab0138da0 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828831295 0 0) 0x7f2ab009a6e0 con 0x7f2ab009c5f0 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): my global_id is 124622 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab00a2e00 con 0x7f2ab009c5f0 2024-09-16T06:41:31.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4098201811 0 0) 0x7f2a940035d0 con 0x7f2ab0138da0 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ab009a6e0 con 0x7f2ab0138da0 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa80044c0 con 0x7f2ab0138da0 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): dump: 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36201291 0 0) 0x7f2ab009a6e0 con 0x7f2ab0138da0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 >> v1:172.21.15.76:6790/0 conn(0x7f2ab009c5f0 legacy=0x7f2ab01cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.432+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 >> v1:172.21.15.76:6789/0 conn(0x7f2ab009bb00 legacy=0x7f2ab01ca1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 monclient: found mon.b 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0094c50 con 0x7f2ab0138da0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 5 monclient: authenticate success, global_id 134335 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _renew_subs 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab0094f30 con 0x7f2ab0138da0 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _renew_subs 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab0094360 con 0x7f2ab0138da0 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aa80047a0 con 0x7f2ab0138da0 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa3fff700 4 set_mon_vals no callback set 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa8004b30 con 0x7f2ab0138da0 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab85a7700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a98005100 con 0x7f2ab0138da0 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 10 monclient: dump: 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.453 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2aa8004d10 con 0x7f2ab0138da0 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7f2aa8043c30 con 0x7f2ab0138da0 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.436+0000 7f2ab6343700 10 monclient: get_auth_request con 0x7f2a94035f80 auth_method 0 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.444+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2aa8014c10 con 0x7f2ab0138da0 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.444+0000 7f2aa2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.444+0000 7f2aa2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:31.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2ab85a7700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T06:41:31.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2ab85a7700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T06:41:31.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2ab85a7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2ab85a7700 1 -- 172.21.15.189:0/3141472802 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) v1 -- 0x7f2a980053b0 con 0x7f2ab0138da0 2024-09-16T06:41:31.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2aa2ffd700 1 -- 172.21.15.189:0/3141472802 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v78) v1 ==== 103+0+0 (unknown 2925398878 0 0) 0x7f2aa8013080 con 0x7f2ab0138da0 2024-09-16T06:41:31.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2aa2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T06:41:31.615 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.608+0000 7f2aa2ffd700 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-16T06:41:31.616 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-16T06:41:31.620 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:31.620 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T06:41:31.620 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:31.620 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.720+0000 7fa486417700 10 monclient: get_monmap_and_config 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.720+0000 7fa486417700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.720+0000 7fa486417700 10 monclient: monmap: 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:31.728343+0000 2024-09-16T06:41:31.727 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:31.728343+0000 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:31.728 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 1 Processor -- start 2024-09-16T06:41:31.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 1 -- start start 2024-09-16T06:41:31.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient: init 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient(hunting): picked mon.noname-c con 0x7fa480073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient(hunting): picked mon.noname-b con 0x7fa480073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient(hunting): picked mon.noname-a con 0x7fa480139840 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa480098910 con 0x7fa480139840 2024-09-16T06:41:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48009e6f0 con 0x7fa480073b60 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48009db30 con 0x7fa480073f40 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient(hunting): authenticate will time out at 1044.242507s 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa480073b60 0x7fa48006b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53658/0 (socket says 172.21.15.189:53658) 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47ffff700 1 -- 172.21.15.189:0/256152053 learned_addr learned my addr 172.21.15.189:0/256152053 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa4849b4700 1 -- 172.21.15.189:0/256152053 >> v1:172.21.15.76:6789/0 conn(0x7fa480139840 legacy=0x7fa48013bc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734517629 0 0) 0x7fa48009e6f0 con 0x7fa480073b60 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): my global_id is 134338 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4680035d0 con 0x7fa480073b60 2024-09-16T06:41:31.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698731165 0 0) 0x7fa48009db30 con 0x7fa480073f40 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient(hunting): my global_id is 124625 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48009e6f0 con 0x7fa480073f40 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103224112 0 0) 0x7fa4680035d0 con 0x7fa480073b60 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 >> v1:172.21.15.76:6790/0 conn(0x7fa480073f40 legacy=0x7fa480074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 >> v1:172.21.15.76:6789/0 conn(0x7fa480139840 legacy=0x7fa48013bc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 monclient: found mon.noname-c 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48009dcc0 con 0x7fa480073b60 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 5 monclient: authenticate success, global_id 134338 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa470001610 con 0x7fa480073b60 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: dump: 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.736 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa470001830 con 0x7fa480073b60 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:31.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 1 -- 172.21.15.189:0/256152053 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa470001bc0 con 0x7fa480073b60 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa486417700 4 set_mon_vals no callback set 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.724+0000 7fa47effd700 10 monclient: dump: 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.738 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 10 monclient: shutdown 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- 172.21.15.189:0/256152053 >> v1:172.21.15.189:6789/0 conn(0x7fa480073b60 legacy=0x7fa48006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- 172.21.15.189:0/256152053 shutdown_connections 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- 172.21.15.189:0/256152053 >> 172.21.15.189:0/256152053 conn(0x7fa4800a1100 msgr2=0x7fa4800a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- 172.21.15.189:0/256152053 shutdown_connections 2024-09-16T06:41:31.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- 172.21.15.189:0/256152053 wait complete. 2024-09-16T06:41:31.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 10 monclient: build_initial_monmap 2024-09-16T06:41:31.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 10 monclient: monmap: 2024-09-16T06:41:31.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 Processor -- start 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 1 -- start start 2024-09-16T06:41:31.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 10 monclient: init 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.728+0000 7fa486417700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient(hunting): picked mon.noname-b con 0x7fa480073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient(hunting): picked mon.noname-c con 0x7fa480073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient(hunting): picked mon.noname-a con 0x7fa480139840 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48009d3e0 con 0x7fa480139840 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4800995a0 con 0x7fa480073b60 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48009b610 con 0x7fa480073f40 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient(hunting): authenticate will time out at 1044.248252s 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa480073b60 0x7fa48009b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53672/0 (socket says 172.21.15.189:53672) 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47ffff700 1 -- 172.21.15.189:0/3779480213 learned_addr learned my addr 172.21.15.189:0/3779480213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:31.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa4849b4700 1 -- 172.21.15.189:0/3779480213 >> v1:172.21.15.76:6789/0 conn(0x7fa480139840 legacy=0x7fa4801d2430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708240634 0 0) 0x7fa4800995a0 con 0x7fa480073b60 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): my global_id is 134341 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c0035d0 con 0x7fa480073b60 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4071114963 0 0) 0x7fa46c0035d0 con 0x7fa480073b60 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4800995a0 con 0x7fa480073b60 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa470001510 con 0x7fa480073b60 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:31.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): dump: 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686910701 0 0) 0x7fa48009b610 con 0x7fa480073f40 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient(hunting): my global_id is 124628 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c0035d0 con 0x7fa480073f40 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2820983341 0 0) 0x7fa4800995a0 con 0x7fa480073b60 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 >> v1:172.21.15.76:6790/0 conn(0x7fa480073f40 legacy=0x7fa48009c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 >> v1:172.21.15.76:6789/0 conn(0x7fa480139840 legacy=0x7fa4801d2430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:31.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 monclient: found mon.b 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48009c740 con 0x7fa480073b60 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 5 monclient: authenticate success, global_id 134341 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _renew_subs 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480092610 con 0x7fa480073b60 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _renew_subs 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa480137b30 con 0x7fa480073b60 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa486417700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48009d590 con 0x7fa480073b60 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa470004c10 con 0x7fa480073b60 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa470004d80 con 0x7fa480073b60 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 10 monclient: dump: 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:31.752 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47dffb700 4 set_mon_vals no callback set 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.732+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa47000f7f0 con 0x7fa480073b60 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.736+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7fa470043830 con 0x7fa480073b60 2024-09-16T06:41:31.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.736+0000 7fa47f7fe700 10 monclient: get_auth_request con 0x7fa46c0361b0 auth_method 0 2024-09-16T06:41:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.744+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa470014620 con 0x7fa480073b60 2024-09-16T06:41:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.744+0000 7fa47cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:31.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.744+0000 7fa47cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:31.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.920+0000 7fa486417700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:31.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.920+0000 7fa486417700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:31.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.920+0000 7fa486417700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:31.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.920+0000 7fa486417700 1 -- 172.21.15.189:0/3779480213 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fa480138140 con 0x7fa480073b60 2024-09-16T06:41:31.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.920+0000 7fa47cff9700 1 -- 172.21.15.189:0/3779480213 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v78) v1 ==== 99+0+0 (unknown 212234542 0 0) 0x7fa470012cc0 con 0x7fa480073b60 2024-09-16T06:41:31.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.924+0000 7fa47cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:31.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:31.924+0000 7fa47cff9700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T06:41:31.929 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T06:41:31.933 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:31.933 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush unlink host2 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.024+0000 7f12b9950700 10 monclient: get_monmap_and_config 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.024+0000 7f12b9950700 10 monclient: build_initial_monmap 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.024+0000 7f12b9950700 10 monclient: monmap: 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:32.031125+0000 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:32.031125+0000 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:32.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.024+0000 7f12b9950700 1 Processor -- start 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- start start 2024-09-16T06:41:32.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: init 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-b con 0x7f12b40a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-c con 0x7f12b40a3d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-a con 0x7f12b40a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:32.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b4099a40 con 0x7f12b40a2350 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b4099ef0 con 0x7f12b40a3d10 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b409a1d0 con 0x7f12b40a27a0 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient(hunting): authenticate will time out at 1044.544746s 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12aa7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12b40a3d10 0x7f12b413a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53684/0 (socket says 172.21.15.189:53684) 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f12b40a2350 legacy=0x7f12b413dcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12aa7fc700 1 -- 172.21.15.189:0/2075924102 learned_addr learned my addr 172.21.15.189:0/2075924102 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246166183 0 0) 0x7f12b4099ef0 con 0x7f12b40a3d10 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:32.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): my global_id is 134344 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a00035d0 con 0x7f12b40a3d10 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039024525 0 0) 0x7f12b409a1d0 con 0x7f12b40a27a0 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient(hunting): my global_id is 124631 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b4099ef0 con 0x7f12b40a27a0 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285253910 0 0) 0x7f12a00035d0 con 0x7f12b40a3d10 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 >> v1:172.21.15.76:6790/0 conn(0x7f12b40a27a0 legacy=0x7f12b40a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 >> v1:172.21.15.76:6789/0 conn(0x7f12b40a2350 legacy=0x7f12b413dcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 monclient: found mon.noname-c 2024-09-16T06:41:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40a3290 con 0x7f12b40a3d10 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f129c001610 con 0x7f12b40a3d10 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 5 monclient: authenticate success, global_id 134344 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: dump: 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f129c001830 con 0x7f12b40a3d10 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 1 -- 172.21.15.189:0/2075924102 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f129c001bc0 con 0x7f12b40a3d10 2024-09-16T06:41:32.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 4 set_mon_vals no callback set 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b27fc700 10 monclient: dump: 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:32.040 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: shutdown 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- 172.21.15.189:0/2075924102 >> v1:172.21.15.189:6789/0 conn(0x7f12b40a3d10 legacy=0x7f12b413a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- 172.21.15.189:0/2075924102 shutdown_connections 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- 172.21.15.189:0/2075924102 >> 172.21.15.189:0/2075924102 conn(0x7f12b40a7d70 msgr2=0x7f12b40a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- 172.21.15.189:0/2075924102 shutdown_connections 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 -- 172.21.15.189:0/2075924102 wait complete. 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: build_initial_monmap 2024-09-16T06:41:32.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 10 monclient: monmap: 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.028+0000 7f12b9950700 1 Processor -- start 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- start start 2024-09-16T06:41:32.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: init 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-b con 0x7f12b40a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-a con 0x7f12b40a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient(hunting): picked mon.noname-c con 0x7f12b413ec10 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b413b650 con 0x7f12b40a27a0 2024-09-16T06:41:32.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b413d060 con 0x7f12b40a2350 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b406bea0 con 0x7f12b413ec10 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient(hunting): authenticate will time out at 1044.548861s 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12aa7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f12b40a27a0 legacy=0x7f12b413af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12b40a2350 0x7f12b413a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53692/0 (socket says 172.21.15.189:53692) 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b2ffd700 1 -- 172.21.15.189:0/881570149 learned_addr learned my addr 172.21.15.189:0/881570149 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472904769 0 0) 0x7f12b413d060 con 0x7f12b40a2350 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:32.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): my global_id is 134347 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12940035d0 con 0x7f12b40a2350 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484371078 0 0) 0x7f12b406bea0 con 0x7f12b413ec10 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): my global_id is 124634 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b413d060 con 0x7f12b413ec10 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 372017437 0 0) 0x7f12940035d0 con 0x7f12b40a2350 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12b406bea0 con 0x7f12b40a2350 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f12a4001970 con 0x7f12b40a2350 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3540426618 0 0) 0x7f12b406bea0 con 0x7f12b40a2350 2024-09-16T06:41:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 >> v1:172.21.15.76:6790/0 conn(0x7f12b413ec10 legacy=0x7f12b41d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 >> v1:172.21.15.76:6789/0 conn(0x7f12b40a27a0 legacy=0x7f12b413af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 monclient: found mon.b 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b409b550 con 0x7f12b40a2350 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 5 monclient: authenticate success, global_id 134347 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _renew_subs 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b413e7f0 con 0x7f12b40a2350 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _renew_subs 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b413e510 con 0x7f12b40a2350 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.032+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b409c240 con 0x7f12b40a2350 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12a4001ce0 con 0x7f12b40a2350 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f12a4003b10 con 0x7f12b40a2350 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 10 monclient: dump: 2024-09-16T06:41:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b1ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f12a4003c70 con 0x7f12b40a2350 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(78..78 src has 50..78) v4 ==== 5181+0+0 (unknown 789217677 0 0) 0x7f12a4043790 con 0x7f12b40a2350 2024-09-16T06:41:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.036+0000 7f12aa7fc700 10 monclient: get_auth_request con 0x7f1294036180 auth_method 0 2024-09-16T06:41:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.040+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f12a40146f0 con 0x7f12b40a2350 2024-09-16T06:41:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.040+0000 7f12b0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.040+0000 7f12b0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:32.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.212+0000 7f12b9950700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:32.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.212+0000 7f12b9950700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:32.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.212+0000 7f12b9950700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:32.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:32.212+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f12b4066fa0 con 0x7f12b40a2350 2024-09-16T06:41:33.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.116+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v79) v1 ==== 128+0+0 (unknown 3388351724 0 0) 0x7f12a40142e0 con 0x7f12b40a2350 2024-09-16T06:41:33.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.116+0000 7f12b0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:33.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.116+0000 7f12b0ff9700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T06:41:33.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.224+0000 7f12b9950700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:33.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.224+0000 7f12b9950700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:33.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.224+0000 7f12b9950700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:33.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:33.224+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f12b406aac0 con 0x7f12b40a2350 2024-09-16T06:41:34.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.124+0000 7f12b0ff9700 1 -- 172.21.15.189:0/881570149 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v80) v1 ==== 128+0+0 (unknown 385943926 0 0) 0x7f12a40144c0 con 0x7f12b40a2350 2024-09-16T06:41:34.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.124+0000 7f12b0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T06:41:34.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.124+0000 7f12b0ff9700 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T06:41:34.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1294036180 legacy=0x7f1294038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:34.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 10 monclient: shutdown 2024-09-16T06:41:34.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:34.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 >> v1:172.21.15.189:6789/0 conn(0x7f12b40a2350 legacy=0x7f12b413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:34.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 shutdown_connections 2024-09-16T06:41:34.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 >> 172.21.15.189:0/881570149 conn(0x7f12b40a7d70 msgr2=0x7f12b40a05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:34.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 shutdown_connections 2024-09-16T06:41:34.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.128+0000 7f12b9950700 1 -- 172.21.15.189:0/881570149 wait complete. 2024-09-16T06:41:34.136 INFO:tasks.workunit.client.0.smithi189.stderr:removed item id -16 name 'host2' from crush map 2024-09-16T06:41:34.152 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-16T06:41:34.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: get_monmap_and_config 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: build_initial_monmap 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: monmap: 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:34.274803+0000 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:34.274803+0000 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:34.274 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:34.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 1 Processor -- start 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 1 -- start start 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: init 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-b con 0x7fefec09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:34.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-a con 0x7fefec099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-c con 0x7fefec1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec09f520 con 0x7fefec099700 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec09bc10 con 0x7fefec1396a0 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec09b810 con 0x7fefec09f140 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7feff2cff700 10 monclient(hunting): authenticate will time out at 1046.787851s 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.268+0000 7fefebfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fefec099700 legacy=0x7fefec09b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff129c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fefec1396a0 0x7fefec13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53706/0 (socket says 172.21.15.189:53706) 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff129c700 1 -- 172.21.15.189:0/3931271935 learned_addr learned my addr 172.21.15.189:0/3931271935 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091415165 0 0) 0x7fefec09bc10 con 0x7fefec1396a0 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:34.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): my global_id is 134350 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd40035d0 con 0x7fefec1396a0 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977299753 0 0) 0x7fefec09b810 con 0x7fefec09f140 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient(hunting): my global_id is 124637 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefec09bc10 con 0x7fefec09f140 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440402325 0 0) 0x7fefd40035d0 con 0x7fefec1396a0 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 >> v1:172.21.15.76:6790/0 conn(0x7fefec09f140 legacy=0x7fefec09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 >> v1:172.21.15.76:6789/0 conn(0x7fefec099700 legacy=0x7fefec09b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:34.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 monclient: found mon.noname-c 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec09ceb0 con 0x7fefec1396a0 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 5 monclient: authenticate success, global_id 134350 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fefe4001990 con 0x7fefec1396a0 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: dump: 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe4001bb0 con 0x7fefec1396a0 2024-09-16T06:41:34.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 4 set_mon_vals no callback set 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 1 -- 172.21.15.189:0/3931271935 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fefe4001f80 con 0x7fefec1396a0 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7fefeb7fe700 10 monclient: dump: 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:34.283 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 10 monclient: shutdown 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 1 -- 172.21.15.189:0/3931271935 >> v1:172.21.15.189:6789/0 conn(0x7fefec1396a0 legacy=0x7fefec13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 1 -- 172.21.15.189:0/3931271935 shutdown_connections 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 1 -- 172.21.15.189:0/3931271935 >> 172.21.15.189:0/3931271935 conn(0x7fefec0a03e0 msgr2=0x7fefec0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 1 -- 172.21.15.189:0/3931271935 shutdown_connections 2024-09-16T06:41:34.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 1 -- 172.21.15.189:0/3931271935 wait complete. 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 10 monclient: build_initial_monmap 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.272+0000 7feff2cff700 10 monclient: monmap: 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:34.288 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 Processor -- start 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- start start 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: init 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-b con 0x7fefec099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-a con 0x7fefec09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:34.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient(hunting): picked mon.noname-c con 0x7fefec1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec09a6e0 con 0x7fefec09f140 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec06c810 con 0x7fefec099700 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff0a9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fefec099700 0x7fefec136ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53720/0 (socket says 172.21.15.189:53720) 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff0a9b700 1 -- 172.21.15.189:0/3898006579 learned_addr learned my addr 172.21.15.189:0/3898006579 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefebfff700 1 -- 172.21.15.189:0/3898006579 >> v1:172.21.15.76:6789/0 conn(0x7fefec09f140 legacy=0x7fefec1375f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec06caf0 con 0x7fefec1396a0 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient(hunting): authenticate will time out at 1046.794324s 2024-09-16T06:41:34.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799691391 0 0) 0x7fefec06c810 con 0x7fefec099700 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): my global_id is 134353 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd80035d0 con 0x7fefec099700 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123066572 0 0) 0x7fefd80035d0 con 0x7fefec099700 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefec06c810 con 0x7fefec099700 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fefe00019d0 con 0x7fefec099700 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): dump: 2024-09-16T06:41:34.291 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187130009 0 0) 0x7fefec06caf0 con 0x7fefec1396a0 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:34.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient(hunting): my global_id is 124640 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd80035d0 con 0x7fefec1396a0 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 936748071 0 0) 0x7fefec06c810 con 0x7fefec099700 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 >> v1:172.21.15.76:6790/0 conn(0x7fefec1396a0 legacy=0x7fefec137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 >> v1:172.21.15.76:6789/0 conn(0x7fefec09f140 legacy=0x7fefec1375f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 monclient: found mon.b 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec06cdd0 con 0x7fefec099700 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:34.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 5 monclient: authenticate success, global_id 134353 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _renew_subs 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec138a30 con 0x7fefec099700 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _renew_subs 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec06bc60 con 0x7fefec099700 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe0001d20 con 0x7fefec099700 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fefe0003b30 con 0x7fefec099700 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefeaffd700 4 set_mon_vals no callback set 2024-09-16T06:41:34.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.276+0000 7fefe97fa700 10 monclient: dump: 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fefe0003c90 con 0x7fefec099700 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7feff2cff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:34.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7feff2cff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7feff2cff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc005100 con 0x7fefec099700 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(80..80 src has 50..80) v4 ==== 5133+0+0 (unknown 353350660 0 0) 0x7fefe00439c0 con 0x7fefec099700 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.280+0000 7fefebfff700 10 monclient: get_auth_request con 0x7fefd8036180 auth_method 0 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.288+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fefe0014950 con 0x7fefec099700 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.288+0000 7fefe97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:34.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.288+0000 7fefe97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:34.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.460+0000 7feff2cff700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:34.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.460+0000 7feff2cff700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:34.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.460+0000 7feff2cff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:34.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:34.460+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7fefcc0053b0 con 0x7fefec099700 2024-09-16T06:41:35.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.128+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v81) v1 ==== 251+0+0 (unknown 3594488747 0 0) 0x7fefe0012dc0 con 0x7fefec099700 2024-09-16T06:41:35.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.128+0000 7fefe97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:35.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.128+0000 7fefe97fa700 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-16T06:41:35.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7feff2cff700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:35.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7feff2cff700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:35.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7feff2cff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7fefcc005c10 con 0x7fefec099700 2024-09-16T06:41:35.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7fefe97fa700 1 -- 172.21.15.189:0/3898006579 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v81) v1 ==== 200+0+0 (unknown 1969133147 0 0) 0x7fefe0014540 con 0x7fefec099700 2024-09-16T06:41:35.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7fefe97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T06:41:35.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.236+0000 7fefe97fa700 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-16T06:41:35.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 >> v1:172.21.15.189:6812/3098810674 conn(0x7fefd8036180 legacy=0x7fefd8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 10 monclient: shutdown 2024-09-16T06:41:35.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:35.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 >> v1:172.21.15.189:6789/0 conn(0x7fefec099700 legacy=0x7fefec136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 shutdown_connections 2024-09-16T06:41:35.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 >> 172.21.15.189:0/3898006579 conn(0x7fefec0a03e0 msgr2=0x7fefec0a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:35.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 shutdown_connections 2024-09-16T06:41:35.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.240+0000 7feff2cff700 1 -- 172.21.15.189:0/3898006579 wait complete. 2024-09-16T06:41:35.246 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'host-for-test' already exists 2024-09-16T06:41:35.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:35.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep host-for-test 2024-09-16T06:41:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.352+0000 7fb92b0de700 10 monclient: get_monmap_and_config 2024-09-16T06:41:35.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.352+0000 7fb92b0de700 10 monclient: build_initial_monmap 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.352+0000 7fb92b0de700 10 monclient: monmap: 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:35.359705+0000 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:35.359705+0000 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:35.359 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 Processor -- start 2024-09-16T06:41:35.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- start start 2024-09-16T06:41:35.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient: init 2024-09-16T06:41:35.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-c con 0x7fb9240984b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-b con 0x7fb924072140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-a con 0x7fb9241384d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92409b2a0 con 0x7fb9241384d0 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb924099940 con 0x7fb9240984b0 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb924099d10 con 0x7fb924072140 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:35.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient(hunting): authenticate will time out at 1047.873101s 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92967b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb9241384d0 legacy=0x7fb92413a8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb928e7a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb9240984b0 0x7fb924098890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53730/0 (socket says 172.21.15.189:53730) 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb928e7a700 1 -- 172.21.15.189:0/1327879917 learned_addr learned my addr 172.21.15.189:0/1327879917 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219615268 0 0) 0x7fb924099940 con 0x7fb9240984b0 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): my global_id is 134356 2024-09-16T06:41:35.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9140057e0 con 0x7fb9240984b0 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860287800 0 0) 0x7fb924099d10 con 0x7fb924072140 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient(hunting): my global_id is 124643 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb924099940 con 0x7fb924072140 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481840018 0 0) 0x7fb9140057e0 con 0x7fb9240984b0 2024-09-16T06:41:35.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 >> v1:172.21.15.76:6790/0 conn(0x7fb924072140 legacy=0x7fb924098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 >> v1:172.21.15.76:6789/0 conn(0x7fb9241384d0 legacy=0x7fb92413a8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 monclient: found mon.noname-c 2024-09-16T06:41:35.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9240996b0 con 0x7fb9240984b0 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb918001610 con 0x7fb9240984b0 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 5 monclient: authenticate success, global_id 134356 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: dump: 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9180018f0 con 0x7fb9240984b0 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 1 -- 172.21.15.189:0/1327879917 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb918001cc0 con 0x7fb9240984b0 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 4 set_mon_vals no callback set 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb9237fe700 10 monclient: dump: 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 10 monclient: shutdown 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- 172.21.15.189:0/1327879917 >> v1:172.21.15.189:6789/0 conn(0x7fb9240984b0 legacy=0x7fb924098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- 172.21.15.189:0/1327879917 shutdown_connections 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- 172.21.15.189:0/1327879917 >> 172.21.15.189:0/1327879917 conn(0x7fb92409fe30 msgr2=0x7fb9240a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:35.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- 172.21.15.189:0/1327879917 shutdown_connections 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.356+0000 7fb92b0de700 1 -- 172.21.15.189:0/1327879917 wait complete. 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient: build_initial_monmap 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient: monmap: 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:35.377 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 1 Processor -- start 2024-09-16T06:41:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 1 -- start start 2024-09-16T06:41:35.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient: init 2024-09-16T06:41:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-b con 0x7fb924072140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-c con 0x7fb9240984b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient(hunting): picked mon.noname-a con 0x7fb9241384d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9240a2c80 con 0x7fb9241384d0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb924095120 con 0x7fb924072140 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92409c720 con 0x7fb9240984b0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92b0de700 10 monclient(hunting): authenticate will time out at 1047.879239s 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb92967b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb9241384d0 legacy=0x7fb9241d1de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb928e7a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb924072140 0x7fb924137cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53742/0 (socket says 172.21.15.189:53742) 2024-09-16T06:41:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb928e7a700 1 -- 172.21.15.189:0/401620573 learned_addr learned my addr 172.21.15.189:0/401620573 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418100270 0 0) 0x7fb924095120 con 0x7fb924072140 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): my global_id is 134359 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9080035d0 con 0x7fb924072140 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789529473 0 0) 0x7fb92409c720 con 0x7fb9240984b0 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 10 monclient(hunting): my global_id is 124646 2024-09-16T06:41:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.360+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb924095120 con 0x7fb9240984b0 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027413994 0 0) 0x7fb9080035d0 con 0x7fb924072140 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb92409c720 con 0x7fb924072140 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb918001c00 con 0x7fb924072140 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient(hunting): dump: 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3217229693 0 0) 0x7fb92409c720 con 0x7fb924072140 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 >> v1:172.21.15.76:6790/0 conn(0x7fb9240984b0 legacy=0x7fb9241ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 >> v1:172.21.15.76:6789/0 conn(0x7fb9241384d0 legacy=0x7fb9241d1de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 monclient: found mon.b 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92409a520 con 0x7fb924072140 2024-09-16T06:41:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 5 monclient: authenticate success, global_id 134359 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _renew_subs 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb924093990 con 0x7fb924072140 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _renew_subs 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb924096e90 con 0x7fb924072140 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb918004ca0 con 0x7fb924072140 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb918005030 con 0x7fb924072140 2024-09-16T06:41:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb922ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 10 monclient: dump: 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb904005100 con 0x7fb924072140 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb918005210 con 0x7fb924072140 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 50..81) v4 ==== 5658+0+0 (unknown 424125532 0 0) 0x7fb918043bf0 con 0x7fb924072140 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.364+0000 7fb923fff700 10 monclient: get_auth_request con 0x7fb908036180 auth_method 0 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.376+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb918014bd0 con 0x7fb924072140 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.376+0000 7fb9217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.376+0000 7fb9217fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:35.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb92b0de700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:35.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb92b0de700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb92b0de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb904005bc0 con 0x7fb924072140 2024-09-16T06:41:35.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fb918013040 con 0x7fb924072140 2024-09-16T06:41:35.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb9217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.548+0000 7fb9217fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:35.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:35.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb904005680 con 0x7fb924072140 2024-09-16T06:41:35.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb9217fa700 1 -- 172.21.15.189:0/401620573 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fb9180147c0 con 0x7fb924072140 2024-09-16T06:41:35.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb9217fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb9217fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb908036180 legacy=0x7fb908038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 10 monclient: shutdown 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 >> v1:172.21.15.189:6789/0 conn(0x7fb924072140 legacy=0x7fb924137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 shutdown_connections 2024-09-16T06:41:35.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 >> 172.21.15.189:0/401620573 conn(0x7fb92409fe30 msgr2=0x7fb92409cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:35.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 shutdown_connections 2024-09-16T06:41:35.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.648+0000 7fb92b0de700 1 -- 172.21.15.189:0/401620573 wait complete. 2024-09-16T06:41:35.665 INFO:tasks.workunit.client.0.smithi189.stdout:-11 0 host host-for-test 2024-09-16T06:41:35.666 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:35.666 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep rack-for-test 2024-09-16T06:41:35.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.760+0000 7fe83ec90700 10 monclient: get_monmap_and_config 2024-09-16T06:41:35.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.760+0000 7fe83ec90700 10 monclient: build_initial_monmap 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.760+0000 7fe83ec90700 10 monclient: monmap: 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:35.767520+0000 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:35.767520+0000 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:35.767 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.760+0000 7fe83ec90700 1 Processor -- start 2024-09-16T06:41:35.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- start start 2024-09-16T06:41:35.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient: init 2024-09-16T06:41:35.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-a con 0x7fe83809b7a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-b con 0x7fe83809c290 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-c con 0x7fe838138a40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe838097b10 con 0x7fe83809b7a0 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe838097550 con 0x7fe838138a40 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83809f990 con 0x7fe83809c290 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient(hunting): authenticate will time out at 1048.280863s 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83dc8e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe83809b7a0 legacy=0x7fe83809bb80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83e48f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe838138a40 0x7fe83813ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53746/0 (socket says 172.21.15.189:53746) 2024-09-16T06:41:35.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83e48f700 1 -- 172.21.15.189:0/4089471788 learned_addr learned my addr 172.21.15.189:0/4089471788 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:35.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456017466 0 0) 0x7fe838097550 con 0x7fe838138a40 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): my global_id is 134362 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8240057e0 con 0x7fe838138a40 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803001416 0 0) 0x7fe83809f990 con 0x7fe83809c290 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient(hunting): my global_id is 124649 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe838097550 con 0x7fe83809c290 2024-09-16T06:41:35.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656294086 0 0) 0x7fe838097550 con 0x7fe83809c290 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 >> v1:172.21.15.189:6789/0 conn(0x7fe838138a40 legacy=0x7fe83813ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 >> v1:172.21.15.76:6789/0 conn(0x7fe83809b7a0 legacy=0x7fe83809bb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: _finish_hunting 0 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 monclient: found mon.noname-b 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838097830 con 0x7fe83809c290 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: _finish_auth 0 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: _check_auth_tickets 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe8280015f0 con 0x7fe83809c290 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 5 monclient: authenticate success, global_id 124649 2024-09-16T06:41:35.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: dump: 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8280018a0 con 0x7fe83809c290 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 1 -- 172.21.15.189:0/4089471788 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe828001c30 con 0x7fe83809c290 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 4 set_mon_vals no callback set 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83cc8c700 10 monclient: dump: 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 10 monclient: shutdown 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- 172.21.15.189:0/4089471788 >> v1:172.21.15.76:6790/0 conn(0x7fe83809c290 legacy=0x7fe838137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- 172.21.15.189:0/4089471788 shutdown_connections 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- 172.21.15.189:0/4089471788 >> 172.21.15.189:0/4089471788 conn(0x7fe83809fe70 msgr2=0x7fe8380a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- 172.21.15.189:0/4089471788 shutdown_connections 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.764+0000 7fe83ec90700 1 -- 172.21.15.189:0/4089471788 wait complete. 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: build_initial_monmap 2024-09-16T06:41:35.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: monmap: 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:35.777 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 Processor -- start 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- start start 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: init 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-c con 0x7fe83809b7a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-a con 0x7fe83809c290 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:35.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient(hunting): picked mon.noname-b con 0x7fe838138a40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83809a380 con 0x7fe83809c290 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83812a940 con 0x7fe838138a40 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe838130ac0 con 0x7fe83809b7a0 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient(hunting): authenticate will time out at 1048.286160s 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83e48f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe838138a40 0x7fe8381d2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53748/0 (socket says 172.21.15.189:53748) 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83e48f700 1 -- 172.21.15.189:0/2054749575 learned_addr learned my addr 172.21.15.189:0/2054749575 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83d48d700 1 -- 172.21.15.189:0/2054749575 >> v1:172.21.15.76:6789/0 conn(0x7fe83809c290 legacy=0x7fe8381ca300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187263824 0 0) 0x7fe83812a940 con 0x7fe838138a40 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): my global_id is 134365 2024-09-16T06:41:35.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81c0035d0 con 0x7fe838138a40 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059315196 0 0) 0x7fe838130ac0 con 0x7fe83809b7a0 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): my global_id is 124652 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe83812a940 con 0x7fe83809b7a0 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60096882 0 0) 0x7fe81c0035d0 con 0x7fe838138a40 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe838130ac0 con 0x7fe838138a40 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe830001d70 con 0x7fe838138a40 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): dump: 2024-09-16T06:41:35.783 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3341126319 0 0) 0x7fe838130ac0 con 0x7fe838138a40 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 >> v1:172.21.15.76:6790/0 conn(0x7fe83809b7a0 legacy=0x7fe8381c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 >> v1:172.21.15.76:6789/0 conn(0x7fe83809c290 legacy=0x7fe8381ca300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:35.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 monclient: found mon.b 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8381360e0 con 0x7fe838138a40 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 5 monclient: authenticate success, global_id 134365 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _renew_subs 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8381caa10 con 0x7fe838138a40 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe830004c10 con 0x7fe838138a40 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _renew_subs 2024-09-16T06:41:35.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83812a660 con 0x7fe838138a40 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe830004f20 con 0x7fe838138a40 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82ffff700 4 set_mon_vals no callback set 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.768+0000 7fe82e7fc700 10 monclient: dump: 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:35.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe83000f990 con 0x7fe838138a40 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 50..81) v4 ==== 5658+0+0 (unknown 424125532 0 0) 0x7fe830043d90 con 0x7fe838138a40 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe83dc8e700 10 monclient: get_auth_request con 0x7fe81c036200 auth_method 0 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe83ec90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe83ec90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe83ec90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.772+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83812aaf0 con 0x7fe838138a40 2024-09-16T06:41:35.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.780+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe83812aaf0 con 0x7fe838138a40 2024-09-16T06:41:35.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.780+0000 7fe82e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:35.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.780+0000 7fe82e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:35.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.952+0000 7fe83ec90700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:35.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.952+0000 7fe83ec90700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.952+0000 7fe83ec90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:35.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.952+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe83809c9b0 con 0x7fe838138a40 2024-09-16T06:41:35.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.956+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fe8300148e0 con 0x7fe838138a40 2024-09-16T06:41:35.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.956+0000 7fe82e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:35.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:35.956+0000 7fe82e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:36.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe83812a080 con 0x7fe838138a40 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe82e7fc700 1 -- 172.21.15.189:0/2054749575 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fe830012f80 con 0x7fe838138a40 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe82e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe82e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:36.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe81c036200 legacy=0x7fe81c0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 10 monclient: shutdown 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 >> v1:172.21.15.189:6789/0 conn(0x7fe838138a40 legacy=0x7fe8381d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 shutdown_connections 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 >> 172.21.15.189:0/2054749575 conn(0x7fe83809fe70 msgr2=0x7fe8381335a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 shutdown_connections 2024-09-16T06:41:36.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.056+0000 7fe83ec90700 1 -- 172.21.15.189:0/2054749575 wait complete. 2024-09-16T06:41:36.072 INFO:tasks.workunit.client.0.smithi189.stdout:-12 0 rack rack-for-test 2024-09-16T06:41:36.072 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:41:36.073 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep root-for-test 2024-09-16T06:41:36.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 10 monclient: get_monmap_and_config 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 10 monclient: build_initial_monmap 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 10 monclient: monmap: 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:36.171588+0000 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:36.171588+0000 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:36.171 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 1 Processor -- start 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 1 -- start start 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.164+0000 7fcc39af8700 10 monclient: init 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-c con 0x7fcc3409ed40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-b con 0x7fcc34097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-a con 0x7fcc3409f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc340973e0 con 0x7fcc3409f830 2024-09-16T06:41:36.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc34098890 con 0x7fcc3409ed40 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc34098b70 con 0x7fcc34097860 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient(hunting): authenticate will time out at 1048.684405s 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc337fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc3409ed40 0x7fcc3409f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53758/0 (socket says 172.21.15.189:53758) 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc33fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fcc3409f830 legacy=0x7fcc3413be10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc337fe700 1 -- 172.21.15.189:0/1794685175 learned_addr learned my addr 172.21.15.189:0/1794685175 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311708366 0 0) 0x7fcc34098890 con 0x7fcc3409ed40 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): my global_id is 134368 2024-09-16T06:41:36.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc200035d0 con 0x7fcc3409ed40 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222852388 0 0) 0x7fcc34098b70 con 0x7fcc34097860 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient(hunting): my global_id is 124655 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc34098890 con 0x7fcc34097860 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656077796 0 0) 0x7fcc200035d0 con 0x7fcc3409ed40 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 >> v1:172.21.15.76:6790/0 conn(0x7fcc34097860 legacy=0x7fcc341386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 >> v1:172.21.15.76:6789/0 conn(0x7fcc3409f830 legacy=0x7fcc3413be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 monclient: found mon.noname-c 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:36.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34098e50 con 0x7fcc3409ed40 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 5 monclient: authenticate success, global_id 134368 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcc24001610 con 0x7fcc3409ed40 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: dump: 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc24001880 con 0x7fcc3409ed40 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 4 set_mon_vals no callback set 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 1 -- 172.21.15.189:0/1794685175 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcc24001c10 con 0x7fcc3409ed40 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc32ffd700 10 monclient: dump: 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: shutdown 2024-09-16T06:41:36.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- 172.21.15.189:0/1794685175 >> v1:172.21.15.189:6789/0 conn(0x7fcc3409ed40 legacy=0x7fcc3409f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- 172.21.15.189:0/1794685175 shutdown_connections 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- 172.21.15.189:0/1794685175 >> 172.21.15.189:0/1794685175 conn(0x7fcc3409fe30 msgr2=0x7fcc340a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- 172.21.15.189:0/1794685175 shutdown_connections 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- 172.21.15.189:0/1794685175 wait complete. 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: build_initial_monmap 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: monmap: 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:36.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 Processor -- start 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 1 -- start start 2024-09-16T06:41:36.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.168+0000 7fcc39af8700 10 monclient: init 2024-09-16T06:41:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:36.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-a con 0x7fcc3409ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-b con 0x7fcc3409f830 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient(hunting): picked mon.noname-c con 0x7fcc341cfd70 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc34099c90 con 0x7fcc3409ed40 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3412b570 con 0x7fcc3409f830 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3412d180 con 0x7fcc341cfd70 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient(hunting): authenticate will time out at 1048.688666s 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc2bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc3409f830 0x7fcc34133700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53770/0 (socket says 172.21.15.189:53770) 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc2bfff700 1 -- 172.21.15.189:0/678712394 learned_addr learned my addr 172.21.15.189:0/678712394 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc337fe700 1 -- 172.21.15.189:0/678712394 >> v1:172.21.15.76:6789/0 conn(0x7fcc3409ed40 legacy=0x7fcc341c9a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522848933 0 0) 0x7fcc3412b570 con 0x7fcc3409f830 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): my global_id is 134371 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc140035d0 con 0x7fcc3409f830 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97645599 0 0) 0x7fcc3412d180 con 0x7fcc341cfd70 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): my global_id is 124658 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3412b570 con 0x7fcc341cfd70 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577150368 0 0) 0x7fcc140035d0 con 0x7fcc3409f830 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc3412d180 con 0x7fcc3409f830 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcc1c001910 con 0x7fcc3409f830 2024-09-16T06:41:36.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:36.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1190155022 0 0) 0x7fcc3412d180 con 0x7fcc3409f830 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 >> v1:172.21.15.76:6790/0 conn(0x7fcc341cfd70 legacy=0x7fcc341d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 >> v1:172.21.15.76:6789/0 conn(0x7fcc3409ed40 legacy=0x7fcc341c9a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 monclient: found mon.b 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3412add0 con 0x7fcc3409f830 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 5 monclient: authenticate success, global_id 134371 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _renew_subs 2024-09-16T06:41:36.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3406bfb0 con 0x7fcc3409f830 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _renew_subs 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3409a520 con 0x7fcc3409f830 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc1c001ca0 con 0x7fcc3409f830 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcc1c003b10 con 0x7fcc3409f830 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 10 monclient: dump: 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.190 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc327fc700 4 set_mon_vals no callback set 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18005100 con 0x7fcc3409f830 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcc1c003c70 con 0x7fcc3409f830 2024-09-16T06:41:36.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.172+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 50..81) v4 ==== 5658+0+0 (unknown 424125532 0 0) 0x7fcc1c043950 con 0x7fcc3409f830 2024-09-16T06:41:36.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.176+0000 7fcc337fe700 10 monclient: get_auth_request con 0x7fcc14036160 auth_method 0 2024-09-16T06:41:36.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.180+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcc1c014850 con 0x7fcc3409f830 2024-09-16T06:41:36.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.180+0000 7fcc30ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.180+0000 7fcc30ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:36.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc39af8700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc39af8700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc39af8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fcc18005bc0 con 0x7fcc3409f830 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fcc1c012cc0 con 0x7fcc3409f830 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc30ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.344+0000 7fcc30ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc39af8700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:41:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc39af8700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc39af8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fcc18005680 con 0x7fcc3409f830 2024-09-16T06:41:36.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc30ff9700 1 -- 172.21.15.189:0/678712394 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1500 (unknown 3717179170 0 2357312645) 0x7fcc1c012cc0 con 0x7fcc3409f830 2024-09-16T06:41:36.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc30ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:41:36.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.440+0000 7fcc30ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:41:36.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcc14036160 legacy=0x7fcc14038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 10 monclient: shutdown 2024-09-16T06:41:36.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:36.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 >> v1:172.21.15.189:6789/0 conn(0x7fcc3409f830 legacy=0x7fcc34133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 shutdown_connections 2024-09-16T06:41:36.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 >> 172.21.15.189:0/678712394 conn(0x7fcc3409fe30 msgr2=0x7fcc341b93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:36.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 shutdown_connections 2024-09-16T06:41:36.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.444+0000 7fcc39af8700 1 -- 172.21.15.189:0/678712394 wait complete. 2024-09-16T06:41:36.458 INFO:tasks.workunit.client.0.smithi189.stdout:-14 0 root root-for-test 2024-09-16T06:41:36.459 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host-for-test 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: get_monmap_and_config 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: build_initial_monmap 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: monmap: 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:36.561884+0000 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:36.561884+0000 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:36.561 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:36.562 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 1 Processor -- start 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 1 -- start start 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: init 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:36.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7acc09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7acc09beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7acc09c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc099ed0 con 0x7f7acc09beb0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc09d2e0 con 0x7f7acc09c2e0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc09ef80 con 0x7f7acc09eba0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acbfff700 10 monclient(hunting): authenticate will time out at 1049.075333s 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7aca7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7acc09beb0 legacy=0x7f7acc137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7acc09c2e0 0x7f7acc13ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53774/0 (socket says 172.21.15.189:53774) 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7acb7fe700 1 -- 172.21.15.189:0/2042967015 learned_addr learned my addr 172.21.15.189:0/2042967015 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942372436 0 0) 0x7f7acc09d2e0 con 0x7f7acc09c2e0 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 10 monclient(hunting): my global_id is 134374 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.556+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab40035d0 con 0x7f7acc09c2e0 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723661984 0 0) 0x7f7acc09ef80 con 0x7f7acc09eba0 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient(hunting): my global_id is 124661 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7acc09d2e0 con 0x7f7acc09eba0 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1547745271 0 0) 0x7f7ab40035d0 con 0x7f7acc09c2e0 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 >> v1:172.21.15.76:6790/0 conn(0x7f7acc09eba0 legacy=0x7f7acc09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 >> v1:172.21.15.76:6789/0 conn(0x7f7acc09beb0 legacy=0x7f7acc137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 monclient: found mon.noname-c 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc09cc70 con 0x7f7acc09c2e0 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ac4001990 con 0x7f7acc09c2e0 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 5 monclient: authenticate success, global_id 134374 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: dump: 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ac4001c00 con 0x7f7acc09c2e0 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 1 -- 172.21.15.189:0/2042967015 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ac4001f90 con 0x7f7acc09c2e0 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 4 set_mon_vals no callback set 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7ac9ffb700 10 monclient: dump: 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.571 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 10 monclient: shutdown 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 1 -- 172.21.15.189:0/2042967015 >> v1:172.21.15.189:6789/0 conn(0x7f7acc09c2e0 legacy=0x7f7acc13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 1 -- 172.21.15.189:0/2042967015 shutdown_connections 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 1 -- 172.21.15.189:0/2042967015 >> 172.21.15.189:0/2042967015 conn(0x7f7acc09fe70 msgr2=0x7f7acc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 1 -- 172.21.15.189:0/2042967015 shutdown_connections 2024-09-16T06:41:36.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 1 -- 172.21.15.189:0/2042967015 wait complete. 2024-09-16T06:41:36.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 10 monclient: build_initial_monmap 2024-09-16T06:41:36.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.560+0000 7f7acbfff700 10 monclient: monmap: 2024-09-16T06:41:36.576 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:36.576 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 Processor -- start 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- start start 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: init 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:36.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7acc09beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7acc09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7acc09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc12d3b0 con 0x7f7acc09c2e0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc09a380 con 0x7f7acc09eba0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc12bc20 con 0x7f7acc09beb0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient(hunting): authenticate will time out at 1049.081613s 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7aca7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7acc09c2e0 legacy=0x7f7acc1ca810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7acc09eba0 0x7f7acc1cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53788/0 (socket says 172.21.15.189:53788) 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acb7fe700 1 -- 172.21.15.189:0/2199937590 learned_addr learned my addr 172.21.15.189:0/2199937590 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760388404 0 0) 0x7f7acc09a380 con 0x7f7acc09eba0 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): my global_id is 134377 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab80035d0 con 0x7f7acc09eba0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552615691 0 0) 0x7f7acc12bc20 con 0x7f7acc09beb0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): my global_id is 124664 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7acc09a380 con 0x7f7acc09beb0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718343500 0 0) 0x7f7ab80035d0 con 0x7f7acc09eba0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7acc12bc20 con 0x7f7acc09eba0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ac4001950 con 0x7f7acc09eba0 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): dump: 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:36.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3268996684 0 0) 0x7f7acc12bc20 con 0x7f7acc09eba0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 >> v1:172.21.15.76:6790/0 conn(0x7f7acc09beb0 legacy=0x7f7acc132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 >> v1:172.21.15.76:6789/0 conn(0x7f7acc09c2e0 legacy=0x7f7acc1ca810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 monclient: found mon.b 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc12b050 con 0x7f7acc09eba0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 5 monclient: authenticate success, global_id 134377 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _renew_subs 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc12cb10 con 0x7f7acc09eba0 2024-09-16T06:41:36.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ac4004f10 con 0x7f7acc09eba0 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _renew_subs 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc12be60 con 0x7f7acc09eba0 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ac40052a0 con 0x7f7acc09eba0 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ac97fa700 4 set_mon_vals no callback set 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.564+0000 7f7ab3fff700 10 monclient: dump: 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:36.582 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7acbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7acbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7acbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc12d030 con 0x7f7acc09eba0 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7ac4005400 con 0x7f7acc09eba0 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(81..81 src has 50..81) v4 ==== 5658+0+0 (unknown 424125532 0 0) 0x7f7ac4043e40 con 0x7f7acc09eba0 2024-09-16T06:41:36.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.568+0000 7f7acaffd700 10 monclient: get_auth_request con 0x7f7ab8036160 auth_method 0 2024-09-16T06:41:36.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.576+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7ac40069f0 con 0x7f7acc09eba0 2024-09-16T06:41:36.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.576+0000 7f7ab3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:36.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.576+0000 7f7ab3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:36.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.736+0000 7f7acbfff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:36.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.736+0000 7f7acbfff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:36.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.736+0000 7f7acbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:36.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:36.736+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f7acc0677c0 con 0x7f7acc09eba0 2024-09-16T06:41:37.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.144+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v82) v1 ==== 140+0+0 (unknown 1658408989 0 0) 0x7f7ac4014b30 con 0x7f7acc09eba0 2024-09-16T06:41:37.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.144+0000 7f7ab3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:37.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.144+0000 7f7ab3fff700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-16T06:41:37.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7acbfff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7acbfff700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7acbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f7acc133030 con 0x7f7acc09eba0 2024-09-16T06:41:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7ab3fff700 1 -- 172.21.15.189:0/2199937590 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v82) v1 ==== 140+0+0 (unknown 2378121819 0 0) 0x7f7ac4014d60 con 0x7f7acc09eba0 2024-09-16T06:41:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7ab3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T06:41:37.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.260+0000 7f7ab3fff700 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7ab8036160 legacy=0x7f7ab8038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 10 monclient: shutdown 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 >> v1:172.21.15.189:6789/0 conn(0x7f7acc09eba0 legacy=0x7f7acc1cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 shutdown_connections 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 >> 172.21.15.189:0/2199937590 conn(0x7f7acc09fe70 msgr2=0x7f7acc09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:37.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 shutdown_connections 2024-09-16T06:41:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.264+0000 7f7acbfff700 1 -- 172.21.15.189:0/2199937590 wait complete. 2024-09-16T06:41:37.270 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host-for-test' does not appear in the crush map 2024-09-16T06:41:37.280 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm rack-for-test 2024-09-16T06:41:37.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.380+0000 7f17bfbdc700 10 monclient: get_monmap_and_config 2024-09-16T06:41:37.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.380+0000 7f17bfbdc700 10 monclient: build_initial_monmap 2024-09-16T06:41:37.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.380+0000 7f17bfbdc700 10 monclient: monmap: 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:37.387287+0000 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:37.387287+0000 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:37.387 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 Processor -- start 2024-09-16T06:41:37.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- start start 2024-09-16T06:41:37.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient: init 2024-09-16T06:41:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-c con 0x7f17b8073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-b con 0x7f17b8073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-a con 0x7f17b813b9b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80728d0 con 0x7f17b813b9b0 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b809c080 con 0x7f17b8073b60 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b8072420 con 0x7f17b8073f40 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient(hunting): authenticate will time out at 1049.901437s 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17be179700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f17b813b9b0 legacy=0x7f17b813dda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bd978700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f17b8073b60 0x7f17b809c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53790/0 (socket says 172.21.15.189:53790) 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bd978700 1 -- 172.21.15.189:0/2413015256 learned_addr learned my addr 172.21.15.189:0/2413015256 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412190810 0 0) 0x7f17b809c080 con 0x7f17b8073b60 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _negotiate 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): my global_id is 134380 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a40035d0 con 0x7f17b8073b60 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971692166 0 0) 0x7f17b8072420 con 0x7f17b8073f40 2024-09-16T06:41:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _negotiate 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient(hunting): my global_id is 124667 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b809c080 con 0x7f17b8073f40 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598682815 0 0) 0x7f17a40035d0 con 0x7f17b8073b60 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 >> v1:172.21.15.76:6790/0 conn(0x7f17b8073f40 legacy=0x7f17b8074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:37.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 >> v1:172.21.15.76:6789/0 conn(0x7f17b813b9b0 legacy=0x7f17b813dda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: _finish_hunting 0 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 monclient: found mon.noname-c 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8071f70 con 0x7f17b8073b60 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: _finish_auth 0 2024-09-16T06:41:37.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: _check_auth_tickets 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 5 monclient: authenticate success, global_id 134380 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17b4001610 con 0x7f17b8073b60 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: dump: 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:37.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b4001880 con 0x7f17b8073b60 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 4 set_mon_vals no callback set 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 1 -- 172.21.15.189:0/2413015256 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17b4001c10 con 0x7f17b8073b60 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bc976700 10 monclient: dump: 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 10 monclient: shutdown 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- 172.21.15.189:0/2413015256 >> v1:172.21.15.189:6789/0 conn(0x7f17b8073b60 legacy=0x7f17b809c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- 172.21.15.189:0/2413015256 shutdown_connections 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- 172.21.15.189:0/2413015256 >> 172.21.15.189:0/2413015256 conn(0x7f17b80a0f20 msgr2=0x7f17b80a1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- 172.21.15.189:0/2413015256 shutdown_connections 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.384+0000 7f17bfbdc700 1 -- 172.21.15.189:0/2413015256 wait complete. 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: build_initial_monmap 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: monmap: 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 Processor -- start 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- start start 2024-09-16T06:41:37.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: init 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-a con 0x7f17b8073b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-c con 0x7f17b8073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient(hunting): picked mon.noname-b con 0x7f17b813b9b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b812d750 con 0x7f17b8073b60 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80a0730 con 0x7f17b813b9b0 2024-09-16T06:41:37.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b812dcd0 con 0x7f17b8073f40 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient(hunting): authenticate will time out at 1049.905956s 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bd978700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f17b8073b60 legacy=0x7f17b8097490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17be179700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f17b813b9b0 0x7f17b81d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53798/0 (socket says 172.21.15.189:53798) 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17be179700 1 -- 172.21.15.189:0/1513105429 learned_addr learned my addr 172.21.15.189:0/1513105429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73292640 0 0) 0x7f17b80a0730 con 0x7f17b813b9b0 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): my global_id is 134383 2024-09-16T06:41:37.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c0035d0 con 0x7f17b813b9b0 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681727050 0 0) 0x7f17b812dcd0 con 0x7f17b8073f40 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): my global_id is 124670 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b80a0730 con 0x7f17b8073f40 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285018334 0 0) 0x7f179c0035d0 con 0x7f17b813b9b0 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17b812dcd0 con 0x7f17b813b9b0 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17b00044f0 con 0x7f17b813b9b0 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): dump: 2024-09-16T06:41:37.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3777215907 0 0) 0x7f17b812dcd0 con 0x7f17b813b9b0 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 >> v1:172.21.15.76:6790/0 conn(0x7f17b8073f40 legacy=0x7f17b81cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 >> v1:172.21.15.76:6789/0 conn(0x7f17b8073b60 legacy=0x7f17b8097490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 monclient: found mon.b 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b812dfb0 con 0x7f17b813b9b0 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 5 monclient: authenticate success, global_id 134383 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _renew_subs 2024-09-16T06:41:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b812c610 con 0x7f17b813b9b0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _renew_subs 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.388+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b812b540 con 0x7f17b813b9b0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b00047d0 con 0x7f17b813b9b0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f17b0004b60 con 0x7f17b813b9b0 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17af7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 10 monclient: dump: 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f17b0004cc0 con 0x7f17b813b9b0 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17bfbdc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17bfbdc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17bfbdc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a0005100 con 0x7f17b813b9b0 2024-09-16T06:41:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(82..82 src has 56..82) v4 ==== 5475+0+0 (unknown 3931892462 0 0) 0x7f17b0043bb0 con 0x7f17b813b9b0 2024-09-16T06:41:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.392+0000 7f17bd978700 10 monclient: get_auth_request con 0x7f179c0361d0 auth_method 0 2024-09-16T06:41:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.396+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f17b00054c0 con 0x7f17b813b9b0 2024-09-16T06:41:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.396+0000 7f17ae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.396+0000 7f17ae7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.556+0000 7f17bfbdc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:37.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.560+0000 7f17bfbdc700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:37.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.560+0000 7f17bfbdc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:37.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:37.560+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f17a00053b0 con 0x7f17b813b9b0 2024-09-16T06:41:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.164+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v83) v1 ==== 140+0+0 (unknown 293873728 0 0) 0x7f17b0012f00 con 0x7f17b813b9b0 2024-09-16T06:41:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.164+0000 7f17ae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:38.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.164+0000 7f17ae7fc700 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-16T06:41:38.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17bfbdc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:38.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17bfbdc700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:38.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17bfbdc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f17a0005650 con 0x7f17b813b9b0 2024-09-16T06:41:38.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17ae7fc700 1 -- 172.21.15.189:0/1513105429 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v83) v1 ==== 140+0+0 (unknown 32484356 0 0) 0x7f17b0014680 con 0x7f17b813b9b0 2024-09-16T06:41:38.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17ae7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T06:41:38.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.268+0000 7f17ae7fc700 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 >> v1:172.21.15.189:6812/3098810674 conn(0x7f179c0361d0 legacy=0x7f179c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 10 monclient: shutdown 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 >> v1:172.21.15.189:6789/0 conn(0x7f17b813b9b0 legacy=0x7f17b81d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 shutdown_connections 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 >> 172.21.15.189:0/1513105429 conn(0x7f17b80a0f20 msgr2=0x7f17b8136980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 shutdown_connections 2024-09-16T06:41:38.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.272+0000 7f17bfbdc700 1 -- 172.21.15.189:0/1513105429 wait complete. 2024-09-16T06:41:38.278 INFO:tasks.workunit.client.0.smithi189.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-16T06:41:38.286 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm root-for-test 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.384+0000 7fc88aad1700 10 monclient: get_monmap_and_config 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.384+0000 7fc88aad1700 10 monclient: build_initial_monmap 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.384+0000 7fc88aad1700 10 monclient: monmap: 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:38.391900+0000 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:38.391900+0000 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:38.391 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:38.392 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 Processor -- start 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- start start 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient: init 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-c con 0x7fc8840a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-a con 0x7fc88409aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-b con 0x7fc8840a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88409a040 con 0x7fc88409aa20 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840a33d0 con 0x7fc8840a2350 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884099d60 con 0x7fc8840a3ca0 2024-09-16T06:41:38.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient(hunting): authenticate will time out at 1050.904884s 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88886d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8840a2350 0x7fc8840a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53812/0 (socket says 172.21.15.189:53812) 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88886d700 1 -- 172.21.15.189:0/2312450717 learned_addr learned my addr 172.21.15.189:0/2312450717 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc883fff700 1 -- 172.21.15.189:0/2312450717 >> v1:172.21.15.76:6789/0 conn(0x7fc88409aa20 legacy=0x7fc88413a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101422765 0 0) 0x7fc8840a33d0 con 0x7fc8840a2350 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient(hunting): my global_id is 134386 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0035d0 con 0x7fc8840a2350 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152176399 0 0) 0x7fc86c0035d0 con 0x7fc8840a2350 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 >> v1:172.21.15.76:6790/0 conn(0x7fc8840a3ca0 legacy=0x7fc88413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:38.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 >> v1:172.21.15.76:6789/0 conn(0x7fc88409aa20 legacy=0x7fc88413a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 monclient: found mon.noname-c 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840a3920 con 0x7fc8840a2350 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc87c001990 con 0x7fc8840a2350 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: dump: 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.397 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 5 monclient: authenticate success, global_id 134386 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc87c001cb0 con 0x7fc8840a2350 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 1 -- 172.21.15.189:0/2312450717 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc87c002040 con 0x7fc8840a2350 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:38.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 4 set_mon_vals no callback set 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc8837fe700 10 monclient: dump: 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 10 monclient: shutdown 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- 172.21.15.189:0/2312450717 >> v1:172.21.15.189:6789/0 conn(0x7fc8840a2350 legacy=0x7fc8840a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- 172.21.15.189:0/2312450717 shutdown_connections 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- 172.21.15.189:0/2312450717 >> 172.21.15.189:0/2312450717 conn(0x7fc8840a7d70 msgr2=0x7fc8840a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- 172.21.15.189:0/2312450717 shutdown_connections 2024-09-16T06:41:38.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.388+0000 7fc88aad1700 1 -- 172.21.15.189:0/2312450717 wait complete. 2024-09-16T06:41:38.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: build_initial_monmap 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: monmap: 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 Processor -- start 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- start start 2024-09-16T06:41:38.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: init 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-c con 0x7fc88409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-b con 0x7fc8840a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient(hunting): picked mon.noname-a con 0x7fc8840a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884122b40 con 0x7fc8840a3ca0 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8841ca830 con 0x7fc8840a2350 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884121620 con 0x7fc88409aa20 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient(hunting): authenticate will time out at 1050.909863s 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88906e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc8840a3ca0 legacy=0x7fc8841d2360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc883fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc8840a2350 0x7fc8841cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53822/0 (socket says 172.21.15.189:53822) 2024-09-16T06:41:38.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc883fff700 1 -- 172.21.15.189:0/3891465078 learned_addr learned my addr 172.21.15.189:0/3891465078 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495486911 0 0) 0x7fc8841ca830 con 0x7fc8840a2350 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): my global_id is 134389 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8700035d0 con 0x7fc8840a2350 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300840712 0 0) 0x7fc884121620 con 0x7fc88409aa20 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): my global_id is 124676 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8841ca830 con 0x7fc88409aa20 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865298326 0 0) 0x7fc8700035d0 con 0x7fc8840a2350 2024-09-16T06:41:38.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc884121620 con 0x7fc8840a2350 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc878004230 con 0x7fc8840a2350 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): dump: 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:38.408 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 588841694 0 0) 0x7fc8841ca830 con 0x7fc88409aa20 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8700035d0 con 0x7fc88409aa20 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc87c0017b0 con 0x7fc88409aa20 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient(hunting): dump: 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:38.409 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3315312647 0 0) 0x7fc884121620 con 0x7fc8840a2350 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 >> v1:172.21.15.76:6790/0 conn(0x7fc88409aa20 legacy=0x7fc884135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 >> v1:172.21.15.76:6789/0 conn(0x7fc8840a3ca0 legacy=0x7fc8841d2360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 monclient: found mon.b 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88409b540 con 0x7fc8840a2350 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:38.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 5 monclient: authenticate success, global_id 134389 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _renew_subs 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88406bed0 con 0x7fc8840a2350 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _renew_subs 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8841213a0 con 0x7fc8840a2350 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc878004510 con 0x7fc8840a2350 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8780048a0 con 0x7fc8840a2350 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc882ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:38.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.392+0000 7fc8817fa700 10 monclient: dump: 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc878004a80 con 0x7fc8840a2350 2024-09-16T06:41:38.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(83..83 src has 56..83) v4 ==== 5292+0+0 (unknown 3216291264 0 0) 0x7fc8780439b0 con 0x7fc8840a2350 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc88886d700 10 monclient: get_auth_request con 0x7fc8700362a0 auth_method 0 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc88aad1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc88aad1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc88aad1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.396+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc864005100 con 0x7fc8840a2350 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.404+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc8780148a0 con 0x7fc8840a2350 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.404+0000 7fc8817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:38.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.404+0000 7fc8817fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.576+0000 7fc88aad1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.576+0000 7fc88aad1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.576+0000 7fc88aad1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:38.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:38.576+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7fc8640053b0 con 0x7fc8840a2350 2024-09-16T06:41:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.164+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v84) v1 ==== 140+0+0 (unknown 4032571887 0 0) 0x7fc878012d10 con 0x7fc8840a2350 2024-09-16T06:41:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.164+0000 7fc8817fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:39.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.164+0000 7fc8817fa700 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-16T06:41:39.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc88aad1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:39.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc88aad1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:39.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc88aad1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7fc8640023c0 con 0x7fc8840a2350 2024-09-16T06:41:39.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc8817fa700 1 -- 172.21.15.189:0/3891465078 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v84) v1 ==== 140+0+0 (unknown 3603915341 0 0) 0x7fc878014490 con 0x7fc8840a2350 2024-09-16T06:41:39.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc8817fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T06:41:39.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.276+0000 7fc8817fa700 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-16T06:41:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc8700362a0 legacy=0x7fc870038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:39.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 10 monclient: shutdown 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 >> v1:172.21.15.189:6789/0 conn(0x7fc8840a2350 legacy=0x7fc8841cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 shutdown_connections 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 >> 172.21.15.189:0/3891465078 conn(0x7fc8840a7d70 msgr2=0x7fc8840a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 shutdown_connections 2024-09-16T06:41:39.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.280+0000 7fc88aad1700 1 -- 172.21.15.189:0/3891465078 wait complete. 2024-09-16T06:41:39.287 INFO:tasks.workunit.client.0.smithi189.stderr:device 'root-for-test' does not appear in the crush map 2024-09-16T06:41:39.300 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: get_monmap_and_config 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: build_initial_monmap 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: monmap: 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:39.408971+0000 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:39.408971+0000 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:39.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:39.409 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 1 Processor -- start 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 1 -- start start 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: init 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-b con 0x7f336c09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-a con 0x7f336c099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-c con 0x7f336c1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c09f520 con 0x7f336c099700 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c09bc10 con 0x7f336c1396a0 2024-09-16T06:41:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c09b810 con 0x7f336c09f140 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient(hunting): authenticate will time out at 1051.922150s 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f336a7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f336c099700 legacy=0x7f336c09b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f336b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f336c1396a0 0x7f336c13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52594/0 (socket says 172.21.15.189:52594) 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f336b7fe700 1 -- 172.21.15.189:0/2096422263 learned_addr learned my addr 172.21.15.189:0/2096422263 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337307320 0 0) 0x7f336c09bc10 con 0x7f336c1396a0 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): my global_id is 134392 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33540035d0 con 0x7f336c1396a0 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216071564 0 0) 0x7f336c09b810 con 0x7f336c09f140 2024-09-16T06:41:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient(hunting): my global_id is 124679 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f336c09bc10 con 0x7f336c09f140 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512828788 0 0) 0x7f33540035d0 con 0x7f336c1396a0 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 >> v1:172.21.15.76:6790/0 conn(0x7f336c09f140 legacy=0x7f336c09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 >> v1:172.21.15.76:6789/0 conn(0x7f336c099700 legacy=0x7f336c09b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 monclient: found mon.noname-c 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c09ceb0 con 0x7f336c1396a0 2024-09-16T06:41:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 5 monclient: authenticate success, global_id 134392 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3364001990 con 0x7f336c1396a0 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: dump: 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3364001c70 con 0x7f336c1396a0 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 1 -- 172.21.15.189:0/2096422263 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3364002000 con 0x7f336c1396a0 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3371a61700 4 set_mon_vals no callback set 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.404+0000 7f3369ffb700 10 monclient: dump: 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:39.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: shutdown 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/2096422263 >> v1:172.21.15.189:6789/0 conn(0x7f336c1396a0 legacy=0x7f336c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/2096422263 shutdown_connections 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/2096422263 >> 172.21.15.189:0/2096422263 conn(0x7f336c0a03e0 msgr2=0x7f336c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/2096422263 shutdown_connections 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/2096422263 wait complete. 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: build_initial_monmap 2024-09-16T06:41:39.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: monmap: 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 Processor -- start 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- start start 2024-09-16T06:41:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: init 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-a con 0x7f336c099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-c con 0x7f336c09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient(hunting): picked mon.noname-b con 0x7f336c1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c090420 con 0x7f336c099700 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c09a6e0 con 0x7f336c1396a0 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c098e60 con 0x7f336c09f140 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient(hunting): authenticate will time out at 1051.926275s 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f336affd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f336c099700 legacy=0x7f336c138c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f336b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f336c1396a0 0x7f336c1cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52596/0 (socket says 172.21.15.189:52596) 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f336b7fe700 1 -- 172.21.15.189:0/433187032 learned_addr learned my addr 172.21.15.189:0/433187032 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934626066 0 0) 0x7f336c09a6e0 con 0x7f336c1396a0 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): my global_id is 134395 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33580035d0 con 0x7f336c1396a0 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497573441 0 0) 0x7f336c098e60 con 0x7f336c09f140 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): my global_id is 124682 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f336c09a6e0 con 0x7f336c09f140 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171657566 0 0) 0x7f33580035d0 con 0x7f336c1396a0 2024-09-16T06:41:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f336c098e60 con 0x7f336c1396a0 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3364001950 con 0x7f336c1396a0 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): dump: 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1124844966 0 0) 0x7f336c098e60 con 0x7f336c1396a0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 >> v1:172.21.15.76:6790/0 conn(0x7f336c09f140 legacy=0x7f336c1ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 >> v1:172.21.15.76:6789/0 conn(0x7f336c099700 legacy=0x7f336c138c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 monclient: found mon.b 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0a2e40 con 0x7f336c1396a0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3353fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 5 monclient: authenticate success, global_id 134395 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _renew_subs 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c08f1e0 con 0x7f336c1396a0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _renew_subs 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.408+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c0909e0 con 0x7f336c1396a0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3364001500 con 0x7f336c1396a0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33640051c0 con 0x7f336c1396a0 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f33697fa700 4 set_mon_vals no callback set 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 10 monclient: dump: 2024-09-16T06:41:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f336400fc30 con 0x7f336c1396a0 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3371a61700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3371a61700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3371a61700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c005100 con 0x7f336c1396a0 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(84..84 src has 56..84) v4 ==== 5133+0+0 (unknown 1467211396 0 0) 0x7f3364043c80 con 0x7f336c1396a0 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.412+0000 7f336affd700 10 monclient: get_auth_request con 0x7f3358036220 auth_method 0 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.420+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3364014ac0 con 0x7f336c1396a0 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.420+0000 7f3353fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:39.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.420+0000 7f3353fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.588+0000 7f3371a61700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.588+0000 7f3371a61700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.588+0000 7f3371a61700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:39.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:39.588+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f334c0053b0 con 0x7f336c1396a0 2024-09-16T06:41:40.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.180+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v85) v1 ==== 145+0+0 (unknown 3680813466 0 0) 0x7f3364013160 con 0x7f336c1396a0 2024-09-16T06:41:40.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.180+0000 7f3353fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:40.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.180+0000 7f3353fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:40.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.292+0000 7f3371a61700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:40.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.292+0000 7f3371a61700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.292+0000 7f3371a61700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.292+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f334c0010a0 con 0x7f336c1396a0 2024-09-16T06:41:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3353fff700 1 -- 172.21.15.189:0/433187032 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v85) v1 ==== 173+0+0 (unknown 1950840413 0 0) 0x7f33640148e0 con 0x7f336c1396a0 2024-09-16T06:41:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3353fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T06:41:40.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3353fff700 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3358036220 legacy=0x7f33580386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 10 monclient: shutdown 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 >> v1:172.21.15.189:6789/0 conn(0x7f336c1396a0 legacy=0x7f336c1cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 shutdown_connections 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 >> 172.21.15.189:0/433187032 conn(0x7f336c0a03e0 msgr2=0x7f336c0a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 shutdown_connections 2024-09-16T06:41:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.296+0000 7f3371a61700 1 -- 172.21.15.189:0/433187032 wait complete. 2024-09-16T06:41:40.304 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-rule already exists 2024-09-16T06:41:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T06:41:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:41:40.314 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T06:41:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.416+0000 7fb737fff700 10 monclient: get_monmap_and_config 2024-09-16T06:41:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.416+0000 7fb737fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.416+0000 7fb737fff700 10 monclient: monmap: 2024-09-16T06:41:40.423 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:40.423749+0000 2024-09-16T06:41:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:40.423749+0000 2024-09-16T06:41:40.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:40.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 Processor -- start 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- start start 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient: init 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb73809eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb73809f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb738139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73809ab80 con 0x7fb73809eba0 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73809a690 con 0x7fb738139a50 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb738072230 con 0x7fb73809f690 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:40.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient(hunting): authenticate will time out at 1052.936881s 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb736ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb73809eba0 legacy=0x7fb73809ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb7377fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb738139a50 0x7fb73813be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52610/0 (socket says 172.21.15.189:52610) 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb7377fe700 1 -- 172.21.15.189:0/230508589 learned_addr learned my addr 172.21.15.189:0/230508589 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813914463 0 0) 0x7fb738072230 con 0x7fb73809f690 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): my global_id is 124685 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7200035d0 con 0x7fb73809f690 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892665219 0 0) 0x7fb73809a690 con 0x7fb738139a50 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient(hunting): my global_id is 134398 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb738072230 con 0x7fb738139a50 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152838596 0 0) 0x7fb7200035d0 con 0x7fb73809f690 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 >> v1:172.21.15.189:6789/0 conn(0x7fb738139a50 legacy=0x7fb73813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 >> v1:172.21.15.76:6789/0 conn(0x7fb73809eba0 legacy=0x7fb73809ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 monclient: found mon.noname-b 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73809be90 con 0x7fb73809f690 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb728001610 con 0x7fb73809f690 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 5 monclient: authenticate success, global_id 124685 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: dump: 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb728001830 con 0x7fb73809f690 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 1 -- 172.21.15.189:0/230508589 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb728001bc0 con 0x7fb73809f690 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb735ffb700 10 monclient: dump: 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 4 set_mon_vals no callback set 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 10 monclient: shutdown 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- 172.21.15.189:0/230508589 >> v1:172.21.15.76:6790/0 conn(0x7fb73809f690 legacy=0x7fb738138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- 172.21.15.189:0/230508589 shutdown_connections 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- 172.21.15.189:0/230508589 >> 172.21.15.189:0/230508589 conn(0x7fb73809fe70 msgr2=0x7fb7380a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- 172.21.15.189:0/230508589 shutdown_connections 2024-09-16T06:41:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.420+0000 7fb737fff700 1 -- 172.21.15.189:0/230508589 wait complete. 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient: build_initial_monmap 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient: monmap: 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 1 Processor -- start 2024-09-16T06:41:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 1 -- start start 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient: init 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb73809eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb73809f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb738139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7380945e0 con 0x7fb73809eba0 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73806fe70 con 0x7fb738139a50 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb738095820 con 0x7fb73809f690 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb736ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb73809eba0 legacy=0x7fb738137b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb737fff700 10 monclient(hunting): authenticate will time out at 1052.943573s 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb7377fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb738139a50 0x7fb7381d2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52612/0 (socket says 172.21.15.189:52612) 2024-09-16T06:41:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.424+0000 7fb7377fe700 1 -- 172.21.15.189:0/1894168378 learned_addr learned my addr 172.21.15.189:0/1894168378 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71480735 0 0) 0x7fb73806fe70 con 0x7fb738139a50 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): my global_id is 134401 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7240035d0 con 0x7fb738139a50 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681996002 0 0) 0x7fb738095820 con 0x7fb73809f690 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): my global_id is 124688 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73806fe70 con 0x7fb73809f690 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462826918 0 0) 0x7fb7240035d0 con 0x7fb738139a50 2024-09-16T06:41:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb738095820 con 0x7fb738139a50 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb730001c30 con 0x7fb738139a50 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281081229 0 0) 0x7fb73806fe70 con 0x7fb73809f690 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7240035d0 con 0x7fb73809f690 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb728001510 con 0x7fb73809f690 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient(hunting): dump: 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1859978476 0 0) 0x7fb738095820 con 0x7fb738139a50 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 >> v1:172.21.15.76:6790/0 conn(0x7fb73809f690 legacy=0x7fb7381cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 >> v1:172.21.15.76:6789/0 conn(0x7fb73809eba0 legacy=0x7fb738137b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 monclient: found mon.b 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73806fad0 con 0x7fb738139a50 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:40.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 5 monclient: authenticate success, global_id 134401 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _renew_subs 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73809a380 con 0x7fb738139a50 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _renew_subs 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb738138210 con 0x7fb738139a50 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb730002000 con 0x7fb738139a50 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb730005060 con 0x7fb738139a50 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb7357fa700 4 set_mon_vals no callback set 2024-09-16T06:41:40.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 10 monclient: dump: 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb73000fad0 con 0x7fb738139a50 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb737fff700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb718005100 con 0x7fb738139a50 2024-09-16T06:41:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(85..85 src has 56..85) v4 ==== 5193+0+0 (unknown 395043415 0 0) 0x7fb730043b30 con 0x7fb738139a50 2024-09-16T06:41:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.428+0000 7fb736ffd700 10 monclient: get_auth_request con 0x7fb7240362a0 auth_method 0 2024-09-16T06:41:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.436+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb730014be0 con 0x7fb738139a50 2024-09-16T06:41:40.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.436+0000 7fb71f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.436+0000 7fb71f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:40.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb737fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:40.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb737fff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:40.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb737fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb737fff700 1 -- 172.21.15.189:0/1894168378 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fb7180053b0 con 0x7fb738139a50 2024-09-16T06:41:40.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb71f7fe700 1 -- 172.21.15.189:0/1894168378 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v85) v1 ==== 103+0+0 (unknown 1496711894 0 0) 0x7fb730013050 con 0x7fb738139a50 2024-09-16T06:41:40.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb71f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:40.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.616+0000 7fb71f7fe700 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-16T06:41:40.625 INFO:tasks.workunit.client.0.smithi189.stderr:Error EBUSY: (16) Device or resource busy 2024-09-16T06:41:40.629 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:41:40.629 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rule rm foo-rule 2024-09-16T06:41:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: get_monmap_and_config 2024-09-16T06:41:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: build_initial_monmap 2024-09-16T06:41:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: monmap: 2024-09-16T06:41:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:40.734369+0000 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:40.734369+0000 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:40.734 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 1 Processor -- start 2024-09-16T06:41:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 1 -- start start 2024-09-16T06:41:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: init 2024-09-16T06:41:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-a con 0x7f6564071370 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-b con 0x7f6564097d90 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-c con 0x7f656413b8d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6564097690 con 0x7f6564071370 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.728+0000 7f656c5da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6564070db0 con 0x7f656413b8d0 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65640982b0 con 0x7f6564097d90 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): authenticate will time out at 1053.248034s 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656a376700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6564071370 legacy=0x7f656406ff90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656ab77700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f656413b8d0 0x7f656413dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52628/0 (socket says 172.21.15.189:52628) 2024-09-16T06:41:40.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656ab77700 1 -- 172.21.15.189:0/653264831 learned_addr learned my addr 172.21.15.189:0/653264831 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812395475 0 0) 0x7f6564070db0 con 0x7f656413b8d0 2024-09-16T06:41:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): my global_id is 134404 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65500035d0 con 0x7f656413b8d0 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585127436 0 0) 0x7f65640982b0 con 0x7f6564097d90 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient(hunting): my global_id is 124691 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6564070db0 con 0x7f6564097d90 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211457221 0 0) 0x7f65500035d0 con 0x7f656413b8d0 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 >> v1:172.21.15.76:6790/0 conn(0x7f6564097d90 legacy=0x7f65640706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 >> v1:172.21.15.76:6789/0 conn(0x7f6564071370 legacy=0x7f656406ff90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: _finish_hunting 0 2024-09-16T06:41:40.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 monclient: found mon.noname-c 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656409f6e0 con 0x7f656413b8d0 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: _finish_auth 0 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: _check_auth_tickets 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f655c001990 con 0x7f656413b8d0 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 5 monclient: authenticate success, global_id 134404 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:40.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: dump: 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c001c70 con 0x7f656413b8d0 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:40.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 4 set_mon_vals no callback set 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 1 -- 172.21.15.189:0/653264831 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f655c002000 con 0x7f656413b8d0 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569374700 10 monclient: dump: 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: shutdown 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- 172.21.15.189:0/653264831 >> v1:172.21.15.189:6789/0 conn(0x7f656413b8d0 legacy=0x7f656413dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- 172.21.15.189:0/653264831 shutdown_connections 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- 172.21.15.189:0/653264831 >> 172.21.15.189:0/653264831 conn(0x7f65640a03e0 msgr2=0x7f65640a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- 172.21.15.189:0/653264831 shutdown_connections 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- 172.21.15.189:0/653264831 wait complete. 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: build_initial_monmap 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: monmap: 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:40.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 Processor -- start 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- start start 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: init 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-c con 0x7f6564071370 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:40.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-a con 0x7f6564097d90 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): picked mon.noname-b con 0x7f656413b8d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6564139b10 con 0x7f6564097d90 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656408c520 con 0x7f656413b8d0 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656409d350 con 0x7f6564071370 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656c5da700 10 monclient(hunting): authenticate will time out at 1053.251627s 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f6569b75700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6564097d90 legacy=0x7f6564138830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656ab77700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f656413b8d0 0x7f65641d2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52636/0 (socket says 172.21.15.189:52636) 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.732+0000 7f656ab77700 1 -- 172.21.15.189:0/753409447 learned_addr learned my addr 172.21.15.189:0/753409447 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367550270 0 0) 0x7f656408c520 con 0x7f656413b8d0 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): my global_id is 134407 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65480035d0 con 0x7f656413b8d0 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499302018 0 0) 0x7f656409d350 con 0x7f6564071370 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): my global_id is 124694 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656408c520 con 0x7f6564071370 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1355623165 0 0) 0x7f65480035d0 con 0x7f656413b8d0 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f656409d350 con 0x7f656413b8d0 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f655c001950 con 0x7f656413b8d0 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): dump: 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055016516 0 0) 0x7f656408c520 con 0x7f6564071370 2024-09-16T06:41:40.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65480035d0 con 0x7f6564071370 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65600041a0 con 0x7f6564071370 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient(hunting): dump: 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.752 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2574535878 0 0) 0x7f656409d350 con 0x7f656413b8d0 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 >> v1:172.21.15.76:6790/0 conn(0x7f6564071370 legacy=0x7f6564138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 >> v1:172.21.15.76:6789/0 conn(0x7f6564097d90 legacy=0x7f6564138830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 monclient: found mon.b 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656413f170 con 0x7f656413b8d0 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: _finish_auth 0 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 5 monclient: authenticate success, global_id 134407 2024-09-16T06:41:40.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _renew_subs 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656409a6e0 con 0x7f656413b8d0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _renew_subs 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f656413a6e0 con 0x7f656413b8d0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c005020 con 0x7f656413b8d0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f655c005330 con 0x7f656413b8d0 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 10 monclient: dump: 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:40.754 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f6568b73700 4 set_mon_vals no callback set 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f655c00fe20 con 0x7f656413b8d0 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(85..85 src has 56..85) v4 ==== 5193+0+0 (unknown 395043415 0 0) 0x7f655c043d10 con 0x7f656413b8d0 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654c005100 con 0x7f656413b8d0 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.736+0000 7f656a376700 10 monclient: get_auth_request con 0x7f65480362d0 auth_method 0 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.744+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f655c014ec0 con 0x7f656413b8d0 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.744+0000 7f655affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:40.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.744+0000 7f655affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.916+0000 7f656c5da700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.916+0000 7f656c5da700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.916+0000 7f656c5da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:40.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:40.916+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f654c0053b0 con 0x7f656413b8d0 2024-09-16T06:41:41.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.188+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v86) v1 ==== 85+0+0 (unknown 3330723634 0 0) 0x7f655c013330 con 0x7f656413b8d0 2024-09-16T06:41:41.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.188+0000 7f655affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:41.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.188+0000 7f655affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f656c5da700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f656c5da700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f656c5da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f654c0023e0 con 0x7f656413b8d0 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f655affd700 1 -- 172.21.15.189:0/753409447 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v86) v1 ==== 113+0+0 (unknown 3789652312 0 0) 0x7f655c014c90 con 0x7f656413b8d0 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f655affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.296+0000 7f655affd700 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 >> v1:172.21.15.189:6812/3098810674 conn(0x7f65480362d0 legacy=0x7f6548038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:41.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 10 monclient: shutdown 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 >> v1:172.21.15.189:6789/0 conn(0x7f656413b8d0 legacy=0x7f65641d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 shutdown_connections 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 >> 172.21.15.189:0/753409447 conn(0x7f65640a03e0 msgr2=0x7f656413ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 shutdown_connections 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.300+0000 7f656c5da700 1 -- 172.21.15.189:0/753409447 wait complete. 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:rule foo-rule does not exist 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm bar 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.420+0000 7f71674fa700 10 monclient: get_monmap_and_config 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.420+0000 7f71674fa700 10 monclient: build_initial_monmap 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.420+0000 7f71674fa700 10 monclient: monmap: 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:41.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:41.428165+0000 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:41.428165+0000 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 Processor -- start 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- start start 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient: init 2024-09-16T06:41:41.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-c con 0x7f7160073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-b con 0x7f7160074650 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-a con 0x7f7160139cd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160098420 con 0x7f7160139cd0 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716006ba10 con 0x7f7160073b60 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716009e830 con 0x7f7160074650 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient(hunting): authenticate will time out at 1053.941479s 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7165a97700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7160139cd0 legacy=0x7f716013c0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7165296700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7160073b60 0x7f7160073f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52646/0 (socket says 172.21.15.189:52646) 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7165296700 1 -- 172.21.15.189:0/942801216 learned_addr learned my addr 172.21.15.189:0/942801216 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:41.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44548729 0 0) 0x7f716006ba10 con 0x7f7160073b60 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): my global_id is 134410 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c0035d0 con 0x7f7160073b60 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406252992 0 0) 0x7f716009e830 con 0x7f7160074650 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient(hunting): my global_id is 124697 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716006ba10 con 0x7f7160074650 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017794047 0 0) 0x7f714c0035d0 con 0x7f7160073b60 2024-09-16T06:41:41.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 >> v1:172.21.15.76:6790/0 conn(0x7f7160074650 legacy=0x7f71601385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 >> v1:172.21.15.76:6789/0 conn(0x7f7160139cd0 legacy=0x7f716013c0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 monclient: found mon.noname-c 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716009b310 con 0x7f7160073b60 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 5 monclient: authenticate success, global_id 134410 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7154001610 con 0x7f7160073b60 2024-09-16T06:41:41.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: dump: 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71540018f0 con 0x7f7160073b60 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 4 set_mon_vals no callback set 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 1 -- 172.21.15.189:0/942801216 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7154001c80 con 0x7f7160073b60 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f7153fff700 10 monclient: dump: 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:41.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 10 monclient: shutdown 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- 172.21.15.189:0/942801216 >> v1:172.21.15.189:6789/0 conn(0x7f7160073b60 legacy=0x7f7160073f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- 172.21.15.189:0/942801216 shutdown_connections 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- 172.21.15.189:0/942801216 >> 172.21.15.189:0/942801216 conn(0x7f71600a1100 msgr2=0x7f71600a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.424+0000 7f71674fa700 1 -- 172.21.15.189:0/942801216 shutdown_connections 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- 172.21.15.189:0/942801216 wait complete. 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: build_initial_monmap 2024-09-16T06:41:41.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: monmap: 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 Processor -- start 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- start start 2024-09-16T06:41:41.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: init 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-b con 0x7f7160073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-a con 0x7f7160074650 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient(hunting): picked mon.noname-c con 0x7f7160139cd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160090eb0 con 0x7f7160074650 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716009d0a0 con 0x7f7160073b60 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160091db0 con 0x7f7160139cd0 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient(hunting): authenticate will time out at 1053.946610s 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7164a95700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7160074650 legacy=0x7f71601cf2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7165296700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7160073b60 0x7f7160135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52650/0 (socket says 172.21.15.189:52650) 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7165296700 1 -- 172.21.15.189:0/4186479568 learned_addr learned my addr 172.21.15.189:0/4186479568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:41.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535397686 0 0) 0x7f716009d0a0 con 0x7f7160073b60 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): my global_id is 134413 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71440035d0 con 0x7f7160073b60 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696760829 0 0) 0x7f7160091db0 con 0x7f7160139cd0 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): my global_id is 124700 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716009d0a0 con 0x7f7160139cd0 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559740996 0 0) 0x7f71440035d0 con 0x7f7160073b60 2024-09-16T06:41:41.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7160091db0 con 0x7f7160073b60 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7154001c00 con 0x7f7160073b60 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:41.472 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2062468368 0 0) 0x7f7160091db0 con 0x7f7160073b60 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 >> v1:172.21.15.76:6790/0 conn(0x7f7160139cd0 legacy=0x7f71601d29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 >> v1:172.21.15.76:6789/0 conn(0x7f7160074650 legacy=0x7f71601cf2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 monclient: found mon.b 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7160071ff0 con 0x7f7160073b60 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f7151ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:41.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 5 monclient: authenticate success, global_id 134413 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: _renew_subs 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.428+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716006ad20 con 0x7f7160073b60 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 10 monclient: _renew_subs 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7160091780 con 0x7f7160073b60 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7154004ca0 con 0x7f7160073b60 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:41.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7154004fb0 con 0x7f7160073b60 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71537fe700 4 set_mon_vals no callback set 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 10 monclient: dump: 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:41.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f715400fa20 con 0x7f7160073b60 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7140005100 con 0x7f7160073b60 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(86..86 src has 56..86) v4 ==== 5133+0+0 (unknown 2197516084 0 0) 0x7f7154043a10 con 0x7f7160073b60 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.432+0000 7f7164a95700 10 monclient: get_auth_request con 0x7f7144035fa0 auth_method 0 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.436+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f71540147e0 con 0x7f7160073b60 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.436+0000 7f7151ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:41.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.436+0000 7f7151ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:41.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.616+0000 7f71674fa700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:41.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.616+0000 7f71674fa700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:41.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.616+0000 7f71674fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:41.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:41.616+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f71400053b0 con 0x7f7160073b60 2024-09-16T06:41:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.196+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v87) v1 ==== 120+0+0 (unknown 643381365 0 0) 0x7f7154012e80 con 0x7f7160073b60 2024-09-16T06:41:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.196+0000 7f7151ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.196+0000 7f7151ffb700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.308+0000 7f71674fa700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.308+0000 7f71674fa700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.308+0000 7f71674fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.308+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f7140005650 con 0x7f7160073b60 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.312+0000 7f7151ffb700 1 -- 172.21.15.189:0/4186479568 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v87) v1 ==== 120+0+0 (unknown 3475446210 0 0) 0x7f7154014600 con 0x7f7160073b60 2024-09-16T06:41:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.312+0000 7f7151ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T06:41:42.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.312+0000 7f7151ffb700 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7144035fa0 legacy=0x7f7144038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 10 monclient: shutdown 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 >> v1:172.21.15.189:6789/0 conn(0x7f7160073b60 legacy=0x7f7160135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 shutdown_connections 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 >> 172.21.15.189:0/4186479568 conn(0x7f71600a1100 msgr2=0x7f71601393a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 shutdown_connections 2024-09-16T06:41:42.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.316+0000 7f71674fa700 1 -- 172.21.15.189:0/4186479568 wait complete. 2024-09-16T06:41:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:device 'bar' does not appear in the crush map 2024-09-16T06:41:42.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T06:41:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.432+0000 7f20af6ef700 10 monclient: get_monmap_and_config 2024-09-16T06:41:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.432+0000 7f20af6ef700 10 monclient: build_initial_monmap 2024-09-16T06:41:42.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.432+0000 7f20af6ef700 10 monclient: monmap: 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:42.439553+0000 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:42.439553+0000 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:42.439 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 Processor -- start 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- start start 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient: init 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-c con 0x7f20a809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-a con 0x7f20a8099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-b con 0x7f20a81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a809f520 con 0x7f20a8099700 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a809bc10 con 0x7f20a809f140 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a809b810 con 0x7f20a81396a0 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient(hunting): authenticate will time out at 1054.953035s 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20ad48b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f20a809f140 0x7f20a809a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52656/0 (socket says 172.21.15.189:52656) 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20ad48b700 1 -- 172.21.15.189:0/2105425817 learned_addr learned my addr 172.21.15.189:0/2105425817 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20acc8a700 1 -- 172.21.15.189:0/2105425817 >> v1:172.21.15.76:6789/0 conn(0x7f20a8099700 legacy=0x7f20a809b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899190097 0 0) 0x7f20a809bc10 con 0x7f20a809f140 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): my global_id is 134416 2024-09-16T06:41:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20940035d0 con 0x7f20a809f140 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056987353 0 0) 0x7f20a809b810 con 0x7f20a81396a0 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient(hunting): my global_id is 124703 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a809bc10 con 0x7f20a81396a0 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550954231 0 0) 0x7f20940035d0 con 0x7f20a809f140 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 >> v1:172.21.15.76:6790/0 conn(0x7f20a81396a0 legacy=0x7f20a813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 >> v1:172.21.15.76:6789/0 conn(0x7f20a8099700 legacy=0x7f20a809b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 monclient: found mon.noname-c 2024-09-16T06:41:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a809ceb0 con 0x7f20a809f140 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: _finish_auth 0 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a4001610 con 0x7f20a809f140 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 5 monclient: authenticate success, global_id 134416 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: dump: 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20a40018f0 con 0x7f20a809f140 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 1 -- 172.21.15.189:0/2105425817 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a4001c80 con 0x7f20a809f140 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 4 set_mon_vals no callback set 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f209ffff700 10 monclient: dump: 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 10 monclient: shutdown 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- 172.21.15.189:0/2105425817 >> v1:172.21.15.189:6789/0 conn(0x7f20a809f140 legacy=0x7f20a809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- 172.21.15.189:0/2105425817 shutdown_connections 2024-09-16T06:41:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- 172.21.15.189:0/2105425817 >> 172.21.15.189:0/2105425817 conn(0x7f20a80a03e0 msgr2=0x7f20a80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- 172.21.15.189:0/2105425817 shutdown_connections 2024-09-16T06:41:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.436+0000 7f20af6ef700 1 -- 172.21.15.189:0/2105425817 wait complete. 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: build_initial_monmap 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: monmap: 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 Processor -- start 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- start start 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: init 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-a con 0x7f20a8099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-b con 0x7f20a809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient(hunting): picked mon.noname-c con 0x7f20a81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80a2bd0 con 0x7f20a8099700 2024-09-16T06:41:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a81394e0 con 0x7f20a809f140 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a8090a50 con 0x7f20a81396a0 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient(hunting): authenticate will time out at 1054.957096s 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20ad48b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f20a8099700 legacy=0x7f20a8095fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20acc8a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f20a809f140 0x7f20a81ca830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52668/0 (socket says 172.21.15.189:52668) 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20acc8a700 1 -- 172.21.15.189:0/597846854 learned_addr learned my addr 172.21.15.189:0/597846854 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353993731 0 0) 0x7f20a81394e0 con 0x7f20a809f140 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): my global_id is 134419 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f208c0035d0 con 0x7f20a809f140 2024-09-16T06:41:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798921222 0 0) 0x7f20a8090a50 con 0x7f20a81396a0 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): my global_id is 124706 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a81394e0 con 0x7f20a81396a0 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2119581808 0 0) 0x7f208c0035d0 con 0x7f20a809f140 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20a8090a50 con 0x7f20a809f140 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2098004230 con 0x7f20a809f140 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): dump: 2024-09-16T06:41:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:42.459 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 527417665 0 0) 0x7f20a81394e0 con 0x7f20a81396a0 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f208c0035d0 con 0x7f20a81396a0 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20a0001c30 con 0x7f20a81396a0 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient(hunting): dump: 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2512504864 0 0) 0x7f20a8090a50 con 0x7f20a809f140 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 >> v1:172.21.15.76:6790/0 conn(0x7f20a81396a0 legacy=0x7f20a81cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 >> v1:172.21.15.76:6789/0 conn(0x7f20a8099700 legacy=0x7f20a8095fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 monclient: found mon.b 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80a26a0 con 0x7f20a809f140 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 5 monclient: authenticate success, global_id 134419 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _renew_subs 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a8090770 con 0x7f20a809f140 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _renew_subs 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a808fe80 con 0x7f20a809f140 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20980044a0 con 0x7f20a809f140 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2098004830 con 0x7f20a809f140 2024-09-16T06:41:42.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209f7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.440+0000 7f209dffb700 10 monclient: dump: 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:42.463 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f20af6ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f20af6ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f20af6ef700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088005100 con 0x7f20a809f140 2024-09-16T06:41:42.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2098004a50 con 0x7f20a809f140 2024-09-16T06:41:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(87..87 src has 61..87) v4 ==== 5004+0+0 (unknown 1729698775 0 0) 0x7f2098043820 con 0x7f20a809f140 2024-09-16T06:41:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.444+0000 7f20ad48b700 10 monclient: get_auth_request con 0x7f208c0362a0 auth_method 0 2024-09-16T06:41:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.448+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2098014830 con 0x7f20a809f140 2024-09-16T06:41:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.448+0000 7f209dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:42.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.448+0000 7f209dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.624+0000 7f20af6ef700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.624+0000 7f20af6ef700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.624+0000 7f20af6ef700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:42.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:42.624+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f20880053b0 con 0x7f20a809f140 2024-09-16T06:41:43.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.200+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v88) v1 ==== 120+0+0 (unknown 4062457904 0 0) 0x7f2098012ca0 con 0x7f20a809f140 2024-09-16T06:41:43.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.200+0000 7f209dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:43.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.200+0000 7f209dffb700 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T06:41:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.308+0000 7f20af6ef700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.308+0000 7f20af6ef700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.308+0000 7f20af6ef700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.308+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f2088005650 con 0x7f20a809f140 2024-09-16T06:41:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.312+0000 7f209dffb700 1 -- 172.21.15.189:0/597846854 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v88) v1 ==== 120+0+0 (unknown 3543709723 0 0) 0x7f2098014420 con 0x7f20a809f140 2024-09-16T06:41:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.312+0000 7f209dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:41:43.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.312+0000 7f209dffb700 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 >> v1:172.21.15.189:6812/3098810674 conn(0x7f208c0362a0 legacy=0x7f208c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 10 monclient: shutdown 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 >> v1:172.21.15.189:6789/0 conn(0x7f20a809f140 legacy=0x7f20a81ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 shutdown_connections 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 >> 172.21.15.189:0/597846854 conn(0x7f20a80a03e0 msgr2=0x7f20a80a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 shutdown_connections 2024-09-16T06:41:43.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.316+0000 7f20af6ef700 1 -- 172.21.15.189:0/597846854 wait complete. 2024-09-16T06:41:43.321 INFO:tasks.workunit.client.0.smithi189.stderr:device 'foo' does not appear in the crush map 2024-09-16T06:41:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.9 host2 2024-09-16T06:41:43.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:41:43.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 10 monclient: build_initial_monmap 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 10 monclient: monmap: 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:43.438577+0000 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:43.438577+0000 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:43.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 1 Processor -- start 2024-09-16T06:41:43.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 1 -- start start 2024-09-16T06:41:43.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.432+0000 7f1f1b59e700 10 monclient: init 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1f1c1317e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1f1c0a0000 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1f1c137da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c09cda0 con 0x7f1f1c137da0 2024-09-16T06:41:43.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c09f940 con 0x7f1f1c1317e0 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c09bb20 con 0x7f1f1c0a0000 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient(hunting): authenticate will time out at 1055.953152s 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1ad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f1f1c137da0 legacy=0x7f1f1c13a190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f1c1317e0 0x7f1f1c131bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52672/0 (socket says 172.21.15.189:52672) 2024-09-16T06:41:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1a59c700 1 -- 172.21.15.189:0/3270204815 learned_addr learned my addr 172.21.15.189:0/3270204815 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442854492 0 0) 0x7f1f1c09f940 con 0x7f1f1c1317e0 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): my global_id is 134422 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f080046e0 con 0x7f1f1c1317e0 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956517914 0 0) 0x7f1f1c09bb20 con 0x7f1f1c0a0000 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient(hunting): my global_id is 124709 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1c09f940 con 0x7f1f1c0a0000 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270481520 0 0) 0x7f1f080046e0 con 0x7f1f1c1317e0 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 >> v1:172.21.15.76:6790/0 conn(0x7f1f1c0a0000 legacy=0x7f1f1c136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 >> v1:172.21.15.76:6789/0 conn(0x7f1f1c137da0 legacy=0x7f1f1c13a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 monclient: found mon.noname-c 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c09be00 con 0x7f1f1c1317e0 2024-09-16T06:41:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: _finish_auth 0 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 5 monclient: authenticate success, global_id 134422 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f0c001610 con 0x7f1f1c1317e0 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: dump: 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f0c001880 con 0x7f1f1c1317e0 2024-09-16T06:41:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 4 set_mon_vals no callback set 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 1 -- 172.21.15.189:0/3270204815 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f0c001c10 con 0x7f1f1c1317e0 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f19d9b700 10 monclient: dump: 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:43.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 10 monclient: shutdown 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3270204815 >> v1:172.21.15.189:6789/0 conn(0x7f1f1c1317e0 legacy=0x7f1f1c131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3270204815 shutdown_connections 2024-09-16T06:41:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3270204815 >> 172.21.15.189:0/3270204815 conn(0x7f1f1c073b60 msgr2=0x7f1f1c073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3270204815 shutdown_connections 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.436+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3270204815 wait complete. 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: build_initial_monmap 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: monmap: 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:43.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 Processor -- start 2024-09-16T06:41:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- start start 2024-09-16T06:41:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: init 2024-09-16T06:41:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1f1c0a0000 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1f1c1317e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1f1c137da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c11b550 con 0x7f1f1c1317e0 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c11d080 con 0x7f1f1c0a0000 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c11bf50 con 0x7f1f1c137da0 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient(hunting): authenticate will time out at 1055.958530s 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f11d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f1f1c1317e0 legacy=0x7f1f1c11a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f1c0a0000 0x7f1f1c11a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52676/0 (socket says 172.21.15.189:52676) 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1a59c700 1 -- 172.21.15.189:0/3323755352 learned_addr learned my addr 172.21.15.189:0/3323755352 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426936168 0 0) 0x7f1f1c11d080 con 0x7f1f1c0a0000 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): my global_id is 134425 2024-09-16T06:41:43.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc0035d0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209889601 0 0) 0x7f1f1c11bf50 con 0x7f1f1c137da0 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): my global_id is 124712 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1c11d080 con 0x7f1f1c137da0 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062783691 0 0) 0x7f1efc0035d0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f1c11bf50 con 0x7f1f1c0a0000 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f0c001c50 con 0x7f1f1c0a0000 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): dump: 2024-09-16T06:41:43.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3393890443 0 0) 0x7f1f1c11bf50 con 0x7f1f1c0a0000 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 >> v1:172.21.15.76:6790/0 conn(0x7f1f1c137da0 legacy=0x7f1f1c1d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 >> v1:172.21.15.76:6789/0 conn(0x7f1f1c1317e0 legacy=0x7f1f1c11a9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 monclient: found mon.b 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c11b0d0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 5 monclient: authenticate success, global_id 134425 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _renew_subs 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c09f120 con 0x7f1f1c0a0000 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _renew_subs 2024-09-16T06:41:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c11d920 con 0x7f1f1c0a0000 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f0c004b90 con 0x7f1f1c0a0000 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.440+0000 7f1f137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f0c004ea0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 10 monclient: dump: 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f1959a700 4 set_mon_vals no callback set 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1f0c005000 con 0x7f1f1c0a0000 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f1b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(88..88 src has 61..88) v4 ==== 4875+0+0 (unknown 1166119773 0 0) 0x7f1f0c043840 con 0x7f1f1c0a0000 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f1b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f1b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c0a2e60 con 0x7f1f1c0a0000 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.444+0000 7f1f11d9b700 10 monclient: get_auth_request con 0x7f1efc036220 auth_method 0 2024-09-16T06:41:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.452+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1f0c0149f0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.452+0000 7f1f137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.452+0000 7f1f137fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:43.571 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:41:43.572 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:41:43.572 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:41:43.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:41:43.575 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:41:43.575 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:41:43.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.624+0000 7f1f1b59e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.624+0000 7f1f1b59e700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.624+0000 7f1f1b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:43.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:43.624+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f1f1c066ed0 con 0x7f1f1c0a0000 2024-09-16T06:41:43.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: get_monmap_and_config 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: build_initial_monmap 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: monmap: 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:43.684284+0000 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:43.684284+0000 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:43.683 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:43.684 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 1 Processor -- start 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 1 -- start start 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: init 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-a con 0x7fc46c12ee50 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-b con 0x7fc46c12ea20 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-c con 0x7fc46c0a06d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c12f940 con 0x7fc46c12ee50 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c09d5b0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c09d890 con 0x7fc46c12ea20 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient(hunting): authenticate will time out at 1050.818409s 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc46c12ee50 0x7fc46c12f230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56870/0 (socket says 172.21.15.76:56870) 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46b7fe700 1 -- 172.21.15.76:0/1459245366 learned_addr learned my addr 172.21.15.76:0/1459245366 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc46c12ea20 0x7fc46c137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43574/0 (socket says 172.21.15.76:43574) 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247446682 0 0) 0x7fc46c09d890 con 0x7fc46c12ea20 2024-09-16T06:41:43.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient(hunting): my global_id is 124715 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4540057e0 con 0x7fc46c12ea20 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892905432 0 0) 0x7fc4540057e0 con 0x7fc46c12ea20 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 >> v1:172.21.15.189:6789/0 conn(0x7fc46c0a06d0 legacy=0x7fc46c13a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 >> v1:172.21.15.76:6789/0 conn(0x7fc46c12ee50 legacy=0x7fc46c12f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 monclient: found mon.noname-b 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:43.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c09db70 con 0x7fc46c12ea20 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc460001610 con 0x7fc46c12ea20 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:43.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: dump: 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 5 monclient: authenticate success, global_id 124715 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc460001930 con 0x7fc46c12ea20 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc471b00700 4 set_mon_vals no callback set 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 1 -- 172.21.15.76:0/1459245366 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc460001cc0 con 0x7fc46c12ea20 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.683+0000 7fc46a7fc700 10 monclient: dump: 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:43.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: shutdown 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- 172.21.15.76:0/1459245366 >> v1:172.21.15.76:6790/0 conn(0x7fc46c12ea20 legacy=0x7fc46c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- 172.21.15.76:0/1459245366 shutdown_connections 2024-09-16T06:41:43.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- 172.21.15.76:0/1459245366 >> 172.21.15.76:0/1459245366 conn(0x7fc46c0a1440 msgr2=0x7fc46c0a1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- 172.21.15.76:0/1459245366 shutdown_connections 2024-09-16T06:41:43.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- 172.21.15.76:0/1459245366 wait complete. 2024-09-16T06:41:43.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: build_initial_monmap 2024-09-16T06:41:43.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: monmap: 2024-09-16T06:41:43.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:43.698 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:43.698 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 Processor -- start 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- start start 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: init 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:43.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-c con 0x7fc46c0a06d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-b con 0x7fc46c12ea20 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient(hunting): picked mon.noname-a con 0x7fc46c12ee50 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c06f550 con 0x7fc46c12ee50 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c0a2d40 con 0x7fc46c12ea20 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46c12e3d0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc471b00700 10 monclient(hunting): authenticate will time out at 1050.823336s 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc46b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc46c0a06d0 0x7fc46c1c6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43588/0 (socket says 172.21.15.76:43588) 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc46b7fe700 1 -- 172.21.15.76:0/1393291540 learned_addr learned my addr 172.21.15.76:0/1393291540 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482955363 0 0) 0x7fc46c12e3d0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): my global_id is 124718 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4580035d0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158004458 0 0) 0x7fc46c0a2d40 con 0x7fc46c12ea20 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 10 monclient(hunting): my global_id is 134431 2024-09-16T06:41:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.687+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c12e3d0 con 0x7fc46c12ea20 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2845647955 0 0) 0x7fc4580035d0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc46c0a2d40 con 0x7fc46c0a06d0 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc45c0017a0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient(hunting): dump: 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:43.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1666633828 0 0) 0x7fc46c0a2d40 con 0x7fc46c0a06d0 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 >> v1:172.21.15.189:6789/0 conn(0x7fc46c12ea20 legacy=0x7fc46c1cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 >> v1:172.21.15.76:6789/0 conn(0x7fc46c12ee50 legacy=0x7fc46c1ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 monclient: found mon.c 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c09b710 con 0x7fc46c0a06d0 2024-09-16T06:41:43.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 5 monclient: authenticate success, global_id 124718 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _renew_subs 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c06ecb0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _renew_subs 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c12bf80 con 0x7fc46c0a06d0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc45c004890 con 0x7fc46c0a06d0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc45c004c20 con 0x7fc46c0a06d0 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:43.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc4697fa700 4 set_mon_vals no callback set 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 10 monclient: dump: 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc45c004e40 con 0x7fc46c0a06d0 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c005100 con 0x7fc46c0a06d0 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(88..88 src has 61..88) v4 ==== 4875+0+0 (unknown 1166119773 0 0) 0x7fc45c0437f0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.691+0000 7fc46affd700 10 monclient: get_auth_request con 0x7fc458036140 auth_method 0 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.699+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc45c014880 con 0x7fc46c0a06d0 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.699+0000 7fc453fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:43.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.699+0000 7fc453fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:43.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc471b00700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:43.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc471b00700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:43.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc471b00700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:43.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc44c005930 con 0x7fc46c0a06d0 2024-09-16T06:41:43.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc453fff700 1 -- 172.21.15.76:0/1393291540 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 495527228) 0x7fc45c012cf0 con 0x7fc46c0a06d0 2024-09-16T06:41:43.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc453fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:43.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.927+0000 7fc453fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:43.930 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:43.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc458036140 legacy=0x7fc458038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 10 monclient: shutdown 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 >> v1:172.21.15.76:6790/0 conn(0x7fc46c0a06d0 legacy=0x7fc46c1c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 shutdown_connections 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 >> 172.21.15.76:0/1393291540 conn(0x7fc46c0a1440 msgr2=0x7fc46c12d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 shutdown_connections 2024-09-16T06:41:43.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:43.931+0000 7fc471b00700 1 -- 172.21.15.76:0/1393291540 wait complete. 2024-09-16T06:41:43.944 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:41:43.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:41:43.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:41:44.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.055+0000 7f967ad94700 10 monclient: get_monmap_and_config 2024-09-16T06:41:44.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.055+0000 7f967ad94700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.055+0000 7f967ad94700 10 monclient: monmap: 2024-09-16T06:41:44.058 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.058 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:44.059510+0000 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:44.059510+0000 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:44.059 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 Processor -- start 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- start start 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient: init 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-a con 0x7f9674097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-c con 0x7f9674099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-b con 0x7f9674139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674098990 con 0x7f9674097b60 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674097f40 con 0x7f9674099090 2024-09-16T06:41:44.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409d610 con 0x7f9674139500 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient(hunting): authenticate will time out at 1051.192830s 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9679d92700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9674097b60 0x7f967409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:58894/0 (socket says 172.21.15.76:58894) 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9679d92700 1 -- 172.21.15.76:0/1827465403 learned_addr learned my addr 172.21.15.76:0/1827465403 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200771562 0 0) 0x7f967409d610 con 0x7f9674139500 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): my global_id is 124721 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96600035d0 con 0x7f9674139500 2024-09-16T06:41:44.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125343325 0 0) 0x7f9674097f40 con 0x7f9674099090 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient(hunting): my global_id is 134434 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967409d610 con 0x7f9674099090 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135988742 0 0) 0x7f967409d610 con 0x7f9674099090 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 >> v1:172.21.15.76:6790/0 conn(0x7f9674139500 legacy=0x7f967413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 >> v1:172.21.15.76:6789/0 conn(0x7f9674097b60 legacy=0x7f967409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 monclient: found mon.noname-c 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740986b0 con 0x7f9674099090 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f96640015f0 con 0x7f9674099090 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: dump: 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 5 monclient: authenticate success, global_id 134434 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96640018a0 con 0x7f9674099090 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 1 -- 172.21.15.76:0/1827465403 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9664001c30 con 0x7f9674099090 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 4 set_mon_vals no callback set 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f9678d90700 10 monclient: dump: 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 10 monclient: shutdown 2024-09-16T06:41:44.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- 172.21.15.76:0/1827465403 >> v1:172.21.15.189:6789/0 conn(0x7f9674099090 legacy=0x7f967409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- 172.21.15.76:0/1827465403 shutdown_connections 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- 172.21.15.76:0/1827465403 >> 172.21.15.76:0/1827465403 conn(0x7f96740a0030 msgr2=0x7f96740a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- 172.21.15.76:0/1827465403 shutdown_connections 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.059+0000 7f967ad94700 1 -- 172.21.15.76:0/1827465403 wait complete. 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: monmap: 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.069 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:44.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:44.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:44.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 Processor -- start 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- start start 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: init 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-a con 0x7f9674097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-b con 0x7f9674099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient(hunting): picked mon.noname-c con 0x7f9674139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409aba0 con 0x7f9674097b60 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409c360 con 0x7f9674099090 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674071b20 con 0x7f9674139500 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient(hunting): authenticate will time out at 1051.197955s 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967a593700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9674139500 0x7f96741d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58676/0 (socket says 172.21.15.76:58676) 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967a593700 1 -- 172.21.15.76:0/119436025 learned_addr learned my addr 172.21.15.76:0/119436025 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044055391 0 0) 0x7f9674071b20 con 0x7f9674139500 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): my global_id is 124724 2024-09-16T06:41:44.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96580035d0 con 0x7f9674139500 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109948405 0 0) 0x7f967409aba0 con 0x7f9674097b60 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): my global_id is 144099 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9674071b20 con 0x7f9674097b60 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463594306 0 0) 0x7f9674071b20 con 0x7f9674097b60 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f967409aba0 con 0x7f9674097b60 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f96700017e0 con 0x7f9674097b60 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): dump: 2024-09-16T06:41:44.076 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864032779 0 0) 0x7f96580035d0 con 0x7f9674139500 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9674071b20 con 0x7f9674139500 2024-09-16T06:41:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f966c001c50 con 0x7f9674139500 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient(hunting): dump: 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759748437 0 0) 0x7f967409aba0 con 0x7f9674097b60 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6790/0 conn(0x7f9674139500 legacy=0x7f96741d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.189:6789/0 conn(0x7f9674099090 legacy=0x7f96740709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 monclient: found mon.a 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740710c0 con 0x7f9674097b60 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 5 monclient: authenticate success, global_id 144099 2024-09-16T06:41:44.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _renew_subs 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967409a540 con 0x7f9674097b60 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _renew_subs 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f967409ba50 con 0x7f9674097b60 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9670001d60 con 0x7f9674097b60 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9670004a30 con 0x7f9674097b60 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966a7fc700 10 monclient: dump: 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.080 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.063+0000 7f966bfff700 4 set_mon_vals no callback set 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f967ad94700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9670004c10 con 0x7f9674097b60 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(88..88 src has 61..88) v4 ==== 4875+0+0 (unknown 1166119773 0 0) 0x7f9670043690 con 0x7f9674097b60 2024-09-16T06:41:44.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f965c006590 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f9679591700 10 monclient: get_auth_request con 0x7f96580362c0 auth_method 0 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897098163 0 0) 0x7f965c006590 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient(hunting): my global_id is 144102 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967409aba0 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1414569107 0 0) 0x7f967409aba0 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f965c006590 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f966c003be0 con 0x7f965c002830 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient: dump: 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3094430735 0 0) 0x7f965c006590 con 0x7f965c002830 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f965c00a190 con 0x7f965c002830 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f966c004100 con 0x7f965c002830 2024-09-16T06:41:44.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:44.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6789/0 conn(0x7f965c002830 legacy=0x7f965c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.067+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6789/0 conn(0x7f965c002830 legacy=0x7f965c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:44.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f965c009e30 con 0x7f965c009150 2024-09-16T06:41:44.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121926428 0 0) 0x7f965c009e30 con 0x7f965c009150 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient(hunting): my global_id is 144105 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c006590 con 0x7f965c009150 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2757368097 0 0) 0x7f965c006590 con 0x7f965c009150 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f965c009e30 con 0x7f965c009150 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f966c004680 con 0x7f965c009150 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient: dump: 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 208289176 0 0) 0x7f965c009e30 con 0x7f965c009150 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f965c00c340 con 0x7f965c009150 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1321988532) 0x7f966c006f30 con 0x7f965c009150 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6789/0 conn(0x7f965c009150 legacy=0x7f965c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f966a7fc700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6789/0 conn(0x7f965c009150 legacy=0x7f965c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:44.088 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.089 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.090 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.091 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.092 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.093 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.095 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:44.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:44.098 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.189:6812/3098810674 conn(0x7f96580362c0 legacy=0x7f9658038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 10 monclient: shutdown 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 >> v1:172.21.15.76:6789/0 conn(0x7f9674097b60 legacy=0x7f96740702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967a593700 1 -- 172.21.15.76:0/119436025 reap_dead start 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 shutdown_connections 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 >> 172.21.15.76:0/119436025 conn(0x7f96740a0030 msgr2=0x7f96740a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 shutdown_connections 2024-09-16T06:41:44.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.075+0000 7f967ad94700 1 -- 172.21.15.76:0/119436025 wait complete. 2024-09-16T06:41:44.101 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:44.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:41:44.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:41:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.208+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v89) v1 ==== 143+0+0 (unknown 142172526 0 0) 0x7f1f0c012e60 con 0x7f1f1c0a0000 2024-09-16T06:41:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.208+0000 7f1f137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:44.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.208+0000 7f1f137fe700 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 10 monclient: get_monmap_and_config 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 10 monclient: monmap: 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:44.217752+0000 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:44.217752+0000 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 1 Processor -- start 2024-09-16T06:41:44.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 1 -- start start 2024-09-16T06:41:44.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.215+0000 7f0632505700 10 monclient: init 2024-09-16T06:41:44.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:44.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient(hunting): picked mon.noname-c con 0x7f0624025590 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient(hunting): picked mon.noname-a con 0x7f06240232e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient(hunting): picked mon.noname-b con 0x7f0624026080 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0624023fd0 con 0x7f06240232e0 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0624004200 con 0x7f0624025590 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0624004390 con 0x7f0624026080 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient(hunting): authenticate will time out at 1051.352789s 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0630d02700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f06240232e0 0x7f06240bd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:58942/0 (socket says 172.21.15.76:58942) 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0630d02700 1 -- 172.21.15.76:0/1132793365 learned_addr learned my addr 172.21.15.76:0/1132793365 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0631d04700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0624026080 0x7f06240c0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58684/0 (socket says 172.21.15.76:58684) 2024-09-16T06:41:44.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032897865 0 0) 0x7f0624023fd0 con 0x7f06240232e0 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): my global_id is 144108 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06180035d0 con 0x7f06240232e0 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759935932 0 0) 0x7f0624004200 con 0x7f0624025590 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient(hunting): my global_id is 134440 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624023fd0 con 0x7f0624025590 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3804244116 0 0) 0x7f06180035d0 con 0x7f06240232e0 2024-09-16T06:41:44.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 >> v1:172.21.15.76:6790/0 conn(0x7f0624026080 legacy=0x7f06240c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 >> v1:172.21.15.189:6789/0 conn(0x7f0624025590 legacy=0x7f0624025970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 monclient: found mon.noname-a 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240228b0 con 0x7f06240232e0 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 5 monclient: authenticate success, global_id 144108 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f061c001610 con 0x7f06240232e0 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: dump: 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f061c001830 con 0x7f06240232e0 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 1 -- 172.21.15.76:0/1132793365 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f061c001bc0 con 0x7f06240232e0 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 4 set_mon_vals no callback set 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0623fff700 10 monclient: dump: 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.231 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 10 monclient: shutdown 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- 172.21.15.76:0/1132793365 >> v1:172.21.15.76:6789/0 conn(0x7f06240232e0 legacy=0x7f06240bd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- 172.21.15.76:0/1132793365 shutdown_connections 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- 172.21.15.76:0/1132793365 >> 172.21.15.76:0/1132793365 conn(0x7f0624006410 msgr2=0x7f0624006820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- 172.21.15.76:0/1132793365 shutdown_connections 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.219+0000 7f0632505700 1 -- 172.21.15.76:0/1132793365 wait complete. 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: monmap: 2024-09-16T06:41:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 Processor -- start 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- start start 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: init 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient(hunting): picked mon.noname-a con 0x7f06240232e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient(hunting): picked mon.noname-c con 0x7f0624025590 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient(hunting): picked mon.noname-b con 0x7f0624026080 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240b7dd0 con 0x7f06240232e0 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240b9cd0 con 0x7f0624026080 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240b9fb0 con 0x7f0624025590 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient(hunting): authenticate will time out at 1051.357080s 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0631503700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f06240232e0 0x7f06240b5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:58956/0 (socket says 172.21.15.76:58956) 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0631503700 1 -- 172.21.15.76:0/2688614362 learned_addr learned my addr 172.21.15.76:0/2688614362 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771914212 0 0) 0x7f06240b7dd0 con 0x7f06240232e0 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): my global_id is 144111 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06100035d0 con 0x7f06240232e0 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925740750 0 0) 0x7f06240b9fb0 con 0x7f0624025590 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): my global_id is 124730 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06240b7dd0 con 0x7f0624025590 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993738690 0 0) 0x7f06240b9cd0 con 0x7f0624026080 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): my global_id is 134443 2024-09-16T06:41:44.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06240b9fb0 con 0x7f0624026080 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116366136 0 0) 0x7f06100035d0 con 0x7f06240232e0 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06240b9cd0 con 0x7f06240232e0 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0628001990 con 0x7f06240232e0 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.236 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666777139 0 0) 0x7f06240b7dd0 con 0x7f0624025590 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06100035d0 con 0x7f0624025590 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f061c0015d0 con 0x7f0624025590 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.237 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4130603210 0 0) 0x7f06240b9fb0 con 0x7f0624026080 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06240b7dd0 con 0x7f0624026080 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f062c069400 con 0x7f0624026080 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.238 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4072410604 0 0) 0x7f06240b9cd0 con 0x7f06240232e0 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6790/0 conn(0x7f0624025590 legacy=0x7f06240b6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.189:6789/0 conn(0x7f0624026080 legacy=0x7f06241587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 monclient: found mon.a 2024-09-16T06:41:44.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240b9150 con 0x7f06240232e0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 5 monclient: authenticate success, global_id 144111 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _renew_subs 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062401f340 con 0x7f06240232e0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _renew_subs 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06240b99f0 con 0x7f06240232e0 2024-09-16T06:41:44.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0628001d20 con 0x7f06240232e0 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0628003b10 con 0x7f06240232e0 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f06237fe700 4 set_mon_vals no callback set 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 10 monclient: dump: 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.241 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f060c006590 con 0x7f060c002830 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0628003c70 con 0x7f06240232e0 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.223+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(89..89 src has 61..89) v4 ==== 4854+0+0 (unknown 1688771278 0 0) 0x7f0628043610 con 0x7f06240232e0 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350667337 0 0) 0x7f060c006590 con 0x7f060c002830 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient(hunting): my global_id is 124733 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06240b9cd0 con 0x7f060c002830 2024-09-16T06:41:44.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0631d04700 10 monclient: get_auth_request con 0x7f0610036520 auth_method 0 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 224973631 0 0) 0x7f06240b9cd0 con 0x7f060c002830 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c006590 con 0x7f060c002830 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f061c004ed0 con 0x7f060c002830 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient: dump: 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1591705609 0 0) 0x7f060c006590 con 0x7f060c002830 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f060c00a190 con 0x7f060c002830 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f061c007110 con 0x7f060c002830 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6790/0 conn(0x7f060c002830 legacy=0x7f060c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.227+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6790/0 conn(0x7f060c002830 legacy=0x7f060c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0632505700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f060c009e30 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919719053 0 0) 0x7f060c009e30 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient(hunting): my global_id is 124736 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c006590 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3925073000 0 0) 0x7f060c006590 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c009e30 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f061c005760 con 0x7f060c009150 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.235+0000 7f0621ffb700 10 monclient: dump: 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1928450891 0 0) 0x7f060c009e30 con 0x7f060c009150 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f060c00c340 con 0x7f060c009150 2024-09-16T06:41:44.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 728876400) 0x7f061c006b40 con 0x7f060c009150 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6790/0 conn(0x7f060c009150 legacy=0x7f060c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0621ffb700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6790/0 conn(0x7f060c009150 legacy=0x7f060c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:44.248 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.249 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.250 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.252 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:44.255 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:44.256 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.258 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.259 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:44.260 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:44.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0610036520 legacy=0x7f06100389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 10 monclient: shutdown 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 >> v1:172.21.15.76:6789/0 conn(0x7f06240232e0 legacy=0x7f06240b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0631d04700 1 -- 172.21.15.76:0/2688614362 reap_dead start 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0631d04700 1 -- 172.21.15.76:0/2688614362 reap_dead start 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 shutdown_connections 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 >> 172.21.15.76:0/2688614362 conn(0x7f0624006410 msgr2=0x7f06240286e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 shutdown_connections 2024-09-16T06:41:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.239+0000 7f0632505700 1 -- 172.21.15.76:0/2688614362 wait complete. 2024-09-16T06:41:44.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:44.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:41:44.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:41:44.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.320+0000 7f1f1b59e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:44.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.320+0000 7f1f1b59e700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:44.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.320+0000 7f1f1b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.320+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f1f1c09acc0 con 0x7f1f1c0a0000 2024-09-16T06:41:44.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f137fe700 1 -- 172.21.15.189:0/3323755352 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v89) v1 ==== 145+0+0 (unknown 659029593 0 0) 0x7f1f0c0147c0 con 0x7f1f1c0a0000 2024-09-16T06:41:44.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f137fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T06:41:44.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f137fe700 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1efc036220 legacy=0x7f1efc0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 10 monclient: shutdown 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 >> v1:172.21.15.189:6789/0 conn(0x7f1f1c0a0000 legacy=0x7f1f1c11a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 shutdown_connections 2024-09-16T06:41:44.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 >> 172.21.15.189:0/3323755352 conn(0x7f1f1c073b60 msgr2=0x7f1f1c0a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 shutdown_connections 2024-09-16T06:41:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.324+0000 7f1f1b59e700 1 -- 172.21.15.189:0/3323755352 wait complete. 2024-09-16T06:41:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T06:41:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm host2 2024-09-16T06:41:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: get_monmap_and_config 2024-09-16T06:41:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: monmap: 2024-09-16T06:41:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:44.377402+0000 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:44.377402+0000 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:44.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 1 Processor -- start 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 1 -- start start 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: init 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:44.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-a con 0x7fd6c406a910 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-c con 0x7fd6c406b400 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-b con 0x7fd6c409c350 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c40a1e10 con 0x7fd6c406a910 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c409e8d0 con 0x7fd6c406b400 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c4072090 con 0x7fd6c409c350 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c99c9700 10 monclient(hunting): authenticate will time out at 1051.511469s 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c89c7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd6c406a910 0x7fd6c406acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:58966/0 (socket says 172.21.15.76:58966) 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c89c7700 1 -- 172.21.15.76:0/1406995257 learned_addr learned my addr 172.21.15.76:0/1406995257 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.375+0000 7fd6c91c8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd6c409c350 0x7fd6c413dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58720/0 (socket says 172.21.15.76:58720) 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480909571 0 0) 0x7fd6c40a1e10 con 0x7fd6c406a910 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): my global_id is 144114 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6ac0035d0 con 0x7fd6c406a910 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752019266 0 0) 0x7fd6c4072090 con 0x7fd6c409c350 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient(hunting): my global_id is 124739 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c40a1e10 con 0x7fd6c409c350 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683426516 0 0) 0x7fd6c40a1e10 con 0x7fd6c409c350 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 >> v1:172.21.15.189:6789/0 conn(0x7fd6c406b400 legacy=0x7fd6c413a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 >> v1:172.21.15.76:6789/0 conn(0x7fd6c406a910 legacy=0x7fd6c406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 monclient: found mon.noname-b 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c409c7e0 con 0x7fd6c409c350 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 5 monclient: authenticate success, global_id 124739 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6bc001990 con 0x7fd6c409c350 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: dump: 2024-09-16T06:41:44.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.385 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6bc001c30 con 0x7fd6c409c350 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 1 -- 172.21.15.76:0/1406995257 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6bc001fc0 con 0x7fd6c409c350 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 4 set_mon_vals no callback set 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c37fe700 10 monclient: dump: 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.386 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 10 monclient: shutdown 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- 172.21.15.76:0/1406995257 >> v1:172.21.15.76:6790/0 conn(0x7fd6c409c350 legacy=0x7fd6c413dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- 172.21.15.76:0/1406995257 shutdown_connections 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- 172.21.15.76:0/1406995257 >> 172.21.15.76:0/1406995257 conn(0x7fd6c40a12f0 msgr2=0x7fd6c40a1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- 172.21.15.76:0/1406995257 shutdown_connections 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- 172.21.15.76:0/1406995257 wait complete. 2024-09-16T06:41:44.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 10 monclient: monmap: 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:44.388 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 Processor -- start 2024-09-16T06:41:44.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 1 -- start start 2024-09-16T06:41:44.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.379+0000 7fd6c99c9700 10 monclient: init 2024-09-16T06:41:44.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-a con 0x7fd6c406a910 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-c con 0x7fd6c406b400 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient(hunting): picked mon.noname-b con 0x7fd6c409c350 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c408e320 con 0x7fd6c406a910 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c40a3360 con 0x7fd6c409c350 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c408be10 con 0x7fd6c406b400 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient(hunting): authenticate will time out at 1051.516291s 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd6c406b400 0x7fd6c408b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58726/0 (socket says 172.21.15.76:58726) 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c3fff700 1 -- 172.21.15.76:0/4254504271 learned_addr learned my addr 172.21.15.76:0/4254504271 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526757761 0 0) 0x7fd6c408be10 con 0x7fd6c406b400 2024-09-16T06:41:44.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): my global_id is 124742 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b00035d0 con 0x7fd6c406b400 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702665790 0 0) 0x7fd6c408e320 con 0x7fd6c406a910 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): my global_id is 144117 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c408be10 con 0x7fd6c406a910 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3065233892 0 0) 0x7fd6c408be10 con 0x7fd6c406a910 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6c408e320 con 0x7fd6c406a910 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017381833 0 0) 0x7fd6c40a3360 con 0x7fd6c409c350 2024-09-16T06:41:44.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): my global_id is 134449 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c408be10 con 0x7fd6c409c350 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6b4001810 con 0x7fd6c406a910 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): dump: 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1706219933 0 0) 0x7fd6b00035d0 con 0x7fd6c406b400 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6c40a3360 con 0x7fd6c406b400 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6b80018f0 con 0x7fd6c406b400 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient(hunting): dump: 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4060336262 0 0) 0x7fd6c408e320 con 0x7fd6c406a910 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.76:6790/0 conn(0x7fd6c406b400 legacy=0x7fd6c408b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6789/0 conn(0x7fd6c409c350 legacy=0x7fd6c41d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 monclient: found mon.a 2024-09-16T06:41:44.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c408c7b0 con 0x7fd6c406a910 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 5 monclient: authenticate success, global_id 144117 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _renew_subs 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c40a2e10 con 0x7fd6c406a910 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _renew_subs 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c408c2a0 con 0x7fd6c406a910 2024-09-16T06:41:44.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6b4001cd0 con 0x7fd6c406a910 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6b4004a30 con 0x7fd6c406a910 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c2ffd700 4 set_mon_vals no callback set 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 10 monclient: dump: 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.400 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd6b4004c10 con 0x7fd6c406a910 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(89..89 src has 61..89) v4 ==== 4854+0+0 (unknown 1688771278 0 0) 0x7fd6b4043660 con 0x7fd6c406a910 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c3fff700 10 monclient: get_auth_request con 0x7fd6b0036470 auth_method 0 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.383+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd6a4006590 con 0x7fd6a4002830 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904657043 0 0) 0x7fd6a4006590 con 0x7fd6a4002830 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient(hunting): my global_id is 134452 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c408e320 con 0x7fd6a4002830 2024-09-16T06:41:44.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1984499354 0 0) 0x7fd6c408e320 con 0x7fd6a4002830 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6a4006590 con 0x7fd6a4002830 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6bc001520 con 0x7fd6a4002830 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient: dump: 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.402 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 458093346 0 0) 0x7fd6a4006590 con 0x7fd6a4002830 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6a400a190 con 0x7fd6a4002830 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd6bc007480 con 0x7fd6a4002830 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6789/0 conn(0x7fd6a4002830 legacy=0x7fd6a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.387+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6789/0 conn(0x7fd6a4002830 legacy=0x7fd6a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c99c9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd6a4009e50 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753356737 0 0) 0x7fd6a4009e50 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient(hunting): my global_id is 134455 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a4006590 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1267760086 0 0) 0x7fd6a4006590 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6a4009e50 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd6bc005410 con 0x7fd6a4009170 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient: dump: 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1507192232 0 0) 0x7fd6a4009e50 con 0x7fd6a4009170 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6a400c340 con 0x7fd6a4009170 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3156051627) 0x7fd6bc0075f0 con 0x7fd6a4009170 2024-09-16T06:41:44.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6789/0 conn(0x7fd6a4009170 legacy=0x7fd6a4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.399+0000 7fd6c17fa700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6789/0 conn(0x7fd6a4009170 legacy=0x7fd6a4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:44.407 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.408 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:44.409 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:44.410 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:44.411 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.412 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.413 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.415 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:44.416 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:44.417 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:44.418 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:44.419 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:44.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd6b0036470 legacy=0x7fd6b0038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 10 monclient: shutdown 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c91c8700 1 -- 172.21.15.76:0/4254504271 reap_dead start 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 >> v1:172.21.15.76:6789/0 conn(0x7fd6c406a910 legacy=0x7fd6c408aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 shutdown_connections 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 >> 172.21.15.76:0/4254504271 conn(0x7fd6c40a12f0 msgr2=0x7fd6c409ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 shutdown_connections 2024-09-16T06:41:44.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.403+0000 7fd6c99c9700 1 -- 172.21.15.76:0/4254504271 wait complete. 2024-09-16T06:41:44.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:44.421 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:41:44.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.448+0000 7f72b25bb700 10 monclient: get_monmap_and_config 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.448+0000 7f72b25bb700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.448+0000 7f72b25bb700 10 monclient: monmap: 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:44.456116+0000 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:44.456116+0000 2024-09-16T06:41:44.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:44.456 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 Processor -- start 2024-09-16T06:41:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- start start 2024-09-16T06:41:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient: init 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-b con 0x7f72ac099c40 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-a con 0x7f72ac098090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-c con 0x7f72ac09a020 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac097690 con 0x7f72ac098090 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac09d770 con 0x7f72ac09a020 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0984a0 con 0x7f72ac099c40 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient(hunting): authenticate will time out at 1056.969192s 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b0b58700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72ac09a020 0x7f72ac09b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52680/0 (socket says 172.21.15.189:52680) 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b0b58700 1 -- 172.21.15.189:0/946002003 learned_addr learned my addr 172.21.15.189:0/946002003 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914053745 0 0) 0x7f72ac09d770 con 0x7f72ac09a020 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient(hunting): my global_id is 134458 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72940046e0 con 0x7f72ac09a020 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075460057 0 0) 0x7f72940046e0 con 0x7f72ac09a020 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 >> v1:172.21.15.76:6790/0 conn(0x7f72ac099c40 legacy=0x7f72ac09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 >> v1:172.21.15.76:6789/0 conn(0x7f72ac098090 legacy=0x7f72ac09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 monclient: found mon.noname-c 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac09fa10 con 0x7f72ac09a020 2024-09-16T06:41:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72a4001990 con 0x7f72ac09a020 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 5 monclient: authenticate success, global_id 134458 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: dump: 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a4001c30 con 0x7f72ac09a020 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 1 -- 172.21.15.189:0/946002003 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72a4002000 con 0x7f72ac09a020 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72aaffd700 10 monclient: dump: 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 4 set_mon_vals no callback set 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 10 monclient: shutdown 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- 172.21.15.189:0/946002003 >> v1:172.21.15.189:6789/0 conn(0x7f72ac09a020 legacy=0x7f72ac09b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- 172.21.15.189:0/946002003 shutdown_connections 2024-09-16T06:41:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- 172.21.15.189:0/946002003 >> 172.21.15.189:0/946002003 conn(0x7f72ac0a03e0 msgr2=0x7f72ac0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.452+0000 7f72b25bb700 1 -- 172.21.15.189:0/946002003 shutdown_connections 2024-09-16T06:41:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 -- 172.21.15.189:0/946002003 wait complete. 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient: monmap: 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:44.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 Processor -- start 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 -- start start 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient: init 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-a con 0x7f72ac098090 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-b con 0x7f72ac099c40 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient(hunting): picked mon.noname-c con 0x7f72ac09a020 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac090890 con 0x7f72ac098090 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac08f3e0 con 0x7f72ac099c40 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0a2d30 con 0x7f72ac09a020 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72b25bb700 10 monclient(hunting): authenticate will time out at 1056.975846s 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72ab7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72ac099c40 0x7f72ac1ca900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52690/0 (socket says 172.21.15.189:52690) 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.456+0000 7f72ab7fe700 1 -- 172.21.15.189:0/3116197641 learned_addr learned my addr 172.21.15.189:0/3116197641 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971781501 0 0) 0x7f72ac08f3e0 con 0x7f72ac099c40 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): my global_id is 134461 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72980057e0 con 0x7f72ac099c40 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683315022 0 0) 0x7f72ac0a2d30 con 0x7f72ac09a020 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): my global_id is 124748 2024-09-16T06:41:44.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac08f3e0 con 0x7f72ac09a020 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450705697 0 0) 0x7f72ac090890 con 0x7f72ac098090 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): my global_id is 144123 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac0a2d30 con 0x7f72ac098090 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902570398 0 0) 0x7f72980057e0 con 0x7f72ac099c40 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ac090890 con 0x7f72ac099c40 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72a0001910 con 0x7f72ac099c40 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046605213 0 0) 0x7f72ac08f3e0 con 0x7f72ac09a020 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72980057e0 con 0x7f72ac09a020 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72a4001f40 con 0x7f72ac09a020 2024-09-16T06:41:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189149702 0 0) 0x7f72ac0a2d30 con 0x7f72ac098090 2024-09-16T06:41:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ac08f3e0 con 0x7f72ac098090 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f729c001830 con 0x7f72ac098090 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.476 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 985188522 0 0) 0x7f72ac090890 con 0x7f72ac099c40 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 >> v1:172.21.15.76:6790/0 conn(0x7f72ac09a020 legacy=0x7f72ac1ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 >> v1:172.21.15.76:6789/0 conn(0x7f72ac098090 legacy=0x7f72ac1c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 monclient: found mon.b 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac09e9f0 con 0x7f72ac099c40 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 5 monclient: authenticate success, global_id 134461 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _renew_subs 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac0905b0 con 0x7f72ac099c40 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _renew_subs 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac08e810 con 0x7f72ac099c40 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a0001cf0 con 0x7f72ac099c40 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72a0003b10 con 0x7f72ac099c40 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 10 monclient: dump: 2024-09-16T06:41:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72aa7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f72a0003cf0 con 0x7f72ac099c40 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c005100 con 0x7f72ac099c40 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(89..89 src has 61..89) v4 ==== 4854+0+0 (unknown 1688771278 0 0) 0x7f72a00436a0 con 0x7f72ac099c40 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.460+0000 7f72abfff700 10 monclient: get_auth_request con 0x7f7298038790 auth_method 0 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.472+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f72a0014660 con 0x7f72ac099c40 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.472+0000 7f72a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.472+0000 7f72a8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: get_monmap_and_config 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: monmap: 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:44.537381+0000 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:44.537381+0000 2024-09-16T06:41:44.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:44.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:44.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 1 Processor -- start 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 1 -- start start 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: init 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-a con 0x7f7c980973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-c con 0x7f7c98097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-b con 0x7f7c981400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c98099350 con 0x7f7c980973c0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9809f690 con 0x7f7c98097eb0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c98099bf0 con 0x7f7c981400d0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient(hunting): authenticate will time out at 1051.670612s 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c97fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7c981400d0 0x7f7c981424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58734/0 (socket says 172.21.15.76:58734) 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c97fff700 1 -- 172.21.15.76:0/352503495 learned_addr learned my addr 172.21.15.76:0/352503495 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142491552 0 0) 0x7f7c98099bf0 con 0x7f7c981400d0 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient(hunting): my global_id is 124751 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c840035d0 con 0x7f7c981400d0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 244257502 0 0) 0x7f7c840035d0 con 0x7f7c981400d0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 >> v1:172.21.15.189:6789/0 conn(0x7f7c98097eb0 legacy=0x7f7c9813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 >> v1:172.21.15.76:6789/0 conn(0x7f7c980973c0 legacy=0x7f7c980977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 monclient: found mon.noname-b 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98098360 con 0x7f7c981400d0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 5 monclient: authenticate success, global_id 124751 2024-09-16T06:41:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c90001990 con 0x7f7c981400d0 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: dump: 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.543 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c90001cb0 con 0x7f7c981400d0 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 1 -- 172.21.15.76:0/352503495 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c90002080 con 0x7f7c981400d0 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c96ffd700 10 monclient: dump: 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.544 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.535+0000 7f7c9ccc0700 4 set_mon_vals no callback set 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: shutdown 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/352503495 >> v1:172.21.15.76:6790/0 conn(0x7f7c981400d0 legacy=0x7f7c981424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/352503495 shutdown_connections 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/352503495 >> 172.21.15.76:0/352503495 conn(0x7f7c980a0030 msgr2=0x7f7c980a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/352503495 shutdown_connections 2024-09-16T06:41:44.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/352503495 wait complete. 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: build_initial_monmap 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: monmap: 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:44.550 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 Processor -- start 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- start start 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: init 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-b con 0x7f7c980973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-c con 0x7f7c98097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient(hunting): picked mon.noname-a con 0x7f7c981400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c98072660 con 0x7f7c981400d0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9806c480 con 0x7f7c980973c0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9809eb50 con 0x7f7c98097eb0 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:44.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c9ccc0700 10 monclient(hunting): authenticate will time out at 1051.675085s 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c97fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7c981400d0 0x7f7c9813d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:58976/0 (socket says 172.21.15.76:58976) 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c97fff700 1 -- 172.21.15.76:0/3441754307 learned_addr learned my addr 172.21.15.76:0/3441754307 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124534173 0 0) 0x7f7c98072660 con 0x7f7c981400d0 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): my global_id is 144129 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c780035d0 con 0x7f7c981400d0 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973052736 0 0) 0x7f7c80001230 con 0x7f7c980973c0 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): my global_id is 134467 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c98072660 con 0x7f7c980973c0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872892551 0 0) 0x7f7c9809eb50 con 0x7f7c98097eb0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): my global_id is 124754 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c80001230 con 0x7f7c98097eb0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396929416 0 0) 0x7f7c780035d0 con 0x7f7c981400d0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c9809eb50 con 0x7f7c981400d0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c90001950 con 0x7f7c981400d0 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:44.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.539+0000 7f7c94ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284395357 0 0) 0x7f7c98072660 con 0x7f7c980973c0 2024-09-16T06:41:44.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c780035d0 con 0x7f7c980973c0 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c80001920 con 0x7f7c980973c0 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient(hunting): dump: 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.555 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270398968 0 0) 0x7f7c9809eb50 con 0x7f7c981400d0 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 >> v1:172.21.15.76:6790/0 conn(0x7f7c98097eb0 legacy=0x7f7c9813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 >> v1:172.21.15.189:6789/0 conn(0x7f7c980973c0 legacy=0x7f7c9813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 monclient: found mon.a 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c980a3090 con 0x7f7c981400d0 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: _finish_auth 0 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 5 monclient: authenticate success, global_id 144129 2024-09-16T06:41:44.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _renew_subs 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9809a540 con 0x7f7c981400d0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _renew_subs 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9813df30 con 0x7f7c981400d0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c980008d0 con 0x7f7c981400d0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c90005020 con 0x7f7c981400d0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c900053b0 con 0x7f7c981400d0 2024-09-16T06:41:44.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 10 monclient: dump: 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c967fc700 4 set_mon_vals no callback set 2024-09-16T06:41:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7c900055d0 con 0x7f7c981400d0 2024-09-16T06:41:44.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(89..89 src has 61..89) v4 ==== 4854+0+0 (unknown 1688771278 0 0) 0x7f7c90043a70 con 0x7f7c981400d0 2024-09-16T06:41:44.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.543+0000 7f7c977fe700 10 monclient: get_auth_request con 0x7f7c780364a0 auth_method 0 2024-09-16T06:41:44.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.551+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7c90014d70 con 0x7f7c981400d0 2024-09-16T06:41:44.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.551+0000 7f7c94ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:44.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.551+0000 7f7c94ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.648+0000 7f72b25bb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.648+0000 7f72b25bb700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.648+0000 7f72b25bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:44.648+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f728c0053b0 con 0x7f72ac099c40 2024-09-16T06:41:44.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.783+0000 7f7c9ccc0700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:41:44.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.783+0000 7f7c9ccc0700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:41:44.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.783+0000 7f7c9ccc0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:44.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.783+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7c9813eb00 con 0x7f7c981400d0 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2890840826,config=1097430493,health=260409864,kv=3833402919,logm=2288518513}) 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=12,logm=55} crc {auth=4281691569,config=1097430493,health=2147088450,kv=790649379,logm=3648355619}) 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2890840826,config=1097430493,health=260409864,kv=3833402919,logm=2288518513}) 2024-09-16T06:41:44.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=12,logm=55} crc {auth=2166970939,config=4178277379,health=3910066772,kv=4212347341,logm=4291718592}) 2024-09-16T06:41:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f7c94ff9700 1 -- 172.21.15.76:0/3441754307 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f7c900131e0 con 0x7f7c981400d0 2024-09-16T06:41:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f7c94ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:41:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.787+0000 7f7c94ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:44.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.793 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=273705553}) 2024-09-16T06:41:44.793 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937372664}) 2024-09-16T06:41:44.793 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.793 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=273705553}) 2024-09-16T06:41:44.793 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.791+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028121806}) 2024-09-16T06:41:44.796 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.796 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3891454190}) 2024-09-16T06:41:44.796 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501528625}) 2024-09-16T06:41:44.796 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7c780364a0 legacy=0x7f7c78038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 10 monclient: shutdown 2024-09-16T06:41:44.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:44.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 >> v1:172.21.15.76:6789/0 conn(0x7f7c981400d0 legacy=0x7f7c9813d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:44.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 shutdown_connections 2024-09-16T06:41:44.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3891454190}) 2024-09-16T06:41:44.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52859852}) 2024-09-16T06:41:44.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 >> 172.21.15.76:0/3441754307 conn(0x7f7c980a0030 msgr2=0x7f7c980a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:44.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 shutdown_connections 2024-09-16T06:41:44.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:44.795+0000 7f7c9ccc0700 1 -- 172.21.15.76:0/3441754307 wait complete. 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1085553146}) 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1699627532}) 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1085553146}) 2024-09-16T06:41:44.800 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2573818332}) 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010348130}) 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608136819}) 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010348130}) 2024-09-16T06:41:44.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045604497}) 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499954738}) 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857659362}) 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499954738}) 2024-09-16T06:41:44.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.799+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748363110}) 2024-09-16T06:41:44.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145275612}) 2024-09-16T06:41:44.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351794578}) 2024-09-16T06:41:44.804 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.804 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145275612}) 2024-09-16T06:41:44.804 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3647407457}) 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624726205}) 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784280146}) 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624726205}) 2024-09-16T06:41:44.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1758168587}) 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302828317}) 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688570921}) 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302828317}) 2024-09-16T06:41:44.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.803+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1477857931}) 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237261320}) 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188685536}) 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237261320}) 2024-09-16T06:41:44.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106852094}) 2024-09-16T06:41:44.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290657862}) 2024-09-16T06:41:44.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197151514}) 2024-09-16T06:41:44.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290657862}) 2024-09-16T06:41:44.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769410660}) 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014956417}) 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460587991}) 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014956417}) 2024-09-16T06:41:44.810 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.807+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553095455}) 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928553873}) 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201423141}) 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928553873}) 2024-09-16T06:41:44.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149873132}) 2024-09-16T06:41:44.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:41:44.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=18,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=619276053,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4158306020,mon_config_key=921397955,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=420077249,osd_snap=2033388227,osdmap=2247263385}) 2024-09-16T06:41:44.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=30} crc {logm=351868966,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4158306020,mon_config_key=4279594037,monmap=1887442722,osd_metadata=1978471115,osd_pg_creating=420077249,osd_snap=2033388227,osdmap=1297263130}) 2024-09-16T06:41:44.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=18,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=619276053,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4158306020,mon_config_key=921397955,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=420077249,osd_snap=2033388227,osdmap=2247263385}) 2024-09-16T06:41:44.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.811+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=30} crc {logm=619276053,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4158306020,mon_config_key=3059854547,monmap=1887442722,osd_metadata=2396766746,osd_pg_creating=420077249,osd_snap=2033388227,osdmap=3044548309}) 2024-09-16T06:41:44.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=1451335049}) 2024-09-16T06:41:44.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=3095968880}) 2024-09-16T06:41:44.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:41:44.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=1451335049}) 2024-09-16T06:41:44.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:41:44.815+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=2038243250}) 2024-09-16T06:41:45.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.236+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v90) v1 ==== 124+0+0 (unknown 911159847 0 0) 0x7f72a0012d00 con 0x7f72ac099c40 2024-09-16T06:41:45.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.236+0000 7f72a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:45.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.236+0000 7f72a8ff9700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T06:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72b25bb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72b25bb700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72b25bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7f728c005650 con 0x7f72ac099c40 2024-09-16T06:41:45.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72a8ff9700 1 -- 172.21.15.189:0/3116197641 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v90) v1 ==== 124+0+0 (unknown 652536795 0 0) 0x7f72a0014480 con 0x7f72ac099c40 2024-09-16T06:41:45.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72a8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T06:41:45.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.352+0000 7f72a8ff9700 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7298038790 legacy=0x7f729803ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 10 monclient: shutdown 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 >> v1:172.21.15.189:6789/0 conn(0x7f72ac099c40 legacy=0x7f72ac1ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 shutdown_connections 2024-09-16T06:41:45.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 >> 172.21.15.189:0/3116197641 conn(0x7f72ac0a03e0 msgr2=0x7f72ac06db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 shutdown_connections 2024-09-16T06:41:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.356+0000 7f72b25bb700 1 -- 172.21.15.189:0/3116197641 wait complete. 2024-09-16T06:41:45.357 INFO:tasks.workunit.client.0.smithi189.stderr:device 'host2' does not appear in the crush map 2024-09-16T06:41:45.366 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket foo host 2024-09-16T06:41:45.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.468+0000 7fc286faa700 10 monclient: get_monmap_and_config 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.468+0000 7fc286faa700 10 monclient: build_initial_monmap 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.468+0000 7fc286faa700 10 monclient: monmap: 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:45.471774+0000 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:45.471774+0000 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:45.471 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:45.472 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 Processor -- start 2024-09-16T06:41:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- start start 2024-09-16T06:41:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient: init 2024-09-16T06:41:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-b con 0x7fc280132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-c con 0x7fc280138460 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-a con 0x7fc28013bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28006abd0 con 0x7fc28013bf70 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28006b170 con 0x7fc280138460 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc280132b70 con 0x7fc280132080 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient(hunting): authenticate will time out at 1057.985223s 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc2857a7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc280138460 0x7fc28013a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52700/0 (socket says 172.21.15.189:52700) 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc2857a7700 1 -- 172.21.15.189:0/1732901373 learned_addr learned my addr 172.21.15.189:0/1732901373 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066490272 0 0) 0x7fc28006b170 con 0x7fc280138460 2024-09-16T06:41:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): my global_id is 134470 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c0035d0 con 0x7fc280138460 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361595035 0 0) 0x7fc28006abd0 con 0x7fc28013bf70 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient(hunting): my global_id is 144132 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28006b170 con 0x7fc28013bf70 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3639594755 0 0) 0x7fc26c0035d0 con 0x7fc280138460 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 >> v1:172.21.15.76:6790/0 conn(0x7fc280132080 legacy=0x7fc280132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 >> v1:172.21.15.76:6789/0 conn(0x7fc28013bf70 legacy=0x7fc28013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: _finish_hunting 0 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 monclient: found mon.noname-c 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2800a38b0 con 0x7fc280138460 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: _finish_auth 0 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: _check_auth_tickets 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc270001610 con 0x7fc280138460 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 5 monclient: authenticate success, global_id 134470 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: dump: 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc270001830 con 0x7fc280138460 2024-09-16T06:41:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 1 -- 172.21.15.189:0/1732901373 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc270001bc0 con 0x7fc280138460 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 4 set_mon_vals no callback set 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc284fa6700 10 monclient: dump: 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 10 monclient: shutdown 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- 172.21.15.189:0/1732901373 >> v1:172.21.15.189:6789/0 conn(0x7fc280138460 legacy=0x7fc28013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- 172.21.15.189:0/1732901373 shutdown_connections 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- 172.21.15.189:0/1732901373 >> 172.21.15.189:0/1732901373 conn(0x7fc2800a4660 msgr2=0x7fc280099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- 172.21.15.189:0/1732901373 shutdown_connections 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.472+0000 7fc286faa700 1 -- 172.21.15.189:0/1732901373 wait complete. 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: build_initial_monmap 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: monmap: 2024-09-16T06:41:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:45.482 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 Processor -- start 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- start start 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: init 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-a con 0x7fc280132080 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-c con 0x7fc280138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:45.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient(hunting): picked mon.noname-b con 0x7fc28013bf70 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800a0110 con 0x7fc280132080 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800a64f0 con 0x7fc28013bf70 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28013b870 con 0x7fc280138460 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient(hunting): authenticate will time out at 1057.989039s 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2867a9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc28013bf70 0x7fc2801d6890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52702/0 (socket says 172.21.15.189:52702) 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2867a9700 1 -- 172.21.15.189:0/4270578750 learned_addr learned my addr 172.21.15.189:0/4270578750 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649289214 0 0) 0x7fc2800a64f0 con 0x7fc28013bf70 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): my global_id is 134473 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2640035d0 con 0x7fc28013bf70 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 15323178 0 0) 0x7fc2640035d0 con 0x7fc28013bf70 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2800a64f0 con 0x7fc28013bf70 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc278001c50 con 0x7fc28013bf70 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): dump: 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845831008 0 0) 0x7fc28013b870 con 0x7fc280138460 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): my global_id is 124760 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2640035d0 con 0x7fc280138460 2024-09-16T06:41:45.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436807020 0 0) 0x7fc2800a0110 con 0x7fc280132080 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient(hunting): my global_id is 144135 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28013b870 con 0x7fc280132080 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3545065011 0 0) 0x7fc2800a64f0 con 0x7fc28013bf70 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 >> v1:172.21.15.76:6790/0 conn(0x7fc280138460 legacy=0x7fc2801d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 >> v1:172.21.15.76:6789/0 conn(0x7fc280132080 legacy=0x7fc2800a1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 monclient: found mon.b 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28009f5b0 con 0x7fc28013bf70 2024-09-16T06:41:45.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 5 monclient: authenticate success, global_id 134473 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _renew_subs 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28009f870 con 0x7fc28013bf70 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _renew_subs 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2800a08a0 con 0x7fc28013bf70 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc278002020 con 0x7fc28013bf70 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc278003f10 con 0x7fc28013bf70 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc2767fc700 10 monclient: dump: 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc277fff700 4 set_mon_vals no callback set 2024-09-16T06:41:45.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.476+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc268005100 con 0x7fc28013bf70 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.480+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc2780040f0 con 0x7fc28013bf70 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.480+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(90..90 src has 61..90) v4 ==== 4719+0+0 (unknown 4242505761 0 0) 0x7fc278043a10 con 0x7fc28013bf70 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.480+0000 7fc285fa8700 10 monclient: get_auth_request con 0x7fc264036390 auth_method 0 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.484+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc2780149f0 con 0x7fc28013bf70 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.484+0000 7fc2767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.484+0000 7fc2767fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:45.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.660+0000 7fc286faa700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:45.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.660+0000 7fc286faa700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:45.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.660+0000 7fc286faa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:45.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:45.660+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fc268005930 con 0x7fc28013bf70 2024-09-16T06:41:45.814 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:41:45.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: get_monmap_and_config 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: build_initial_monmap 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: monmap: 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:45.930201+0000 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:45.930201+0000 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:45.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 1 Processor -- start 2024-09-16T06:41:45.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 1 -- start start 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: init 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient(hunting): picked mon.noname-a con 0x7fa20c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient(hunting): picked mon.noname-b con 0x7fa20c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 10 monclient(hunting): picked mon.noname-c con 0x7fa20c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.926+0000 7fa211919700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c06c280 con 0x7fa20c132330 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c06bfa0 con 0x7fa20c06cf50 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c09ace0 con 0x7fa20c06d480 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient(hunting): authenticate will time out at 1053.064034s 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa20c06d480 0x7fa20c13a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58758/0 (socket says 172.21.15.76:58758) 2024-09-16T06:41:45.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20bfff700 1 -- 172.21.15.76:0/4068783566 learned_addr learned my addr 172.21.15.76:0/4068783566 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696229166 0 0) 0x7fa20c09ace0 con 0x7fa20c06d480 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): my global_id is 124763 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f40035d0 con 0x7fa20c06d480 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810908456 0 0) 0x7fa20c06c280 con 0x7fa20c132330 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): my global_id is 144138 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c09ace0 con 0x7fa20c132330 2024-09-16T06:41:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903824447 0 0) 0x7fa20c06bfa0 con 0x7fa20c06cf50 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient(hunting): my global_id is 134476 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c06c280 con 0x7fa20c06cf50 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3153036772 0 0) 0x7fa20c09ace0 con 0x7fa20c132330 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 >> v1:172.21.15.76:6790/0 conn(0x7fa20c06d480 legacy=0x7fa20c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 >> v1:172.21.15.189:6789/0 conn(0x7fa20c06cf50 legacy=0x7fa20c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c09ae70 con 0x7fa20c132330 2024-09-16T06:41:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 5 monclient: authenticate success, global_id 144138 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1fc0015f0 con 0x7fa20c132330 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: dump: 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1fc001910 con 0x7fa20c132330 2024-09-16T06:41:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 1 -- 172.21.15.76:0/4068783566 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1fc001b40 con 0x7fa20c132330 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 4 set_mon_vals no callback set 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa20b7fe700 10 monclient: dump: 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient: shutdown 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- 172.21.15.76:0/4068783566 >> v1:172.21.15.76:6789/0 conn(0x7fa20c132330 legacy=0x7fa20c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- 172.21.15.76:0/4068783566 shutdown_connections 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- 172.21.15.76:0/4068783566 >> 172.21.15.76:0/4068783566 conn(0x7fa20c0a41c0 msgr2=0x7fa20c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- 172.21.15.76:0/4068783566 shutdown_connections 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 1 -- 172.21.15.76:0/4068783566 wait complete. 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient: build_initial_monmap 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.930+0000 7fa211919700 10 monclient: monmap: 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 Processor -- start 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- start start 2024-09-16T06:41:45.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: init 2024-09-16T06:41:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient(hunting): picked mon.noname-c con 0x7fa20c06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient(hunting): picked mon.noname-a con 0x7fa20c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient(hunting): picked mon.noname-b con 0x7fa20c132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c131140 con 0x7fa20c06d480 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c09e8b0 con 0x7fa20c132330 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c1ca7c0 con 0x7fa20c06cf50 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient(hunting): authenticate will time out at 1053.069185s 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa210917700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa20c06cf50 0x7fa20c1c98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58774/0 (socket says 172.21.15.76:58774) 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa210917700 1 -- 172.21.15.76:0/2631558434 learned_addr learned my addr 172.21.15.76:0/2631558434 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539309231 0 0) 0x7fa20c1ca7c0 con 0x7fa20c06cf50 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): my global_id is 124766 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f80035d0 con 0x7fa20c06cf50 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139442982 0 0) 0x7fa20c131140 con 0x7fa20c06d480 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): my global_id is 144141 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c1ca7c0 con 0x7fa20c06d480 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639135663 0 0) 0x7fa20c09e8b0 con 0x7fa20c132330 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): my global_id is 134479 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c131140 con 0x7fa20c132330 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300466312 0 0) 0x7fa20c1ca7c0 con 0x7fa20c06d480 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa20c09e8b0 con 0x7fa20c06d480 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa200001850 con 0x7fa20c06d480 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): dump: 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3720837939 0 0) 0x7fa1f80035d0 con 0x7fa20c06cf50 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa20c1ca7c0 con 0x7fa20c06cf50 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1fc0036a0 con 0x7fa20c06cf50 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient(hunting): dump: 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2899074207 0 0) 0x7fa20c09e8b0 con 0x7fa20c06d480 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 >> v1:172.21.15.76:6790/0 conn(0x7fa20c06cf50 legacy=0x7fa20c1c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 >> v1:172.21.15.189:6789/0 conn(0x7fa20c132330 legacy=0x7fa20c1d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 monclient: found mon.a 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c131670 con 0x7fa20c06d480 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 5 monclient: authenticate success, global_id 144141 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _renew_subs 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20c12edd0 con 0x7fa20c06d480 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _renew_subs 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20c12fb30 con 0x7fa20c06d480 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa200004890 con 0x7fa20c06d480 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa200004ba0 con 0x7fa20c06d480 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa20affd700 4 set_mon_vals no callback set 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 10 monclient: dump: 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:45.951 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.934+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa20000f610 con 0x7fa20c06d480 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa211919700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa211919700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa211919700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec005100 con 0x7fa20c06d480 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(90..90 src has 61..90) v4 ==== 4719+0+0 (unknown 4242505761 0 0) 0x7fa200043550 con 0x7fa20c06d480 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.938+0000 7fa210917700 10 monclient: get_auth_request con 0x7fa1f80363c0 auth_method 0 2024-09-16T06:41:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.946+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa200014450 con 0x7fa20c06d480 2024-09-16T06:41:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.946+0000 7fa2097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:45.946+0000 7fa2097fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:46.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.178+0000 7fa211919700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:46.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.178+0000 7fa211919700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:46.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.178+0000 7fa211919700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.178+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa1ec005930 con 0x7fa20c06d480 2024-09-16T06:41:46.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa2097fa700 1 -- 172.21.15.76:0/2631558434 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1770515891) 0x7fa200012af0 con 0x7fa20c06d480 2024-09-16T06:41:46.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa2097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:46.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa2097fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:46.183 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa1f80363c0 legacy=0x7fa1f8038880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 10 monclient: shutdown 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 >> v1:172.21.15.76:6789/0 conn(0x7fa20c06d480 legacy=0x7fa20c13e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 shutdown_connections 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 >> 172.21.15.76:0/2631558434 conn(0x7fa20c0a41c0 msgr2=0x7fa20c09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 shutdown_connections 2024-09-16T06:41:46.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.182+0000 7fa211919700 1 -- 172.21.15.76:0/2631558434 wait complete. 2024-09-16T06:41:46.197 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:41:46.197 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:41:46.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:41:46.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:41:46.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.248+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v91) v1 ==== 138+0+0 (unknown 1810020012 0 0) 0x7fc278013090 con 0x7fc28013bf70 2024-09-16T06:41:46.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.248+0000 7fc2767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:46.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.248+0000 7fc2767fc700 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: get_monmap_and_config 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: monmap: 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.315 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:46.316387+0000 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:46.316387+0000 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:46.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 1 Processor -- start 2024-09-16T06:41:46.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 1 -- start start 2024-09-16T06:41:46.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: init 2024-09-16T06:41:46.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2409b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-c con 0x7f3b240964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-b con 0x7f3b24138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24098560 con 0x7f3b2409b1c0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24099ec0 con 0x7f3b240964f0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24097030 con 0x7f3b24138870 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient(hunting): authenticate will time out at 1053.450129s 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b29ae3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3b24138870 0x7f3b2413ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58780/0 (socket says 172.21.15.76:58780) 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b29ae3700 1 -- 172.21.15.76:0/749363124 learned_addr learned my addr 172.21.15.76:0/749363124 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155321983 0 0) 0x7f3b24098560 con 0x7f3b2409b1c0 2024-09-16T06:41:46.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): my global_id is 144144 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b100035d0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293519099 0 0) 0x7f3b24097030 con 0x7f3b24138870 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): my global_id is 124769 2024-09-16T06:41:46.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24098560 con 0x7f3b24138870 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743617354 0 0) 0x7f3b24099ec0 con 0x7f3b240964f0 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient(hunting): my global_id is 134482 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24097030 con 0x7f3b240964f0 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2530328880 0 0) 0x7f3b100035d0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 >> v1:172.21.15.76:6790/0 conn(0x7f3b24138870 legacy=0x7f3b2413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 >> v1:172.21.15.189:6789/0 conn(0x7f3b240964f0 legacy=0x7f3b24096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 monclient: found mon.noname-a 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2409b5a0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 5 monclient: authenticate success, global_id 144144 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20001610 con 0x7f3b2409b1c0 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: dump: 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b200018f0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b2a2e4700 4 set_mon_vals no callback set 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 1 -- 172.21.15.76:0/749363124 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20001c80 con 0x7f3b2409b1c0 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.314+0000 7f3b1bfff700 10 monclient: dump: 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: shutdown 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/749363124 >> v1:172.21.15.76:6789/0 conn(0x7f3b2409b1c0 legacy=0x7f3b2409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/749363124 shutdown_connections 2024-09-16T06:41:46.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/749363124 >> 172.21.15.76:0/749363124 conn(0x7f3b240a0030 msgr2=0x7f3b240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/749363124 shutdown_connections 2024-09-16T06:41:46.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/749363124 wait complete. 2024-09-16T06:41:46.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: monmap: 2024-09-16T06:41:46.331 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 Processor -- start 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- start start 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: init 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-a con 0x7f3b240964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-c con 0x7f3b2409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient(hunting): picked mon.noname-b con 0x7f3b24138870 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24071b40 con 0x7f3b240964f0 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2409a540 con 0x7f3b24138870 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2409bd50 con 0x7f3b2409b1c0 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b2a2e4700 10 monclient(hunting): authenticate will time out at 1053.454871s 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b28ae1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3b2409b1c0 0x7f3b24095a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58794/0 (socket says 172.21.15.76:58794) 2024-09-16T06:41:46.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b28ae1700 1 -- 172.21.15.76:0/4140836928 learned_addr learned my addr 172.21.15.76:0/4140836928 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b292e2700 1 --1- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b240964f0 0x7f3b24095300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59032/0 (socket says 172.21.15.76:59032) 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088554925 0 0) 0x7f3b2409bd50 con 0x7f3b2409b1c0 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): my global_id is 124772 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b080035d0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350868566 0 0) 0x7f3b24071b40 con 0x7f3b240964f0 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): my global_id is 144147 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2409bd50 con 0x7f3b240964f0 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121564748 0 0) 0x7f3b2409a540 con 0x7f3b24138870 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): my global_id is 134485 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24071b40 con 0x7f3b24138870 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336046248 0 0) 0x7f3b2409bd50 con 0x7f3b240964f0 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b2409a540 con 0x7f3b240964f0 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20001c00 con 0x7f3b240964f0 2024-09-16T06:41:46.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:46.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2947881964 0 0) 0x7f3b080035d0 con 0x7f3b2409b1c0 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b2409bd50 con 0x7f3b2409b1c0 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b14001990 con 0x7f3b2409b1c0 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.318+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 751795614 0 0) 0x7f3b2409a540 con 0x7f3b240964f0 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6790/0 conn(0x7f3b2409b1c0 legacy=0x7f3b24095a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.189:6789/0 conn(0x7f3b24138870 legacy=0x7f3b241cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 monclient: found mon.a 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240a2d00 con 0x7f3b240964f0 2024-09-16T06:41:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 5 monclient: authenticate success, global_id 144147 2024-09-16T06:41:46.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 10 monclient: _renew_subs 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240712a0 con 0x7f3b240964f0 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 10 monclient: _renew_subs 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24132340 con 0x7f3b240964f0 2024-09-16T06:41:46.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b20001200 con 0x7f3b240964f0 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20003c40 con 0x7f3b240964f0 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b1b7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: dump: 2024-09-16T06:41:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3b04006590 con 0x7f3b04002830 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671579913 0 0) 0x7f3b04006590 con 0x7f3b04002830 2024-09-16T06:41:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient(hunting): my global_id is 144150 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2409a540 con 0x7f3b04002830 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3b20003da0 con 0x7f3b240964f0 2024-09-16T06:41:46.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(91..91 src has 61..91) v4 ==== 4746+0+0 (unknown 4055880498 0 0) 0x7f3b200436b0 con 0x7f3b240964f0 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3802238824 0 0) 0x7f3b2409a540 con 0x7f3b04002830 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b04006590 con 0x7f3b04002830 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b14003860 con 0x7f3b04002830 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: dump: 2024-09-16T06:41:46.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b29ae3700 10 monclient: get_auth_request con 0x7f3b08038410 auth_method 0 2024-09-16T06:41:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1141756243 0 0) 0x7f3b04006590 con 0x7f3b04002830 2024-09-16T06:41:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b0400a190 con 0x7f3b04002830 2024-09-16T06:41:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3b14005870 con 0x7f3b04002830 2024-09-16T06:41:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b04002830 legacy=0x7f3b04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.322+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b04002830 legacy=0x7f3b04004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3b04009e30 con 0x7f3b04009150 2024-09-16T06:41:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081636908 0 0) 0x7f3b04009e30 con 0x7f3b04009150 2024-09-16T06:41:46.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient(hunting): my global_id is 144153 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04006590 con 0x7f3b04009150 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 725174079 0 0) 0x7f3b04006590 con 0x7f3b04009150 2024-09-16T06:41:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b04009e30 con 0x7f3b04009150 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b14004140 con 0x7f3b04009150 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient: dump: 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.353 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.354 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 851031362 0 0) 0x7f3b04009e30 con 0x7f3b04009150 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b0400c340 con 0x7f3b04009150 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3588176397) 0x7f3b14004370 con 0x7f3b04009150 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:46.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b04009150 legacy=0x7f3b040095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b19ffb700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b04009150 legacy=0x7f3b040095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.356 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:41:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:46.379 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3b08038410 legacy=0x7f3b0803a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 10 monclient: shutdown 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 >> v1:172.21.15.76:6789/0 conn(0x7f3b240964f0 legacy=0x7f3b24095300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b29ae3700 1 -- 172.21.15.76:0/4140836928 reap_dead start 2024-09-16T06:41:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b29ae3700 1 -- 172.21.15.76:0/4140836928 reap_dead start 2024-09-16T06:41:46.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 shutdown_connections 2024-09-16T06:41:46.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 >> 172.21.15.76:0/4140836928 conn(0x7f3b240a0030 msgr2=0x7f3b2409ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 shutdown_connections 2024-09-16T06:41:46.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.338+0000 7f3b2a2e4700 1 -- 172.21.15.76:0/4140836928 wait complete. 2024-09-16T06:41:46.383 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:46.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:41:46.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:41:46.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc286faa700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:46.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc286faa700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:46.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc286faa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fc2680053f0 con 0x7fc28013bf70 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc2767fc700 1 -- 172.21.15.189:0/4270578750 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v91) v1 ==== 126+0+0 (unknown 2753640235 0 0) 0x7fc278014810 con 0x7fc28013bf70 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc2767fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.364+0000 7fc2767fc700 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc264036390 legacy=0x7fc264038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 10 monclient: shutdown 2024-09-16T06:41:46.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 >> v1:172.21.15.189:6789/0 conn(0x7fc28013bf70 legacy=0x7fc2801d6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 shutdown_connections 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 >> 172.21.15.189:0/4270578750 conn(0x7fc2800a4660 msgr2=0x7fc280137bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 shutdown_connections 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.368+0000 7fc286faa700 1 -- 172.21.15.189:0/4270578750 wait complete. 2024-09-16T06:41:46.388 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'foo' already exists 2024-09-16T06:41:46.389 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-16T06:41:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 10 monclient: get_monmap_and_config 2024-09-16T06:41:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 10 monclient: monmap: 2024-09-16T06:41:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:41:46.495212+0000 2024-09-16T06:41:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:41:46.495212+0000 2024-09-16T06:41:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:46.497 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 1 Processor -- start 2024-09-16T06:41:46.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 1 -- start start 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.492+0000 7fb8c76bb700 10 monclient: init 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-c con 0x7fb8c0132230 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-a con 0x7fb8c00a1340 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-b con 0x7fb8c0132610 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c0132a80 con 0x7fb8c00a1340 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c0071550 con 0x7fb8c0132230 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c009fa80 con 0x7fb8c0132610 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): authenticate will time out at 1059.008217s 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c5457700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8c0132230 0x7fb8c0070730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52714/0 (socket says 172.21.15.189:52714) 2024-09-16T06:41:46.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c5457700 1 -- 172.21.15.189:0/1355681081 learned_addr learned my addr 172.21.15.189:0/1355681081 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402390400 0 0) 0x7fb8c0071550 con 0x7fb8c0132230 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): my global_id is 134488 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a80035d0 con 0x7fb8c0132230 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599962653 0 0) 0x7fb8c009fa80 con 0x7fb8c0132610 2024-09-16T06:41:46.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): my global_id is 124775 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c0071550 con 0x7fb8c0132610 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869805161 0 0) 0x7fb8c0132a80 con 0x7fb8c00a1340 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient(hunting): my global_id is 144156 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c009fa80 con 0x7fb8c00a1340 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099240708 0 0) 0x7fb8a80035d0 con 0x7fb8c0132230 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 >> v1:172.21.15.76:6790/0 conn(0x7fb8c0132610 legacy=0x7fb8c013e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 >> v1:172.21.15.76:6789/0 conn(0x7fb8c00a1340 legacy=0x7fb8c0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 monclient: found mon.noname-c 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c009e5d0 con 0x7fb8c0132230 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 5 monclient: authenticate success, global_id 134488 2024-09-16T06:41:46.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8b0001610 con 0x7fb8c0132230 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: dump: 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.520 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.521 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b00018f0 con 0x7fb8c0132230 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 4 set_mon_vals no callback set 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 1 -- 172.21.15.189:0/1355681081 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8b0001c80 con 0x7fb8c0132230 2024-09-16T06:41:46.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8bffff700 10 monclient: dump: 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.523 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: shutdown 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- 172.21.15.189:0/1355681081 >> v1:172.21.15.189:6789/0 conn(0x7fb8c0132230 legacy=0x7fb8c0070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- 172.21.15.189:0/1355681081 shutdown_connections 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- 172.21.15.189:0/1355681081 >> 172.21.15.189:0/1355681081 conn(0x7fb8c00a44f0 msgr2=0x7fb8c00a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- 172.21.15.189:0/1355681081 shutdown_connections 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- 172.21.15.189:0/1355681081 wait complete. 2024-09-16T06:41:46.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: monmap: 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:41:46.526 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 Processor -- start 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- start start 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: init 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:46.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-c con 0x7fb8c00a1340 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-a con 0x7fb8c0132230 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): picked mon.noname-b con 0x7fb8c0132610 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c012f940 con 0x7fb8c0132230 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c0123020 con 0x7fb8c0132610 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c012fe90 con 0x7fb8c00a1340 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c76bb700 10 monclient(hunting): authenticate will time out at 1059.011817s 2024-09-16T06:41:46.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c5c58700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8c0132610 0x7fb8c01d20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52720/0 (socket says 172.21.15.189:52720) 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.496+0000 7fb8c5c58700 1 -- 172.21.15.189:0/986157653 learned_addr learned my addr 172.21.15.189:0/986157653 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960523794 0 0) 0x7fb8c0123020 con 0x7fb8c0132610 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): my global_id is 134491 2024-09-16T06:41:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac0035d0 con 0x7fb8c0132610 2024-09-16T06:41:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905014901 0 0) 0x7fb8c012fe90 con 0x7fb8c00a1340 2024-09-16T06:41:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): my global_id is 124778 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c0123020 con 0x7fb8c00a1340 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282349259 0 0) 0x7fb8c012f940 con 0x7fb8c0132230 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): my global_id is 144159 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c012fe90 con 0x7fb8c0132230 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3544410603 0 0) 0x7fb8ac0035d0 con 0x7fb8c0132610 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c012f940 con 0x7fb8c0132610 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8b8001c30 con 0x7fb8c0132610 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): dump: 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035684487 0 0) 0x7fb8c0123020 con 0x7fb8c00a1340 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8ac0035d0 con 0x7fb8c00a1340 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8b00015d0 con 0x7fb8c00a1340 2024-09-16T06:41:46.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient(hunting): dump: 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45855262 0 0) 0x7fb8c012f940 con 0x7fb8c0132610 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 >> v1:172.21.15.76:6790/0 conn(0x7fb8c00a1340 legacy=0x7fb8c01c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 >> v1:172.21.15.76:6789/0 conn(0x7fb8c0132230 legacy=0x7fb8c012e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 monclient: found mon.b 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c006b260 con 0x7fb8c0132610 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 5 monclient: authenticate success, global_id 134491 2024-09-16T06:41:46.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _renew_subs 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c0122780 con 0x7fb8c0132610 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _renew_subs 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c0120cc0 con 0x7fb8c0132610 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8003a40 con 0x7fb8c0132610 2024-09-16T06:41:46.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8b8003d50 con 0x7fb8c0132610 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 10 monclient: dump: 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.539 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bf7fe700 4 set_mon_vals no callback set 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0005100 con 0x7fb8c0132610 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb8b8003fb0 con 0x7fb8c0132610 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(91..91 src has 61..91) v4 ==== 4746+0+0 (unknown 4055880498 0 0) 0x7fb8b80438c0 con 0x7fb8c0132610 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.500+0000 7fb8c5457700 10 monclient: get_auth_request con 0x7fb8ac036390 auth_method 0 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.508+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb8b8014ab0 con 0x7fb8c0132610 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.508+0000 7fb8bdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.508+0000 7fb8bdffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:46.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: get_monmap_and_config 2024-09-16T06:41:46.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: monmap: 2024-09-16T06:41:46.555 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:46.512987+0000 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:46.512987+0000 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:46.556 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 Processor -- start 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 -- start start 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: init 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-c con 0x7fe66009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-a con 0x7fe660098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-b con 0x7fe660099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66009d560 con 0x7fe660098be0 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66009f140 con 0x7fe66009ed60 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66009d790 con 0x7fe660099030 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient(hunting): authenticate will time out at 1053.646452s 2024-09-16T06:41:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe665118700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe660098be0 0x7fe6601374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59060/0 (socket says 172.21.15.76:59060) 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe665118700 1 -- 172.21.15.76:0/4195318317 learned_addr learned my addr 172.21.15.76:0/4195318317 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081689811 0 0) 0x7fe66009d560 con 0x7fe660098be0 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): my global_id is 144162 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64c0035d0 con 0x7fe660098be0 2024-09-16T06:41:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024809025 0 0) 0x7fe66009d790 con 0x7fe660099030 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): my global_id is 124781 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66009d560 con 0x7fe660099030 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153305577 0 0) 0x7fe66009f140 con 0x7fe66009ed60 2024-09-16T06:41:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient(hunting): my global_id is 134494 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66009d790 con 0x7fe66009ed60 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447722574 0 0) 0x7fe64c0035d0 con 0x7fe660098be0 2024-09-16T06:41:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 >> v1:172.21.15.76:6790/0 conn(0x7fe660099030 legacy=0x7fe66013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 >> v1:172.21.15.189:6789/0 conn(0x7fe66009ed60 legacy=0x7fe6600984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 monclient: found mon.noname-a 2024-09-16T06:41:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66009f620 con 0x7fe660098be0 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe650001610 con 0x7fe660098be0 2024-09-16T06:41:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: dump: 2024-09-16T06:41:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.566 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 5 monclient: authenticate success, global_id 144162 2024-09-16T06:41:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6500018f0 con 0x7fe660098be0 2024-09-16T06:41:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 1 -- 172.21.15.76:0/4195318317 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe650001c80 con 0x7fe660098be0 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 4 set_mon_vals no callback set 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe664917700 10 monclient: dump: 2024-09-16T06:41:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 10 monclient: shutdown 2024-09-16T06:41:46.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.510+0000 7fe66691b700 1 -- 172.21.15.76:0/4195318317 >> v1:172.21.15.76:6789/0 conn(0x7fe660098be0 legacy=0x7fe6601374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/4195318317 shutdown_connections 2024-09-16T06:41:46.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/4195318317 >> 172.21.15.76:0/4195318317 conn(0x7fe6600a0030 msgr2=0x7fe6600a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/4195318317 shutdown_connections 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/4195318317 wait complete. 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: monmap: 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 Processor -- start 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- start start 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: init 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-a con 0x7fe660098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-c con 0x7fe660099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient(hunting): picked mon.noname-b con 0x7fe66009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6600a1470 con 0x7fe660098be0 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6601c85d0 con 0x7fe66009ed60 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe660092c90 con 0x7fe660099030 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient(hunting): authenticate will time out at 1053.649745s 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe665118700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe660099030 0x7fe6601cc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58826/0 (socket says 172.21.15.76:58826) 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe665118700 1 -- 172.21.15.76:0/3443228189 learned_addr learned my addr 172.21.15.76:0/3443228189 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe665919700 1 --1- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6789/0 conn(0x7fe660098be0 0x7fe6600964c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59066/0 (socket says 172.21.15.76:59066) 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161662214 0 0) 0x7fe6600a1470 con 0x7fe660098be0 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): my global_id is 144165 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6440035d0 con 0x7fe660098be0 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489493223 0 0) 0x7fe6601c85d0 con 0x7fe66009ed60 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): my global_id is 134497 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6600a1470 con 0x7fe66009ed60 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095947355 0 0) 0x7fe660092c90 con 0x7fe660099030 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): my global_id is 124784 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6601c85d0 con 0x7fe660099030 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584150418 0 0) 0x7fe6440035d0 con 0x7fe660098be0 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe660092c90 con 0x7fe660098be0 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe65c0018d0 con 0x7fe660098be0 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): dump: 2024-09-16T06:41:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1612560172 0 0) 0x7fe6601c85d0 con 0x7fe660099030 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6440035d0 con 0x7fe660099030 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe650001610 con 0x7fe660099030 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): dump: 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051036932 0 0) 0x7fe6600a1470 con 0x7fe66009ed60 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6601c85d0 con 0x7fe66009ed60 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe658001c30 con 0x7fe66009ed60 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient(hunting): dump: 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1293711734 0 0) 0x7fe660092c90 con 0x7fe660098be0 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6790/0 conn(0x7fe660099030 legacy=0x7fe6601cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.189:6789/0 conn(0x7fe66009ed60 legacy=0x7fe6601d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 monclient: found mon.a 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600910b0 con 0x7fe660098be0 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 5 monclient: authenticate success, global_id 144165 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _renew_subs 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66009ea50 con 0x7fe660098be0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _renew_subs 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6600929b0 con 0x7fe660098be0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe65c001c60 con 0x7fe660098be0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe65c003b10 con 0x7fe660098be0 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe6567fc700 10 monclient: dump: 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.514+0000 7fe657fff700 4 set_mon_vals no callback set 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe65c003d70 con 0x7fe660098be0 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe66691b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(91..91 src has 61..91) v4 ==== 4746+0+0 (unknown 4055880498 0 0) 0x7fe65c0435a0 con 0x7fe660098be0 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe66006dea0 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe665118700 10 monclient: get_auth_request con 0x7fe64403a9c0 auth_method 0 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528439302 0 0) 0x7fe66006dea0 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient(hunting): my global_id is 124787 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660092c90 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3724074733 0 0) 0x7fe660092c90 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe66006dea0 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe658003a20 con 0x7fe6601c7920 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient: dump: 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3952589708 0 0) 0x7fe66006dea0 con 0x7fe6601c7920 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe66009aaa0 con 0x7fe6601c7920 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe658003f80 con 0x7fe6601c7920 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6790/0 conn(0x7fe6601c7920 legacy=0x7fe66008aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.518+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6790/0 conn(0x7fe6601c7920 legacy=0x7fe66008aff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe660090ba0 con 0x7fe66002c6b0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689259043 0 0) 0x7fe660090ba0 con 0x7fe66002c6b0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient(hunting): my global_id is 124790 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66006dea0 con 0x7fe66002c6b0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3567268890 0 0) 0x7fe66006dea0 con 0x7fe66002c6b0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe660090ba0 con 0x7fe66002c6b0 2024-09-16T06:41:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe658004510 con 0x7fe66002c6b0 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient: dump: 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1857918380 0 0) 0x7fe660090ba0 con 0x7fe66002c6b0 2024-09-16T06:41:46.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe66009b0f0 con 0x7fe66002c6b0 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2127100610) 0x7fe658004920 con 0x7fe66002c6b0 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6790/0 conn(0x7fe66002c6b0 legacy=0x7fe66008b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe6567fc700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6790/0 conn(0x7fe66002c6b0 legacy=0x7fe66008b700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:46.601 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:46.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe64403a9c0 legacy=0x7fe64403ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 10 monclient: shutdown 2024-09-16T06:41:46.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66611a700 1 -- 172.21.15.76:0/3443228189 reap_dead start 2024-09-16T06:41:46.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 >> v1:172.21.15.76:6789/0 conn(0x7fe660098be0 legacy=0x7fe6600964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 shutdown_connections 2024-09-16T06:41:46.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 >> 172.21.15.76:0/3443228189 conn(0x7fe6600a0030 msgr2=0x7fe6600973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 shutdown_connections 2024-09-16T06:41:46.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.530+0000 7fe66691b700 1 -- 172.21.15.76:0/3443228189 wait complete. 2024-09-16T06:41:46.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:46.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:41:46.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:41:46.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.684+0000 7fb8c76bb700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:41:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.684+0000 7fb8c76bb700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:41:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.684+0000 7fb8c76bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:41:46.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:46.684+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7fb8a0005930 con 0x7fb8c0132610 2024-09-16T06:41:46.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 10 monclient: get_monmap_and_config 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 10 monclient: monmap: 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:46.727017+0000 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:46.727017+0000 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:46.726 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 1 Processor -- start 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 1 -- start start 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.722+0000 7fa80169b700 10 monclient: init 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-a con 0x7fa7fc09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-c con 0x7fa7fc098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-b con 0x7fa7fc099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc09d560 con 0x7fa7fc09ed60 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc09f140 con 0x7fa7fc098be0 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc09d790 con 0x7fa7fc099030 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient(hunting): authenticate will time out at 1053.860685s 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fbfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa7fc09ed60 0x7fa7fc0984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59082/0 (socket says 172.21.15.76:59082) 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fbfff700 1 -- 172.21.15.76:0/478305597 learned_addr learned my addr 172.21.15.76:0/478305597 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541105066 0 0) 0x7fa7fc09d560 con 0x7fa7fc09ed60 2024-09-16T06:41:46.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): my global_id is 144168 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e80035d0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369524860 0 0) 0x7fa7fc09d790 con 0x7fa7fc099030 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): my global_id is 124793 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc09d560 con 0x7fa7fc099030 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057151655 0 0) 0x7fa7fc09f140 con 0x7fa7fc098be0 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient(hunting): my global_id is 134500 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc09d790 con 0x7fa7fc098be0 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871405941 0 0) 0x7fa7e80035d0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 >> v1:172.21.15.76:6790/0 conn(0x7fa7fc099030 legacy=0x7fa7fc13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 >> v1:172.21.15.189:6789/0 conn(0x7fa7fc098be0 legacy=0x7fa7fc1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 monclient: found mon.noname-a 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc09f620 con 0x7fa7fc09ed60 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 5 monclient: authenticate success, global_id 144168 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7ec001610 con 0x7fa7fc09ed60 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: dump: 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.734 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7ec001830 con 0x7fa7fc09ed60 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 1 -- 172.21.15.76:0/478305597 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7ec001c00 con 0x7fa7fc09ed60 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa7fb7fe700 10 monclient: dump: 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.735 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 4 set_mon_vals no callback set 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 10 monclient: shutdown 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- 172.21.15.76:0/478305597 >> v1:172.21.15.76:6789/0 conn(0x7fa7fc09ed60 legacy=0x7fa7fc0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- 172.21.15.76:0/478305597 shutdown_connections 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- 172.21.15.76:0/478305597 >> 172.21.15.76:0/478305597 conn(0x7fa7fc0a0030 msgr2=0x7fa7fc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- 172.21.15.76:0/478305597 shutdown_connections 2024-09-16T06:41:46.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.726+0000 7fa80169b700 1 -- 172.21.15.76:0/478305597 wait complete. 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: monmap: 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:46.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 Processor -- start 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- start start 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: init 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-a con 0x7fa7fc098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-b con 0x7fa7fc099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient(hunting): picked mon.noname-c con 0x7fa7fc09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc090cc0 con 0x7fa7fc098be0 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc090a50 con 0x7fa7fc099030 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc0a2cc0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient(hunting): authenticate will time out at 1053.866241s 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa800e9a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa7fc09ed60 0x7fa7fc1cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58862/0 (socket says 172.21.15.76:58862) 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa800e9a700 1 -- 172.21.15.76:0/201277227 learned_addr learned my addr 172.21.15.76:0/201277227 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653179378 0 0) 0x7fa7fc0a2cc0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): my global_id is 124796 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7dc0035d0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432749843 0 0) 0x7fa7fc090cc0 con 0x7fa7fc098be0 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): my global_id is 144171 2024-09-16T06:41:46.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc0a2cc0 con 0x7fa7fc098be0 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049366107 0 0) 0x7fa7fc090a50 con 0x7fa7fc099030 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): my global_id is 134503 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc090cc0 con 0x7fa7fc099030 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1844603028 0 0) 0x7fa7dc0035d0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7fc090a50 con 0x7fa7fc09ed60 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7f4001d50 con 0x7fa7fc09ed60 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): dump: 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403217977 0 0) 0x7fa7fc0a2cc0 con 0x7fa7fc098be0 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7dc0035d0 con 0x7fa7fc098be0 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7ec001b40 con 0x7fa7fc098be0 2024-09-16T06:41:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient(hunting): dump: 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 214682037 0 0) 0x7fa7fc090a50 con 0x7fa7fc09ed60 2024-09-16T06:41:46.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6789/0 conn(0x7fa7fc099030 legacy=0x7fa7fc1ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.76:6789/0 conn(0x7fa7fc098be0 legacy=0x7fa7fc096230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 monclient: found mon.c 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0984d0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 5 monclient: authenticate success, global_id 124796 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _renew_subs 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc09a540 con 0x7fa7fc09ed60 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _renew_subs 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc1c55f0 con 0x7fa7fc09ed60 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7f4003a20 con 0x7fa7fc09ed60 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7f4003d30 con 0x7fa7fc09ed60 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7fa7fc700 4 set_mon_vals no callback set 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.730+0000 7fa7f97fa700 10 monclient: dump: 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.749 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa7f4003e90 con 0x7fa7fc09ed60 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa80169b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa7e0006590 con 0x7fa7e0002830 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 61..91) v4 ==== 4746+0+0 (unknown 4055880498 0 0) 0x7fa7f4043880 con 0x7fa7fc09ed60 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f37fe700 10 monclient: get_auth_request con 0x7fa7dc036230 auth_method 0 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230353574 0 0) 0x7fa7e0006590 con 0x7fa7e0002830 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient(hunting): my global_id is 134506 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc090a50 con 0x7fa7e0002830 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3132519176 0 0) 0x7fa7fc090a50 con 0x7fa7e0002830 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7e0006590 con 0x7fa7e0002830 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7ec004ca0 con 0x7fa7e0002830 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient: dump: 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4113894141 0 0) 0x7fa7e0006590 con 0x7fa7e0002830 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7e000a190 con 0x7fa7e0002830 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa7ec006ff0 con 0x7fa7e0002830 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6789/0 conn(0x7fa7e0002830 legacy=0x7fa7e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.734+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6789/0 conn(0x7fa7e0002830 legacy=0x7fa7e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa7e0009e50 con 0x7fa7e0009170 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717418268 0 0) 0x7fa7e0009e50 con 0x7fa7e0009170 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient(hunting): my global_id is 134509 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e0006590 con 0x7fa7e0009170 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 862004482 0 0) 0x7fa7e0006590 con 0x7fa7e0009170 2024-09-16T06:41:46.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7e0009e50 con 0x7fa7e0009170 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7ec005580 con 0x7fa7e0009170 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient: dump: 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.755 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2199705184 0 0) 0x7fa7e0009e50 con 0x7fa7e0009170 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7e000c340 con 0x7fa7e0009170 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2425927789) 0x7fa7ec006920 con 0x7fa7e0009170 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6789/0 conn(0x7fa7e0009170 legacy=0x7fa7e0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 38, 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:41:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:41:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:41:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:41:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:41:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:41:46.767 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa7f97fa700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6789/0 conn(0x7fa7e0009170 legacy=0x7fa7e0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa7dc036230 legacy=0x7fa7dc0386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 10 monclient: shutdown 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa800e9a700 1 -- 172.21.15.76:0/201277227 reap_dead start 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 >> v1:172.21.15.76:6790/0 conn(0x7fa7fc09ed60 legacy=0x7fa7fc1cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 shutdown_connections 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 >> 172.21.15.76:0/201277227 conn(0x7fa7fc0a0030 msgr2=0x7fa7fc06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 shutdown_connections 2024-09-16T06:41:46.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.746+0000 7fa80169b700 1 -- 172.21.15.76:0/201277227 wait complete. 2024-09-16T06:41:46.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:41:46.770 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:41:46.770 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:41:46.770 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:41:46.770 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:41:46.770 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:41:46.770 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:41:46.823 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:41:46.823 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:41:46.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:46.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: get_monmap_and_config 2024-09-16T06:41:46.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: monmap: 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:46.940665+0000 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:46.940665+0000 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:46.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 1 Processor -- start 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 1 -- start start 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: init 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient(hunting): picked mon.noname-b con 0x7efe6c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient(hunting): picked mon.noname-c con 0x7efe6c06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient(hunting): picked mon.noname-a con 0x7efe6c06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c06c280 con 0x7efe6c06cf50 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c06bfa0 con 0x7efe6c06d480 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c09ace0 con 0x7efe6c132330 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 10 monclient(hunting): authenticate will time out at 1054.074761s 2024-09-16T06:41:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7efe6c06cf50 legacy=0x7efe6c13da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7efe6c132330 0x7efe6c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58864/0 (socket says 172.21.15.76:58864) 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6b7fe700 1 -- 172.21.15.76:0/3747965399 learned_addr learned my addr 172.21.15.76:0/3747965399 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054839132 0 0) 0x7efe6c09ace0 con 0x7efe6c132330 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): my global_id is 124799 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe540035d0 con 0x7efe6c132330 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076976407 0 0) 0x7efe6c06bfa0 con 0x7efe6c06d480 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient(hunting): my global_id is 134512 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c09ace0 con 0x7efe6c06d480 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4016591926 0 0) 0x7efe540035d0 con 0x7efe6c132330 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 >> v1:172.21.15.189:6789/0 conn(0x7efe6c06d480 legacy=0x7efe6c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 >> v1:172.21.15.76:6789/0 conn(0x7efe6c06cf50 legacy=0x7efe6c13da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 monclient: found mon.noname-b 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c09ae70 con 0x7efe6c132330 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe70982700 5 monclient: authenticate success, global_id 124799 2024-09-16T06:41:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe60001610 con 0x7efe6c132330 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.938+0000 7efe6a7fc700 10 monclient: dump: 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe600018f0 con 0x7efe6c132330 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 1 -- 172.21.15.76:0/3747965399 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe60001ae0 con 0x7efe6c132330 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 4 set_mon_vals no callback set 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe6a7fc700 10 monclient: dump: 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 10 monclient: shutdown 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- 172.21.15.76:0/3747965399 >> v1:172.21.15.76:6790/0 conn(0x7efe6c132330 legacy=0x7efe6c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- 172.21.15.76:0/3747965399 shutdown_connections 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- 172.21.15.76:0/3747965399 >> 172.21.15.76:0/3747965399 conn(0x7efe6c0a41c0 msgr2=0x7efe6c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- 172.21.15.76:0/3747965399 shutdown_connections 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- 172.21.15.76:0/3747965399 wait complete. 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 10 monclient: build_initial_monmap 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 10 monclient: monmap: 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:46.953 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 Processor -- start 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 1 -- start start 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.942+0000 7efe70982700 10 monclient: init 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient(hunting): picked mon.noname-c con 0x7efe6c06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient(hunting): picked mon.noname-a con 0x7efe6c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient(hunting): picked mon.noname-b con 0x7efe6c132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c12e180 con 0x7efe6c06d480 2024-09-16T06:41:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe6affd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7efe6c06d480 legacy=0x7efe6c1cebe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c12dbc0 con 0x7efe6c132330 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c12dea0 con 0x7efe6c06cf50 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient(hunting): authenticate will time out at 1054.080731s 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe6b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7efe6c06cf50 0x7efe6c1306c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58880/0 (socket says 172.21.15.76:58880) 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe6b7fe700 1 -- 172.21.15.76:0/3383516660 learned_addr learned my addr 172.21.15.76:0/3383516660 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044628821 0 0) 0x7efe6c12dea0 con 0x7efe6c06cf50 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): my global_id is 124802 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe580035d0 con 0x7efe6c06cf50 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353520110 0 0) 0x7efe6c12dbc0 con 0x7efe6c132330 2024-09-16T06:41:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): my global_id is 134515 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c12dea0 con 0x7efe6c132330 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1881916732 0 0) 0x7efe580035d0 con 0x7efe6c06cf50 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe6c12dbc0 con 0x7efe6c06cf50 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe60004890 con 0x7efe6c06cf50 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): dump: 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310762707 0 0) 0x7efe6c12dea0 con 0x7efe6c132330 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe580035d0 con 0x7efe6c132330 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe64004390 con 0x7efe6c132330 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient(hunting): dump: 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3683119614 0 0) 0x7efe6c12dbc0 con 0x7efe6c06cf50 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 >> v1:172.21.15.189:6789/0 conn(0x7efe6c132330 legacy=0x7efe6c1d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 >> v1:172.21.15.76:6789/0 conn(0x7efe6c06d480 legacy=0x7efe6c1cebe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 monclient: found mon.c 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c12d8e0 con 0x7efe6c06cf50 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 5 monclient: authenticate success, global_id 124802 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _renew_subs 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c0a1520 con 0x7efe6c06cf50 2024-09-16T06:41:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _renew_subs 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c12c9f0 con 0x7efe6c06cf50 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe60004ca0 con 0x7efe6c06cf50 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efe60005030 con 0x7efe6c06cf50 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe69ffb700 4 set_mon_vals no callback set 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe53fff700 10 monclient: dump: 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.946+0000 7efe70982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.950+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c005100 con 0x7efe6c06cf50 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.950+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efe60005210 con 0x7efe6c06cf50 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.950+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 61..91) v4 ==== 4746+0+0 (unknown 4055880498 0 0) 0x7efe600439d0 con 0x7efe6c06cf50 2024-09-16T06:41:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.950+0000 7efe6affd700 10 monclient: get_auth_request con 0x7efe580362a0 auth_method 0 2024-09-16T06:41:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.962+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efe60014910 con 0x7efe6c06cf50 2024-09-16T06:41:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.962+0000 7efe53fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:46.962+0000 7efe53fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe4c005930 con 0x7efe6c06cf50 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe53fff700 1 -- 172.21.15.76:0/3383516660 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3514217100) 0x7efe60012fb0 con 0x7efe6c06cf50 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe53fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe53fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:47.201 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 >> v1:172.21.15.189:6812/3098810674 conn(0x7efe580362a0 legacy=0x7efe58038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 10 monclient: shutdown 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.198+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 >> v1:172.21.15.76:6790/0 conn(0x7efe6c06cf50 legacy=0x7efe6c1306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.202+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 shutdown_connections 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.202+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 >> 172.21.15.76:0/3383516660 conn(0x7efe6c0a41c0 msgr2=0x7efe6c12a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.202+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 shutdown_connections 2024-09-16T06:41:47.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:47.202+0000 7efe70982700 1 -- 172.21.15.76:0/3383516660 wait complete. 2024-09-16T06:41:49.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:49.496+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:49.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:49.496+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:49.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:49.496+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:49.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:49.496+0000 7fb8bcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:50.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: get_monmap_and_config 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: build_initial_monmap 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: monmap: 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:50.338064+0000 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:50.338064+0000 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:50.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 1 Processor -- start 2024-09-16T06:41:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 1 -- start start 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: init 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient(hunting): picked mon.noname-c con 0x7febb0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient(hunting): picked mon.noname-b con 0x7febb0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient(hunting): picked mon.noname-a con 0x7febb0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb0098990 con 0x7febb0139500 2024-09-16T06:41:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb0097f40 con 0x7febb0097b60 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb009d610 con 0x7febb0099090 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb6532700 10 monclient(hunting): authenticate will time out at 1057.471322s 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb5d31700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7febb0139500 legacy=0x7febb013b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb4d2f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7febb0099090 0x7febb009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58882/0 (socket says 172.21.15.76:58882) 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7febb4d2f700 1 -- 172.21.15.76:0/1776776848 learned_addr learned my addr 172.21.15.76:0/1776776848 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835062693 0 0) 0x7febb009d610 con 0x7febb0099090 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:41:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 10 monclient(hunting): my global_id is 124805 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.334+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c0035d0 con 0x7febb0099090 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953299593 0 0) 0x7feb9c0035d0 con 0x7febb0099090 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 >> v1:172.21.15.189:6789/0 conn(0x7febb0097b60 legacy=0x7febb009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 >> v1:172.21.15.76:6789/0 conn(0x7febb0139500 legacy=0x7febb013b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 monclient: found mon.noname-b 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00986b0 con 0x7febb0099090 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: _finish_auth 0 2024-09-16T06:41:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7febac001610 con 0x7febb0099090 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 5 monclient: authenticate success, global_id 124805 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: dump: 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac001930 con 0x7febb0099090 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 1 -- 172.21.15.76:0/1776776848 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7febac001cc0 con 0x7febb0099090 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7feba7fff700 10 monclient: dump: 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: get_monmap_and_config success 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 4 set_mon_vals no callback set 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: shutdown 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- 172.21.15.76:0/1776776848 >> v1:172.21.15.76:6790/0 conn(0x7febb0099090 legacy=0x7febb009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- 172.21.15.76:0/1776776848 shutdown_connections 2024-09-16T06:41:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- 172.21.15.76:0/1776776848 >> 172.21.15.76:0/1776776848 conn(0x7febb00a0030 msgr2=0x7febb00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- 172.21.15.76:0/1776776848 shutdown_connections 2024-09-16T06:41:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- 172.21.15.76:0/1776776848 wait complete. 2024-09-16T06:41:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: build_initial_monmap 2024-09-16T06:41:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: monmap: 2024-09-16T06:41:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:41:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 Processor -- start 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- start start 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: init 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient(hunting): picked mon.noname-c con 0x7febb0097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient(hunting): picked mon.noname-b con 0x7febb0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient(hunting): picked mon.noname-a con 0x7febb0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb008f4d0 con 0x7febb0139500 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb013c270 con 0x7febb0099090 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00a2b80 con 0x7febb0097b60 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb6532700 10 monclient(hunting): authenticate will time out at 1057.475427s 2024-09-16T06:41:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb5d31700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7febb0139500 legacy=0x7febb01d0600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb5530700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7febb0097b60 0x7febb0138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58890/0 (socket says 172.21.15.76:58890) 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.338+0000 7febb5530700 1 -- 172.21.15.76:0/4179239967 learned_addr learned my addr 172.21.15.76:0/4179239967 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983477549 0 0) 0x7febb00a2b80 con 0x7febb0097b60 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): my global_id is 124808 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb940035d0 con 0x7febb0097b60 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718982271 0 0) 0x7feb940035d0 con 0x7febb0097b60 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb00a2b80 con 0x7febb0097b60 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba0001930 con 0x7febb0097b60 2024-09-16T06:41:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): dump: 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 774675144 0 0) 0x7febb00a2b80 con 0x7febb0097b60 2024-09-16T06:41:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 >> v1:172.21.15.189:6789/0 conn(0x7febb0099090 legacy=0x7febb01cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 >> v1:172.21.15.76:6789/0 conn(0x7febb0139500 legacy=0x7febb01d0600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 monclient: found mon.c 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb009a540 con 0x7febb0097b60 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 5 monclient: authenticate success, global_id 124808 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _renew_subs 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb008f1f0 con 0x7febb0097b60 2024-09-16T06:41:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _renew_subs 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febb008d170 con 0x7febb0097b60 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba0001d00 con 0x7febb0097b60 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba0003b10 con 0x7febb0097b60 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba77fe700 4 set_mon_vals no callback set 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 10 monclient: dump: 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:41:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90005100 con 0x7febb0097b60 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7feba0003c70 con 0x7febb0097b60 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 66..91) v4 ==== 4746+0+0 (unknown 2790596778 0 0) 0x7feba0043790 con 0x7febb0097b60 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.342+0000 7febb4d2f700 10 monclient: get_auth_request con 0x7feb94036120 auth_method 0 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.350+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feba00148a0 con 0x7febb0097b60 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.350+0000 7feba5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:41:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.350+0000 7feba5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:41:50.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:50.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:50.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:50.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:50.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:50.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7febb6532700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:41:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7febb6532700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7febb6532700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:41:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feb90005930 con 0x7febb0097b60 2024-09-16T06:41:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7feba5ffb700 1 -- 172.21.15.76:0/4179239967 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2174385325) 0x7feba0012d10 con 0x7febb0097b60 2024-09-16T06:41:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7feba5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:41:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.582+0000 7feba5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:41:50.586 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:41:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 >> v1:172.21.15.189:6812/3098810674 conn(0x7feb94036120 legacy=0x7feb940385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 10 monclient: shutdown 2024-09-16T06:41:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:41:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 >> v1:172.21.15.76:6790/0 conn(0x7febb0097b60 legacy=0x7febb0138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 shutdown_connections 2024-09-16T06:41:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 >> 172.21.15.76:0/4179239967 conn(0x7febb00a0030 msgr2=0x7febb00a1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:41:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 shutdown_connections 2024-09-16T06:41:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:50.586+0000 7febb6532700 1 -- 172.21.15.76:0/4179239967 wait complete. 2024-09-16T06:41:51.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:51.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:51.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:51.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:51.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:51.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:52.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:52.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:52.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:52.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:52.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:52.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:53.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:53.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:53.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:53.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:53.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:41:53.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: get_monmap_and_config 2024-09-16T06:41:53.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: build_initial_monmap 2024-09-16T06:41:53.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: monmap: 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:41:53.724796+0000 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:41:53.724796+0000 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:41:53.724 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:41:53.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 1 Processor -- start 2024-09-16T06:41:53.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 1 -- start start 2024-09-16T06:41:53.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: init 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: _reopen_session rank -1 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-b con 0x7f844c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-c con 0x7f844c09c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-a con 0x7f844c09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09a040 con 0x7f844c09fd20 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09ac80 con 0x7f844c09c460 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09def0 con 0x7f844c09f230 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f84525ba700 10 monclient(hunting): authenticate will time out at 1060.858127s 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f8450b57700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f844c09fd20 legacy=0x7f844c142160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f844bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f844c09f230 0x7f844c09f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58898/0 (socket says 172.21.15.76:58898) 2024-09-16T06:41:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.722+0000 7f844bfff700 1 -- 172.21.15.76:0/1193996822 learned_addr learned my addr 172.21.15.76:0/1193996822 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:41:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:53.926+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f844c09fd20 legacy=0x7f844c142160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:54.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:54.330+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f844c09fd20 legacy=0x7f844c142160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:54.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:54.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:54.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:54.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:54.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:54.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:55.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:55.134+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f844c09fd20 legacy=0x7f844c142160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:55.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:55.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:55.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:55.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:56.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:56.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:56.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:56.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:56.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:56.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:56.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): tick 2024-09-16T06:41:56.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6790/0 conn(0x7f844c09f230 legacy=0x7f844c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.189:6789/0 conn(0x7f844c09c460 legacy=0x7f844c13ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f844c09fd20 legacy=0x7f844c142160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f8438003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f844c09c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f844c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f844bfff700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09a410 con 0x7f8438003740 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09a040 con 0x7f844c09c460 2024-09-16T06:41:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438010450 con 0x7f844c09f230 2024-09-16T06:41:56.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.722+0000 7f8449ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:56.926+0000 7f844bfff700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:57.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:57.330+0000 7f844bfff700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:57.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:57.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:57.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:58.134+0000 7f844bfff700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:58.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:58.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:58.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:58.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:58.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:58.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:59.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:59.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:41:59.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:59.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:41:59.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:59.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:41:59.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:41:59.500+0000 7fb8bcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): tick 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6790/0 conn(0x7f844c09f230 legacy=0x7f843800cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.189:6789/0 conn(0x7f844c09c460 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:41:59.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f843800f4d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f84380105e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f8438003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438011660 con 0x7f8438003740 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c09a410 con 0x7f843800f4d0 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8438011170 con 0x7f84380105e0 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8449ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:41:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.726+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:41:59.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:41:59.930+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:00.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:00.334+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:00.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:00.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:42:00.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:00.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:00.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:00.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.138+0000 7f8450b57700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020122564 0 0) 0x7f844c09a410 con 0x7f843800f4d0 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 10 monclient(hunting): my global_id is 134518 2024-09-16T06:42:01.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.298+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84340035d0 con 0x7f843800f4d0 2024-09-16T06:42:01.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70034023 0 0) 0x7f8438011170 con 0x7f84380105e0 2024-09-16T06:42:01.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient(hunting): my global_id is 124811 2024-09-16T06:42:01.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c09a410 con 0x7f84380105e0 2024-09-16T06:42:01.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042042310 0 0) 0x7f844c09a410 con 0x7f84380105e0 2024-09-16T06:42:01.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.189:6789/0 conn(0x7f843800f4d0 legacy=0x7f843800f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f8438009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:01.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 monclient: found mon.noname-b 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438011300 con 0x7f84380105e0 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8444001830 con 0x7f84380105e0 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f844affd700 10 monclient: dump: 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f84525ba700 5 monclient: authenticate success, global_id 124811 2024-09-16T06:42:01.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.302+0000 7f84525ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8444001a20 con 0x7f84380105e0 2024-09-16T06:42:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 1 -- 172.21.15.76:0/1193996822 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8444001df0 con 0x7f84380105e0 2024-09-16T06:42:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f844affd700 10 monclient: dump: 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.311 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.311 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f84525ba700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:01.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.306+0000 7f84525ba700 4 set_mon_vals no callback set 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: shutdown 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- 172.21.15.76:0/1193996822 >> v1:172.21.15.76:6790/0 conn(0x7f84380105e0 legacy=0x7f8438010a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- 172.21.15.76:0/1193996822 shutdown_connections 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- 172.21.15.76:0/1193996822 >> 172.21.15.76:0/1193996822 conn(0x7f844c0a3e50 msgr2=0x7f844c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- 172.21.15.76:0/1193996822 shutdown_connections 2024-09-16T06:42:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- 172.21.15.76:0/1193996822 wait complete. 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: build_initial_monmap 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: monmap: 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:01.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 Processor -- start 2024-09-16T06:42:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 1 -- start start 2024-09-16T06:42:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: init 2024-09-16T06:42:01.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-a con 0x7f8438003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.310+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-c con 0x7f843800f4d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient(hunting): picked mon.noname-b con 0x7f84380105e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0a2bc0 con 0x7f8438003740 2024-09-16T06:42:01.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f844bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f844c138b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0a2670 con 0x7f84380105e0 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c072450 con 0x7f843800f4d0 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient(hunting): authenticate will time out at 1068.448147s 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f844b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f843800f4d0 0x7f844c1d31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44778/0 (socket says 172.21.15.76:44778) 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f844b7fe700 1 -- 172.21.15.76:0/2357761997 learned_addr learned my addr 172.21.15.76:0/2357761997 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208143189 0 0) 0x7f844c072450 con 0x7f843800f4d0 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): my global_id is 124814 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438011490 con 0x7f843800f4d0 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898185228 0 0) 0x7f844c0a2670 con 0x7f84380105e0 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): my global_id is 134521 2024-09-16T06:42:01.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c072450 con 0x7f84380105e0 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079084446 0 0) 0x7f8438011490 con 0x7f843800f4d0 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c0a2670 con 0x7f843800f4d0 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8444001fa0 con 0x7f843800f4d0 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1146226763 0 0) 0x7f844c072450 con 0x7f84380105e0 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8438011490 con 0x7f84380105e0 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f843c004290 con 0x7f84380105e0 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2315101128 0 0) 0x7f844c0a2670 con 0x7f843800f4d0 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 >> v1:172.21.15.189:6789/0 conn(0x7f84380105e0 legacy=0x7f844c1d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 >> v1:172.21.15.76:6789/0 conn(0x7f8438003740 legacy=0x7f844c138b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 monclient: found mon.c 2024-09-16T06:42:01.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c126260 con 0x7f843800f4d0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 5 monclient: authenticate success, global_id 124814 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _renew_subs 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c139290 con 0x7f843800f4d0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _renew_subs 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c124d40 con 0x7f843800f4d0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8444003cd0 con 0x7f843800f4d0 2024-09-16T06:42:01.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8444004060 con 0x7f843800f4d0 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8449ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 10 monclient: dump: 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8444004240 con 0x7f843800f4d0 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.314+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c005100 con 0x7f843800f4d0 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.318+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 66..91) v4 ==== 4746+0+0 (unknown 2790596778 0 0) 0x7f8444043b90 con 0x7f843800f4d0 2024-09-16T06:42:01.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.318+0000 7f844bfff700 10 monclient: get_auth_request con 0x7f8438037700 auth_method 0 2024-09-16T06:42:01.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.322+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8444043e60 con 0x7f843800f4d0 2024-09-16T06:42:01.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.322+0000 7f8448ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.322+0000 7f8448ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:01.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:01.500+0000 7fb8bcff9700 10 monclient: tick 2024-09-16T06:42:01.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:01.500+0000 7fb8bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:01.500+0000 7fb8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f84525ba700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:42:01.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f84525ba700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:01.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f84525ba700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f842c005930 con 0x7f843800f4d0 2024-09-16T06:42:01.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f8448ff9700 1 -- 172.21.15.76:0/2357761997 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3379681615) 0x7f8444014d60 con 0x7f843800f4d0 2024-09-16T06:42:01.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f8448ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:01.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.558+0000 7f8448ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:01.563 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:42:01.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8438037700 legacy=0x7f8438039bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 10 monclient: shutdown 2024-09-16T06:42:01.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 >> v1:172.21.15.76:6790/0 conn(0x7f843800f4d0 legacy=0x7f844c1d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 shutdown_connections 2024-09-16T06:42:01.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 >> 172.21.15.76:0/2357761997 conn(0x7f844c0a3e50 msgr2=0x7f844c09c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 shutdown_connections 2024-09-16T06:42:01.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.562+0000 7f84525ba700 1 -- 172.21.15.76:0/2357761997 wait complete. 2024-09-16T06:42:01.582 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:42:01.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:42:01.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: get_monmap_and_config 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: build_initial_monmap 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: monmap: 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:01.700952+0000 2024-09-16T06:42:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:01.700952+0000 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 1 Processor -- start 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 1 -- start start 2024-09-16T06:42:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: init 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-c con 0x7f4d2c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-a con 0x7f4d2c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c098990 con 0x7f4d2c099090 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c097f40 con 0x7f4d2c097b60 2024-09-16T06:42:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c09d610 con 0x7f4d2c139500 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient(hunting): authenticate will time out at 1068.834421s 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d30ec1700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4d2c099090 legacy=0x7f4d2c09f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d31ec3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d2c139500 0x7f4d2c13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44788/0 (socket says 172.21.15.76:44788) 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d31ec3700 1 -- 172.21.15.76:0/1788035948 learned_addr learned my addr 172.21.15.76:0/1788035948 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442992920 0 0) 0x7f4d2c09d610 con 0x7f4d2c139500 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient(hunting): my global_id is 124817 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d140035d0 con 0x7f4d2c139500 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205183011 0 0) 0x7f4d140035d0 con 0x7f4d2c139500 2024-09-16T06:42:01.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 >> v1:172.21.15.189:6789/0 conn(0x7f4d2c097b60 legacy=0x7f4d2c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 >> v1:172.21.15.76:6789/0 conn(0x7f4d2c099090 legacy=0x7f4d2c09f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 monclient: found mon.noname-b 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0986b0 con 0x7f4d2c139500 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 5 monclient: authenticate success, global_id 124817 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d1c001610 con 0x7f4d2c139500 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: dump: 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d1c001930 con 0x7f4d2c139500 2024-09-16T06:42:01.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 1 -- 172.21.15.76:0/1788035948 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d1c001d00 con 0x7f4d2c139500 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d326c4700 4 set_mon_vals no callback set 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.698+0000 7f4d23fff700 10 monclient: dump: 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 10 monclient: shutdown 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- 172.21.15.76:0/1788035948 >> v1:172.21.15.76:6790/0 conn(0x7f4d2c139500 legacy=0x7f4d2c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- 172.21.15.76:0/1788035948 shutdown_connections 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- 172.21.15.76:0/1788035948 >> 172.21.15.76:0/1788035948 conn(0x7f4d2c0a0030 msgr2=0x7f4d2c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- 172.21.15.76:0/1788035948 shutdown_connections 2024-09-16T06:42:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- 172.21.15.76:0/1788035948 wait complete. 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 10 monclient: build_initial_monmap 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 10 monclient: monmap: 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:01.714 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 Processor -- start 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 1 -- start start 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.702+0000 7f4d326c4700 10 monclient: init 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-c con 0x7f4d2c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-b con 0x7f4d2c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient(hunting): picked mon.noname-a con 0x7f4d2c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c06d3f0 con 0x7f4d2c139500 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c06ec30 con 0x7f4d2c099090 2024-09-16T06:42:01.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c06fb20 con 0x7f4d2c097b60 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient(hunting): authenticate will time out at 1068.840452s 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d31ec3700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4d2c139500 legacy=0x7f4d2c1cfef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d316c2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4d2c097b60 0x7f4d2c06c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44790/0 (socket says 172.21.15.76:44790) 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d316c2700 1 -- 172.21.15.76:0/682955018 learned_addr learned my addr 172.21.15.76:0/682955018 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976427878 0 0) 0x7f4d2c06fb20 con 0x7f4d2c097b60 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): my global_id is 124820 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d180035d0 con 0x7f4d2c097b60 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503174563 0 0) 0x7f4d2c06ec30 con 0x7f4d2c099090 2024-09-16T06:42:01.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): my global_id is 134527 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c06fb20 con 0x7f4d2c099090 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62318686 0 0) 0x7f4d180035d0 con 0x7f4d2c097b60 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d2c06ec30 con 0x7f4d2c097b60 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d28001cb0 con 0x7f4d2c097b60 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): dump: 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.717 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3708760663 0 0) 0x7f4d2c06ec30 con 0x7f4d2c097b60 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.189:6789/0 conn(0x7f4d2c099090 legacy=0x7f4d2c06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6789/0 conn(0x7f4d2c139500 legacy=0x7f4d2c1cfef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 monclient: found mon.c 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0700e0 con 0x7f4d2c097b60 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 5 monclient: authenticate success, global_id 124820 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _renew_subs 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c06fe00 con 0x7f4d2c097b60 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _renew_subs 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c06e120 con 0x7f4d2c097b60 2024-09-16T06:42:01.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d28002040 con 0x7f4d2c097b60 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d28003e90 con 0x7f4d2c097b60 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d237fe700 4 set_mon_vals no callback set 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient: dump: 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.720 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d0c006590 con 0x7f4d0c002830 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4d28003ff0 con 0x7f4d2c097b60 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 66..91) v4 ==== 4746+0+0 (unknown 2790596778 0 0) 0x7f4d28043930 con 0x7f4d2c097b60 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288109189 0 0) 0x7f4d0c006590 con 0x7f4d0c002830 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 10 monclient(hunting): my global_id is 124823 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d2c06ec30 con 0x7f4d0c002830 2024-09-16T06:42:01.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.706+0000 7f4d31ec3700 10 monclient: get_auth_request con 0x7f4d18035fd0 auth_method 0 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 142039489 0 0) 0x7f4d2c06ec30 con 0x7f4d0c002830 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d0c006590 con 0x7f4d0c002830 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d24004800 con 0x7f4d0c002830 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 10 monclient: dump: 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.722 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1887444814 0 0) 0x7f4d0c006590 con 0x7f4d0c002830 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d0c00a190 con 0x7f4d0c002830 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4d24007480 con 0x7f4d0c002830 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6790/0 conn(0x7f4d0c002830 legacy=0x7f4d0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.710+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6790/0 conn(0x7f4d0c002830 legacy=0x7f4d0c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:01.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d0c009e30 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586734715 0 0) 0x7f4d0c009e30 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient(hunting): my global_id is 124826 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c006590 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 147478157 0 0) 0x7f4d0c006590 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d0c009e30 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d240054a0 con 0x7f4d0c009150 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient: dump: 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2953021464 0 0) 0x7f4d0c009e30 con 0x7f4d0c009150 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d0c00c340 con 0x7f4d0c009150 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 2931101967) 0x7f4d240075f0 con 0x7f4d0c009150 2024-09-16T06:42:01.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6790/0 conn(0x7f4d0c009150 legacy=0x7f4d0c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d21ffb700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6790/0 conn(0x7f4d0c009150 legacy=0x7f4d0c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 42, 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.731 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.733 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.734 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:01.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:01.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.738 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:01.739 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4d18035fd0 legacy=0x7f4d18038490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 10 monclient: shutdown 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 >> v1:172.21.15.76:6790/0 conn(0x7f4d2c097b60 legacy=0x7f4d2c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d31ec3700 1 -- 172.21.15.76:0/682955018 reap_dead start 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d31ec3700 1 -- 172.21.15.76:0/682955018 reap_dead start 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 shutdown_connections 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 >> 172.21.15.76:0/682955018 conn(0x7f4d2c0a0030 msgr2=0x7f4d2c13ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 shutdown_connections 2024-09-16T06:42:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.718+0000 7f4d326c4700 1 -- 172.21.15.76:0/682955018 wait complete. 2024-09-16T06:42:01.741 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:42:01.741 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:42:01.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.846+0000 7fc1b2def700 10 monclient: get_monmap_and_config 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.846+0000 7fc1b2def700 10 monclient: build_initial_monmap 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.846+0000 7fc1b2def700 10 monclient: monmap: 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:01.851471+0000 2024-09-16T06:42:01.850 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:01.851471+0000 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:01.851 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 Processor -- start 2024-09-16T06:42:01.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- start start 2024-09-16T06:42:01.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient: init 2024-09-16T06:42:01.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-b con 0x7fc1ac132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-a con 0x7fc1ac06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-c con 0x7fc1ac06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac06c280 con 0x7fc1ac06d480 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac06bfa0 con 0x7fc1ac06cf50 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac09ace0 con 0x7fc1ac132330 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient(hunting): authenticate will time out at 1068.985527s 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b15ec700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc1ac06d480 legacy=0x7fc1ac13a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b1ded700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1ac132330 0x7fc1ac132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44822/0 (socket says 172.21.15.76:44822) 2024-09-16T06:42:01.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b1ded700 1 -- 172.21.15.76:0/4084804373 learned_addr learned my addr 172.21.15.76:0/4084804373 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918218606 0 0) 0x7fc1ac09ace0 con 0x7fc1ac132330 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): my global_id is 124829 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1980035d0 con 0x7fc1ac132330 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188156325 0 0) 0x7fc1ac06bfa0 con 0x7fc1ac06cf50 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient(hunting): my global_id is 134530 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac09ace0 con 0x7fc1ac06cf50 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725204826 0 0) 0x7fc1980035d0 con 0x7fc1ac132330 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac06cf50 legacy=0x7fc1ac13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 >> v1:172.21.15.76:6789/0 conn(0x7fc1ac06d480 legacy=0x7fc1ac13a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 monclient: found mon.noname-b 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac09ae70 con 0x7fc1ac132330 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 5 monclient: authenticate success, global_id 124829 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a8001610 con 0x7fc1ac132330 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: dump: 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.858 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a80018f0 con 0x7fc1ac132330 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 4 set_mon_vals no callback set 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 1 -- 172.21.15.76:0/4084804373 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a8001ae0 con 0x7fc1ac132330 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b0deb700 10 monclient: dump: 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 10 monclient: shutdown 2024-09-16T06:42:01.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- 172.21.15.76:0/4084804373 >> v1:172.21.15.76:6790/0 conn(0x7fc1ac132330 legacy=0x7fc1ac132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- 172.21.15.76:0/4084804373 shutdown_connections 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- 172.21.15.76:0/4084804373 >> 172.21.15.76:0/4084804373 conn(0x7fc1ac0a41c0 msgr2=0x7fc1ac0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- 172.21.15.76:0/4084804373 shutdown_connections 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.850+0000 7fc1b2def700 1 -- 172.21.15.76:0/4084804373 wait complete. 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: build_initial_monmap 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: monmap: 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:01.861 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:01.862 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:01.862 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:01.862 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:01.862 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 Processor -- start 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- start start 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: init 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-a con 0x7fc1ac06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-b con 0x7fc1ac06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient(hunting): picked mon.noname-c con 0x7fc1ac132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:01.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac12de10 con 0x7fc1ac06cf50 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac131650 con 0x7fc1ac06d480 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac1ca590 con 0x7fc1ac132330 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient(hunting): authenticate will time out at 1068.988960s 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b1ded700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc1ac06cf50 legacy=0x7fc1ac1c9770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b25ee700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1ac132330 0x7fc1ac1d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44830/0 (socket says 172.21.15.76:44830) 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b25ee700 1 -- 172.21.15.76:0/113144776 learned_addr learned my addr 172.21.15.76:0/113144776 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239497208 0 0) 0x7fc1ac1ca590 con 0x7fc1ac132330 2024-09-16T06:42:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): my global_id is 124832 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1900035d0 con 0x7fc1ac132330 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717198315 0 0) 0x7fc1ac131650 con 0x7fc1ac06d480 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): my global_id is 134533 2024-09-16T06:42:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac1ca590 con 0x7fc1ac06d480 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079498425 0 0) 0x7fc1900035d0 con 0x7fc1ac132330 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1ac131650 con 0x7fc1ac132330 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a4001c30 con 0x7fc1ac132330 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): dump: 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710525 0 0) 0x7fc1ac1ca590 con 0x7fc1ac06d480 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1900035d0 con 0x7fc1ac06d480 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc19c004170 con 0x7fc1ac06d480 2024-09-16T06:42:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient(hunting): dump: 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.871 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4288192248 0 0) 0x7fc1ac131650 con 0x7fc1ac132330 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac06d480 legacy=0x7fc1ac1c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.76:6789/0 conn(0x7fc1ac06cf50 legacy=0x7fc1ac1c9770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 monclient: found mon.c 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac09e8b0 con 0x7fc1ac132330 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 5 monclient: authenticate success, global_id 124832 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _renew_subs 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac12e9a0 con 0x7fc1ac132330 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _renew_subs 2024-09-16T06:42:01.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1310b0 con 0x7fc1ac132330 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a4003a40 con 0x7fc1ac132330 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a3fff700 4 set_mon_vals no callback set 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a4003dd0 con 0x7fc1ac132330 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1a27fc700 10 monclient: dump: 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.874 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.854+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc1ac12f890 con 0x7fc1ac09ea40 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc1a4003fb0 con 0x7fc1ac132330 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(91..91 src has 66..91) v4 ==== 4746+0+0 (unknown 2790596778 0 0) 0x7fc1a4043850 con 0x7fc1ac132330 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538393970 0 0) 0x7fc1ac12f890 con 0x7fc1ac09ea40 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient(hunting): my global_id is 134536 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac131650 con 0x7fc1ac09ea40 2024-09-16T06:42:01.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1b15ec700 10 monclient: get_auth_request con 0x7fc1900362c0 auth_method 0 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 481217954 0 0) 0x7fc1ac131650 con 0x7fc1ac09ea40 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1ac12f890 con 0x7fc1ac09ea40 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a8001a60 con 0x7fc1ac09ea40 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient: dump: 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.877 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 301999584 0 0) 0x7fc1ac12f890 con 0x7fc1ac09ea40 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1ac12fad0 con 0x7fc1ac09ea40 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc1a8004950 con 0x7fc1ac09ea40 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:01.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:01.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac09ea40 legacy=0x7fc1ac09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.858+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac09ea40 legacy=0x7fc1ac09ee20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1b2def700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc1ac0a3430 con 0x7fc1ac09f530 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338158964 0 0) 0x7fc1a8004ac0 con 0x7fc1ac09f530 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient(hunting): my global_id is 134539 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac12f890 con 0x7fc1ac09f530 2024-09-16T06:42:01.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1728504546 0 0) 0x7fc1ac12f890 con 0x7fc1ac09f530 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a8004ac0 con 0x7fc1ac09f530 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1a8004e80 con 0x7fc1ac09f530 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient: dump: 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2016225494 0 0) 0x7fc1a8004ac0 con 0x7fc1ac09f530 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc1ac0672a0 con 0x7fc1ac09f530 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 202608606) 0x7fc1a80071b0 con 0x7fc1ac09f530 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.870+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac09f530 legacy=0x7fc1ac02c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1a27fc700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6789/0 conn(0x7fc1ac09f530 legacy=0x7fc1ac02c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:01.882 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 42, 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:01.894 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc1900362c0 legacy=0x7fc190038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 10 monclient: shutdown 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 >> v1:172.21.15.76:6790/0 conn(0x7fc1ac132330 legacy=0x7fc1ac1d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b25ee700 1 -- 172.21.15.76:0/113144776 reap_dead start 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b25ee700 1 -- 172.21.15.76:0/113144776 reap_dead start 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 shutdown_connections 2024-09-16T06:42:01.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 >> 172.21.15.76:0/113144776 conn(0x7fc1ac0a41c0 msgr2=0x7fc1ac09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:01.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 shutdown_connections 2024-09-16T06:42:01.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:01.874+0000 7fc1b2def700 1 -- 172.21.15.76:0/113144776 wait complete. 2024-09-16T06:42:01.896 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:42:01.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:42:02.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.292+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 204+0+0 (unknown 1464787419 0 0) 0x7fb8b8012f20 con 0x7fb8c0132610 2024-09-16T06:42:02.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.292+0000 7fb8bdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:42:02.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.292+0000 7fb8bdffb700 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T06:42:02.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8c76bb700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:42:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8c76bb700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:42:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8c76bb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7fb8a00053b0 con 0x7fb8c0132610 2024-09-16T06:42:02.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8bdffb700 1 -- 172.21.15.189:0/986157653 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v92) v1 ==== 214+0+0 (unknown 4238273258 0 0) 0x7fb8b80146a0 con 0x7fb8c0132610 2024-09-16T06:42:02.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8bdffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T06:42:02.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.400+0000 7fb8bdffb700 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb8ac036390 legacy=0x7fb8ac038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 10 monclient: shutdown 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 >> v1:172.21.15.189:6789/0 conn(0x7fb8c0132610 legacy=0x7fb8c01d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 shutdown_connections 2024-09-16T06:42:02.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 >> 172.21.15.189:0/986157653 conn(0x7fb8c00a44f0 msgr2=0x7fb8c00a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:02.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 shutdown_connections 2024-09-16T06:42:02.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.404+0000 7fb8c76bb700 1 -- 172.21.15.189:0/986157653 wait complete. 2024-09-16T06:42:02.406 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T06:42:02.415 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 10 monclient: get_monmap_and_config 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 10 monclient: build_initial_monmap 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 10 monclient: monmap: 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:02.526969+0000 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:02.526969+0000 2024-09-16T06:42:02.526 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:02.527 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:02.527 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 1 Processor -- start 2024-09-16T06:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 1 -- start start 2024-09-16T06:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.524+0000 7f9d12edf700 10 monclient: init 2024-09-16T06:42:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c0a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-b con 0x7f9d0c09aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c0a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c09a040 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0a33d0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c099d60 con 0x7f9d0c09aa20 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient(hunting): authenticate will time out at 1075.040330s 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d1147c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9d0c0a3ca0 legacy=0x7f9d0c13dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d10c7b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0a2350 0x7f9d0c0a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44232/0 (socket says 172.21.15.189:44232) 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d10c7b700 1 -- 172.21.15.189:0/2467169555 learned_addr learned my addr 172.21.15.189:0/2467169555 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494894499 0 0) 0x7f9d0c0a33d0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): my global_id is 134542 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf40035d0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609928674 0 0) 0x7f9d0c099d60 con 0x7f9d0c09aa20 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient(hunting): my global_id is 124835 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c0a33d0 con 0x7f9d0c09aa20 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467516468 0 0) 0x7f9cf40035d0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 >> v1:172.21.15.76:6790/0 conn(0x7f9d0c09aa20 legacy=0x7f9d0c13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 >> v1:172.21.15.76:6789/0 conn(0x7f9d0c0a3ca0 legacy=0x7f9d0c13dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0a3920 con 0x7f9d0c0a2350 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 5 monclient: authenticate success, global_id 134542 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d00001610 con 0x7f9d0c0a2350 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: dump: 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d000018f0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 4 set_mon_vals no callback set 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 1 -- 172.21.15.189:0/2467169555 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d00001c80 con 0x7f9d0c0a2350 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d0b7fe700 10 monclient: dump: 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: shutdown 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- 172.21.15.189:0/2467169555 >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0a2350 legacy=0x7f9d0c0a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- 172.21.15.189:0/2467169555 shutdown_connections 2024-09-16T06:42:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- 172.21.15.189:0/2467169555 >> 172.21.15.189:0/2467169555 conn(0x7f9d0c0a7d70 msgr2=0x7f9d0c0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- 172.21.15.189:0/2467169555 shutdown_connections 2024-09-16T06:42:02.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 1 -- 172.21.15.189:0/2467169555 wait complete. 2024-09-16T06:42:02.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: build_initial_monmap 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.528+0000 7f9d12edf700 10 monclient: monmap: 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:02.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 Processor -- start 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- start start 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: init 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c09aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c0a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient(hunting): picked mon.noname-b con 0x7f9d0c0a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:02.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c120c50 con 0x7f9d0c09aa20 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c09b7b0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c1ca9a0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient(hunting): authenticate will time out at 1075.045512s 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d10c7b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9d0c09aa20 legacy=0x7f9d0c1355d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d1147c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0a3ca0 0x7f9d0c1d24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44234/0 (socket says 172.21.15.189:44234) 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d1147c700 1 -- 172.21.15.189:0/3106247429 learned_addr learned my addr 172.21.15.189:0/3106247429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881738773 0 0) 0x7f9d04003e70 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): my global_id is 134545 2024-09-16T06:42:02.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf80035d0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706748985 0 0) 0x7f9d0c1ca9a0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): my global_id is 124838 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d04003e70 con 0x7f9d0c0a2350 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970466544 0 0) 0x7f9cf80035d0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d0c1ca9a0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d040044e0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): dump: 2024-09-16T06:42:02.547 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749245590 0 0) 0x7f9d04003e70 con 0x7f9d0c0a2350 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cf80035d0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9cfc0018b0 con 0x7f9d0c0a2350 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient(hunting): dump: 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:02.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277210670 0 0) 0x7f9d0c1ca9a0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 >> v1:172.21.15.76:6790/0 conn(0x7f9d0c0a2350 legacy=0x7f9d0c1ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 >> v1:172.21.15.76:6789/0 conn(0x7f9d0c09aa20 legacy=0x7f9d0c1355d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: _finish_hunting 0 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 monclient: found mon.b 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c122cb0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: _finish_auth 0 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: _check_auth_tickets 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 5 monclient: authenticate success, global_id 134545 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _renew_subs 2024-09-16T06:42:02.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c121de0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _renew_subs 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c06bed0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d04004750 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d04004ae0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d0affd700 4 set_mon_vals no callback set 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.532+0000 7f9d097fa700 10 monclient: dump: 2024-09-16T06:42:02.551 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d12edf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d12edf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:02.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d12edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec005100 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d04004d00 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(92..92 src has 66..92) v4 ==== 5043+0+0 (unknown 1622423787 0 0) 0x7f9d040439c0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.536+0000 7f9d10c7b700 10 monclient: get_auth_request con 0x7f9cf80362a0 auth_method 0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.540+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d04014aa0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.540+0000 7f9d097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:02.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.540+0000 7f9d097fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:02.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.716+0000 7f9d12edf700 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:02.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.716+0000 7f9d12edf700 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:02.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.716+0000 7f9d12edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:02.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:02.716+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f9cec005930 con 0x7f9d0c0a3ca0 2024-09-16T06:42:03.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.292+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 212+0+0 (unknown 2234209046 0 0) 0x7f9d0401d0f0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:03.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.292+0000 7f9d097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:03.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.292+0000 7f9d097fa700 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T06:42:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d12edf700 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:03.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d12edf700 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:03.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d12edf700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7f9cec0053b0 con 0x7f9d0c0a3ca0 2024-09-16T06:42:03.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d097fa700 1 -- 172.21.15.189:0/3106247429 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v93) v1 ==== 211+0+0 (unknown 1298689654 0 0) 0x7f9d04012f10 con 0x7f9d0c0a3ca0 2024-09-16T06:42:03.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d097fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T06:42:03.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.404+0000 7f9d097fa700 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9cf80362a0 legacy=0x7f9cf8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 10 monclient: shutdown 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 >> v1:172.21.15.189:6789/0 conn(0x7f9d0c0a3ca0 legacy=0x7f9d0c1d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 shutdown_connections 2024-09-16T06:42:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 >> 172.21.15.189:0/3106247429 conn(0x7f9d0c0a7d70 msgr2=0x7f9d0c1bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:03.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 shutdown_connections 2024-09-16T06:42:03.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.408+0000 7f9d12edf700 1 -- 172.21.15.189:0/3106247429 wait complete. 2024-09-16T06:42:03.410 INFO:tasks.workunit.client.0.smithi189.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T06:42:03.419 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: get_monmap_and_config 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: build_initial_monmap 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: monmap: 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:03.528714+0000 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:03.528714+0000 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:03.529 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 Processor -- start 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- start start 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: init 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-c con 0x7f2b1409f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-a con 0x7f2b14099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-b con 0x7f2b141396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1409f520 con 0x7f2b14099700 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1409bc10 con 0x7f2b1409f140 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1409b810 con 0x7f2b141396a0 2024-09-16T06:42:03.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient(hunting): authenticate will time out at 1076.041489s 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b12ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b1409f140 0x7f2b1409a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44246/0 (socket says 172.21.15.189:44246) 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b12ffd700 1 -- 172.21.15.189:0/2568184055 learned_addr learned my addr 172.21.15.189:0/2568184055 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b127fc700 1 -- 172.21.15.189:0/2568184055 >> v1:172.21.15.76:6789/0 conn(0x7f2b14099700 legacy=0x7f2b1409b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232463926 0 0) 0x7f2b1409bc10 con 0x7f2b1409f140 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient(hunting): my global_id is 134548 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc0035d0 con 0x7f2b1409f140 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3752423822 0 0) 0x7f2afc0035d0 con 0x7f2b1409f140 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 >> v1:172.21.15.76:6790/0 conn(0x7f2b141396a0 legacy=0x7f2b1413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 >> v1:172.21.15.76:6789/0 conn(0x7f2b14099700 legacy=0x7f2b1409b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:03.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 monclient: found mon.noname-c 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1409ceb0 con 0x7f2b1409f140 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 5 monclient: authenticate success, global_id 134548 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b08001610 con 0x7f2b1409f140 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: dump: 2024-09-16T06:42:03.539 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b08001930 con 0x7f2b1409f140 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 4 set_mon_vals no callback set 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 1 -- 172.21.15.189:0/2568184055 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b08001cc0 con 0x7f2b1409f140 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b11ffb700 10 monclient: dump: 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 10 monclient: shutdown 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- 172.21.15.189:0/2568184055 >> v1:172.21.15.189:6789/0 conn(0x7f2b1409f140 legacy=0x7f2b1409a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- 172.21.15.189:0/2568184055 shutdown_connections 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- 172.21.15.189:0/2568184055 >> 172.21.15.189:0/2568184055 conn(0x7f2b140a03e0 msgr2=0x7f2b140a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- 172.21.15.189:0/2568184055 shutdown_connections 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.528+0000 7f2b1981f700 1 -- 172.21.15.189:0/2568184055 wait complete. 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: build_initial_monmap 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: monmap: 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 Processor -- start 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- start start 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: init 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-a con 0x7f2b14099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-c con 0x7f2b1409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient(hunting): picked mon.noname-b con 0x7f2b141396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1409a6e0 con 0x7f2b14099700 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140714a0 con 0x7f2b141396a0 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140a2d60 con 0x7f2b1409f140 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient(hunting): authenticate will time out at 1076.045694s 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b12ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2b14099700 legacy=0x7f2b14096630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b137fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b141396a0 0x7f2b141d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44258/0 (socket says 172.21.15.189:44258) 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b137fe700 1 -- 172.21.15.189:0/3915316433 learned_addr learned my addr 172.21.15.189:0/3915316433 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783300691 0 0) 0x7f2b140714a0 con 0x7f2b141396a0 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): my global_id is 134551 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b000035d0 con 0x7f2b141396a0 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4207842725 0 0) 0x7f2b000035d0 con 0x7f2b141396a0 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b140714a0 con 0x7f2b141396a0 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b0c001c70 con 0x7f2b141396a0 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): dump: 2024-09-16T06:42:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298593301 0 0) 0x7f2b140a2d60 con 0x7f2b1409f140 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient(hunting): my global_id is 124844 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b000035d0 con 0x7f2b1409f140 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267112256 0 0) 0x7f2b140714a0 con 0x7f2b141396a0 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 >> v1:172.21.15.76:6790/0 conn(0x7f2b1409f140 legacy=0x7f2b141ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 >> v1:172.21.15.76:6789/0 conn(0x7f2b14099700 legacy=0x7f2b14096630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 monclient: found mon.b 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14094510 con 0x7f2b141396a0 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient: _finish_auth 0 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 5 monclient: authenticate success, global_id 134551 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _renew_subs 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b14094db0 con 0x7f2b141396a0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _renew_subs 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14092a50 con 0x7f2b141396a0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b0c003a80 con 0x7f2b141396a0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.532+0000 7f2afbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b0c003d90 con 0x7f2b141396a0 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 10 monclient: dump: 2024-09-16T06:42:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b1981f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2b0c003f70 con 0x7f2b141396a0 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b117fa700 4 set_mon_vals no callback set 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(93..93 src has 66..93) v4 ==== 5064+0+0 (unknown 3453249005 0 0) 0x7f2b0c043c00 con 0x7f2b141396a0 2024-09-16T06:42:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b1981f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b1981f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1408deb0 con 0x7f2b141396a0 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.536+0000 7f2b12ffd700 10 monclient: get_auth_request con 0x7f2b000362c0 auth_method 0 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.540+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b0c014bf0 con 0x7f2b141396a0 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.540+0000 7f2afbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.540+0000 7f2afbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:03.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.712+0000 7f2b1981f700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:03.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.712+0000 7f2b1981f700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:03.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.712+0000 7f2b1981f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:03.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:03.712+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f2b14094bd0 con 0x7f2b141396a0 2024-09-16T06:42:04.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.312+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 165+0+0 (unknown 995047785 0 0) 0x7f2b0c0147e0 con 0x7f2b141396a0 2024-09-16T06:42:04.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.312+0000 7f2afbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:04.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.312+0000 7f2afbfff700 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T06:42:04.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2b1981f700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:04.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2b1981f700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:04.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2b1981f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7f2b140008d0 con 0x7f2b141396a0 2024-09-16T06:42:04.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2afbfff700 1 -- 172.21.15.189:0/3915316433 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v94) v1 ==== 175+0+0 (unknown 2455362110 0 0) 0x7f2b0c0149c0 con 0x7f2b141396a0 2024-09-16T06:42:04.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2afbfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T06:42:04.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.420+0000 7f2afbfff700 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2b000362c0 legacy=0x7f2b00038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 10 monclient: shutdown 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 >> v1:172.21.15.189:6789/0 conn(0x7f2b141396a0 legacy=0x7f2b141d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 shutdown_connections 2024-09-16T06:42:04.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 >> 172.21.15.189:0/3915316433 conn(0x7f2b140a03e0 msgr2=0x7f2b1408afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:04.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 shutdown_connections 2024-09-16T06:42:04.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.424+0000 7f2b1981f700 1 -- 172.21.15.189:0/3915316433 wait complete. 2024-09-16T06:42:04.425 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T06:42:04.435 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd find osd.7 2024-09-16T06:42:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep host 2024-09-16T06:42:04.436 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep foo 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.532+0000 7f252eb32700 10 monclient: get_monmap_and_config 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.532+0000 7f252eb32700 10 monclient: build_initial_monmap 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.532+0000 7f252eb32700 10 monclient: monmap: 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:04.536166+0000 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:04.536166+0000 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 1 Processor -- start 2024-09-16T06:42:04.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 1 -- start start 2024-09-16T06:42:04.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient: init 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-c con 0x7f2528132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-b con 0x7f2528138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-a con 0x7f252813bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252806abd0 con 0x7f252813bf70 2024-09-16T06:42:04.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252806b170 con 0x7f2528132080 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528132b70 con 0x7f2528138460 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient(hunting): authenticate will time out at 1077.050534s 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252db30700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2528132080 0x7f2528132460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44262/0 (socket says 172.21.15.189:44262) 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252db30700 1 -- 172.21.15.189:0/4276908212 learned_addr learned my addr 172.21.15.189:0/4276908212 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252e331700 1 -- 172.21.15.189:0/4276908212 >> v1:172.21.15.76:6789/0 conn(0x7f252813bf70 legacy=0x7f252813e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713387715 0 0) 0x7f252806b170 con 0x7f2528132080 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient(hunting): my global_id is 134554 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25140035d0 con 0x7f2528132080 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787740255 0 0) 0x7f25140035d0 con 0x7f2528132080 2024-09-16T06:42:04.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 >> v1:172.21.15.76:6790/0 conn(0x7f2528138460 legacy=0x7f252813a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 >> v1:172.21.15.76:6789/0 conn(0x7f252813bf70 legacy=0x7f252813e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: _finish_hunting 0 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 monclient: found mon.noname-c 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280a38b0 con 0x7f2528132080 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: _finish_auth 0 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: _check_auth_tickets 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 5 monclient: authenticate success, global_id 134554 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2524001610 con 0x7f2528132080 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: dump: 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2524001930 con 0x7f2528132080 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252eb32700 4 set_mon_vals no callback set 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 1 -- 172.21.15.189:0/4276908212 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2524001d00 con 0x7f2528132080 2024-09-16T06:42:04.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.536+0000 7f252cb2e700 10 monclient: dump: 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: shutdown 2024-09-16T06:42:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- 172.21.15.189:0/4276908212 >> v1:172.21.15.189:6789/0 conn(0x7f2528132080 legacy=0x7f2528132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- 172.21.15.189:0/4276908212 shutdown_connections 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- 172.21.15.189:0/4276908212 >> 172.21.15.189:0/4276908212 conn(0x7f25280a4660 msgr2=0x7f2528099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- 172.21.15.189:0/4276908212 shutdown_connections 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- 172.21.15.189:0/4276908212 wait complete. 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: build_initial_monmap 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: monmap: 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 Processor -- start 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- start start 2024-09-16T06:42:04.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: init 2024-09-16T06:42:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-c con 0x7f2528132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:04.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-b con 0x7f2528138460 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient(hunting): picked mon.noname-a con 0x7f252813bf70 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252809d9e0 con 0x7f252813bf70 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252813ba00 con 0x7f2528138460 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252809e720 con 0x7f2528132080 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252eb32700 10 monclient(hunting): authenticate will time out at 1077.054680s 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252e331700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f252813bf70 legacy=0x7f25281d2140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252db30700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2528132080 0x7f252809bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:50118/0 (socket says 172.21.15.189:50118) 2024-09-16T06:42:04.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252d32f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2528138460 0x7f252809c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44274/0 (socket says 172.21.15.189:44274) 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f252db30700 1 -- 172.21.15.189:0/1035089052 learned_addr learned my addr 172.21.15.189:0/1035089052 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483327739 0 0) 0x7f2528132b70 con 0x7f2528138460 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): my global_id is 134557 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c0035d0 con 0x7f2528138460 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672043050 0 0) 0x7f252809e720 con 0x7f2528132080 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): my global_id is 124850 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528132b70 con 0x7f2528132080 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687807937 0 0) 0x7f250c0035d0 con 0x7f2528138460 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f252809e720 con 0x7f2528138460 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2518001710 con 0x7f2528138460 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:04.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:04.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238531657 0 0) 0x7f2528132b70 con 0x7f2528132080 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f250c0035d0 con 0x7f2528132080 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25240015d0 con 0x7f2528132080 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.540+0000 7f251e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.557 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2830682621 0 0) 0x7f252809e720 con 0x7f2528138460 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 >> v1:172.21.15.76:6790/0 conn(0x7f2528132080 legacy=0x7f252809bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 >> v1:172.21.15.76:6789/0 conn(0x7f252813bf70 legacy=0x7f25281d2140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 monclient: found mon.b 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252809cce0 con 0x7f2528138460 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:04.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 5 monclient: authenticate success, global_id 134557 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _renew_subs 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2528137fe0 con 0x7f2528138460 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _renew_subs 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25280a0170 con 0x7f2528138460 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2518001d50 con 0x7f2528138460 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2518003860 con 0x7f2528138460 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251ffff700 4 set_mon_vals no callback set 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:04.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 10 monclient: dump: 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.560 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f25180039c0 con 0x7f2528138460 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2510005100 con 0x7f2528138460 2024-09-16T06:42:04.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(94..94 src has 66..94) v4 ==== 5234+0+0 (unknown 4260897437 0 0) 0x7f2518043650 con 0x7f2528138460 2024-09-16T06:42:04.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.544+0000 7f252db30700 10 monclient: get_auth_request con 0x7f250c0362c0 auth_method 0 2024-09-16T06:42:04.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.552+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f25180041c0 con 0x7f2528138460 2024-09-16T06:42:04.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.552+0000 7f251e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.552+0000 7f251e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:04.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f252eb32700 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f252eb32700 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f252eb32700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f2510005930 con 0x7f2528138460 2024-09-16T06:42:04.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (unknown 188148463 0 2157906930) 0x7f25180129e0 con 0x7f2528138460 2024-09-16T06:42:04.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f251e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.724+0000 7f251e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f252eb32700 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f252eb32700 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f252eb32700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7f2510005420 con 0x7f2528138460 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f251e7fc700 1 -- 172.21.15.189:0/1035089052 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v94) v1 ==== 65+0+226 (unknown 188148463 0 2157906930) 0x7f2518014160 con 0x7f2528138460 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f251e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T06:42:04.834 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.832+0000 7f251e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 >> v1:172.21.15.189:6812/3098810674 conn(0x7f250c0362c0 legacy=0x7f250c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 10 monclient: shutdown 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 >> v1:172.21.15.189:6789/0 conn(0x7f2528138460 legacy=0x7f252809c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 shutdown_connections 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 >> 172.21.15.189:0/1035089052 conn(0x7f25280a4660 msgr2=0x7f2528137810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 shutdown_connections 2024-09-16T06:42:04.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.836+0000 7f252eb32700 1 -- 172.21.15.189:0/1035089052 wait complete. 2024-09-16T06:42:04.847 INFO:tasks.workunit.client.0.smithi189.stdout: "host": "foo", 2024-09-16T06:42:04.847 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.7 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: get_monmap_and_config 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: build_initial_monmap 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: monmap: 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:04.949076+0000 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:04.949076+0000 2024-09-16T06:42:04.948 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 1 Processor -- start 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 1 -- start start 2024-09-16T06:42:04.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: init 2024-09-16T06:42:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-a con 0x7f76480a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-b con 0x7f76480a3d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:04.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-c con 0x7f76480a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7648099a40 con 0x7f76480a27a0 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7648099ef0 con 0x7f76480a2350 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764809a1d0 con 0x7f76480a3d10 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 10 monclient(hunting): authenticate will time out at 1077.462770s 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ca26700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f76480a27a0 legacy=0x7f76480a2b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764d227700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76480a2350 0x7f764813dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44286/0 (socket says 172.21.15.189:44286) 2024-09-16T06:42:04.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764d227700 1 -- 172.21.15.189:0/1166241101 learned_addr learned my addr 172.21.15.189:0/1166241101 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:04.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680699443 0 0) 0x7f7648099ef0 con 0x7f76480a2350 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): my global_id is 134560 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76300035d0 con 0x7f76480a2350 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261970178 0 0) 0x7f764809a1d0 con 0x7f76480a3d10 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient(hunting): my global_id is 124853 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7648099ef0 con 0x7f76480a3d10 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3558134941 0 0) 0x7f76300035d0 con 0x7f76480a2350 2024-09-16T06:42:04.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 >> v1:172.21.15.76:6790/0 conn(0x7f76480a3d10 legacy=0x7f764813a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 >> v1:172.21.15.76:6789/0 conn(0x7f76480a27a0 legacy=0x7f76480a2b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 monclient: found mon.noname-c 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480a3290 con 0x7f76480a2350 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 5 monclient: authenticate success, global_id 134560 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7640001990 con 0x7f76480a2350 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f76477fe700 10 monclient: dump: 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.948+0000 7f764ec8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7640001bb0 con 0x7f76480a2350 2024-09-16T06:42:04.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 4 set_mon_vals no callback set 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 1 -- 172.21.15.189:0/1166241101 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7640001f40 con 0x7f76480a2350 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76477fe700 10 monclient: dump: 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.961 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: shutdown 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- 172.21.15.189:0/1166241101 >> v1:172.21.15.189:6789/0 conn(0x7f76480a2350 legacy=0x7f764813dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- 172.21.15.189:0/1166241101 shutdown_connections 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- 172.21.15.189:0/1166241101 >> 172.21.15.189:0/1166241101 conn(0x7f76480a7d70 msgr2=0x7f76480a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- 172.21.15.189:0/1166241101 shutdown_connections 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- 172.21.15.189:0/1166241101 wait complete. 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: build_initial_monmap 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: monmap: 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:04.962 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 Processor -- start 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- start start 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: init 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:04.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-b con 0x7f76480a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-c con 0x7f76480a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient(hunting): picked mon.noname-a con 0x7f76481cfee0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764809b520 con 0x7f76481cfee0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480a0060 con 0x7f76480a2350 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76481ca730 con 0x7f76480a27a0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ec8a700 10 monclient(hunting): authenticate will time out at 1077.467044s 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764d227700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f76481cfee0 legacy=0x7f76481d22d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ca26700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76480a2350 0x7f76481c9910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44288/0 (socket says 172.21.15.189:44288) 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f764ca26700 1 -- 172.21.15.189:0/788093530 learned_addr learned my addr 172.21.15.189:0/788093530 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501280575 0 0) 0x7f76480a0060 con 0x7f76480a2350 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): my global_id is 134563 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76340035d0 con 0x7f76480a2350 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904200640 0 0) 0x7f76481ca730 con 0x7f76480a27a0 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): my global_id is 124856 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76480a0060 con 0x7f76480a27a0 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063508167 0 0) 0x7f76340035d0 con 0x7f76480a2350 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76481ca730 con 0x7f76480a2350 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7638004230 con 0x7f76480a2350 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.952+0000 7f76457fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient(hunting): dump: 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1146651105 0 0) 0x7f76481ca730 con 0x7f76480a2350 2024-09-16T06:42:04.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 >> v1:172.21.15.76:6790/0 conn(0x7f76480a27a0 legacy=0x7f76481ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 >> v1:172.21.15.76:6789/0 conn(0x7f76481cfee0 legacy=0x7f76481d22d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: _finish_hunting 0 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 monclient: found mon.b 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764806bea0 con 0x7f76480a2350 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: _finish_auth 0 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: _check_auth_tickets 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 5 monclient: authenticate success, global_id 134563 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _renew_subs 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764813c230 con 0x7f76480a2350 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _renew_subs 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f764813d020 con 0x7f76480a2350 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7638004450 con 0x7f76480a2350 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7638004760 con 0x7f76480a2350 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 10 monclient: dump: 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:04.968 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f7646ffd700 4 set_mon_vals no callback set 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7638004940 con 0x7f76480a2350 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(94..94 src has 66..94) v4 ==== 5234+0+0 (unknown 4260897437 0 0) 0x7f7638043870 con 0x7f76480a2350 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f7647fff700 10 monclient: get_auth_request con 0x7f7634035fa0 auth_method 0 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:04.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.956+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628005100 con 0x7f76480a2350 2024-09-16T06:42:04.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.964+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7638014730 con 0x7f76480a2350 2024-09-16T06:42:04.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.964+0000 7f76457fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:04.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:04.964+0000 7f76457fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:05.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.140+0000 7f764ec8a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.140+0000 7f764ec8a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.140+0000 7f764ec8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.140+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f76280053b0 con 0x7f76480a2350 2024-09-16T06:42:05.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.316+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v95) v1 ==== 122+0+0 (unknown 3479235143 0 0) 0x7f7638012ba0 con 0x7f76480a2350 2024-09-16T06:42:05.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.316+0000 7f76457fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.316+0000 7f76457fa700 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T06:42:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.424+0000 7f764ec8a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.424+0000 7f764ec8a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.424+0000 7f764ec8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.424+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f76280023c0 con 0x7f76480a2350 2024-09-16T06:42:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f76457fa700 1 -- 172.21.15.189:0/788093530 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 157926631 0 0) 0x7f7638014320 con 0x7f76480a2350 2024-09-16T06:42:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f76457fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T06:42:05.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f76457fa700 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7634035fa0 legacy=0x7f7634038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 10 monclient: shutdown 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 >> v1:172.21.15.189:6789/0 conn(0x7f76480a2350 legacy=0x7f76481c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 shutdown_connections 2024-09-16T06:42:05.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 >> 172.21.15.189:0/788093530 conn(0x7f76480a7d70 msgr2=0x7f76480a05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 shutdown_connections 2024-09-16T06:42:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.428+0000 7f764ec8a700 1 -- 172.21.15.189:0/788093530 wait complete. 2024-09-16T06:42:05.431 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T06:42:05.440 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.9 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: get_monmap_and_config 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: build_initial_monmap 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: monmap: 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:05.546867+0000 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:05.546867+0000 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.546 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:05.547 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:05.547 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 1 Processor -- start 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 1 -- start start 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: init 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.544+0000 7f2701d8a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-b con 0x7f26fc099c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-c con 0x7f26fc135410 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-a con 0x7f26fc138f20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0998c0 con 0x7f26fc138f20 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc09bf60 con 0x7f26fc135410 2024-09-16T06:42:05.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc09d410 con 0x7f26fc099c80 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient(hunting): authenticate will time out at 1078.060105s 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701589700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f26fc138f20 legacy=0x7f26fc13b310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f26fc135410 0x7f26fc137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44302/0 (socket says 172.21.15.189:44302) 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fbfff700 1 -- 172.21.15.189:0/2072991928 learned_addr learned my addr 172.21.15.189:0/2072991928 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193112146 0 0) 0x7f26fc09bf60 con 0x7f26fc135410 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient(hunting): my global_id is 134566 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e40035d0 con 0x7f26fc135410 2024-09-16T06:42:05.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963510918 0 0) 0x7f26e40035d0 con 0x7f26fc135410 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 >> v1:172.21.15.76:6790/0 conn(0x7f26fc099c80 legacy=0x7f26fc09eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 >> v1:172.21.15.76:6789/0 conn(0x7f26fc138f20 legacy=0x7f26fc13b310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 monclient: found mon.noname-c 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc09c160 con 0x7f26fc135410 2024-09-16T06:42:05.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 5 monclient: authenticate success, global_id 134566 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26ec001610 con 0x7f26fc135410 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: dump: 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.552 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26ec001930 con 0x7f26fc135410 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 4 set_mon_vals no callback set 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 1 -- 172.21.15.189:0/2072991928 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26ec001cc0 con 0x7f26fc135410 2024-09-16T06:42:05.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f26fb7fe700 10 monclient: dump: 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient: shutdown 2024-09-16T06:42:05.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- 172.21.15.189:0/2072991928 >> v1:172.21.15.189:6789/0 conn(0x7f26fc135410 legacy=0x7f26fc137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- 172.21.15.189:0/2072991928 shutdown_connections 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- 172.21.15.189:0/2072991928 >> 172.21.15.189:0/2072991928 conn(0x7f26fc09fe70 msgr2=0x7f26fc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- 172.21.15.189:0/2072991928 shutdown_connections 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 -- 172.21.15.189:0/2072991928 wait complete. 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient: build_initial_monmap 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 10 monclient: monmap: 2024-09-16T06:42:05.560 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.548+0000 7f2701d8a700 1 Processor -- start 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- start start 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: init 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:05.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-c con 0x7f26fc099c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-a con 0x7f26fc135410 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient(hunting): picked mon.noname-b con 0x7f26fc138f20 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0707e0 con 0x7f26fc135410 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0983a0 con 0x7f26fc138f20 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc138ab0 con 0x7f26fc099c80 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient(hunting): authenticate will time out at 1078.065150s 2024-09-16T06:42:05.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26fbfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f26fc135410 legacy=0x7f26fc06ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701589700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f26fc138f20 0x7f26fc1ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44316/0 (socket says 172.21.15.189:44316) 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701589700 1 -- 172.21.15.189:0/280270645 learned_addr learned my addr 172.21.15.189:0/280270645 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404461066 0 0) 0x7f26fc0983a0 con 0x7f26fc138f20 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): my global_id is 134569 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e80035d0 con 0x7f26fc138f20 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225534956 0 0) 0x7f26fc138ab0 con 0x7f26fc099c80 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): my global_id is 124862 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26fc0983a0 con 0x7f26fc099c80 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705577125 0 0) 0x7f26e80035d0 con 0x7f26fc138f20 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26fc138ab0 con 0x7f26fc138f20 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26f4004390 con 0x7f26fc138f20 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): dump: 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.564 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560367149 0 0) 0x7f26fc138ab0 con 0x7f26fc138f20 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 >> v1:172.21.15.76:6790/0 conn(0x7f26fc099c80 legacy=0x7f26fc06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 >> v1:172.21.15.76:6789/0 conn(0x7f26fc135410 legacy=0x7f26fc06ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 monclient: found mon.b 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc09a380 con 0x7f26fc138f20 2024-09-16T06:42:05.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: _finish_auth 0 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 5 monclient: authenticate success, global_id 134569 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _renew_subs 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc06f7d0 con 0x7f26fc138f20 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _renew_subs 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc06f210 con 0x7f26fc138f20 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f4004600 con 0x7f26fc138f20 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f26f4004910 con 0x7f26fc138f20 2024-09-16T06:42:05.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26faffd700 4 set_mon_vals no callback set 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f26f97fa700 10 monclient: dump: 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.552+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc005100 con 0x7f26fc138f20 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.556+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f26f4004b30 con 0x7f26fc138f20 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.556+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(95..95 src has 66..95) v4 ==== 5188+0+0 (unknown 1496236569 0 0) 0x7f26f4043a40 con 0x7f26fc138f20 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.556+0000 7f2700d88700 10 monclient: get_auth_request con 0x7f26e80361d0 auth_method 0 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.564+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f26f4005350 con 0x7f26fc138f20 2024-09-16T06:42:05.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.564+0000 7f26f97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.564+0000 7f26f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.736+0000 7f2701d8a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.736+0000 7f2701d8a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.736+0000 7f2701d8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.736+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7f26dc0053b0 con 0x7f26fc138f20 2024-09-16T06:42:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.740+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 1574543391 0 0) 0x7f26f4012e00 con 0x7f26fc138f20 2024-09-16T06:42:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.740+0000 7f26f97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.740+0000 7f26f97fa700 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T06:42:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.852+0000 7f2701d8a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.852+0000 7f2701d8a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.852+0000 7f2701d8a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.852+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7f26dc005650 con 0x7f26fc138f20 2024-09-16T06:42:05.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f26f97fa700 1 -- 172.21.15.189:0/280270645 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v95) v1 ==== 124+0+0 (unknown 1574543391 0 0) 0x7f26f4014580 con 0x7f26fc138f20 2024-09-16T06:42:05.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f26f97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T06:42:05.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f26f97fa700 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T06:42:05.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 >> v1:172.21.15.189:6812/3098810674 conn(0x7f26e80361d0 legacy=0x7f26e8038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 10 monclient: shutdown 2024-09-16T06:42:05.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:05.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 >> v1:172.21.15.189:6789/0 conn(0x7f26fc138f20 legacy=0x7f26fc1ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 shutdown_connections 2024-09-16T06:42:05.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 >> 172.21.15.189:0/280270645 conn(0x7f26fc09fe70 msgr2=0x7f26fc137800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:05.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 shutdown_connections 2024-09-16T06:42:05.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.856+0000 7f2701d8a700 1 -- 172.21.15.189:0/280270645 wait complete. 2024-09-16T06:42:05.859 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T06:42:05.869 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm foo 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: get_monmap_and_config 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: build_initial_monmap 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: monmap: 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:05.972834+0000 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:05.972834+0000 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.972 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:05.973 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:05.973 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:05.973 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 1 Processor -- start 2024-09-16T06:42:05.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 1 -- start start 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: init 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-a con 0x7f2e3c0a04b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-b con 0x7f2e3c132880 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c09cb70 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c0a0890 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c09bea0 con 0x7f2e3c09cb70 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c131d60 con 0x7f2e3c132880 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient(hunting): authenticate will time out at 1078.485910s 2024-09-16T06:42:05.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e40c35700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2e3c09cb70 0x7f2e3c13a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44332/0 (socket says 172.21.15.189:44332) 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2e3c0a04b0 legacy=0x7f2e3c132170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e40c35700 1 -- 172.21.15.189:0/1765540683 learned_addr learned my addr 172.21.15.189:0/1765540683 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309201328 0 0) 0x7f2e3c09bea0 con 0x7f2e3c09cb70 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): my global_id is 134572 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e280035d0 con 0x7f2e3c09cb70 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699407170 0 0) 0x7f2e3c131d60 con 0x7f2e3c132880 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient(hunting): my global_id is 124865 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e3c09bea0 con 0x7f2e3c132880 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1424242720 0 0) 0x7f2e280035d0 con 0x7f2e3c09cb70 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 >> v1:172.21.15.76:6790/0 conn(0x7f2e3c132880 legacy=0x7f2e3c136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 >> v1:172.21.15.76:6789/0 conn(0x7f2e3c0a04b0 legacy=0x7f2e3c132170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 monclient: found mon.noname-c 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c071bb0 con 0x7f2e3c09cb70 2024-09-16T06:42:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: _finish_auth 0 2024-09-16T06:42:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:05.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 5 monclient: authenticate success, global_id 134572 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e34001990 con 0x7f2e3c09cb70 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: dump: 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.980 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e34001c70 con 0x7f2e3c09cb70 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 1 -- 172.21.15.189:0/1765540683 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e34002000 con 0x7f2e3c09cb70 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e41436700 4 set_mon_vals no callback set 2024-09-16T06:42:05.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.972+0000 7f2e3affd700 10 monclient: dump: 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: shutdown 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/1765540683 >> v1:172.21.15.189:6789/0 conn(0x7f2e3c09cb70 legacy=0x7f2e3c13a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/1765540683 shutdown_connections 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/1765540683 >> 172.21.15.189:0/1765540683 conn(0x7f2e3c0a3950 msgr2=0x7f2e3c09fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/1765540683 shutdown_connections 2024-09-16T06:42:05.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/1765540683 wait complete. 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: build_initial_monmap 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: monmap: 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:05.990 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 Processor -- start 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- start start 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: init 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-a con 0x7f2e3c09cb70 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:05.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-b con 0x7f2e3c0a04b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c132880 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c09a490 con 0x7f2e3c09cb70 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c11cad0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c11ddc0 con 0x7f2e3c132880 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient(hunting): authenticate will time out at 1078.489606s 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e3bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2e3c09cb70 legacy=0x7f2e3c11bcb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e3b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2e3c0a04b0 0x7f2e3c11c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44348/0 (socket says 172.21.15.189:44348) 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e3b7fe700 1 -- 172.21.15.189:0/2872080641 learned_addr learned my addr 172.21.15.189:0/2872080641 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603641415 0 0) 0x7f2e3c11cad0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): my global_id is 134575 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e1c0035d0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4223900388 0 0) 0x7f2e1c0035d0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e3c11cad0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e24001970 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.993 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107990791 0 0) 0x7f2e3c11ddc0 con 0x7f2e3c132880 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient(hunting): my global_id is 124868 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e1c0035d0 con 0x7f2e3c132880 2024-09-16T06:42:05.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108389958 0 0) 0x7f2e3c11cad0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 >> v1:172.21.15.76:6790/0 conn(0x7f2e3c132880 legacy=0x7f2e3c1cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 >> v1:172.21.15.76:6789/0 conn(0x7f2e3c09cb70 legacy=0x7f2e3c11bcb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 monclient: found mon.b 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c070340 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 5 monclient: authenticate success, global_id 134575 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _renew_subs 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c0a18c0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _renew_subs 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c11fc10 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e24001cc0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e24003b90 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e3a7fc700 4 set_mon_vals no callback set 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 10 monclient: dump: 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:05.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2e24003d70 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.976+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e20005100 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.980+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(95..95 src has 66..95) v4 ==== 5188+0+0 (unknown 1496236569 0 0) 0x7f2e240438e0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.980+0000 7f2e3bfff700 10 monclient: get_auth_request con 0x7f2e1c036180 auth_method 0 2024-09-16T06:42:05.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.984+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2e240146e0 con 0x7f2e3c0a04b0 2024-09-16T06:42:05.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.984+0000 7f2e38ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:05.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:05.984+0000 7f2e38ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:06.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.144+0000 7f2e41436700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.144+0000 7f2e41436700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.144+0000 7f2e41436700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.144+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f2e200053b0 con 0x7f2e3c0a04b0 2024-09-16T06:42:06.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.316+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v96) v1 ==== 120+0+0 (unknown 3939064113 0 0) 0x7f2e24012d80 con 0x7f2e3c0a04b0 2024-09-16T06:42:06.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.316+0000 7f2e38ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.316+0000 7f2e38ff9700 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T06:42:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.424+0000 7f2e41436700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.424+0000 7f2e41436700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.424+0000 7f2e41436700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.424+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f2e20005650 con 0x7f2e3c0a04b0 2024-09-16T06:42:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e38ff9700 1 -- 172.21.15.189:0/2872080641 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v96) v1 ==== 120+0+0 (unknown 3419787546 0 0) 0x7f2e24014500 con 0x7f2e3c0a04b0 2024-09-16T06:42:06.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e38ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T06:42:06.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e38ff9700 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2e1c036180 legacy=0x7f2e1c038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 10 monclient: shutdown 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 >> v1:172.21.15.189:6789/0 conn(0x7f2e3c0a04b0 legacy=0x7f2e3c11c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 shutdown_connections 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 >> 172.21.15.189:0/2872080641 conn(0x7f2e3c0a3950 msgr2=0x7f2e3c070700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 shutdown_connections 2024-09-16T06:42:06.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.428+0000 7f2e41436700 1 -- 172.21.15.189:0/2872080641 wait complete. 2024-09-16T06:42:06.432 INFO:tasks.workunit.client.0.smithi189.stderr:device 'foo' does not appear in the crush map 2024-09-16T06:42:06.442 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 10 monclient: get_monmap_and_config 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 10 monclient: build_initial_monmap 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 10 monclient: monmap: 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:06.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:06.543674+0000 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:06.543674+0000 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 1 Processor -- start 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 1 -- start start 2024-09-16T06:42:06.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.540+0000 7fa12408c700 10 monclient: init 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-b con 0x7fa11c09ed40 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-c con 0x7fa11c097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-a con 0x7fa11c09f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c099b10 con 0x7fa11c09f830 2024-09-16T06:42:06.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c098890 con 0x7fa11c097860 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c098b70 con 0x7fa11c09ed40 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): authenticate will time out at 1079.056357s 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa121627700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa11c097860 0x7fa11c1386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44358/0 (socket says 172.21.15.189:44358) 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa121627700 1 -- 172.21.15.189:0/2295586726 learned_addr learned my addr 172.21.15.189:0/2295586726 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa122629700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa11c09f830 legacy=0x7fa11c13be10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796219864 0 0) 0x7fa11c098890 con 0x7fa11c097860 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient(hunting): _negotiate 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:06.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient(hunting): my global_id is 134578 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1080035d0 con 0x7fa11c097860 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800761777 0 0) 0x7fa1080035d0 con 0x7fa11c097860 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 >> v1:172.21.15.76:6790/0 conn(0x7fa11c09ed40 legacy=0x7fa11c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 >> v1:172.21.15.76:6789/0 conn(0x7fa11c09f830 legacy=0x7fa11c13be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: _finish_hunting 0 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 monclient: found mon.noname-c 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0973a0 con 0x7fa11c097860 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: _finish_auth 0 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: _check_auth_tickets 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa110001610 con 0x7fa11c097860 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 5 monclient: authenticate success, global_id 134578 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:06.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: dump: 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:06.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa110001930 con 0x7fa11c097860 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 1 -- 172.21.15.189:0/2295586726 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa110001cc0 con 0x7fa11c097860 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 4 set_mon_vals no callback set 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa120e26700 10 monclient: dump: 2024-09-16T06:42:06.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: shutdown 2024-09-16T06:42:06.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- 172.21.15.189:0/2295586726 >> v1:172.21.15.189:6789/0 conn(0x7fa11c097860 legacy=0x7fa11c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- 172.21.15.189:0/2295586726 shutdown_connections 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- 172.21.15.189:0/2295586726 >> 172.21.15.189:0/2295586726 conn(0x7fa11c09fe30 msgr2=0x7fa11c0a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- 172.21.15.189:0/2295586726 shutdown_connections 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- 172.21.15.189:0/2295586726 wait complete. 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: build_initial_monmap 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: monmap: 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:06.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 Processor -- start 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- start start 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: init 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-b con 0x7fa11c097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-a con 0x7fa11c09ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): picked mon.noname-c con 0x7fa11c09f830 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c1302b0 con 0x7fa11c09ed40 2024-09-16T06:42:06.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c09a520 con 0x7fa11c097860 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c12ef70 con 0x7fa11c09f830 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.544+0000 7fa12408c700 10 monclient(hunting): authenticate will time out at 1079.059951s 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa121627700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa11c09ed40 legacy=0x7fa11c1cca50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa121e28700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa11c097860 0x7fa11c133800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44374/0 (socket says 172.21.15.189:44374) 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa121e28700 1 -- 172.21.15.189:0/937664818 learned_addr learned my addr 172.21.15.189:0/937664818 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059552569 0 0) 0x7fa11c09a520 con 0x7fa11c097860 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:06.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): my global_id is 134581 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1000035d0 con 0x7fa11c097860 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012149496 0 0) 0x7fa11c12ef70 con 0x7fa11c09f830 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): my global_id is 124874 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c09a520 con 0x7fa11c09f830 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908387852 0 0) 0x7fa1000035d0 con 0x7fa11c097860 2024-09-16T06:42:06.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa11c12ef70 con 0x7fa11c097860 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa118001930 con 0x7fa11c097860 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:06.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1228805809 0 0) 0x7fa11c12ef70 con 0x7fa11c097860 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 >> v1:172.21.15.76:6790/0 conn(0x7fa11c09f830 legacy=0x7fa11c1d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 >> v1:172.21.15.76:6789/0 conn(0x7fa11c09ed40 legacy=0x7fa11c1cca50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 monclient: found mon.b 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c12fa10 con 0x7fa11c097860 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:06.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 5 monclient: authenticate success, global_id 134581 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _renew_subs 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c12eab0 con 0x7fa11c097860 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _renew_subs 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c133f10 con 0x7fa11c097860 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa118001cc0 con 0x7fa11c097860 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa118003b10 con 0x7fa11c097860 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:06.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10ffff700 4 set_mon_vals no callback set 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 10 monclient: dump: 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:06.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104005100 con 0x7fa11c097860 2024-09-16T06:42:06.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.548+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa118003c70 con 0x7fa11c097860 2024-09-16T06:42:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.552+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(96..96 src has 66..96) v4 ==== 4890+0+0 (unknown 447951916 0 0) 0x7fa118043710 con 0x7fa11c097860 2024-09-16T06:42:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.552+0000 7fa121627700 10 monclient: get_auth_request con 0x7fa100036180 auth_method 0 2024-09-16T06:42:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.556+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa1180147b0 con 0x7fa11c097860 2024-09-16T06:42:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.556+0000 7fa10e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:06.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.556+0000 7fa10e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:06.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.740+0000 7fa12408c700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.740+0000 7fa12408c700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T06:42:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.740+0000 7fa12408c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:06.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:06.740+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa104005930 con 0x7fa11c097860 2024-09-16T06:42:07.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.315+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v97) v1 ==== 58+0+3 (unknown 2995259560 0 520826459) 0x7fa118012c20 con 0x7fa11c097860 2024-09-16T06:42:07.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.315+0000 7fa10e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T06:42:07.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.315+0000 7fa10e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.435+0000 7fa12408c700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.435+0000 7fa12408c700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T06:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.435+0000 7fa12408c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:07.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:07.435+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa104005420 con 0x7fa11c097860 2024-09-16T06:42:08.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.343+0000 7fa10e7fc700 1 -- 172.21.15.189:0/937664818 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v98) v1 ==== 58+0+3 (unknown 1761221957 0 212448812) 0x7fa118012c20 con 0x7fa11c097860 2024-09-16T06:42:08.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.343+0000 7fa10e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T06:42:08.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.343+0000 7fa10e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa100036180 legacy=0x7fa100038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 10 monclient: shutdown 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 >> v1:172.21.15.189:6789/0 conn(0x7fa11c097860 legacy=0x7fa11c133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 shutdown_connections 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 >> 172.21.15.189:0/937664818 conn(0x7fa11c09fe30 msgr2=0x7fa11c06f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 shutdown_connections 2024-09-16T06:42:08.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.347+0000 7fa12408c700 1 -- 172.21.15.189:0/937664818 wait complete. 2024-09-16T06:42:08.365 INFO:tasks.workunit.client.0.smithi189.stderr:+ o3=11 2024-09-16T06:42:08.365 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 11 123 root=default 2024-09-16T06:42:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: get_monmap_and_config 2024-09-16T06:42:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: build_initial_monmap 2024-09-16T06:42:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: monmap: 2024-09-16T06:42:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:08.477192+0000 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:08.477192+0000 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:08.477 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 1 Processor -- start 2024-09-16T06:42:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 1 -- start start 2024-09-16T06:42:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: init 2024-09-16T06:42:08.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:08.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:08.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-c con 0x7f9d1809d880 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:08.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-a con 0x7f9d1809b1d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-b con 0x7f9d18131950 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d181323d0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1809f480 con 0x7f9d1809d880 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d1809d880 0x7f9d1809e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44376/0 (socket says 172.21.15.189:44376) 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1659c700 1 -- 172.21.15.189:0/1722420194 learned_addr learned my addr 172.21.15.189:0/1722420194 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d0dd9b700 1 -- 172.21.15.189:0/1722420194 >> v1:172.21.15.76:6789/0 conn(0x7f9d1809b1d0 legacy=0x7f9d1809ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1809bb60 con 0x7f9d18131950 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.475+0000 7f9d1cf55700 10 monclient(hunting): authenticate will time out at 1080.991969s 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040079250 0 0) 0x7f9d1809f480 con 0x7f9d1809d880 2024-09-16T06:42:08.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): my global_id is 134584 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d040035d0 con 0x7f9d1809d880 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917621569 0 0) 0x7f9d1809bb60 con 0x7f9d18131950 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient(hunting): my global_id is 124877 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d1809f480 con 0x7f9d18131950 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072964433 0 0) 0x7f9d040035d0 con 0x7f9d1809d880 2024-09-16T06:42:08.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 >> v1:172.21.15.76:6790/0 conn(0x7f9d18131950 legacy=0x7f9d1813a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 >> v1:172.21.15.76:6789/0 conn(0x7f9d1809b1d0 legacy=0x7f9d1809ed70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 monclient: found mon.noname-c 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1809be40 con 0x7f9d1809d880 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: _finish_auth 0 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 5 monclient: authenticate success, global_id 134584 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:08.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d08001610 con 0x7f9d1809d880 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: dump: 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:08.486 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d080018f0 con 0x7f9d1809d880 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 4 set_mon_vals no callback set 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 1 -- 172.21.15.189:0/1722420194 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d08001c80 con 0x7f9d1809d880 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d15d9b700 10 monclient: dump: 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.487 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: shutdown 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 >> v1:172.21.15.189:6789/0 conn(0x7f9d1809d880 legacy=0x7f9d1809e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 shutdown_connections 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 >> 172.21.15.189:0/1722420194 conn(0x7f9d180a42f0 msgr2=0x7f9d180a0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 shutdown_connections 2024-09-16T06:42:08.488 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- 172.21.15.189:0/1722420194 wait complete. 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: build_initial_monmap 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: monmap: 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 Processor -- start 2024-09-16T06:42:08.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- start start 2024-09-16T06:42:08.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: init 2024-09-16T06:42:08.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-b con 0x7f9d1809b1d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-a con 0x7f9d1809d880 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient(hunting): picked mon.noname-c con 0x7f9d18131950 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1806b0d0 con 0x7f9d1809d880 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1811f200 con 0x7f9d1809b1d0 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1811f7c0 con 0x7f9d18131950 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1cf55700 10 monclient(hunting): authenticate will time out at 1080.995457s 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9d1809b1d0 0x7f9d1811b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:44386/0 (socket says 172.21.15.189:44386) 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d1659c700 1 -- 172.21.15.189:0/806993596 learned_addr learned my addr 172.21.15.189:0/806993596 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:08.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0dd9b700 1 -- 172.21.15.189:0/806993596 >> v1:172.21.15.76:6789/0 conn(0x7f9d1809d880 legacy=0x7f9d1811bf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713439137 0 0) 0x7f9d1811f200 con 0x7f9d1809b1d0 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.479+0000 7f9d0f7fe700 10 monclient(hunting): my global_id is 134587 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf80035d0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467067773 0 0) 0x7f9d1811f7c0 con 0x7f9d18131950 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): my global_id is 124880 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d1811f200 con 0x7f9d18131950 2024-09-16T06:42:08.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1463715900 0 0) 0x7f9cf80035d0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d1811f7c0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d08001c00 con 0x7f9d1809b1d0 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:08.495 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394797443 0 0) 0x7f9d1811f7c0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 >> v1:172.21.15.76:6790/0 conn(0x7f9d18131950 legacy=0x7f9d1811c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 >> v1:172.21.15.76:6789/0 conn(0x7f9d1809d880 legacy=0x7f9d1811bf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 monclient: found mon.b 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d181311c0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:08.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 5 monclient: authenticate success, global_id 134587 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _renew_subs 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1809ef10 con 0x7f9d1809b1d0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _renew_subs 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1811d3f0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d08004ca0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d08004fb0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1559a700 4 set_mon_vals no callback set 2024-09-16T06:42:08.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 10 monclient: dump: 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:08.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc005100 con 0x7f9d1809b1d0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d08005190 con 0x7f9d1809b1d0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(98..98 src has 66..98) v4 ==== 5120+0+0 (unknown 3247141499 0 0) 0x7f9d08043970 con 0x7f9d1809b1d0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.483+0000 7f9d0dd9b700 10 monclient: get_auth_request con 0x7f9cf8036180 auth_method 0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.491+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d08014ab0 con 0x7f9d1809b1d0 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.491+0000 7f9d0f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:08.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.491+0000 7f9d0f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:08.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.667+0000 7f9d1cf55700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:08.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.667+0000 7f9d1cf55700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:08.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.667+0000 7f9d1cf55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:08.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:08.667+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f9cfc005930 con 0x7f9d1809b1d0 2024-09-16T06:42:09.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.347+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v99) v1 ==== 193+0+0 (unknown 1204463796 0 0) 0x7f9d08012f20 con 0x7f9d1809b1d0 2024-09-16T06:42:09.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.347+0000 7f9d0f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:09.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.347+0000 7f9d0f7fe700 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-16T06:42:09.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d1cf55700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:09.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d1cf55700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:09.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d1cf55700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f9cfc0053b0 con 0x7f9d1809b1d0 2024-09-16T06:42:09.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d0f7fe700 1 -- 172.21.15.189:0/806993596 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v99) v1 ==== 191+0+0 (unknown 1675519427 0 0) 0x7f9d080146a0 con 0x7f9d1809b1d0 2024-09-16T06:42:09.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d0f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T06:42:09.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.459+0000 7f9d0f7fe700 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9cf8036180 legacy=0x7f9cf8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 10 monclient: shutdown 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 >> v1:172.21.15.189:6789/0 conn(0x7f9d1809b1d0 legacy=0x7f9d1811b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 shutdown_connections 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 >> 172.21.15.189:0/806993596 conn(0x7f9d180a42f0 msgr2=0x7f9d18099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 shutdown_connections 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.463+0000 7f9d1cf55700 1 -- 172.21.15.189:0/806993596 wait complete. 2024-09-16T06:42:09.465 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T06:42:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.11 2024-09-16T06:42:09.475 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T06:42:09.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: get_monmap_and_config 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: build_initial_monmap 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: monmap: 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:09.578617+0000 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:09.578617+0000 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:09.578 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:09.579 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 1 Processor -- start 2024-09-16T06:42:09.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 1 -- start start 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: init 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-b con 0x7f181c09ed40 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-a con 0x7f181c135bd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-c con 0x7f181c135fe0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c097960 con 0x7f181c135bd0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0710c0 con 0x7f181c135fe0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c09f120 con 0x7f181c09ed40 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f18234f6700 10 monclient(hunting): authenticate will time out at 1082.091241s 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f1820a91700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f181c135bd0 legacy=0x7f181c13a700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:09.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f1821a93700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f181c135fe0 0x7f181c13de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37000/0 (socket says 172.21.15.189:37000) 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.575+0000 7f1821a93700 1 -- 172.21.15.189:0/2361570062 learned_addr learned my addr 172.21.15.189:0/2361570062 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804146159 0 0) 0x7f181c0710c0 con 0x7f181c135fe0 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): my global_id is 134590 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18080035d0 con 0x7f181c135fe0 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120470762 0 0) 0x7f181c09f120 con 0x7f181c09ed40 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:09.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient(hunting): my global_id is 124883 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c0710c0 con 0x7f181c09ed40 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705245594 0 0) 0x7f18080035d0 con 0x7f181c135fe0 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 >> v1:172.21.15.76:6790/0 conn(0x7f181c09ed40 legacy=0x7f181c1354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 >> v1:172.21.15.76:6789/0 conn(0x7f181c135bd0 legacy=0x7f181c13a700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 monclient: found mon.noname-c 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c071d60 con 0x7f181c135fe0 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 5 monclient: authenticate success, global_id 134590 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1814001990 con 0x7f181c135fe0 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: dump: 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.586 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1814001bb0 con 0x7f181c135fe0 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 4 set_mon_vals no callback set 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 1 -- 172.21.15.189:0/2361570062 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1814001ec0 con 0x7f181c135fe0 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f1813fff700 10 monclient: dump: 2024-09-16T06:42:09.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 10 monclient: shutdown 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:09.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- 172.21.15.189:0/2361570062 >> v1:172.21.15.189:6789/0 conn(0x7f181c135fe0 legacy=0x7f181c13de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- 172.21.15.189:0/2361570062 shutdown_connections 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- 172.21.15.189:0/2361570062 >> 172.21.15.189:0/2361570062 conn(0x7f181c09fe30 msgr2=0x7f181c0a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- 172.21.15.189:0/2361570062 shutdown_connections 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- 172.21.15.189:0/2361570062 wait complete. 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 10 monclient: build_initial_monmap 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 10 monclient: monmap: 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:09.589 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 Processor -- start 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 1 -- start start 2024-09-16T06:42:09.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.579+0000 7f18234f6700 10 monclient: init 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-b con 0x7f181c09ed40 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-a con 0x7f181c135bd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient(hunting): picked mon.noname-c con 0x7f181c135fe0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c08f0e0 con 0x7f181c135bd0 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c09a520 con 0x7f181c09ed40 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0900d0 con 0x7f181c135fe0 2024-09-16T06:42:09.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1821292700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f181c09ed40 0x7f181c0964a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37010/0 (socket says 172.21.15.189:37010) 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1821292700 1 -- 172.21.15.189:0/1157090005 learned_addr learned my addr 172.21.15.189:0/1157090005 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient(hunting): authenticate will time out at 1082.096305s 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1820a91700 1 -- 172.21.15.189:0/1157090005 >> v1:172.21.15.76:6789/0 conn(0x7f181c135bd0 legacy=0x7f181c1cecb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255328644 0 0) 0x7f181c09a520 con 0x7f181c09ed40 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): my global_id is 134593 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18000035d0 con 0x7f181c09ed40 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282470239 0 0) 0x7f181c0900d0 con 0x7f181c135fe0 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:09.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): my global_id is 124886 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c09a520 con 0x7f181c135fe0 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4263471582 0 0) 0x7f18000035d0 con 0x7f181c09ed40 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f181c0900d0 con 0x7f181c09ed40 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1818001870 con 0x7f181c09ed40 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): dump: 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.596 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2022886220 0 0) 0x7f181c0900d0 con 0x7f181c09ed40 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 >> v1:172.21.15.76:6790/0 conn(0x7f181c135fe0 legacy=0x7f181c1d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 >> v1:172.21.15.76:6789/0 conn(0x7f181c135bd0 legacy=0x7f181c1cecb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 monclient: found mon.b 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:09.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0a1c90 con 0x7f181c09ed40 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 5 monclient: authenticate success, global_id 134593 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _renew_subs 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c090c50 con 0x7f181c09ed40 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _renew_subs 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c096bb0 con 0x7f181c09ed40 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18180036c0 con 0x7f181c09ed40 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1818003a50 con 0x7f181c09ed40 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18137fe700 4 set_mon_vals no callback set 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 10 monclient: dump: 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:09.599 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:09.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc005100 con 0x7f181c09ed40 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1818003bb0 con 0x7f181c09ed40 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(99..99 src has 66..99) v4 ==== 5142+0+0 (unknown 604494212 0 0) 0x7f1818043740 con 0x7f181c09ed40 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.583+0000 7f1820a91700 10 monclient: get_auth_request con 0x7f1800036220 auth_method 0 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.591+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1818004330 con 0x7f181c09ed40 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.591+0000 7f1811ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:09.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.591+0000 7f1811ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.767+0000 7f18234f6700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.767+0000 7f18234f6700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.767+0000 7f18234f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.767+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f17fc005bc0 con 0x7f181c09ed40 2024-09-16T06:42:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.771+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v99) v1 ==== 56+0+1173 (unknown 3011264473 0 2068243082) 0x7f181801ccb0 con 0x7f181c09ed40 2024-09-16T06:42:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.771+0000 7f1811ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:09.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.771+0000 7f1811ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.867+0000 7f18234f6700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.867+0000 7f18234f6700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.867+0000 7f18234f6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.867+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f17fc005680 con 0x7f181c09ed40 2024-09-16T06:42:09.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f1811ffb700 1 -- 172.21.15.189:0/1157090005 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v99) v1 ==== 56+0+1173 (unknown 3011264473 0 2068243082) 0x7f1818012ad0 con 0x7f181c09ed40 2024-09-16T06:42:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f1811ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:09.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f1811ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1800036220 legacy=0x7f18000386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 10 monclient: shutdown 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 >> v1:172.21.15.189:6789/0 conn(0x7f181c09ed40 legacy=0x7f181c0964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 shutdown_connections 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 >> 172.21.15.189:0/1157090005 conn(0x7f181c09fe30 msgr2=0x7f181c06fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 shutdown_connections 2024-09-16T06:42:09.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.871+0000 7f18234f6700 1 -- 172.21.15.189:0/1157090005 wait complete. 2024-09-16T06:42:09.883 INFO:tasks.workunit.client.0.smithi189.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-16T06:42:09.884 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight osd.11 113 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.991+0000 7fa774bdb700 10 monclient: get_monmap_and_config 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.991+0000 7fa774bdb700 10 monclient: build_initial_monmap 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.991+0000 7fa774bdb700 10 monclient: monmap: 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:09.996076+0000 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:09.996076+0000 2024-09-16T06:42:09.995 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:09.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:09.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:09.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:09.996 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:09.996 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 Processor -- start 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- start start 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: init 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-c con 0x7fa770099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:10.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-a con 0x7fa77009ada0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-b con 0x7fa77009b1b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7700998c0 con 0x7fa77009ada0 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77009bf60 con 0x7fa770099c80 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77009d410 con 0x7fa77009b1b0 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient(hunting): authenticate will time out at 1082.508982s 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa770099c80 0x7fa77009a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37012/0 (socket says 172.21.15.189:37012) 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa77009ada0 legacy=0x7fa770137320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76f7fe700 1 -- 172.21.15.189:0/4108829840 learned_addr learned my addr 172.21.15.189:0/4108829840 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021811328 0 0) 0x7fa77009bf60 con 0x7fa770099c80 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:10.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient(hunting): my global_id is 134596 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7580057e0 con 0x7fa770099c80 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1800844161 0 0) 0x7fa7580057e0 con 0x7fa770099c80 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 >> v1:172.21.15.76:6790/0 conn(0x7fa77009b1b0 legacy=0x7fa77013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 >> v1:172.21.15.76:6789/0 conn(0x7fa77009ada0 legacy=0x7fa770137320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77009c160 con 0x7fa770099c80 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:10.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 5 monclient: authenticate success, global_id 134596 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa760001610 con 0x7fa770099c80 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: dump: 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:10.008 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa760001930 con 0x7fa770099c80 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 1 -- 172.21.15.189:0/4108829840 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa760001cc0 con 0x7fa770099c80 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 4 set_mon_vals no callback set 2024-09-16T06:42:10.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa76e7fc700 10 monclient: dump: 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:10.010 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: shutdown 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- 172.21.15.189:0/4108829840 >> v1:172.21.15.189:6789/0 conn(0x7fa770099c80 legacy=0x7fa77009a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- 172.21.15.189:0/4108829840 shutdown_connections 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- 172.21.15.189:0/4108829840 >> 172.21.15.189:0/4108829840 conn(0x7fa77009fe70 msgr2=0x7fa7700a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- 172.21.15.189:0/4108829840 shutdown_connections 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 1 -- 172.21.15.189:0/4108829840 wait complete. 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: build_initial_monmap 2024-09-16T06:42:10.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.995+0000 7fa774bdb700 10 monclient: monmap: 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 Processor -- start 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- start start 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: init 2024-09-16T06:42:10.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-b con 0x7fa770099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-a con 0x7fa77009ada0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient(hunting): picked mon.noname-c con 0x7fa77009b1b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa770090bf0 con 0x7fa77009ada0 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa770098710 con 0x7fa770099c80 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77009ead0 con 0x7fa77009b1b0 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient(hunting): authenticate will time out at 1082.513316s 2024-09-16T06:42:10.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa76f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa770099c80 0x7fa770096440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37018/0 (socket says 172.21.15.189:37018) 2024-09-16T06:42:10.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa76f7fe700 1 -- 172.21.15.189:0/3300835932 learned_addr learned my addr 172.21.15.189:0/3300835932 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa76effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa77009ada0 legacy=0x7fa7701ca840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416955258 0 0) 0x7fa770098710 con 0x7fa770099c80 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): my global_id is 134599 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c0035d0 con 0x7fa770099c80 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1186381316 0 0) 0x7fa75c0035d0 con 0x7fa770099c80 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa770098710 con 0x7fa770099c80 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7600015d0 con 0x7fa770099c80 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:10.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): dump: 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111861563 0 0) 0x7fa77009ead0 con 0x7fa77009b1b0 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:10.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient(hunting): my global_id is 124892 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c0035d0 con 0x7fa77009b1b0 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1684986328 0 0) 0x7fa770098710 con 0x7fa770099c80 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 >> v1:172.21.15.76:6790/0 conn(0x7fa77009b1b0 legacy=0x7fa7701cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 >> v1:172.21.15.76:6789/0 conn(0x7fa77009ada0 legacy=0x7fa7701ca840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 monclient: found mon.b 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77009a380 con 0x7fa770099c80 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:10.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 5 monclient: authenticate success, global_id 134599 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _renew_subs 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa770090910 con 0x7fa770099c80 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa760004de0 con 0x7fa770099c80 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _renew_subs 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa770096b50 con 0x7fa770099c80 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa760005170 con 0x7fa770099c80 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:10.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 10 monclient: dump: 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa76dffb700 4 set_mon_vals no callback set 2024-09-16T06:42:10.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa760005350 con 0x7fa770099c80 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:09.999+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(99..99 src has 66..99) v4 ==== 5142+0+0 (unknown 604494212 0 0) 0x7fa760043a30 con 0x7fa770099c80 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.003+0000 7fa76effd700 10 monclient: get_auth_request con 0x7fa75c0361b0 auth_method 0 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.003+0000 7fa774bdb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.003+0000 7fa774bdb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.003+0000 7fa774bdb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.003+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7700705b0 con 0x7fa770099c80 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.007+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa760001150 con 0x7fa770099c80 2024-09-16T06:42:10.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.007+0000 7fa757fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:10.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.007+0000 7fa757fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.179+0000 7fa774bdb700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.179+0000 7fa774bdb700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.179+0000 7fa774bdb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.179+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fa77006a610 con 0x7fa770099c80 2024-09-16T06:42:10.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.355+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v100) v1 ==== 156+0+0 (unknown 1860376160 0 0) 0x7fa760014980 con 0x7fa770099c80 2024-09-16T06:42:10.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.355+0000 7fa757fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:10.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.355+0000 7fa757fff700 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T06:42:10.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.467+0000 7fa774bdb700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:10.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.467+0000 7fa774bdb700 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:10.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.467+0000 7fa774bdb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:10.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:10.467+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7fa770090710 con 0x7fa770099c80 2024-09-16T06:42:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.367+0000 7fa757fff700 1 -- 172.21.15.189:0/3300835932 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v101) v1 ==== 156+0+0 (unknown 1361214950 0 0) 0x7fa760014bb0 con 0x7fa770099c80 2024-09-16T06:42:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.367+0000 7fa757fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T06:42:11.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.367+0000 7fa757fff700 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa75c0361b0 legacy=0x7fa75c038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 10 monclient: shutdown 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 >> v1:172.21.15.189:6789/0 conn(0x7fa770099c80 legacy=0x7fa770096440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 shutdown_connections 2024-09-16T06:42:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 >> 172.21.15.189:0/3300835932 conn(0x7fa77009fe70 msgr2=0x7fa77006d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 shutdown_connections 2024-09-16T06:42:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.371+0000 7fa774bdb700 1 -- 172.21.15.189:0/3300835932 wait complete. 2024-09-16T06:42:11.374 INFO:tasks.workunit.client.0.smithi189.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T06:42:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-16T06:42:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:42:11.387 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.491+0000 7f111be35700 10 monclient: get_monmap_and_config 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.491+0000 7f111be35700 10 monclient: build_initial_monmap 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.491+0000 7f111be35700 10 monclient: monmap: 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:11.495888+0000 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:11.495888+0000 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:11.495 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:11.496 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:11.496 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 Processor -- start 2024-09-16T06:42:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- start start 2024-09-16T06:42:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient: init 2024-09-16T06:42:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:11.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient(hunting): picked mon.noname-a con 0x7f111409f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient(hunting): picked mon.noname-b con 0x7f1114099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient(hunting): picked mon.noname-c con 0x7f11141396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409f520 con 0x7f111409f140 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409bc10 con 0x7f11141396a0 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111409b810 con 0x7f1114099700 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient(hunting): authenticate will time out at 1084.008787s 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1119bd1700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f111409f140 legacy=0x7f111409a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111a3d2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f11141396a0 0x7f111413ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37028/0 (socket says 172.21.15.189:37028) 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111a3d2700 1 -- 172.21.15.189:0/2970481237 learned_addr learned my addr 172.21.15.189:0/2970481237 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:11.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184332666 0 0) 0x7f111409bc10 con 0x7f11141396a0 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): my global_id is 134602 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10fc0035d0 con 0x7f11141396a0 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783991617 0 0) 0x7f111409b810 con 0x7f1114099700 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient(hunting): my global_id is 124895 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111409bc10 con 0x7f1114099700 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1120906313 0 0) 0x7f10fc0035d0 con 0x7f11141396a0 2024-09-16T06:42:11.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 >> v1:172.21.15.76:6790/0 conn(0x7f1114099700 legacy=0x7f111409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 >> v1:172.21.15.76:6789/0 conn(0x7f111409f140 legacy=0x7f111409a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: _finish_hunting 0 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 monclient: found mon.noname-c 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111409ceb0 con 0x7f11141396a0 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: _finish_auth 0 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: _check_auth_tickets 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f110c001990 con 0x7f11141396a0 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 5 monclient: authenticate success, global_id 134602 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:11.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: dump: 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f110c001c70 con 0x7f11141396a0 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 1 -- 172.21.15.189:0/2970481237 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f110c002000 con 0x7f11141396a0 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 4 set_mon_vals no callback set 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f1118bcf700 10 monclient: dump: 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.505 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.506 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 10 monclient: shutdown 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- 172.21.15.189:0/2970481237 >> v1:172.21.15.189:6789/0 conn(0x7f11141396a0 legacy=0x7f111413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- 172.21.15.189:0/2970481237 shutdown_connections 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- 172.21.15.189:0/2970481237 >> 172.21.15.189:0/2970481237 conn(0x7f11140a03e0 msgr2=0x7f11140a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- 172.21.15.189:0/2970481237 shutdown_connections 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.495+0000 7f111be35700 1 -- 172.21.15.189:0/2970481237 wait complete. 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: build_initial_monmap 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: monmap: 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:11.507 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 Processor -- start 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- start start 2024-09-16T06:42:11.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: init 2024-09-16T06:42:11.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient(hunting): picked mon.noname-b con 0x7f1114099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient(hunting): picked mon.noname-a con 0x7f111409f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient(hunting): picked mon.noname-c con 0x7f11141396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1114094ba0 con 0x7f111409f140 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1114071260 con 0x7f1114099700 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11140a1410 con 0x7f11141396a0 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:11.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient(hunting): authenticate will time out at 1084.013688s 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11193d0700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f111409f140 legacy=0x7f11141ce870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f1119bd1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1114099700 0x7f1114096520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37030/0 (socket says 172.21.15.189:37030) 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f1119bd1700 1 -- 172.21.15.189:0/1913128529 learned_addr learned my addr 172.21.15.189:0/1913128529 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228671861 0 0) 0x7f1114071260 con 0x7f1114099700 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): my global_id is 134605 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11000035d0 con 0x7f1114099700 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662419220 0 0) 0x7f11140a1410 con 0x7f11141396a0 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): my global_id is 124898 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1114071260 con 0x7f11141396a0 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336653964 0 0) 0x7f11000035d0 con 0x7f1114099700 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11140a1410 con 0x7f1114099700 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1104004230 con 0x7f1114099700 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): dump: 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.515 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136891439 0 0) 0x7f1114071260 con 0x7f11141396a0 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11000035d0 con 0x7f11141396a0 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f110c0017b0 con 0x7f11141396a0 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient(hunting): dump: 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.516 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2831249336 0 0) 0x7f11140a1410 con 0x7f1114099700 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 >> v1:172.21.15.76:6790/0 conn(0x7f11141396a0 legacy=0x7f11141d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 >> v1:172.21.15.76:6789/0 conn(0x7f111409f140 legacy=0x7f11141ce870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 monclient: found mon.b 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:11.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111409a6e0 con 0x7f1114099700 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 5 monclient: authenticate success, global_id 134605 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _renew_subs 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11140948c0 con 0x7f1114099700 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _renew_subs 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f111be35700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1114092840 con 0x7f1114099700 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11040044a0 con 0x7f1114099700 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:11.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1104004830 con 0x7f1114099700 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11127fc700 10 monclient: dump: 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.499+0000 7f11137fe700 4 set_mon_vals no callback set 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f111be35700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f111be35700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f111be35700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f111be35700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f4005100 con 0x7f1114099700 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1104004a90 con 0x7f1114099700 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 66..101) v4 ==== 5171+0+0 (unknown 437838425 0 0) 0x7f1104043770 con 0x7f1114099700 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.503+0000 7f11193d0700 10 monclient: get_auth_request con 0x7f11000362a0 auth_method 0 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.507+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1104014830 con 0x7f1114099700 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.507+0000 7f11127fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.507+0000 7f11127fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:11.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.679+0000 7f111be35700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T06:42:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f111be35700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T06:42:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f111be35700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f111be35700 1 -- 172.21.15.189:0/1913128529 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) v1 -- 0x7f10f4005930 con 0x7f1114099700 2024-09-16T06:42:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f11127fc700 1 -- 172.21.15.189:0/1913128529 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v101) v1 ==== 146+0+0 (unknown 1716379202 0 0) 0x7f110401ce20 con 0x7f1114099700 2024-09-16T06:42:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f11127fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T06:42:11.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.683+0000 7f11127fc700 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T06:42:11.688 INFO:tasks.workunit.client.0.smithi189.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T06:42:11.692 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:42:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.11 2024-09-16T06:42:11.693 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 113 2024-09-16T06:42:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.787+0000 7f60d981d700 10 monclient: get_monmap_and_config 2024-09-16T06:42:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.787+0000 7f60d981d700 10 monclient: build_initial_monmap 2024-09-16T06:42:11.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.787+0000 7f60d981d700 10 monclient: monmap: 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:11.792270+0000 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:11.792270+0000 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:11.792 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 1 Processor -- start 2024-09-16T06:42:11.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 1 -- start start 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient: init 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-b con 0x7f60d406b760 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-c con 0x7f60d4135b50 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-a con 0x7f60d4135f30 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d406bb40 con 0x7f60d4135f30 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d406c310 con 0x7f60d4135b50 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d406ac80 con 0x7f60d406b760 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient(hunting): authenticate will time out at 1084.305645s 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f60d4135f30 legacy=0x7f60d413de90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:11.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f60d4135b50 0x7f60d413a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37036/0 (socket says 172.21.15.189:37036) 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d27fc700 1 -- 172.21.15.189:0/604010177 learned_addr learned my addr 172.21.15.189:0/604010177 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088881311 0 0) 0x7f60d406c310 con 0x7f60d4135b50 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): my global_id is 134608 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60bc0035d0 con 0x7f60d4135b50 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071148273 0 0) 0x7f60d406ac80 con 0x7f60d406b760 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient(hunting): my global_id is 124901 2024-09-16T06:42:11.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d406c310 con 0x7f60d406b760 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027489672 0 0) 0x7f60bc0035d0 con 0x7f60d4135b50 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 >> v1:172.21.15.76:6790/0 conn(0x7f60d406b760 legacy=0x7f60d4135440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 >> v1:172.21.15.76:6789/0 conn(0x7f60d4135f30 legacy=0x7f60d413de90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 monclient: found mon.noname-c 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d406af60 con 0x7f60d4135b50 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:11.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 5 monclient: authenticate success, global_id 134608 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f60c8001610 con 0x7f60d4135b50 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: dump: 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.799 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60c80018f0 con 0x7f60d4135b50 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 1 -- 172.21.15.189:0/604010177 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f60c8001c80 con 0x7f60d4135b50 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d981d700 4 set_mon_vals no callback set 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.791+0000 7f60d1ffb700 10 monclient: dump: 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.801 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 10 monclient: shutdown 2024-09-16T06:42:11.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:11.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- 172.21.15.189:0/604010177 >> v1:172.21.15.189:6789/0 conn(0x7f60d4135b50 legacy=0x7f60d413a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- 172.21.15.189:0/604010177 shutdown_connections 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- 172.21.15.189:0/604010177 >> 172.21.15.189:0/604010177 conn(0x7f60d40a4660 msgr2=0x7f60d40a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- 172.21.15.189:0/604010177 shutdown_connections 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- 172.21.15.189:0/604010177 wait complete. 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 10 monclient: build_initial_monmap 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 10 monclient: monmap: 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:11.806 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 Processor -- start 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 1 -- start start 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.795+0000 7f60d981d700 10 monclient: init 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-c con 0x7f60d406b760 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-b con 0x7f60d4135b50 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:11.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient(hunting): picked mon.noname-a con 0x7f60d4135f30 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d412f3f0 con 0x7f60d4135f30 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d409e930 con 0x7f60d4135b50 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d40a71e0 con 0x7f60d406b760 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f60d4135f30 legacy=0x7f60d41d1cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient(hunting): authenticate will time out at 1084.312784s 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f60d4135b50 0x7f60d4132940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37050/0 (socket says 172.21.15.189:37050) 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d27fc700 1 -- 172.21.15.189:0/4010494680 learned_addr learned my addr 172.21.15.189:0/4010494680 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653884163 0 0) 0x7f60d409e930 con 0x7f60d4135b50 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): my global_id is 134611 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c00035d0 con 0x7f60d4135b50 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695112738 0 0) 0x7f60d40a71e0 con 0x7f60d406b760 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): my global_id is 124904 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d409e930 con 0x7f60d406b760 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723272160 0 0) 0x7f60c00035d0 con 0x7f60d4135b50 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60d40a71e0 con 0x7f60d4135b50 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f60c80015d0 con 0x7f60d4135b50 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): dump: 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1775761955 0 0) 0x7f60d40a71e0 con 0x7f60d4135b50 2024-09-16T06:42:11.810 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 >> v1:172.21.15.76:6790/0 conn(0x7f60d406b760 legacy=0x7f60d41c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 >> v1:172.21.15.76:6789/0 conn(0x7f60d4135f30 legacy=0x7f60d41d1cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 monclient: found mon.b 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40a6be0 con 0x7f60d4135b50 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: _finish_auth 0 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 5 monclient: authenticate success, global_id 134611 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _renew_subs 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d409e4d0 con 0x7f60d4135b50 2024-09-16T06:42:11.811 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _renew_subs 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4133230 con 0x7f60d4135b50 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60c8004ca0 con 0x7f60d4135b50 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d17fa700 4 set_mon_vals no callback set 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f60c8004fb0 con 0x7f60d4135b50 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 10 monclient: dump: 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:11.812 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f60c800fa20 con 0x7f60d4135b50 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 66..101) v4 ==== 5171+0+0 (unknown 437838425 0 0) 0x7f60c8043a50 con 0x7f60d4135b50 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d2ffd700 10 monclient: get_auth_request con 0x7f60c0035fa0 auth_method 0 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:11.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.799+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b4005100 con 0x7f60d4135b50 2024-09-16T06:42:11.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.807+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f60c8014ad0 con 0x7f60d4135b50 2024-09-16T06:42:11.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.807+0000 7f60bbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:11.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.807+0000 7f60bbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:11.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60d981d700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60d981d700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60d981d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f60b4005bc0 con 0x7f60d4135b50 2024-09-16T06:42:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v101) v1 ==== 56+0+1173 (unknown 4176898266 0 1017396475) 0x7f60c8012f40 con 0x7f60d4135b50 2024-09-16T06:42:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60bbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:11.975+0000 7f60bbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.067+0000 7f60d981d700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.067+0000 7f60d981d700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.067+0000 7f60d981d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.067+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f60b4005680 con 0x7f60d4135b50 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60bbfff700 1 -- 172.21.15.189:0/4010494680 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v101) v1 ==== 56+0+1173 (unknown 4176898266 0 1017396475) 0x7f60c80146c0 con 0x7f60d4135b50 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60bbfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60bbfff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 >> v1:172.21.15.189:6812/3098810674 conn(0x7f60c0035fa0 legacy=0x7f60c0038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 10 monclient: shutdown 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 >> v1:172.21.15.189:6789/0 conn(0x7f60d4135b50 legacy=0x7f60d4132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 shutdown_connections 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 >> 172.21.15.189:0/4010494680 conn(0x7f60d40a4660 msgr2=0x7f60d412bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 shutdown_connections 2024-09-16T06:42:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.071+0000 7f60d981d700 1 -- 172.21.15.189:0/4010494680 wait complete. 2024-09-16T06:42:12.093 INFO:tasks.workunit.client.0.smithi189.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-16T06:42:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.11 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.187+0000 7fb2339ac700 10 monclient: get_monmap_and_config 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.187+0000 7fb2339ac700 10 monclient: build_initial_monmap 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.187+0000 7fb2339ac700 10 monclient: monmap: 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:12.191 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:12.192237+0000 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:12.192237+0000 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 1 Processor -- start 2024-09-16T06:42:12.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 1 -- start start 2024-09-16T06:42:12.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient: init 2024-09-16T06:42:12.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-c con 0x7fb22c133380 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-b con 0x7fb22c09f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-a con 0x7fb22c0a1190 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c132ec0 con 0x7fb22c0a1190 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c09e4c0 con 0x7fb22c133380 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c09e7a0 con 0x7fb22c09f760 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient(hunting): authenticate will time out at 1084.706432s 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb231748700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb22c133380 0x7fb22c133760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37062/0 (socket says 172.21.15.189:37062) 2024-09-16T06:42:12.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb231748700 1 -- 172.21.15.189:0/3737356101 learned_addr learned my addr 172.21.15.189:0/3737356101 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb231f49700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb22c0a1190 legacy=0x7fb22c13b970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098608378 0 0) 0x7fb22c09e4c0 con 0x7fb22c133380 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): my global_id is 134614 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2180035d0 con 0x7fb22c133380 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717450750 0 0) 0x7fb22c09e7a0 con 0x7fb22c09f760 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient(hunting): my global_id is 124907 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c09e4c0 con 0x7fb22c09f760 2024-09-16T06:42:12.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011568702 0 0) 0x7fb2180035d0 con 0x7fb22c133380 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 >> v1:172.21.15.76:6790/0 conn(0x7fb22c09f760 legacy=0x7fb22c138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 >> v1:172.21.15.76:6789/0 conn(0x7fb22c0a1190 legacy=0x7fb22c13b970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 monclient: found mon.noname-c 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0a1cb0 con 0x7fb22c133380 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 5 monclient: authenticate success, global_id 134614 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb228001610 con 0x7fb22c133380 2024-09-16T06:42:12.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: dump: 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2280018f0 con 0x7fb22c133380 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 1 -- 172.21.15.189:0/3737356101 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb228001c80 con 0x7fb22c133380 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb2339ac700 4 set_mon_vals no callback set 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.191+0000 7fb223fff700 10 monclient: dump: 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.201 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: shutdown 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- 172.21.15.189:0/3737356101 >> v1:172.21.15.189:6789/0 conn(0x7fb22c133380 legacy=0x7fb22c133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- 172.21.15.189:0/3737356101 shutdown_connections 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- 172.21.15.189:0/3737356101 >> 172.21.15.189:0/3737356101 conn(0x7fb22c0a21f0 msgr2=0x7fb22c0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- 172.21.15.189:0/3737356101 shutdown_connections 2024-09-16T06:42:12.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- 172.21.15.189:0/3737356101 wait complete. 2024-09-16T06:42:12.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: build_initial_monmap 2024-09-16T06:42:12.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: monmap: 2024-09-16T06:42:12.207 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:12.207 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 Processor -- start 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- start start 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: init 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:12.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-b con 0x7fb22c09f760 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-a con 0x7fb22c0a1190 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient(hunting): picked mon.noname-c con 0x7fb22c133380 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c072250 con 0x7fb22c0a1190 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c06f390 con 0x7fb22c09f760 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c12b630 con 0x7fb22c133380 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.195+0000 7fb2339ac700 10 monclient(hunting): authenticate will time out at 1084.711917s 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb230f47700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fb22c0a1190 legacy=0x7fb22c12af20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb231748700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb22c09f760 0x7fb22c12a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37066/0 (socket says 172.21.15.189:37066) 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb231748700 1 -- 172.21.15.189:0/2834777660 learned_addr learned my addr 172.21.15.189:0/2834777660 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459120546 0 0) 0x7fb22c06f390 con 0x7fb22c09f760 2024-09-16T06:42:12.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): my global_id is 134617 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2100035d0 con 0x7fb22c09f760 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381924334 0 0) 0x7fb22c12b630 con 0x7fb22c133380 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): my global_id is 124910 2024-09-16T06:42:12.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c06f390 con 0x7fb22c133380 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 9523063 0 0) 0x7fb2100035d0 con 0x7fb22c09f760 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22c12b630 con 0x7fb22c09f760 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2280015d0 con 0x7fb22c09f760 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): dump: 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:12.211 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3053866789 0 0) 0x7fb22c12b630 con 0x7fb22c09f760 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 >> v1:172.21.15.76:6790/0 conn(0x7fb22c133380 legacy=0x7fb22c1d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 >> v1:172.21.15.76:6789/0 conn(0x7fb22c0a1190 legacy=0x7fb22c12af20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 monclient: found mon.b 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c12c7e0 con 0x7fb22c09f760 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 5 monclient: authenticate success, global_id 134617 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _renew_subs 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c0a4fa0 con 0x7fb22c09f760 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _renew_subs 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c06f0b0 con 0x7fb22c09f760 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb228004ca0 con 0x7fb22c09f760 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:12.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb228005030 con 0x7fb22c09f760 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2237fe700 4 set_mon_vals no callback set 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 10 monclient: dump: 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:12.214 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c005100 con 0x7fb22c09f760 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb228005210 con 0x7fb22c09f760 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(101..101 src has 66..101) v4 ==== 5171+0+0 (unknown 437838425 0 0) 0x7fb228043a10 con 0x7fb22c09f760 2024-09-16T06:42:12.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.199+0000 7fb230f47700 10 monclient: get_auth_request con 0x7fb210036180 auth_method 0 2024-09-16T06:42:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.207+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb228014bf0 con 0x7fb22c09f760 2024-09-16T06:42:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.207+0000 7fb221ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:12.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.207+0000 7fb221ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:12.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.383+0000 7fb2339ac700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:12.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.383+0000 7fb2339ac700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:12.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.383+0000 7fb2339ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:12.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:12.383+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7fb20c0053b0 con 0x7fb22c09f760 2024-09-16T06:42:13.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.387+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v103) v1 ==== 125+0+0 (unknown 87793557 0 0) 0x7fb228013060 con 0x7fb22c09f760 2024-09-16T06:42:13.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.387+0000 7fb221ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:13.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.387+0000 7fb221ffb700 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-16T06:42:13.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.495+0000 7fb2339ac700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:13.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.495+0000 7fb2339ac700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:13.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.495+0000 7fb2339ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.499+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7fb20c005650 con 0x7fb22c09f760 2024-09-16T06:42:13.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.499+0000 7fb221ffb700 1 -- 172.21.15.189:0/2834777660 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v103) v1 ==== 126+0+0 (unknown 1997963734 0 0) 0x7fb2280147e0 con 0x7fb22c09f760 2024-09-16T06:42:13.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.499+0000 7fb221ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T06:42:13.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.499+0000 7fb221ffb700 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb210036180 legacy=0x7fb210038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 10 monclient: shutdown 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 >> v1:172.21.15.189:6789/0 conn(0x7fb22c09f760 legacy=0x7fb22c12a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 shutdown_connections 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 >> 172.21.15.189:0/2834777660 conn(0x7fb22c0a21f0 msgr2=0x7fb22c09b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 shutdown_connections 2024-09-16T06:42:13.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.503+0000 7fb2339ac700 1 -- 172.21.15.189:0/2834777660 wait complete. 2024-09-16T06:42:13.506 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.11' does not appear in the crush map 2024-09-16T06:42:13.517 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.11 2024-09-16T06:42:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: get_monmap_and_config 2024-09-16T06:42:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: build_initial_monmap 2024-09-16T06:42:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: monmap: 2024-09-16T06:42:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:13.616 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:13.617136+0000 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:13.617136+0000 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:13.617 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 1 Processor -- start 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 1 -- start start 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: init 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-a con 0x7fd5a809f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-b con 0x7fd5a8099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-c con 0x7fd5a81396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a809f520 con 0x7fd5a809f140 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a809bc10 con 0x7fd5a81396a0 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a809b810 con 0x7fd5a8099700 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:13.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient(hunting): authenticate will time out at 1086.130081s 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5ae4ec700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5a81396a0 0x7fd5a813ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37068/0 (socket says 172.21.15.189:37068) 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5ae4ec700 1 -- 172.21.15.189:0/2008931191 learned_addr learned my addr 172.21.15.189:0/2008931191 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5adceb700 1 -- 172.21.15.189:0/2008931191 >> v1:172.21.15.76:6789/0 conn(0x7fd5a809f140 legacy=0x7fd5a809a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832571855 0 0) 0x7fd5a809bc10 con 0x7fd5a81396a0 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient(hunting): my global_id is 134620 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5940035d0 con 0x7fd5a81396a0 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314130450 0 0) 0x7fd5940035d0 con 0x7fd5a81396a0 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 >> v1:172.21.15.76:6790/0 conn(0x7fd5a8099700 legacy=0x7fd5a809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 >> v1:172.21.15.76:6789/0 conn(0x7fd5a809f140 legacy=0x7fd5a809a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:13.622 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 monclient: found mon.noname-c 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a809ceb0 con 0x7fd5a81396a0 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: _finish_auth 0 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 5 monclient: authenticate success, global_id 134620 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd5a0001610 con 0x7fd5a81396a0 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: dump: 2024-09-16T06:42:13.623 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:13.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a0001930 con 0x7fd5a81396a0 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 1 -- 172.21.15.189:0/2008931191 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd5a0001cc0 con 0x7fd5a81396a0 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5aff4f700 4 set_mon_vals no callback set 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.615+0000 7fd5acce9700 10 monclient: dump: 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.625 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: shutdown 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- 172.21.15.189:0/2008931191 >> v1:172.21.15.189:6789/0 conn(0x7fd5a81396a0 legacy=0x7fd5a813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- 172.21.15.189:0/2008931191 shutdown_connections 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- 172.21.15.189:0/2008931191 >> 172.21.15.189:0/2008931191 conn(0x7fd5a80a03e0 msgr2=0x7fd5a80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- 172.21.15.189:0/2008931191 shutdown_connections 2024-09-16T06:42:13.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- 172.21.15.189:0/2008931191 wait complete. 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: build_initial_monmap 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: monmap: 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:13.631 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 Processor -- start 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- start start 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: init 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-c con 0x7fd5a8099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-a con 0x7fd5a809f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient(hunting): picked mon.noname-b con 0x7fd5a81396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a8090bd0 con 0x7fd5a809f140 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a8098ef0 con 0x7fd5a81396a0 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a808f4b0 con 0x7fd5a8099700 2024-09-16T06:42:13.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5aff4f700 10 monclient(hunting): authenticate will time out at 1086.135820s 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5ad4ea700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd5a809f140 legacy=0x7fd5a81ca830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5ae4ec700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd5a81396a0 0x7fd5a81cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37084/0 (socket says 172.21.15.189:37084) 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5adceb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd5a8099700 0x7fd5a8096120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:44690/0 (socket says 172.21.15.189:44690) 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.619+0000 7fd5ae4ec700 1 -- 172.21.15.189:0/568508340 learned_addr learned my addr 172.21.15.189:0/568508340 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882164552 0 0) 0x7fd5a808f4b0 con 0x7fd5a8099700 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): my global_id is 124916 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c0035d0 con 0x7fd5a8099700 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485782840 0 0) 0x7fd5a8098ef0 con 0x7fd5a81396a0 2024-09-16T06:42:13.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): my global_id is 134623 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a808f4b0 con 0x7fd5a81396a0 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170046252 0 0) 0x7fd5a808f4b0 con 0x7fd5a81396a0 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a8098ef0 con 0x7fd5a81396a0 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749163004 0 0) 0x7fd58c0035d0 con 0x7fd5a8099700 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a808f4b0 con 0x7fd5a8099700 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd5a0001430 con 0x7fd5a81396a0 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:13.634 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd5a4004520 con 0x7fd5a8099700 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:13.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2472256544 0 0) 0x7fd5a8098ef0 con 0x7fd5a81396a0 2024-09-16T06:42:13.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 >> v1:172.21.15.76:6790/0 conn(0x7fd5a8099700 legacy=0x7fd5a8096120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 >> v1:172.21.15.76:6789/0 conn(0x7fd5a809f140 legacy=0x7fd5a81ca830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 monclient: found mon.b 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80a2710 con 0x7fd5a81396a0 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 5 monclient: authenticate success, global_id 134623 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _renew_subs 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a80908f0 con 0x7fd5a81396a0 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _renew_subs 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a808f1c0 con 0x7fd5a81396a0 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a0004cf0 con 0x7fd5a81396a0 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd5a0001660 con 0x7fd5a81396a0 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59ffff700 4 set_mon_vals no callback set 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 10 monclient: dump: 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd5a000f910 con 0x7fd5a81396a0 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd590005100 con 0x7fd5a81396a0 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(103..103 src has 66..103) v4 ==== 5149+0+0 (unknown 4241378393 0 0) 0x7fd5a00438e0 con 0x7fd5a81396a0 2024-09-16T06:42:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.623+0000 7fd5adceb700 10 monclient: get_auth_request con 0x7fd58c0362b0 auth_method 0 2024-09-16T06:42:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.631+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd5a0011470 con 0x7fd5a81396a0 2024-09-16T06:42:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.631+0000 7fd59e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:13.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.631+0000 7fd59e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:13.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.803+0000 7fd5aff4f700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:13.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.803+0000 7fd5aff4f700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:13.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.803+0000 7fd5aff4f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:13.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:13.803+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7fd590005930 con 0x7fd5a81396a0 2024-09-16T06:42:13.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:13.806+0000 7f2941141700 -1 mon.c@2(leader).osd e103 definitely_dead 0 2024-09-16T06:42:14.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.399+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v104) v1 ==== 87+0+0 (unknown 630198649 0 0) 0x7fd5a0012ef0 con 0x7fd5a81396a0 2024-09-16T06:42:14.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.399+0000 7fd59e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:14.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.399+0000 7fd59e7fc700 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-16T06:42:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd5aff4f700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd5aff4f700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd5aff4f700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7fd590005420 con 0x7fd5a81396a0 2024-09-16T06:42:14.519 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:14.518+0000 7f2941141700 -1 mon.c@2(leader).osd e104 definitely_dead 0 2024-09-16T06:42:14.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd59e7fc700 1 -- 172.21.15.189:0/568508340 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v104) v1 ==== 96+0+0 (unknown 1800114499 0 0) 0x7fd5a0014a80 con 0x7fd5a81396a0 2024-09-16T06:42:14.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd59e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T06:42:14.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.515+0000 7fd59e7fc700 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-16T06:42:14.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd58c0362b0 legacy=0x7fd58c038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:14.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 10 monclient: shutdown 2024-09-16T06:42:14.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:14.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 >> v1:172.21.15.189:6789/0 conn(0x7fd5a81396a0 legacy=0x7fd5a81cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:14.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 shutdown_connections 2024-09-16T06:42:14.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 >> 172.21.15.189:0/568508340 conn(0x7fd5a80a03e0 msgr2=0x7fd5a80a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:14.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 shutdown_connections 2024-09-16T06:42:14.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.519+0000 7fd5aff4f700 1 -- 172.21.15.189:0/568508340 wait complete. 2024-09-16T06:42:14.524 INFO:tasks.workunit.client.0.smithi189.stderr:osd.11 does not exist. 2024-09-16T06:42:14.536 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T06:42:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.627+0000 7fc39a0f4700 10 monclient: get_monmap_and_config 2024-09-16T06:42:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.627+0000 7fc39a0f4700 10 monclient: build_initial_monmap 2024-09-16T06:42:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.627+0000 7fc39a0f4700 10 monclient: monmap: 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:14.632328+0000 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:14.632328+0000 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:14.632 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 Processor -- start 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- start start 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient: init 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-a con 0x7fc39409ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-b con 0x7fc394135bd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-c con 0x7fc394135fe0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394097960 con 0x7fc39409ed40 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940710c0 con 0x7fc394135fe0 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39409f120 con 0x7fc394135bd0 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient(hunting): authenticate will time out at 1087.145158s 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3937fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc39409ed40 legacy=0x7fc3941354c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc393fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc394135fe0 0x7fc39413de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37096/0 (socket says 172.21.15.189:37096) 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc393fff700 1 -- 172.21.15.189:0/3677863114 learned_addr learned my addr 172.21.15.189:0/3677863114 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356623465 0 0) 0x7fc3940710c0 con 0x7fc394135fe0 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): my global_id is 134626 2024-09-16T06:42:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c0035d0 con 0x7fc394135fe0 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770823347 0 0) 0x7fc39409f120 con 0x7fc394135bd0 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient(hunting): my global_id is 124919 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3940710c0 con 0x7fc394135bd0 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3344316477 0 0) 0x7fc37c0035d0 con 0x7fc394135fe0 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 >> v1:172.21.15.76:6790/0 conn(0x7fc394135bd0 legacy=0x7fc39413a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 >> v1:172.21.15.76:6789/0 conn(0x7fc39409ed40 legacy=0x7fc3941354c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 monclient: found mon.noname-c 2024-09-16T06:42:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc394071d60 con 0x7fc394135fe0 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc38c001990 con 0x7fc394135fe0 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: dump: 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 5 monclient: authenticate success, global_id 134626 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc38c001c00 con 0x7fc394135fe0 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 1 -- 172.21.15.189:0/3677863114 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc38c001f90 con 0x7fc394135fe0 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc3927fc700 10 monclient: dump: 2024-09-16T06:42:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 4 set_mon_vals no callback set 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 10 monclient: shutdown 2024-09-16T06:42:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3677863114 >> v1:172.21.15.189:6789/0 conn(0x7fc394135fe0 legacy=0x7fc39413de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3677863114 shutdown_connections 2024-09-16T06:42:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3677863114 >> 172.21.15.189:0/3677863114 conn(0x7fc39409fe30 msgr2=0x7fc3940a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3677863114 shutdown_connections 2024-09-16T06:42:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.631+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3677863114 wait complete. 2024-09-16T06:42:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: build_initial_monmap 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: monmap: 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 Processor -- start 2024-09-16T06:42:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- start start 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: init 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-a con 0x7fc39409ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-b con 0x7fc394135bd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient(hunting): picked mon.noname-c con 0x7fc394135fe0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394090fd0 con 0x7fc39409ed40 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39409a520 con 0x7fc394135bd0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39409d320 con 0x7fc394135fe0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient(hunting): authenticate will time out at 1087.150264s 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc3937fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc39409ed40 legacy=0x7fc394096540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:14.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc392ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc394135bd0 0x7fc3941ceda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37108/0 (socket says 172.21.15.189:37108) 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc392ffd700 1 -- 172.21.15.189:0/3076680809 learned_addr learned my addr 172.21.15.189:0/3076680809 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678954344 0 0) 0x7fc39409a520 con 0x7fc394135bd0 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): my global_id is 134629 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3800035d0 con 0x7fc394135bd0 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803143783 0 0) 0x7fc39409d320 con 0x7fc394135fe0 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): my global_id is 124922 2024-09-16T06:42:14.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39409a520 con 0x7fc394135fe0 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3055128655 0 0) 0x7fc3800035d0 con 0x7fc394135bd0 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc39409d320 con 0x7fc394135bd0 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc388001890 con 0x7fc394135bd0 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): dump: 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:14.648 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546040387 0 0) 0x7fc39409d320 con 0x7fc394135bd0 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 >> v1:172.21.15.76:6790/0 conn(0x7fc394135fe0 legacy=0x7fc3941d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 >> v1:172.21.15.76:6789/0 conn(0x7fc39409ed40 legacy=0x7fc394096540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 monclient: found mon.b 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39408f140 con 0x7fc394135bd0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: _finish_auth 0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 5 monclient: authenticate success, global_id 134629 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _renew_subs 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc394090730 con 0x7fc394135bd0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _renew_subs 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc394096c50 con 0x7fc394135bd0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3880036c0 con 0x7fc394135bd0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3880039d0 con 0x7fc394135bd0 2024-09-16T06:42:14.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc391ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.635+0000 7fc37bfff700 10 monclient: dump: 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:14.651 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc39a0f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc39a0f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc39a0f4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc394072970 con 0x7fc394135bd0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc388003b30 con 0x7fc394135bd0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(104..104 src has 66..104) v4 ==== 5168+0+0 (unknown 916779314 0 0) 0x7fc3880436e0 con 0x7fc394135bd0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.639+0000 7fc3937fe700 10 monclient: get_auth_request con 0x7fc380036220 auth_method 0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.643+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc3880042e0 con 0x7fc394135bd0 2024-09-16T06:42:14.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.643+0000 7fc37bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:14.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.643+0000 7fc37bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:14.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.823+0000 7fc39a0f4700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:14.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.823+0000 7fc39a0f4700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T06:42:14.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.823+0000 7fc39a0f4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:14.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:14.823+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fc394096f00 con 0x7fc394135bd0 2024-09-16T06:42:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.399+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v105) v1 ==== 58+0+3 (unknown 192631357 0 212448812) 0x7fc388012b40 con 0x7fc394135bd0 2024-09-16T06:42:15.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.399+0000 7fc37bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T06:42:15.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.399+0000 7fc37bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:15.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.519+0000 7fc39a0f4700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:15.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.519+0000 7fc39a0f4700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T06:42:15.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.519+0000 7fc39a0f4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:15.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:15.519+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fc394097710 con 0x7fc394135bd0 2024-09-16T06:42:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.419+0000 7fc37bfff700 1 -- 172.21.15.189:0/3076680809 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v106) v1 ==== 58+0+3 (unknown 3507046352 0 944643765) 0x7fc3880144a0 con 0x7fc394135bd0 2024-09-16T06:42:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.419+0000 7fc37bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T06:42:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.419+0000 7fc37bfff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:16.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc380036220 legacy=0x7fc3800386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 10 monclient: shutdown 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 >> v1:172.21.15.189:6789/0 conn(0x7fc394135bd0 legacy=0x7fc3941ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 shutdown_connections 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 >> 172.21.15.189:0/3076680809 conn(0x7fc39409fe30 msgr2=0x7fc3940a1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 shutdown_connections 2024-09-16T06:42:16.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.423+0000 7fc39a0f4700 1 -- 172.21.15.189:0/3076680809 wait complete. 2024-09-16T06:42:16.445 INFO:tasks.workunit.client.0.smithi189.stderr:+ o4=12 2024-09-16T06:42:16.445 INFO:tasks.workunit.client.0.smithi189.stderr:++ ceph osd create 2024-09-16T06:42:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: get_monmap_and_config 2024-09-16T06:42:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: build_initial_monmap 2024-09-16T06:42:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: monmap: 2024-09-16T06:42:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:16.553366+0000 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:16.553366+0000 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:16.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 1 Processor -- start 2024-09-16T06:42:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 1 -- start start 2024-09-16T06:42:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: init 2024-09-16T06:42:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-b con 0x7f9e9806a890 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-a con 0x7f9e980a2f20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-c con 0x7f9e980a3de0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9806ac70 con 0x7f9e980a2f20 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9806b550 con 0x7f9e980a3de0 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980a23c0 con 0x7f9e9806a890 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 10 monclient(hunting): authenticate will time out at 1089.066618s 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9de81700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e980a3de0 0x7f9e9813dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37116/0 (socket says 172.21.15.189:37116) 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9de81700 1 -- 172.21.15.189:0/1968419914 learned_addr learned my addr 172.21.15.189:0/1968419914 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9ce7f700 1 -- 172.21.15.189:0/1968419914 >> v1:172.21.15.76:6789/0 conn(0x7f9e980a2f20 legacy=0x7f9e9813a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382390737 0 0) 0x7f9e9806b550 con 0x7f9e980a3de0 2024-09-16T06:42:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): my global_id is 134632 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e840035d0 con 0x7f9e980a3de0 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819981047 0 0) 0x7f9e980a23c0 con 0x7f9e9806a890 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient(hunting): my global_id is 124925 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9806b550 con 0x7f9e9806a890 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021082055 0 0) 0x7f9e840035d0 con 0x7f9e980a3de0 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 >> v1:172.21.15.76:6790/0 conn(0x7f9e9806a890 legacy=0x7f9e980a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 >> v1:172.21.15.76:6789/0 conn(0x7f9e980a2f20 legacy=0x7f9e9813a500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 monclient: found mon.noname-c 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98099dd0 con 0x7f9e980a3de0 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: _finish_auth 0 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e9f8e4700 5 monclient: authenticate success, global_id 134632 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e90001990 con 0x7f9e980a3de0 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.551+0000 7f9e8ffff700 10 monclient: dump: 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e90001c70 con 0x7f9e980a3de0 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 1 -- 172.21.15.189:0/1968419914 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e90002000 con 0x7f9e980a3de0 2024-09-16T06:42:16.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 4 set_mon_vals no callback set 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e8ffff700 10 monclient: dump: 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:16.561 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: shutdown 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1968419914 >> v1:172.21.15.189:6789/0 conn(0x7f9e980a3de0 legacy=0x7f9e9813dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1968419914 shutdown_connections 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1968419914 >> 172.21.15.189:0/1968419914 conn(0x7f9e980a7c60 msgr2=0x7f9e980a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1968419914 shutdown_connections 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1968419914 wait complete. 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: build_initial_monmap 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: monmap: 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:16.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:16.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:16.563 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:16.563 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:16.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:16.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 Processor -- start 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- start start 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: init 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-b con 0x7f9e9806a890 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-c con 0x7f9e980a2f20 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient(hunting): picked mon.noname-a con 0x7f9e980a3de0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9813d7c0 con 0x7f9e980a3de0 2024-09-16T06:42:16.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9809fda0 con 0x7f9e9806a890 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9809b1f0 con 0x7f9e980a2f20 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9f8e4700 10 monclient(hunting): authenticate will time out at 1089.071717s 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9d680700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e9806a890 0x7f9e981c6780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37126/0 (socket says 172.21.15.189:37126) 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9d680700 1 -- 172.21.15.189:0/1411693815 learned_addr learned my addr 172.21.15.189:0/1411693815 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.555+0000 7f9e9de81700 1 -- 172.21.15.189:0/1411693815 >> v1:172.21.15.76:6789/0 conn(0x7f9e980a3de0 legacy=0x7f9e981d22a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489108216 0 0) 0x7f9e9809fda0 con 0x7f9e9806a890 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): my global_id is 134635 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e7c0035d0 con 0x7f9e9806a890 2024-09-16T06:42:16.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147963245 0 0) 0x7f9e9809b1f0 con 0x7f9e980a2f20 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): my global_id is 124928 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9809fda0 con 0x7f9e980a2f20 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811431753 0 0) 0x7f9e7c0035d0 con 0x7f9e9806a890 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e9809b1f0 con 0x7f9e9806a890 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e94001850 con 0x7f9e9806a890 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): dump: 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:16.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 167659781 0 0) 0x7f9e9809b1f0 con 0x7f9e9806a890 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 >> v1:172.21.15.76:6790/0 conn(0x7f9e980a2f20 legacy=0x7f9e981ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 >> v1:172.21.15.76:6789/0 conn(0x7f9e980a3de0 legacy=0x7f9e981d22a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:16.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 monclient: found mon.b 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9806bdd0 con 0x7f9e9806a890 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 5 monclient: authenticate success, global_id 134635 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _renew_subs 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9813b460 con 0x7f9e9806a890 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _renew_subs 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9813c250 con 0x7f9e9806a890 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e940036c0 con 0x7f9e9806a890 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e94003a50 con 0x7f9e9806a890 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 10 monclient: dump: 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:16.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8f7fe700 4 set_mon_vals no callback set 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9e94003bb0 con 0x7f9e9806a890 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(106..106 src has 66..106) v4 ==== 5287+0+0 (unknown 4236273987 0 0) 0x7f9e940437e0 con 0x7f9e9806a890 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e78005100 con 0x7f9e9806a890 2024-09-16T06:42:16.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.559+0000 7f9e9ce7f700 10 monclient: get_auth_request con 0x7f9e7c036220 auth_method 0 2024-09-16T06:42:16.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.567+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9e940146d0 con 0x7f9e9806a890 2024-09-16T06:42:16.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.567+0000 7f9e8dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:16.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.567+0000 7f9e8dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.747+0000 7f9e9f8e4700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.747+0000 7f9e9f8e4700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T06:42:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.747+0000 7f9e9f8e4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:16.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:16.747+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f9e78005930 con 0x7f9e9806a890 2024-09-16T06:42:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.079+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) v1 ==== 58+0+3 (unknown 1685887268 0 736921282) 0x7f9e94012b40 con 0x7f9e9806a890 2024-09-16T06:42:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.079+0000 7f9e8dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T06:42:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.079+0000 7f9e8dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:17.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.195+0000 7f9e9f8e4700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T06:42:17.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.195+0000 7f9e9f8e4700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T06:42:17.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.195+0000 7f9e9f8e4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:17.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:17.195+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f9e78001140 con 0x7f9e9806a890 2024-09-16T06:42:18.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.087+0000 7f9e8dffb700 1 -- 172.21.15.189:0/1411693815 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v108) v1 ==== 58+0+3 (unknown 1610685179 0 1367428999) 0x7f9e94012b40 con 0x7f9e9806a890 2024-09-16T06:42:18.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.087+0000 7f9e8dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T06:42:18.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.087+0000 7f9e8dffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9e7c036220 legacy=0x7f9e7c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 10 monclient: shutdown 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 >> v1:172.21.15.189:6789/0 conn(0x7f9e9806a890 legacy=0x7f9e981c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 shutdown_connections 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 >> 172.21.15.189:0/1411693815 conn(0x7f9e980a7c60 msgr2=0x7f9e9813e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 shutdown_connections 2024-09-16T06:42:18.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.095+0000 7f9e9f8e4700 1 -- 172.21.15.189:0/1411693815 wait complete. 2024-09-16T06:42:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:+ o5=14 2024-09-16T06:42:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-16T06:42:18.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.215+0000 7f01754fc700 10 monclient: get_monmap_and_config 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: build_initial_monmap 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: monmap: 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:18.220558+0000 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:18.220558+0000 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:18.220 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 Processor -- start 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- start start 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: init 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-b con 0x7f0170098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-c con 0x7f0170099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-a con 0x7f0170138a80 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017009c460 con 0x7f0170138a80 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017009c1e0 con 0x7f0170099a60 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017009bd60 con 0x7f0170098070 2024-09-16T06:42:18.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient(hunting): authenticate will time out at 1090.733794s 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0170138a80 legacy=0x7f017013ae70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016e7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0170099a60 0x7f0170099e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37138/0 (socket says 172.21.15.189:37138) 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016e7fc700 1 -- 172.21.15.189:0/112332201 learned_addr learned my addr 172.21.15.189:0/112332201 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350785706 0 0) 0x7f017009c1e0 con 0x7f0170099a60 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): my global_id is 134638 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f015c0035d0 con 0x7f0170099a60 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996879592 0 0) 0x7f017009bd60 con 0x7f0170098070 2024-09-16T06:42:18.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient(hunting): my global_id is 124931 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017009c1e0 con 0x7f0170098070 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2140678795 0 0) 0x7f015c0035d0 con 0x7f0170099a60 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 >> v1:172.21.15.76:6790/0 conn(0x7f0170098070 legacy=0x7f017009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 >> v1:172.21.15.76:6789/0 conn(0x7f0170138a80 legacy=0x7f017013ae70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 monclient: found mon.noname-c 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017009d490 con 0x7f0170099a60 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 5 monclient: authenticate success, global_id 134638 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0160001610 con 0x7f0170099a60 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: dump: 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:18.227 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0160001800 con 0x7f0170099a60 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 4 set_mon_vals no callback set 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 1 -- 172.21.15.189:0/112332201 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0160001bd0 con 0x7f0170099a60 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f016dffb700 10 monclient: dump: 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:18.228 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 10 monclient: shutdown 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- 172.21.15.189:0/112332201 >> v1:172.21.15.189:6789/0 conn(0x7f0170099a60 legacy=0x7f0170099e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- 172.21.15.189:0/112332201 shutdown_connections 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.219+0000 7f01754fc700 1 -- 172.21.15.189:0/112332201 >> 172.21.15.189:0/112332201 conn(0x7f01700a03e0 msgr2=0x7f01700a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- 172.21.15.189:0/112332201 shutdown_connections 2024-09-16T06:42:18.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- 172.21.15.189:0/112332201 wait complete. 2024-09-16T06:42:18.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient: build_initial_monmap 2024-09-16T06:42:18.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient: monmap: 2024-09-16T06:42:18.233 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:18.233 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:18.233 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 Processor -- start 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- start start 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient: init 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-a con 0x7f0170098070 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-c con 0x7f0170099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:18.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient(hunting): picked mon.noname-b con 0x7f0170138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01701296a0 con 0x7f0170098070 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017012aae0 con 0x7f0170138a80 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017012adc0 con 0x7f0170099a60 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f016f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0170138a80 0x7f01701d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:37154/0 (socket says 172.21.15.189:37154) 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f016f7fe700 1 -- 172.21.15.189:0/228949784 learned_addr learned my addr 172.21.15.189:0/228949784 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f016effd700 1 -- 172.21.15.189:0/228949784 >> v1:172.21.15.76:6789/0 conn(0x7f0170098070 legacy=0x7f01701c7e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01754fc700 10 monclient(hunting): authenticate will time out at 1090.738769s 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642247277 0 0) 0x7f017012aae0 con 0x7f0170138a80 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:18.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): my global_id is 134641 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01500035d0 con 0x7f0170138a80 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673489796 0 0) 0x7f017012adc0 con 0x7f0170099a60 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): my global_id is 124934 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017012aae0 con 0x7f0170099a60 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320071061 0 0) 0x7f01500035d0 con 0x7f0170138a80 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f017012adc0 con 0x7f0170138a80 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f01680044f0 con 0x7f0170138a80 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): dump: 2024-09-16T06:42:18.236 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.223+0000 7f01677fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1427589392 0 0) 0x7f017012adc0 con 0x7f0170138a80 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 >> v1:172.21.15.76:6790/0 conn(0x7f0170099a60 legacy=0x7f01701cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 >> v1:172.21.15.76:6789/0 conn(0x7f0170098070 legacy=0x7f01701c7e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 monclient: found mon.b 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170128f70 con 0x7f0170138a80 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 5 monclient: authenticate success, global_id 134641 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _renew_subs 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01700a3090 con 0x7f0170138a80 2024-09-16T06:42:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _renew_subs 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0170128aa0 con 0x7f0170138a80 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0168004760 con 0x7f0170138a80 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0168004af0 con 0x7f0170138a80 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f016d7fa700 4 set_mon_vals no callback set 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 10 monclient: dump: 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0168004c50 con 0x7f0170138a80 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(108..108 src has 66..108) v4 ==== 5525+0+0 (unknown 2286394304 0 0) 0x7f0168043c70 con 0x7f0170138a80 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f016effd700 10 monclient: get_auth_request con 0x7f0150035fa0 auth_method 0 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.227+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0154005100 con 0x7f0170138a80 2024-09-16T06:42:18.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.235+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0168005450 con 0x7f0170138a80 2024-09-16T06:42:18.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.235+0000 7f01677fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:18.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.235+0000 7f01677fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:18.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.407+0000 7f01754fc700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:18.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.407+0000 7f01754fc700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:18.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.407+0000 7f01754fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:18.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:18.407+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f0154005930 con 0x7f0170138a80 2024-09-16T06:42:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.095+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v109) v1 ==== 220+0+0 (unknown 3059984712 0 0) 0x7f0168012e80 con 0x7f0170138a80 2024-09-16T06:42:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.095+0000 7f01677fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.095+0000 7f01677fe700 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T06:42:19.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01754fc700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01754fc700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01754fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f01540053b0 con 0x7f0170138a80 2024-09-16T06:42:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01677fe700 1 -- 172.21.15.189:0/228949784 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v109) v1 ==== 218+0+0 (unknown 1700362389 0 0) 0x7f01680147e0 con 0x7f0170138a80 2024-09-16T06:42:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01677fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.207+0000 7f01677fe700 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T06:42:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0150035fa0 legacy=0x7f0150038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 10 monclient: shutdown 2024-09-16T06:42:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 >> v1:172.21.15.189:6789/0 conn(0x7f0170138a80 legacy=0x7f01701d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 shutdown_connections 2024-09-16T06:42:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 >> 172.21.15.189:0/228949784 conn(0x7f01700a03e0 msgr2=0x7f017009b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 shutdown_connections 2024-09-16T06:42:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.211+0000 7f01754fc700 1 -- 172.21.15.189:0/228949784 wait complete. 2024-09-16T06:42:19.215 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T06:42:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-16T06:42:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.331+0000 7fd2c98c9700 10 monclient: get_monmap_and_config 2024-09-16T06:42:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.331+0000 7fd2c98c9700 10 monclient: build_initial_monmap 2024-09-16T06:42:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.331+0000 7fd2c98c9700 10 monclient: monmap: 2024-09-16T06:42:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:19.335 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:19.335793+0000 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:19.335793+0000 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:19.336 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 Processor -- start 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- start start 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: init 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2c409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2c409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c4138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409cca0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409f750 con 0x7fd2c409bb00 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4097620 con 0x7fd2c4138da0 2024-09-16T06:42:19.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient(hunting): authenticate will time out at 1091.848920s 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd2c409bb00 0x7fd2c409bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46436/0 (socket says 172.21.15.189:46436) 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c2ffd700 1 -- 172.21.15.189:0/2489033914 learned_addr learned my addr 172.21.15.189:0/2489033914 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c27fc700 1 -- 172.21.15.189:0/2489033914 >> v1:172.21.15.76:6789/0 conn(0x7fd2c409c5f0 legacy=0x7fd2c4137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055526010 0 0) 0x7fd2c409f750 con 0x7fd2c409bb00 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient(hunting): my global_id is 134644 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac0035d0 con 0x7fd2c409bb00 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261730562 0 0) 0x7fd2ac0035d0 con 0x7fd2c409bb00 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 >> v1:172.21.15.76:6790/0 conn(0x7fd2c4138da0 legacy=0x7fd2c413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:19.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 >> v1:172.21.15.76:6789/0 conn(0x7fd2c409c5f0 legacy=0x7fd2c4137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 monclient: found mon.noname-c 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c4097900 con 0x7fd2c409bb00 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 5 monclient: authenticate success, global_id 134644 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b4001610 con 0x7fd2c409bb00 2024-09-16T06:42:19.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: dump: 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b4001930 con 0x7fd2c409bb00 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 4 set_mon_vals no callback set 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 1 -- 172.21.15.189:0/2489033914 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b4001d00 con 0x7fd2c409bb00 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c1ffb700 10 monclient: dump: 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: shutdown 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- 172.21.15.189:0/2489033914 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409bb00 legacy=0x7fd2c409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- 172.21.15.189:0/2489033914 shutdown_connections 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- 172.21.15.189:0/2489033914 >> 172.21.15.189:0/2489033914 conn(0x7fd2c40a03e0 msgr2=0x7fd2c40a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- 172.21.15.189:0/2489033914 shutdown_connections 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 -- 172.21.15.189:0/2489033914 wait complete. 2024-09-16T06:42:19.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: build_initial_monmap 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 10 monclient: monmap: 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.335+0000 7fd2c98c9700 1 Processor -- start 2024-09-16T06:42:19.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- start start 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: init 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2c409bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c409c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2c4138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40a2c50 con 0x7fd2c409bb00 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40994f0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4093b00 con 0x7fd2c4138da0 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:19.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient(hunting): authenticate will time out at 1091.852627s 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c2ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd2c409bb00 legacy=0x7fd2c408fe10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c27fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd2c409c5f0 0x7fd2c4090520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46450/0 (socket says 172.21.15.189:46450) 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c27fc700 1 -- 172.21.15.189:0/18688893 learned_addr learned my addr 172.21.15.189:0/18688893 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172130639 0 0) 0x7fd2c40994f0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): my global_id is 134647 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b00035d0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038961369 0 0) 0x7fd2b00035d0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c40994f0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b8004230 con 0x7fd2c409c5f0 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:19.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): dump: 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:19.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3404158276 0 0) 0x7fd2c40994f0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 >> v1:172.21.15.76:6790/0 conn(0x7fd2c4138da0 legacy=0x7fd2c41d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 >> v1:172.21.15.76:6789/0 conn(0x7fd2c409bb00 legacy=0x7fd2c408fe10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 monclient: found mon.b 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c409a6e0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: _finish_auth 0 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 5 monclient: authenticate success, global_id 134647 2024-09-16T06:42:19.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _renew_subs 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c4092370 con 0x7fd2c409c5f0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _renew_subs 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4096290 con 0x7fd2c409c5f0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b8004550 con 0x7fd2c409c5f0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b80048e0 con 0x7fd2c409c5f0 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c17fa700 4 set_mon_vals no callback set 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 10 monclient: dump: 2024-09-16T06:42:19.353 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:19.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:19.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd2b8004a40 con 0x7fd2c409c5f0 2024-09-16T06:42:19.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:19.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a4005100 con 0x7fd2c409c5f0 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(109..109 src has 66..109) v4 ==== 5709+0+0 (unknown 981434505 0 0) 0x7fd2b8043a60 con 0x7fd2c409c5f0 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.339+0000 7fd2c2ffd700 10 monclient: get_auth_request con 0x7fd2b0035f10 auth_method 0 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.347+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd2b8005210 con 0x7fd2c409c5f0 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.347+0000 7fd2abfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:19.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.347+0000 7fd2abfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:19.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.523+0000 7fd2c98c9700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.523+0000 7fd2c98c9700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:19.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.523+0000 7fd2c98c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:19.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:19.523+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fd2a4005930 con 0x7fd2c409c5f0 2024-09-16T06:42:20.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.115+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v110) v1 ==== 220+0+0 (unknown 4093542432 0 0) 0x7fd2b801ce00 con 0x7fd2c409c5f0 2024-09-16T06:42:20.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.115+0000 7fd2abfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:20.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.115+0000 7fd2abfff700 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T06:42:20.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.223+0000 7fd2c98c9700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:20.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.223+0000 7fd2c98c9700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:20.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.223+0000 7fd2c98c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.223+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fd2a40053b0 con 0x7fd2c409c5f0 2024-09-16T06:42:20.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.227+0000 7fd2abfff700 1 -- 172.21.15.189:0/18688893 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v110) v1 ==== 218+0+0 (unknown 211588743 0 0) 0x7fd2b8012c80 con 0x7fd2c409c5f0 2024-09-16T06:42:20.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.227+0000 7fd2abfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T06:42:20.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.227+0000 7fd2abfff700 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T06:42:20.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd2b0035f10 legacy=0x7fd2b00383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 10 monclient: shutdown 2024-09-16T06:42:20.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:20.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409c5f0 legacy=0x7fd2c4090520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 shutdown_connections 2024-09-16T06:42:20.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 >> 172.21.15.189:0/18688893 conn(0x7fd2c40a03e0 msgr2=0x7fd2c4096810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:20.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 shutdown_connections 2024-09-16T06:42:20.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.231+0000 7fd2c98c9700 1 -- 172.21.15.189:0/18688893 wait complete. 2024-09-16T06:42:20.234 INFO:tasks.workunit.client.0.smithi189.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T06:42:20.243 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:20.243 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.12 2024-09-16T06:42:20.243 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: get_monmap_and_config 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: build_initial_monmap 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: monmap: 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:20.345631+0000 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:20.345631+0000 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:20.345 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:20.346 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 1 Processor -- start 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 1 -- start start 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: init 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-c con 0x7f757406b760 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-a con 0x7f7574135b50 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-b con 0x7f7574135f30 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757406bb40 con 0x7f7574135b50 2024-09-16T06:42:20.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757406c310 con 0x7f757406b760 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757406ac80 con 0x7f7574135f30 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient(hunting): authenticate will time out at 1092.858639s 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f75792ef700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7574135b50 legacy=0x7f757413a760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7579af0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f757406b760 0x7f7574135440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46454/0 (socket says 172.21.15.189:46454) 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7579af0700 1 -- 172.21.15.189:0/556702824 learned_addr learned my addr 172.21.15.189:0/556702824 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838131969 0 0) 0x7f757406c310 con 0x7f757406b760 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient(hunting): my global_id is 134650 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75600035d0 con 0x7f757406b760 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3373021902 0 0) 0x7f75600035d0 con 0x7f757406b760 2024-09-16T06:42:20.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 >> v1:172.21.15.76:6790/0 conn(0x7f7574135f30 legacy=0x7f757413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 >> v1:172.21.15.76:6789/0 conn(0x7f7574135b50 legacy=0x7f757413a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: _finish_hunting 0 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 monclient: found mon.noname-c 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757406af60 con 0x7f757406b760 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: _finish_auth 0 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: _check_auth_tickets 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7564001610 con 0x7f757406b760 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 5 monclient: authenticate success, global_id 134650 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:20.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: dump: 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7564001930 con 0x7f757406b760 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 1 -- 172.21.15.189:0/556702824 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7564001cc0 con 0x7f757406b760 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f757bd54700 4 set_mon_vals no callback set 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.343+0000 7f7578aee700 10 monclient: dump: 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: shutdown 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:20.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- 172.21.15.189:0/556702824 >> v1:172.21.15.189:6789/0 conn(0x7f757406b760 legacy=0x7f7574135440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- 172.21.15.189:0/556702824 shutdown_connections 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- 172.21.15.189:0/556702824 >> 172.21.15.189:0/556702824 conn(0x7f75740a4660 msgr2=0x7f75740a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- 172.21.15.189:0/556702824 shutdown_connections 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- 172.21.15.189:0/556702824 wait complete. 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: build_initial_monmap 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: monmap: 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:20.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:20.356 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:20.356 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 Processor -- start 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- start start 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: init 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-a con 0x7f757406b760 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-c con 0x7f7574135b50 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient(hunting): picked mon.noname-b con 0x7f7574135f30 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:20.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740a7290 con 0x7f757406b760 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757413e890 con 0x7f7574135f30 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757409e930 con 0x7f7574135b50 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757bd54700 10 monclient(hunting): authenticate will time out at 1092.862860s 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f7579af0700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f757406b760 legacy=0x7f757413ede0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757a2f1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7574135f30 0x7f75741d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46464/0 (socket says 172.21.15.189:46464) 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f757a2f1700 1 -- 172.21.15.189:0/299255404 learned_addr learned my addr 172.21.15.189:0/299255404 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:20.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756797526 0 0) 0x7f757413e890 con 0x7f7574135f30 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): my global_id is 134653 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75580035d0 con 0x7f7574135f30 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127015553 0 0) 0x7f757409e930 con 0x7f7574135b50 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): my global_id is 124946 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757413e890 con 0x7f7574135b50 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988664242 0 0) 0x7f75580035d0 con 0x7f7574135f30 2024-09-16T06:42:20.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757409e930 con 0x7f7574135f30 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75700018f0 con 0x7f7574135f30 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.363 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.347+0000 7f756a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2516951365 0 0) 0x7f757409e930 con 0x7f7574135f30 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 >> v1:172.21.15.76:6790/0 conn(0x7f7574135b50 legacy=0x7f75741ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 >> v1:172.21.15.76:6789/0 conn(0x7f757406b760 legacy=0x7f757413ede0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 monclient: found mon.b 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757413bb90 con 0x7f7574135f30 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:20.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 5 monclient: authenticate success, global_id 134653 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _renew_subs 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75741397c0 con 0x7f7574135f30 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _renew_subs 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75741d3490 con 0x7f7574135f30 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7570001c80 con 0x7f7574135f30 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7570003b10 con 0x7f7574135f30 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756bfff700 4 set_mon_vals no callback set 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:20.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 10 monclient: dump: 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7570003cf0 con 0x7f7574135f30 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 66..110) v4 ==== 5731+0+0 (unknown 2301019725 0 0) 0x7f7570043a30 con 0x7f7574135f30 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c005100 con 0x7f7574135f30 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.351+0000 7f7579af0700 10 monclient: get_auth_request con 0x7f7558036010 auth_method 0 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.359+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7570014670 con 0x7f7574135f30 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.359+0000 7f756a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.359+0000 7f756a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:20.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f757bd54700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f757bd54700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f757bd54700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f755c005bc0 con 0x7f7574135f30 2024-09-16T06:42:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (unknown 3122809895 0 948571019) 0x7f7570014490 con 0x7f7574135f30 2024-09-16T06:42:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f756a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.531+0000 7f756a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f755c005680 con 0x7f7574135f30 2024-09-16T06:42:20.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f756a7fc700 1 -- 172.21.15.189:0/299255404 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (unknown 3122809895 0 948571019) 0x7f7570014490 con 0x7f7574135f30 2024-09-16T06:42:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f756a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f756a7fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7558036010 legacy=0x7f75580384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 10 monclient: shutdown 2024-09-16T06:42:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 >> v1:172.21.15.189:6789/0 conn(0x7f7574135f30 legacy=0x7f75741d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 shutdown_connections 2024-09-16T06:42:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 >> 172.21.15.189:0/299255404 conn(0x7f75740a4660 msgr2=0x7f7574135400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 shutdown_connections 2024-09-16T06:42:20.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.635+0000 7f757bd54700 1 -- 172.21.15.189:0/299255404 wait complete. 2024-09-16T06:42:20.649 INFO:tasks.workunit.client.0.smithi189.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-16T06:42:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:20.650 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.14 2024-09-16T06:42:20.651 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 123 2024-09-16T06:42:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 10 monclient: get_monmap_and_config 2024-09-16T06:42:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 10 monclient: build_initial_monmap 2024-09-16T06:42:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 10 monclient: monmap: 2024-09-16T06:42:20.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:20.751134+0000 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:20.751134+0000 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:20.751 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 1 Processor -- start 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 1 -- start start 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 10 monclient: init 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.747+0000 7f6881f90700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-b con 0x7f687c09ed40 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-c con 0x7f687c097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:20.754 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-a con 0x7f687c09f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c099b10 con 0x7f687c09f830 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c098890 con 0x7f687c097860 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c098b70 con 0x7f687c09ed40 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient(hunting): authenticate will time out at 1093.264107s 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f687c09f830 legacy=0x7f687c13be10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f687c097860 0x7f687c1386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46466/0 (socket says 172.21.15.189:46466) 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687affd700 1 -- 172.21.15.189:0/3972581400 learned_addr learned my addr 172.21.15.189:0/3972581400 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882685713 0 0) 0x7f687c098890 con 0x7f687c097860 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): my global_id is 134656 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f685c0035d0 con 0x7f687c097860 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850274311 0 0) 0x7f687c098b70 con 0x7f687c09ed40 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient(hunting): my global_id is 124949 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687c098890 con 0x7f687c09ed40 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560894388 0 0) 0x7f685c0035d0 con 0x7f687c097860 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 >> v1:172.21.15.76:6790/0 conn(0x7f687c09ed40 legacy=0x7f687c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 >> v1:172.21.15.76:6789/0 conn(0x7f687c09f830 legacy=0x7f687c13be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:20.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 monclient: found mon.noname-c 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c0973a0 con 0x7f687c097860 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 5 monclient: authenticate success, global_id 134656 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f686c001610 con 0x7f687c097860 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: dump: 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.757 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f686c0018f0 con 0x7f687c097860 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:20.758 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 4 set_mon_vals no callback set 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 1 -- 172.21.15.189:0/3972581400 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f686c001c80 con 0x7f687c097860 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f687a7fc700 10 monclient: dump: 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.759 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient: shutdown 2024-09-16T06:42:20.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:20.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- 172.21.15.189:0/3972581400 >> v1:172.21.15.189:6789/0 conn(0x7f687c097860 legacy=0x7f687c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- 172.21.15.189:0/3972581400 shutdown_connections 2024-09-16T06:42:20.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- 172.21.15.189:0/3972581400 >> 172.21.15.189:0/3972581400 conn(0x7f687c09fe30 msgr2=0x7f687c0a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:20.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- 172.21.15.189:0/3972581400 shutdown_connections 2024-09-16T06:42:20.761 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 1 -- 172.21.15.189:0/3972581400 wait complete. 2024-09-16T06:42:20.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient: build_initial_monmap 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.751+0000 7f6881f90700 10 monclient: monmap: 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 Processor -- start 2024-09-16T06:42:20.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- start start 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: init 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-c con 0x7f687c097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-a con 0x7f687c09ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient(hunting): picked mon.noname-b con 0x7f687c09f830 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c09a520 con 0x7f687c09ed40 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c1c86f0 con 0x7f687c09f830 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c130040 con 0x7f687c097860 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient(hunting): authenticate will time out at 1093.269103s 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f687bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f687c09f830 0x7f687c1d01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46478/0 (socket says 172.21.15.189:46478) 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f687bfff700 1 -- 172.21.15.189:0/4040402478 learned_addr learned my addr 172.21.15.189:0/4040402478 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:20.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f687affd700 1 -- 172.21.15.189:0/4040402478 >> v1:172.21.15.76:6789/0 conn(0x7f687c09ed40 legacy=0x7f687c1ccac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031396166 0 0) 0x7f6874003aa0 con 0x7f687c09f830 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): my global_id is 134659 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68600035d0 con 0x7f687c09f830 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680742079 0 0) 0x7f68600035d0 con 0x7f687c09f830 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6874003aa0 con 0x7f687c09f830 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68740040d0 con 0x7f687c09f830 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): dump: 2024-09-16T06:42:20.768 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:20.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525450362 0 0) 0x7f687c130040 con 0x7f687c097860 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient(hunting): my global_id is 124952 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68600035d0 con 0x7f687c097860 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4235128764 0 0) 0x7f6874003aa0 con 0x7f687c09f830 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 >> v1:172.21.15.76:6790/0 conn(0x7f687c097860 legacy=0x7f687c133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 >> v1:172.21.15.76:6789/0 conn(0x7f687c09ed40 legacy=0x7f687c1ccac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 monclient: found mon.b 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c130320 con 0x7f687c09f830 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 5 monclient: authenticate success, global_id 134659 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _renew_subs 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c12dfb0 con 0x7f687c09f830 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _renew_subs 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c12fa80 con 0x7f687c09f830 2024-09-16T06:42:20.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6874004380 con 0x7f687c09f830 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6874004570 con 0x7f687c09f830 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6879ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 10 monclient: dump: 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:20.772 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6874004750 con 0x7f687c09f830 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 66..110) v4 ==== 5731+0+0 (unknown 2301019725 0 0) 0x7f68740438e0 con 0x7f687c09f830 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.755+0000 7f6881f90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.759+0000 7f6881f90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.759+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6854005100 con 0x7f687c09f830 2024-09-16T06:42:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.759+0000 7f687b7fe700 10 monclient: get_auth_request con 0x7f6860036190 auth_method 0 2024-09-16T06:42:20.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.763+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6874004fa0 con 0x7f687c09f830 2024-09-16T06:42:20.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.763+0000 7f6873fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:20.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.763+0000 7f6873fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:20.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6881f90700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:20.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6881f90700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6881f90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:20.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f6854005bc0 con 0x7f687c09f830 2024-09-16T06:42:20.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (unknown 3122809895 0 948571019) 0x7f687401cc40 con 0x7f687c09f830 2024-09-16T06:42:20.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6873fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:20.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:20.935+0000 7f6873fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6881f90700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6881f90700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6881f90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f6854005680 con 0x7f687c09f830 2024-09-16T06:42:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6873fff700 1 -- 172.21.15.189:0/4040402478 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v110) v1 ==== 56+0+1449 (unknown 3122809895 0 948571019) 0x7f6874012a60 con 0x7f687c09f830 2024-09-16T06:42:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6873fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:21.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.035+0000 7f6873fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6860036190 legacy=0x7f6860038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 10 monclient: shutdown 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 >> v1:172.21.15.189:6789/0 conn(0x7f687c09f830 legacy=0x7f687c1d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 shutdown_connections 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 >> 172.21.15.189:0/4040402478 conn(0x7f687c09fe30 msgr2=0x7f687c06f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 shutdown_connections 2024-09-16T06:42:21.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.039+0000 7f6881f90700 1 -- 172.21.15.189:0/4040402478 wait complete. 2024-09-16T06:42:21.052 INFO:tasks.workunit.client.0.smithi189.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-16T06:42:21.052 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: build_initial_monmap 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: monmap: 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:21.159079+0000 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:21.159079+0000 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:21.158 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:21.159 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:21.159 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:21.159 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:21.159 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 1 Processor -- start 2024-09-16T06:42:21.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 1 -- start start 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: init 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f242009eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f242009f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f2420139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009ab80 con 0x7f2420139a50 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009a690 con 0x7f242009eba0 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420072230 con 0x7f242009f690 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.155+0000 7f241f59e700 10 monclient(hunting): authenticate will time out at 1093.671821s 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f242009eba0 0x7f242009ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46484/0 (socket says 172.21.15.189:46484) 2024-09-16T06:42:21.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241e59c700 1 -- 172.21.15.189:0/3496847648 learned_addr learned my addr 172.21.15.189:0/3496847648 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241ed9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2420139a50 legacy=0x7f242013be40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755719374 0 0) 0x7f242009a690 con 0x7f242009eba0 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): my global_id is 134662 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c0035d0 con 0x7f242009eba0 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420331558 0 0) 0x7f2420072230 con 0x7f242009f690 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:21.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient(hunting): my global_id is 124955 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242009a690 con 0x7f242009f690 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1964490033 0 0) 0x7f240c0035d0 con 0x7f242009eba0 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 >> v1:172.21.15.76:6790/0 conn(0x7f242009f690 legacy=0x7f2420138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 >> v1:172.21.15.76:6789/0 conn(0x7f2420139a50 legacy=0x7f242013be40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 monclient: found mon.noname-c 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242009be90 con 0x7f242009eba0 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:42:21.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 5 monclient: authenticate success, global_id 134662 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2410001610 con 0x7f242009eba0 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: dump: 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:21.166 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24100018f0 con 0x7f242009eba0 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 4 set_mon_vals no callback set 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 1 -- 172.21.15.189:0/3496847648 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2410001c80 con 0x7f242009eba0 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241dd9b700 10 monclient: dump: 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:21.167 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: shutdown 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- 172.21.15.189:0/3496847648 >> v1:172.21.15.189:6789/0 conn(0x7f242009eba0 legacy=0x7f242009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- 172.21.15.189:0/3496847648 shutdown_connections 2024-09-16T06:42:21.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- 172.21.15.189:0/3496847648 >> 172.21.15.189:0/3496847648 conn(0x7f242009fe70 msgr2=0x7f24200a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- 172.21.15.189:0/3496847648 shutdown_connections 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- 172.21.15.189:0/3496847648 wait complete. 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: build_initial_monmap 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: monmap: 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:21.169 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 Processor -- start 2024-09-16T06:42:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 1 -- start start 2024-09-16T06:42:21.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: init 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f242009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f242009f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.159+0000 7f241f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f2420139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200929e0 con 0x7f2420139a50 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009e750 con 0x7f242009f690 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242009a380 con 0x7f242009eba0 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient(hunting): authenticate will time out at 1093.676043s 2024-09-16T06:42:21.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241ed9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2420139a50 legacy=0x7f24201d0540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2417fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f242009f690 0x7f24201cce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46496/0 (socket says 172.21.15.189:46496) 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2417fff700 1 -- 172.21.15.189:0/3006271617 learned_addr learned my addr 172.21.15.189:0/3006271617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59925647 0 0) 0x7f242009e750 con 0x7f242009f690 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): my global_id is 134665 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24000035d0 con 0x7f242009f690 2024-09-16T06:42:21.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664081701 0 0) 0x7f242009a380 con 0x7f242009eba0 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): my global_id is 124958 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242009e750 con 0x7f242009eba0 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686341150 0 0) 0x7f24000035d0 con 0x7f242009f690 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f242009a380 con 0x7f242009f690 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2408001910 con 0x7f242009f690 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:21.175 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1414553673 0 0) 0x7f242009a380 con 0x7f242009f690 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 >> v1:172.21.15.76:6790/0 conn(0x7f242009eba0 legacy=0x7f24201c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 >> v1:172.21.15.76:6789/0 conn(0x7f2420139a50 legacy=0x7f24201d0540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:21.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 monclient: found mon.b 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242006b7c0 con 0x7f242009f690 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 5 monclient: authenticate success, global_id 134665 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _renew_subs 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2420092140 con 0x7f242009f690 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _renew_subs 2024-09-16T06:42:21.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2420091d60 con 0x7f242009f690 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24080036c0 con 0x7f242009f690 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f24080039d0 con 0x7f242009f690 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 10 monclient: dump: 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:21.178 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241d59a700 4 set_mon_vals no callback set 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2408003bb0 con 0x7f242009f690 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 66..110) v4 ==== 5731+0+0 (unknown 2301019725 0 0) 0x7f24080439c0 con 0x7f242009f690 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2404005100 con 0x7f242009f690 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.163+0000 7f241e59c700 10 monclient: get_auth_request con 0x7f2400035fa0 auth_method 0 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.171+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2408004400 con 0x7f242009f690 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.171+0000 7f2416ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:21.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.171+0000 7f2416ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:21.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.347+0000 7f241f59e700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:21.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.347+0000 7f241f59e700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:21.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.347+0000 7f241f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:21.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:21.347+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f2404005930 con 0x7f242009f690 2024-09-16T06:42:21.896 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:42:21.897 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:42:21.897 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:42:21.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:42:21.900 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:42:21.900 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:42:21.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:42:22.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.009+0000 7f0ebd291700 10 monclient: get_monmap_and_config 2024-09-16T06:42:22.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.009+0000 7f0ebd291700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.009+0000 7f0ebd291700 10 monclient: monmap: 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:22.014691+0000 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:22.014691+0000 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:22.014 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 Processor -- start 2024-09-16T06:42:22.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- start start 2024-09-16T06:42:22.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient: init 2024-09-16T06:42:22.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:22.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-b con 0x7f0eb001fe40 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-a con 0x7f0eb001e160 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-c con 0x7f0eb00bebf0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb001d780 con 0x7f0eb001e160 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00226e0 con 0x7f0eb00bebf0 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb001d4a0 con 0x7f0eb001fe40 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient(hunting): authenticate will time out at 1089.148946s 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb7fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0eb001fe40 0x7f0eb00218c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51450/0 (socket says 172.21.15.76:51450) 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb7fff700 1 -- 172.21.15.76:0/717885628 learned_addr learned my addr 172.21.15.76:0/717885628 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146466590 0 0) 0x7f0eb001d4a0 con 0x7f0eb001fe40 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient(hunting): my global_id is 124961 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea80035d0 con 0x7f0eb001fe40 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098937831 0 0) 0x7f0ea80035d0 con 0x7f0eb001fe40 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 >> v1:172.21.15.189:6789/0 conn(0x7f0eb00bebf0 legacy=0x7f0eb00c0fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 >> v1:172.21.15.76:6789/0 conn(0x7f0eb001e160 legacy=0x7f0eb0021fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 monclient: found mon.noname-b 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb001da60 con 0x7f0eb001fe40 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0eac001610 con 0x7f0eb001fe40 2024-09-16T06:42:22.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: dump: 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.023 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 5 monclient: authenticate success, global_id 124961 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eac001900 con 0x7f0eb001fe40 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 1 -- 172.21.15.76:0/717885628 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0eac001c90 con 0x7f0eb001fe40 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 4 set_mon_vals no callback set 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0eb6ffd700 10 monclient: dump: 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.025 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 10 monclient: shutdown 2024-09-16T06:42:22.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- 172.21.15.76:0/717885628 >> v1:172.21.15.76:6790/0 conn(0x7f0eb001fe40 legacy=0x7f0eb00218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- 172.21.15.76:0/717885628 shutdown_connections 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- 172.21.15.76:0/717885628 >> 172.21.15.76:0/717885628 conn(0x7f0eb0005b40 msgr2=0x7f0eb0005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- 172.21.15.76:0/717885628 shutdown_connections 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.013+0000 7f0ebd291700 1 -- 172.21.15.76:0/717885628 wait complete. 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient: monmap: 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.028 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 1 Processor -- start 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 1 -- start start 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient: init 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-b con 0x7f0eb001e160 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-c con 0x7f0eb001fe40 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient(hunting): picked mon.noname-a con 0x7f0eb00bebf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00a0db0 con 0x7f0eb00bebf0 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0022080 con 0x7f0eb001e160 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0028500 con 0x7f0eb001fe40 2024-09-16T06:42:22.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebd291700 10 monclient(hunting): authenticate will time out at 1089.153249s 2024-09-16T06:42:22.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0eb77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0eb001fe40 0x7f0eb00bc6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51464/0 (socket says 172.21.15.76:51464) 2024-09-16T06:42:22.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0ebca90700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0eb00bebf0 0x7f0eb0153db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38048/0 (socket says 172.21.15.76:38048) 2024-09-16T06:42:22.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.017+0000 7f0eb77fe700 1 -- 172.21.15.76:0/748868078 learned_addr learned my addr 172.21.15.76:0/748868078 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674561733 0 0) 0x7f0eb0022080 con 0x7f0eb001e160 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): my global_id is 134668 2024-09-16T06:42:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea80035d0 con 0x7f0eb001e160 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431884134 0 0) 0x7f0ea80035d0 con 0x7f0eb001e160 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eb0022080 con 0x7f0eb001e160 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0eac0015d0 con 0x7f0eb001e160 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3503679520 0 0) 0x7f0eb0022080 con 0x7f0eb001e160 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 >> v1:172.21.15.76:6790/0 conn(0x7f0eb001fe40 legacy=0x7f0eb00bc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 >> v1:172.21.15.76:6789/0 conn(0x7f0eb00bebf0 legacy=0x7f0eb0153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 monclient: found mon.b 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00bce00 con 0x7f0eb001e160 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 5 monclient: authenticate success, global_id 134668 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 10 monclient: _renew_subs 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb00be8c0 con 0x7f0eb001e160 2024-09-16T06:42:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 10 monclient: _renew_subs 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb00bdbf0 con 0x7f0eb001e160 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eac004cf0 con 0x7f0eb001e160 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0eac004e60 con 0x7f0eb001e160 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb4ff9700 10 monclient: dump: 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.045 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.037+0000 7f0eb67fc700 4 set_mon_vals no callback set 2024-09-16T06:42:22.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0eac00f8d0 con 0x7f0eb001e160 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(110..110 src has 66..110) v4 ==== 5731+0+0 (unknown 2301019725 0 0) 0x7f0eac043b40 con 0x7f0eb001e160 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0ebd291700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0ebd291700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0ebd291700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98005100 con 0x7f0eb001e160 2024-09-16T06:42:22.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.041+0000 7f0eb77fe700 10 monclient: get_auth_request con 0x7f0ea80384c0 auth_method 0 2024-09-16T06:42:22.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.053+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0eac014910 con 0x7f0eb001e160 2024-09-16T06:42:22.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.053+0000 7f0eb4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.053+0000 7f0eb4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:22.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.079+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v111) v1 ==== 168+0+0 (unknown 1314649835 0 0) 0x7f240801ce20 con 0x7f242009f690 2024-09-16T06:42:22.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.079+0000 7f2416ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:22.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.079+0000 7f2416ffd700 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T06:42:22.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.195+0000 7f241f59e700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.195+0000 7f241f59e700 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.195+0000 7f241f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:22.195+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f24040023d0 con 0x7f242009f690 2024-09-16T06:42:22.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:42:22.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:22.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:22.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0e98005930 con 0x7f0eb001e160 2024-09-16T06:42:22.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0eb4ff9700 1 -- 172.21.15.76:0/748868078 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1917225192) 0x7f0eac012d80 con 0x7f0eb001e160 2024-09-16T06:42:22.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0eb4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:22.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0eb4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:22.296 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:42:22.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ea80384c0 legacy=0x7f0ea803a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 10 monclient: shutdown 2024-09-16T06:42:22.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 >> v1:172.21.15.189:6789/0 conn(0x7f0eb001e160 legacy=0x7f0eb00bbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 shutdown_connections 2024-09-16T06:42:22.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 >> 172.21.15.76:0/748868078 conn(0x7f0eb0005b40 msgr2=0x7f0eb0028a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 shutdown_connections 2024-09-16T06:42:22.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.293+0000 7f0ebd291700 1 -- 172.21.15.76:0/748868078 wait complete. 2024-09-16T06:42:22.311 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:42:22.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:42:22.311 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:42:22.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: get_monmap_and_config 2024-09-16T06:42:22.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: monmap: 2024-09-16T06:42:22.424 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:22.425348+0000 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:22.425348+0000 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:22.425 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 1 Processor -- start 2024-09-16T06:42:22.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 1 -- start start 2024-09-16T06:42:22.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: init 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-c con 0x7f10b8097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-b con 0x7f10b8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-a con 0x7f10b8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b8098990 con 0x7f10b8139500 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b8097f40 con 0x7f10b8097b60 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b809d610 con 0x7f10b8099090 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient(hunting): authenticate will time out at 1089.558561s 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10bd81c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f10b8139500 0x7f10b813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38060/0 (socket says 172.21.15.76:38060) 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10bd81c700 1 -- 172.21.15.76:0/2248801417 learned_addr learned my addr 172.21.15.76:0/2248801417 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989902731 0 0) 0x7f10b8098990 con 0x7f10b8139500 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): my global_id is 154098 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a40035d0 con 0x7f10b8139500 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740272317 0 0) 0x7f10b809d610 con 0x7f10b8099090 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): my global_id is 124964 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b8098990 con 0x7f10b8099090 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643947734 0 0) 0x7f10b8097f40 con 0x7f10b8097b60 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient(hunting): my global_id is 134671 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b809d610 con 0x7f10b8097b60 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354795406 0 0) 0x7f10a40035d0 con 0x7f10b8139500 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 >> v1:172.21.15.76:6790/0 conn(0x7f10b8099090 legacy=0x7f10b809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 >> v1:172.21.15.189:6789/0 conn(0x7f10b8097b60 legacy=0x7f10b809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 monclient: found mon.noname-a 2024-09-16T06:42:22.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80986b0 con 0x7f10b8139500 2024-09-16T06:42:22.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 5 monclient: authenticate success, global_id 154098 2024-09-16T06:42:22.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10b0001910 con 0x7f10b8139500 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: dump: 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b0001b30 con 0x7f10b8139500 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10be01d700 4 set_mon_vals no callback set 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 1 -- 172.21.15.76:0/2248801417 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10b0001ec0 con 0x7f10b8139500 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.421+0000 7f10affff700 10 monclient: dump: 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.434 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: shutdown 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- 172.21.15.76:0/2248801417 >> v1:172.21.15.76:6789/0 conn(0x7f10b8139500 legacy=0x7f10b813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- 172.21.15.76:0/2248801417 shutdown_connections 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- 172.21.15.76:0/2248801417 >> 172.21.15.76:0/2248801417 conn(0x7f10b80a0030 msgr2=0x7f10b80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- 172.21.15.76:0/2248801417 shutdown_connections 2024-09-16T06:42:22.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- 172.21.15.76:0/2248801417 wait complete. 2024-09-16T06:42:22.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: monmap: 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 Processor -- start 2024-09-16T06:42:22.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- start start 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: init 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-c con 0x7f10b8097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-a con 0x7f10b8099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient(hunting): picked mon.noname-b con 0x7f10b8139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80a2cd0 con 0x7f10b8099090 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b809c830 con 0x7f10b8139500 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b813c0e0 con 0x7f10b8097b60 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10be01d700 10 monclient(hunting): authenticate will time out at 1089.562821s 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10bd01b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f10b8097b60 0x7f10b8070510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51492/0 (socket says 172.21.15.76:51492) 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10bd01b700 1 -- 172.21.15.76:0/3919930431 learned_addr learned my addr 172.21.15.76:0/3919930431 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299881393 0 0) 0x7f10b813c0e0 con 0x7f10b8097b60 2024-09-16T06:42:22.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): my global_id is 124967 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f109c0035d0 con 0x7f10b8097b60 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960541484 0 0) 0x7f10b80a2cd0 con 0x7f10b8099090 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): my global_id is 154101 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b813c0e0 con 0x7f10b8099090 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924373881 0 0) 0x7f10b809c830 con 0x7f10b8139500 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): my global_id is 134674 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b80a2cd0 con 0x7f10b8139500 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884221206 0 0) 0x7f109c0035d0 con 0x7f10b8097b60 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10b809c830 con 0x7f10b8097b60 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10b4001990 con 0x7f10b8097b60 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): dump: 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.444 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433855199 0 0) 0x7f10b813c0e0 con 0x7f10b8099090 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f109c0035d0 con 0x7f10b8099090 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10a80019d0 con 0x7f10b8099090 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.425+0000 7f10adffb700 10 monclient(hunting): dump: 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.445 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054924063 0 0) 0x7f10b80a2cd0 con 0x7f10b8139500 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10b813c0e0 con 0x7f10b8139500 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2266082196 0 0) 0x7f10b809c830 con 0x7f10b8097b60 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.189:6789/0 conn(0x7f10b8139500 legacy=0x7f10b81d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6789/0 conn(0x7f10b8099090 legacy=0x7f10b8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 monclient: found mon.c 2024-09-16T06:42:22.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b809a540 con 0x7f10b8097b60 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 5 monclient: authenticate success, global_id 124967 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 10 monclient: _renew_subs 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8136d50 con 0x7f10b8097b60 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b4001d20 con 0x7f10b8097b60 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10b4003b90 con 0x7f10b8097b60 2024-09-16T06:42:22.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10af7fe700 4 set_mon_vals no callback set 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: dump: 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.449 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 10 monclient: _renew_subs 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b809a850 con 0x7f10b8097b60 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10b80994e0 con 0x7f10b8071330 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f10b4003cf0 con 0x7f10b8097b60 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(111..111 src has 66..111) v4 ==== 5731+0+0 (unknown 1678976670 0 0) 0x7f10b4043aa0 con 0x7f10b8097b60 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103512055 0 0) 0x7f10a80036a0 con 0x7f10b8071330 2024-09-16T06:42:22.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient(hunting): my global_id is 154104 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b809c830 con 0x7f10b8071330 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10bd81c700 10 monclient: get_auth_request con 0x7f109c0364b0 auth_method 0 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3032862879 0 0) 0x7f10b809c830 con 0x7f10b8071330 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10a80036a0 con 0x7f10b8071330 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10a8003a20 con 0x7f10b8071330 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: dump: 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.451 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1797667638 0 0) 0x7f10a80036a0 con 0x7f10b8071330 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10b80668a0 con 0x7f10b8071330 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f10a8003f00 con 0x7f10b8071330 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:22.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6789/0 conn(0x7f10b8071330 legacy=0x7f10b8133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.429+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6789/0 conn(0x7f10b8071330 legacy=0x7f10b8133960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10b80994e0 con 0x7f10b802c6b0 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629033987 0 0) 0x7f10a80041d0 con 0x7f10b802c6b0 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient(hunting): my global_id is 154107 2024-09-16T06:42:22.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a80036a0 con 0x7f10b802c6b0 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3706405476 0 0) 0x7f10a80036a0 con 0x7f10b802c6b0 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10a80041d0 con 0x7f10b802c6b0 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f10a80045d0 con 0x7f10b802c6b0 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient: dump: 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1167762702 0 0) 0x7f10a80041d0 con 0x7f10b802c6b0 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10b809fb80 con 0x7f10b802c6b0 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1744294724) 0x7f10a8004a50 con 0x7f10b802c6b0 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6789/0 conn(0x7f10b802c6b0 legacy=0x7f10b8134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10adffb700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6789/0 conn(0x7f10b802c6b0 legacy=0x7f10b8134070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.456 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.457 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.458 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:22.460 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.461 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.462 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.464 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.465 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:22.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:22.467 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:22.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.189:6812/3098810674 conn(0x7f109c0364b0 legacy=0x7f109c038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 10 monclient: shutdown 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 >> v1:172.21.15.76:6790/0 conn(0x7f10b8097b60 legacy=0x7f10b8070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10bd81c700 1 -- 172.21.15.76:0/3919930431 reap_dead start 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10bd81c700 1 -- 172.21.15.76:0/3919930431 reap_dead start 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 shutdown_connections 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 >> 172.21.15.76:0/3919930431 conn(0x7f10b80a0030 msgr2=0x7f10b80a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 shutdown_connections 2024-09-16T06:42:22.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.441+0000 7f10be01d700 1 -- 172.21.15.76:0/3919930431 wait complete. 2024-09-16T06:42:22.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:22.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:42:22.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:42:22.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.585+0000 7f55c3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:42:22.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.585+0000 7f55c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.585+0000 7f55c3fff700 10 monclient: monmap: 2024-09-16T06:42:22.590 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:22.591707+0000 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:22.591707+0000 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:22.591 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 Processor -- start 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- start start 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient: init 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f55c4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f55c4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f55c4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4098990 con 0x7f55c4099090 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4097f40 con 0x7f55c4139500 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c409d610 con 0x7f55c4097b60 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient(hunting): authenticate will time out at 1089.725033s 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f55c4097b60 0x7f55c409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51496/0 (socket says 172.21.15.76:51496) 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c2ffd700 1 -- 172.21.15.76:0/436175478 learned_addr learned my addr 172.21.15.76:0/436175478 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924565600 0 0) 0x7f55c409d610 con 0x7f55c4097b60 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): my global_id is 124970 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b00035d0 con 0x7f55c4097b60 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551340577 0 0) 0x7f55c4098990 con 0x7f55c4099090 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): my global_id is 154110 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c409d610 con 0x7f55c4099090 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232585800 0 0) 0x7f55c4097f40 con 0x7f55c4139500 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient(hunting): my global_id is 134677 2024-09-16T06:42:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4098990 con 0x7f55c4139500 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600611370 0 0) 0x7f55c409d610 con 0x7f55c4099090 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 >> v1:172.21.15.76:6790/0 conn(0x7f55c4097b60 legacy=0x7f55c409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 >> v1:172.21.15.189:6789/0 conn(0x7f55c4139500 legacy=0x7f55c413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 monclient: found mon.noname-a 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40986b0 con 0x7f55c4099090 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 5 monclient: authenticate success, global_id 154110 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55ac0015f0 con 0x7f55c4099090 2024-09-16T06:42:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: dump: 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.599 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55ac001890 con 0x7f55c4099090 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 4 set_mon_vals no callback set 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 1 -- 172.21.15.76:0/436175478 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55ac001c20 con 0x7f55c4099090 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c27fc700 10 monclient: dump: 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 10 monclient: shutdown 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- 172.21.15.76:0/436175478 >> v1:172.21.15.76:6789/0 conn(0x7f55c4099090 legacy=0x7f55c409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- 172.21.15.76:0/436175478 shutdown_connections 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- 172.21.15.76:0/436175478 >> 172.21.15.76:0/436175478 conn(0x7f55c40a0030 msgr2=0x7f55c40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- 172.21.15.76:0/436175478 shutdown_connections 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.589+0000 7f55c3fff700 1 -- 172.21.15.76:0/436175478 wait complete. 2024-09-16T06:42:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: monmap: 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 Processor -- start 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- start start 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: init 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f55c4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f55c4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f55c4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4139330 con 0x7f55c4099090 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40914f0 con 0x7f55c4139500 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c413c090 con 0x7f55c4097b60 2024-09-16T06:42:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f55c4097b60 0x7f55c4138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51512/0 (socket says 172.21.15.76:51512) 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient(hunting): authenticate will time out at 1089.730278s 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c2ffd700 1 -- 172.21.15.76:0/1690745505 learned_addr learned my addr 172.21.15.76:0/1690745505 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050409039 0 0) 0x7f55c4139330 con 0x7f55c4099090 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): my global_id is 154113 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a40035d0 con 0x7f55c4099090 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192477802 0 0) 0x7f55c413c090 con 0x7f55c4097b60 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): my global_id is 124973 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4139330 con 0x7f55c4097b60 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596276973 0 0) 0x7f55c40914f0 con 0x7f55c4139500 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): my global_id is 134680 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c413c090 con 0x7f55c4139500 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608743898 0 0) 0x7f55a40035d0 con 0x7f55c4099090 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c40914f0 con 0x7f55c4099090 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55ac001ba0 con 0x7f55c4099090 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): dump: 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:22.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196818789 0 0) 0x7f55c413c090 con 0x7f55c4139500 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55a40035d0 con 0x7f55c4139500 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2853404516 0 0) 0x7f55c40914f0 con 0x7f55c4099090 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6790/0 conn(0x7f55c4097b60 legacy=0x7f55c4138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.189:6789/0 conn(0x7f55c4139500 legacy=0x7f55c41d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 monclient: found mon.a 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40a2c10 con 0x7f55c4099090 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 5 monclient: authenticate success, global_id 154113 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _renew_subs 2024-09-16T06:42:22.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c409a540 con 0x7f55c4099090 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _renew_subs 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c408fae0 con 0x7f55c4099090 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55ac003b00 con 0x7f55c4099090 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55ac003e90 con 0x7f55c4099090 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55bbfff700 10 monclient: dump: 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.593+0000 7f55c1ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55c3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f55ac00e980 con 0x7f55c4099090 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f55c40994e0 con 0x7f55c408e5a0 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(111..111 src has 66..111) v4 ==== 5731+0+0 (unknown 1678976670 0 0) 0x7f55ac0429a0 con 0x7f55c4099090 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409563360 0 0) 0x7f55b4001a80 con 0x7f55c408e5a0 2024-09-16T06:42:22.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55c37fe700 10 monclient: get_auth_request con 0x7f55a4036170 auth_method 0 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient(hunting): my global_id is 124976 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40914f0 con 0x7f55c408e5a0 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3590902864 0 0) 0x7f55c40914f0 con 0x7f55c408e5a0 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b4001a80 con 0x7f55c408e5a0 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55b40048d0 con 0x7f55c408e5a0 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient: dump: 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.614 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2271512174 0 0) 0x7f55b4001a80 con 0x7f55c408e5a0 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55c4090c50 con 0x7f55c408e5a0 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f55b4004f00 con 0x7f55c408e5a0 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6790/0 conn(0x7f55c408e5a0 legacy=0x7f55c40702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.597+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6790/0 conn(0x7f55c408e5a0 legacy=0x7f55c40702a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f55c40994e0 con 0x7f55c402c6b0 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596111859 0 0) 0x7f55b4005070 con 0x7f55c402c6b0 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient(hunting): my global_id is 124979 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b4001a80 con 0x7f55c402c6b0 2024-09-16T06:42:22.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3339889956 0 0) 0x7f55b4001a80 con 0x7f55c402c6b0 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b4005070 con 0x7f55c402c6b0 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55b4005470 con 0x7f55c402c6b0 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient: dump: 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4088155391 0 0) 0x7f55b4005070 con 0x7f55c402c6b0 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55c409fb80 con 0x7f55c402c6b0 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 3577360096) 0x7f55b4006950 con 0x7f55c402c6b0 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6790/0 conn(0x7f55c402c6b0 legacy=0x7f55c40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55bbfff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6790/0 conn(0x7f55c402c6b0 legacy=0x7f55c40709b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.619 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.620 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:22.622 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:22.623 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:22.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.628 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.629 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:42:22.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:22.631 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.189:6812/3098810674 conn(0x7f55a4036170 legacy=0x7f55a4038630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 10 monclient: shutdown 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 >> v1:172.21.15.76:6789/0 conn(0x7f55c4099090 legacy=0x7f55c4138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c37fe700 1 -- 172.21.15.76:0/1690745505 reap_dead start 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c37fe700 1 -- 172.21.15.76:0/1690745505 reap_dead start 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 shutdown_connections 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 >> 172.21.15.76:0/1690745505 conn(0x7f55c40a0030 msgr2=0x7f55c40a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 shutdown_connections 2024-09-16T06:42:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.609+0000 7f55c3fff700 1 -- 172.21.15.76:0/1690745505 wait complete. 2024-09-16T06:42:22.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:22.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:42:22.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:42:22.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: get_monmap_and_config 2024-09-16T06:42:22.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: monmap: 2024-09-16T06:42:22.760 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:22.760641+0000 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:22.760641+0000 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:22.761 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 1 Processor -- start 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 1 -- start start 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: init 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-a con 0x7f7068097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-b con 0x7f7068099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-c con 0x7f7068139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7068098990 con 0x7f7068097b60 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7068097f40 con 0x7f7068139500 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706809d610 con 0x7f7068099090 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient(hunting): authenticate will time out at 1089.893882s 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f7066ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7068099090 0x7f706809f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51530/0 (socket says 172.21.15.76:51530) 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f7066ffd700 1 -- 172.21.15.76:0/3033469269 learned_addr learned my addr 172.21.15.76:0/3033469269 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134460501 0 0) 0x7f7068098990 con 0x7f7068097b60 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): my global_id is 154116 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70500035d0 con 0x7f7068097b60 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149950958 0 0) 0x7f706809d610 con 0x7f7068099090 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): my global_id is 124982 2024-09-16T06:42:22.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7068098990 con 0x7f7068099090 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024013927 0 0) 0x7f7068097f40 con 0x7f7068139500 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient(hunting): my global_id is 134683 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706809d610 con 0x7f7068139500 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381202307 0 0) 0x7f70500035d0 con 0x7f7068097b60 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 >> v1:172.21.15.76:6790/0 conn(0x7f7068099090 legacy=0x7f706809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 >> v1:172.21.15.189:6789/0 conn(0x7f7068139500 legacy=0x7f706813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 monclient: found mon.noname-a 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680986b0 con 0x7f7068097b60 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 5 monclient: authenticate success, global_id 154116 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f705c001610 con 0x7f7068097b60 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: dump: 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.768 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f705c0018f0 con 0x7f7068097b60 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:22.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 1 -- 172.21.15.76:0/3033469269 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f705c001c80 con 0x7f7068097b60 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f706cdfd700 4 set_mon_vals no callback set 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.757+0000 7f70667fc700 10 monclient: dump: 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: shutdown 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- 172.21.15.76:0/3033469269 >> v1:172.21.15.76:6789/0 conn(0x7f7068097b60 legacy=0x7f706809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- 172.21.15.76:0/3033469269 shutdown_connections 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- 172.21.15.76:0/3033469269 >> 172.21.15.76:0/3033469269 conn(0x7f70680a0030 msgr2=0x7f70680a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- 172.21.15.76:0/3033469269 shutdown_connections 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- 172.21.15.76:0/3033469269 wait complete. 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: monmap: 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.771 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 Processor -- start 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- start start 2024-09-16T06:42:22.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: init 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-a con 0x7f7068097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-c con 0x7f7068099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient(hunting): picked mon.noname-b con 0x7f7068139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706813c340 con 0x7f7068097b60 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706809a540 con 0x7f7068139500 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70680962d0 con 0x7f7068099090 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f706cdfd700 10 monclient(hunting): authenticate will time out at 1089.899138s 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f7066ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7068099090 0x7f70681cced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51538/0 (socket says 172.21.15.76:51538) 2024-09-16T06:42:22.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f7066ffd700 1 -- 172.21.15.76:0/3994891408 learned_addr learned my addr 172.21.15.76:0/3994891408 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474741699 0 0) 0x7f70680962d0 con 0x7f7068099090 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): my global_id is 124985 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70540035d0 con 0x7f7068099090 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915937249 0 0) 0x7f70680986b0 con 0x7f7068097b60 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): my global_id is 154119 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70680962d0 con 0x7f7068097b60 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523915326 0 0) 0x7f706809a540 con 0x7f7068139500 2024-09-16T06:42:22.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 10 monclient(hunting): my global_id is 134686 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.761+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70680986b0 con 0x7f7068139500 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429639743 0 0) 0x7f70680962d0 con 0x7f7068097b60 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706809a540 con 0x7f7068097b60 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f705c001430 con 0x7f7068097b60 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): dump: 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.778 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1349209723 0 0) 0x7f70540035d0 con 0x7f7068099090 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70680962d0 con 0x7f7068099090 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7060001bf0 con 0x7f7068099090 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): dump: 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3562540749 0 0) 0x7f70680986b0 con 0x7f7068139500 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70540035d0 con 0x7f7068139500 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f70580018b0 con 0x7f7068139500 2024-09-16T06:42:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): dump: 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2275714906 0 0) 0x7f706809a540 con 0x7f7068097b60 2024-09-16T06:42:22.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.76:6790/0 conn(0x7f7068099090 legacy=0x7f70681cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6789/0 conn(0x7f7068139500 legacy=0x7f70681d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 monclient: found mon.a 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680a2c30 con 0x7f7068097b60 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 5 monclient: authenticate success, global_id 154119 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 10 monclient: _renew_subs 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706808f5b0 con 0x7f7068097b60 2024-09-16T06:42:22.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f705c003950 con 0x7f7068097b60 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f705c003c60 con 0x7f7068097b60 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: dump: 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.783 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f7065ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 10 monclient: _renew_subs 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7068096d90 con 0x7f7068097b60 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f706808caa0 con 0x7f706808c6c0 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f705c00e6d0 con 0x7f7068097b60 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(111..111 src has 66..111) v4 ==== 5731+0+0 (unknown 1678976670 0 0) 0x7f705c042880 con 0x7f7068097b60 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f7067fff700 10 monclient: get_auth_request con 0x7f70540364e0 auth_method 0 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031190782 0 0) 0x7f706808caa0 con 0x7f706808c6c0 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient(hunting): my global_id is 134689 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706809a540 con 0x7f706808c6c0 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 384428360 0 0) 0x7f706809a540 con 0x7f706808c6c0 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706808caa0 con 0x7f706808c6c0 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7060004db0 con 0x7f706808c6c0 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 10 monclient: dump: 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1038631605 0 0) 0x7f706808caa0 con 0x7f706808c6c0 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.765+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f706808ed10 con 0x7f706808c6c0 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.769+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f70600020e0 con 0x7f706808c6c0 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.769+0000 7f704ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.769+0000 7f704ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:22.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.769+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6789/0 conn(0x7f706808c6c0 legacy=0x7f7068061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.769+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6789/0 conn(0x7f706808c6c0 legacy=0x7f7068061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f706cdfd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:22.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f706809fb00 con 0x7f706802c820 2024-09-16T06:42:22.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125659663 0 0) 0x7f706809fb00 con 0x7f706802c820 2024-09-16T06:42:22.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient(hunting): my global_id is 134692 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706808caa0 con 0x7f706802c820 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1546617540 0 0) 0x7f706808caa0 con 0x7f706802c820 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706809fb00 con 0x7f706802c820 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7060005570 con 0x7f706802c820 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient: dump: 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4192332 0 0) 0x7f706809fb00 con 0x7f706802c820 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7068099500 con 0x7f706802c820 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1733626431) 0x7f7060006910 con 0x7f706802c820 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6789/0 conn(0x7f706802c820 legacy=0x7f706806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.777+0000 7f704ffff700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6789/0 conn(0x7f706802c820 legacy=0x7f706806eeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.792 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.793 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:22.795 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:22.796 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.798 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:22.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.802 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:22.803 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.189:6812/3098810674 conn(0x7f70540364e0 legacy=0x7f70540389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 10 monclient: shutdown 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 >> v1:172.21.15.76:6789/0 conn(0x7f7068097b60 legacy=0x7f7068138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f7067fff700 1 -- 172.21.15.76:0/3994891408 reap_dead start 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f7067fff700 1 -- 172.21.15.76:0/3994891408 reap_dead start 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 shutdown_connections 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 >> 172.21.15.76:0/3994891408 conn(0x7f70680a0030 msgr2=0x7f70680a1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 shutdown_connections 2024-09-16T06:42:22.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.781+0000 7f706cdfd700 1 -- 172.21.15.76:0/3994891408 wait complete. 2024-09-16T06:42:22.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:22.806 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:42:22.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 10 monclient: get_monmap_and_config 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 10 monclient: monmap: 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:22.923150+0000 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:22.923150+0000 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:22.923 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:22.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 1 Processor -- start 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 1 -- start start 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.917+0000 7fcb54307700 10 monclient: init 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-b con 0x7fcb4c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-c con 0x7fcb4c09fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-a con 0x7fcb4c139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c09cd80 con 0x7fcb4c139b70 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c099fd0 con 0x7fcb4c09fd20 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c09a410 con 0x7fcb4c09f230 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient(hunting): authenticate will time out at 1090.056438s 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb520a3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcb4c09f230 0x7fcb4c09f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51548/0 (socket says 172.21.15.76:51548) 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb520a3700 1 -- 172.21.15.76:0/3397024920 learned_addr learned my addr 172.21.15.76:0/3397024920 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995218572 0 0) 0x7fcb4c09a410 con 0x7fcb4c09f230 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): my global_id is 124988 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb380035d0 con 0x7fcb4c09f230 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692271197 0 0) 0x7fcb4c09cd80 con 0x7fcb4c139b70 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): my global_id is 154122 2024-09-16T06:42:22.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c09a410 con 0x7fcb4c139b70 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345402349 0 0) 0x7fcb4c099fd0 con 0x7fcb4c09fd20 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient(hunting): my global_id is 134695 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c09cd80 con 0x7fcb4c09fd20 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815093599 0 0) 0x7fcb380035d0 con 0x7fcb4c09f230 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 >> v1:172.21.15.189:6789/0 conn(0x7fcb4c09fd20 legacy=0x7fcb4c138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 >> v1:172.21.15.76:6789/0 conn(0x7fcb4c139b70 legacy=0x7fcb4c13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 monclient: found mon.noname-b 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c09de90 con 0x7fcb4c09f230 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb48001610 con 0x7fcb4c09f230 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 5 monclient: authenticate success, global_id 124988 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: dump: 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001880 con 0x7fcb4c09f230 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 4 set_mon_vals no callback set 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 1 -- 172.21.15.76:0/3397024920 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb48001c50 con 0x7fcb4c09f230 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb510a1700 10 monclient: dump: 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: shutdown 2024-09-16T06:42:22.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:22.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- 172.21.15.76:0/3397024920 >> v1:172.21.15.76:6790/0 conn(0x7fcb4c09f230 legacy=0x7fcb4c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- 172.21.15.76:0/3397024920 shutdown_connections 2024-09-16T06:42:22.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- 172.21.15.76:0/3397024920 >> 172.21.15.76:0/3397024920 conn(0x7fcb4c0a3e50 msgr2=0x7fcb4c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:22.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- 172.21.15.76:0/3397024920 shutdown_connections 2024-09-16T06:42:22.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 1 -- 172.21.15.76:0/3397024920 wait complete. 2024-09-16T06:42:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: build_initial_monmap 2024-09-16T06:42:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.921+0000 7fcb54307700 10 monclient: monmap: 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 Processor -- start 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- start start 2024-09-16T06:42:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: init 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-b con 0x7fcb4c09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-a con 0x7fcb4c09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient(hunting): picked mon.noname-c con 0x7fcb4c139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c13a0e0 con 0x7fcb4c09fd20 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c072450 con 0x7fcb4c09f230 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c134030 con 0x7fcb4c139b70 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient(hunting): authenticate will time out at 1090.061192s 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb518a2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcb4c09fd20 0x7fcb4c1ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38102/0 (socket says 172.21.15.76:38102) 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb518a2700 1 -- 172.21.15.76:0/4062866666 learned_addr learned my addr 172.21.15.76:0/4062866666 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521880221 0 0) 0x7fcb4c134030 con 0x7fcb4c139b70 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): my global_id is 124991 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb300035d0 con 0x7fcb4c139b70 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233581752 0 0) 0x7fcb4c072450 con 0x7fcb4c09f230 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): my global_id is 134698 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c134030 con 0x7fcb4c09f230 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780190570 0 0) 0x7fcb3c0011d0 con 0x7fcb4c09fd20 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:22.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): my global_id is 154125 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4c072450 con 0x7fcb4c09fd20 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1408499486 0 0) 0x7fcb300035d0 con 0x7fcb4c139b70 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb3c0011d0 con 0x7fcb4c139b70 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799220578 0 0) 0x7fcb4c134030 con 0x7fcb4c09f230 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb300035d0 con 0x7fcb4c09f230 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb44001c30 con 0x7fcb4c139b70 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.939 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb480015d0 con 0x7fcb4c09f230 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:22.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228197188 0 0) 0x7fcb4c072450 con 0x7fcb4c09fd20 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb4c134030 con 0x7fcb4c09fd20 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb3c0018f0 con 0x7fcb4c09fd20 2024-09-16T06:42:22.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 57687279 0 0) 0x7fcb3c0011d0 con 0x7fcb4c139b70 2024-09-16T06:42:22.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 >> v1:172.21.15.189:6789/0 conn(0x7fcb4c09f230 legacy=0x7fcb4c133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 >> v1:172.21.15.76:6789/0 conn(0x7fcb4c09fd20 legacy=0x7fcb4c1ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 monclient: found mon.c 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c11ca00 con 0x7fcb4c139b70 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 5 monclient: authenticate success, global_id 124991 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _renew_subs 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c11cce0 con 0x7fcb4c139b70 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _renew_subs 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c11bcc0 con 0x7fcb4c139b70 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb44003a40 con 0x7fcb4c139b70 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb44003d50 con 0x7fcb4c139b70 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb42ffd700 10 monclient: dump: 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:22.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.925+0000 7fcb43fff700 4 set_mon_vals no callback set 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb54307700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcb4400e7c0 con 0x7fcb4c139b70 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb54307700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb54307700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb34005100 con 0x7fcb4c139b70 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(111..111 src has 66..111) v4 ==== 5731+0+0 (unknown 1678976670 0 0) 0x7fcb44042ab0 con 0x7fcb4c139b70 2024-09-16T06:42:22.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.929+0000 7fcb520a3700 10 monclient: get_auth_request con 0x7fcb30036550 auth_method 0 2024-09-16T06:42:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.933+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcb440135b0 con 0x7fcb4c139b70 2024-09-16T06:42:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.933+0000 7fcb42ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:22.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:22.933+0000 7fcb42ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:23.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.127+0000 7f2416ffd700 1 -- 172.21.15.189:0/3006271617 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v112) v1 ==== 168+0+0 (unknown 435931192 0 0) 0x7f2408012ca0 con 0x7f242009f690 2024-09-16T06:42:23.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.127+0000 7f2416ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T06:42:23.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.127+0000 7f2416ffd700 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2400035fa0 legacy=0x7f2400038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 10 monclient: shutdown 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 >> v1:172.21.15.189:6789/0 conn(0x7f242009f690 legacy=0x7f24201cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 shutdown_connections 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 >> 172.21.15.189:0/3006271617 conn(0x7f242009fe70 msgr2=0x7f24200a2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 shutdown_connections 2024-09-16T06:42:23.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.131+0000 7f241f59e700 1 -- 172.21.15.189:0/3006271617 wait complete. 2024-09-16T06:42:23.134 INFO:tasks.workunit.client.0.smithi189.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T06:42:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-16T06:42:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:42:23.150 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-16T06:42:23.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb54307700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:42:23.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb54307700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:42:23.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb54307700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:23.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fcb34005bc0 con 0x7fcb4c139b70 2024-09-16T06:42:23.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb42ffd700 1 -- 172.21.15.76:0/4062866666 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fcb440135b0 con 0x7fcb4c139b70 2024-09-16T06:42:23.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb42ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:42:23.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.169+0000 7fcb42ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:23.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=905155413,config=1097430493,health=1361790972,kv=1588170259,logm=2837904873}) 2024-09-16T06:42:23.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=10,kv=12,logm=55} crc {auth=2175202686,config=1097430493,health=1308591922,kv=1008284096,logm=137723923}) 2024-09-16T06:42:23.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=905155413,config=1097430493,health=1361790972,kv=1588170259,logm=2837904873}) 2024-09-16T06:42:23.176 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=1,health=12,kv=11,logm=55} crc {auth=2232065025,config=1228700967,health=1361790972,kv=3850320945,logm=2085448231}) 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999399449}) 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694964094}) 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999399449}) 2024-09-16T06:42:23.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005685587}) 2024-09-16T06:42:23.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcb30036550 legacy=0x7fcb30038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 10 monclient: shutdown 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 >> v1:172.21.15.76:6790/0 conn(0x7fcb4c139b70 legacy=0x7fcb4c1d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 shutdown_connections 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 >> 172.21.15.76:0/4062866666 conn(0x7fcb4c0a3e50 msgr2=0x7fcb4c138840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 shutdown_connections 2024-09-16T06:42:23.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:23.173+0000 7fcb54307700 1 -- 172.21.15.76:0/4062866666 wait complete. 2024-09-16T06:42:23.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072173714}) 2024-09-16T06:42:23.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104835656}) 2024-09-16T06:42:23.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072173714}) 2024-09-16T06:42:23.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.173+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376307912}) 2024-09-16T06:42:23.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678524993}) 2024-09-16T06:42:23.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=526682451}) 2024-09-16T06:42:23.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678524993}) 2024-09-16T06:42:23.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.177+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=310326453}) 2024-09-16T06:42:23.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836061804}) 2024-09-16T06:42:23.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599529585}) 2024-09-16T06:42:23.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836061804}) 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915968198}) 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=356156212}) 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928728384}) 2024-09-16T06:42:23.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=356156212}) 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3950715496}) 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517749808}) 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421328215}) 2024-09-16T06:42:23.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517749808}) 2024-09-16T06:42:23.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.181+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763865136}) 2024-09-16T06:42:23.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686129146}) 2024-09-16T06:42:23.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499967821}) 2024-09-16T06:42:23.192 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.192 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686129146}) 2024-09-16T06:42:23.192 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=816147698}) 2024-09-16T06:42:23.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838729473}) 2024-09-16T06:42:23.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592964834}) 2024-09-16T06:42:23.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838729473}) 2024-09-16T06:42:23.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848336250}) 2024-09-16T06:42:23.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282908066}) 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351206902}) 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282908066}) 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.185+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1455051686}) 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=516893909}) 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358280787}) 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=516893909}) 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91848717}) 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046619913}) 2024-09-16T06:42:23.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974124267}) 2024-09-16T06:42:23.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046619913}) 2024-09-16T06:42:23.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214974597}) 2024-09-16T06:42:23.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:42:23.206 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.206 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670955583}) 2024-09-16T06:42:23.206 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537180178}) 2024-09-16T06:42:23.206 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.206 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670955583}) 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.189+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282623941}) 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001336206}) 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2850583949}) 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.207 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001336206}) 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3873762218}) 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094964710}) 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032935117}) 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094964710}) 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788109424}) 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257987346}) 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281299768}) 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257987346}) 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018699755}) 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814150307}) 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766246867}) 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814150307}) 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.193+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812283813}) 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=1011037794,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3696258066,mon_config_key=953308755,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3497992868,osd_snap=2033388227,osdmap=3901218877}) 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=9} crc {logm=2587860412,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2845912146,mon_config_key=3652152013,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3497992868,osd_snap=2033388227,osdmap=4188910316}) 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.211 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=1011037794,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3696258066,mon_config_key=953308755,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3497992868,osd_snap=2033388227,osdmap=3901218877}) 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.197+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=38,mdsmap=2,mgr=8,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=307030713,mdsmap=3140514009,mgr=1752639018,mgrstat=2845912146,mon_config_key=4279594037,monmap=1887442723,osd_metadata=1269664135,osd_pg_creating=3497992868,osd_snap=2033388227,osdmap=4070113882}) 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=75} crc {osdmap=2255750943}) 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=2169674843}) 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=75} crc {osdmap=2255750943}) 2024-09-16T06:42:23.212 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:23.201+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=72} crc {osdmap=3359580276}) 2024-09-16T06:42:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: get_monmap_and_config 2024-09-16T06:42:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: monmap: 2024-09-16T06:42:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:23.265128+0000 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:23.265128+0000 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 1 Processor -- start 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 1 -- start start 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: init 2024-09-16T06:42:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-c con 0x7f40e01352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-a con 0x7f40e0135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-b con 0x7f40e00a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e009b050 con 0x7f40e0135db0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e006ab20 con 0x7f40e01352c0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e006a840 con 0x7f40e00a2030 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient(hunting): authenticate will time out at 1095.778064s 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e6594700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f40e01352c0 0x7f40e01356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46498/0 (socket says 172.21.15.189:46498) 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e6594700 1 -- 172.21.15.189:0/3299690133 learned_addr learned my addr 172.21.15.189:0/3299690133 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426350737 0 0) 0x7f40e006ab20 con 0x7f40e01352c0 2024-09-16T06:42:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): my global_id is 134701 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40cc0035d0 con 0x7f40e01352c0 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565414282 0 0) 0x7f40e006a840 con 0x7f40e00a2030 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): my global_id is 124994 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e006ab20 con 0x7f40e00a2030 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222865870 0 0) 0x7f40e009b050 con 0x7f40e0135db0 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient(hunting): my global_id is 154128 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e006a840 con 0x7f40e0135db0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281174381 0 0) 0x7f40cc0035d0 con 0x7f40e01352c0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 >> v1:172.21.15.76:6790/0 conn(0x7f40e00a2030 legacy=0x7f40e013df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 >> v1:172.21.15.76:6789/0 conn(0x7f40e0135db0 legacy=0x7f40e013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 monclient: found mon.noname-c 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e006c1b0 con 0x7f40e01352c0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f40d4001590 con 0x7f40e01352c0 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 5 monclient: authenticate success, global_id 134701 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: dump: 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40d40017b0 con 0x7f40e01352c0 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 1 -- 172.21.15.189:0/3299690133 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f40d4001b40 con 0x7f40e01352c0 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e7596700 4 set_mon_vals no callback set 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.263+0000 7f40e5592700 10 monclient: dump: 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: shutdown 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- 172.21.15.189:0/3299690133 >> v1:172.21.15.189:6789/0 conn(0x7f40e01352c0 legacy=0x7f40e01356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- 172.21.15.189:0/3299690133 shutdown_connections 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- 172.21.15.189:0/3299690133 >> 172.21.15.189:0/3299690133 conn(0x7f40e00a4550 msgr2=0x7f40e00a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- 172.21.15.189:0/3299690133 shutdown_connections 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- 172.21.15.189:0/3299690133 wait complete. 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: monmap: 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 Processor -- start 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- start start 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: init 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-a con 0x7f40e00a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-c con 0x7f40e01352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient(hunting): picked mon.noname-b con 0x7f40e0135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e00a6aa0 con 0x7f40e00a2030 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e01ca8c0 con 0x7f40e0135db0 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e00a64e0 con 0x7f40e01352c0 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e7596700 10 monclient(hunting): authenticate will time out at 1095.782561s 2024-09-16T06:42:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e6d95700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f40e0135db0 0x7f40e01d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46502/0 (socket says 172.21.15.189:46502) 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e6594700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f40e00a2030 0x7f40e01323d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:52540/0 (socket says 172.21.15.189:52540) 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40e6d95700 1 -- 172.21.15.189:0/1621687762 learned_addr learned my addr 172.21.15.189:0/1621687762 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809434094 0 0) 0x7f40e01ca8c0 con 0x7f40e0135db0 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): my global_id is 134704 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c40035d0 con 0x7f40e0135db0 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979307482 0 0) 0x7f40e006c1b0 con 0x7f40e00a2030 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): my global_id is 154131 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e01ca8c0 con 0x7f40e00a2030 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093895431 0 0) 0x7f40e00a64e0 con 0x7f40e01352c0 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): my global_id is 124997 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e006c1b0 con 0x7f40e01352c0 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1144182360 0 0) 0x7f40c40035d0 con 0x7f40e0135db0 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40e00a64e0 con 0x7f40e0135db0 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f40d8001c70 con 0x7f40e0135db0 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301498279 0 0) 0x7f40e01ca8c0 con 0x7f40e00a2030 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c40035d0 con 0x7f40e00a2030 2024-09-16T06:42:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f40d40010b0 con 0x7f40e00a2030 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.267+0000 7f40d2ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 971306903 0 0) 0x7f40e00a64e0 con 0x7f40e0135db0 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 >> v1:172.21.15.76:6790/0 conn(0x7f40e01352c0 legacy=0x7f40e01cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 >> v1:172.21.15.76:6789/0 conn(0x7f40e00a2030 legacy=0x7f40e01323d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 monclient: found mon.b 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e009e820 con 0x7f40e0135db0 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 5 monclient: authenticate success, global_id 134704 2024-09-16T06:42:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _renew_subs 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e012fcf0 con 0x7f40e0135db0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _renew_subs 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e012f710 con 0x7f40e0135db0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40d8002000 con 0x7f40e0135db0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f40d8003e90 con 0x7f40e0135db0 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e4d91700 4 set_mon_vals no callback set 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 10 monclient: dump: 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.289 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f40d8003ff0 con 0x7f40e0135db0 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7f40d8043de0 con 0x7f40e0135db0 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e7596700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8005100 con 0x7f40e0135db0 2024-09-16T06:42:23.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.271+0000 7f40e6594700 10 monclient: get_auth_request con 0x7f40c4036230 auth_method 0 2024-09-16T06:42:23.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.279+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f40d80042f0 con 0x7f40e0135db0 2024-09-16T06:42:23.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.279+0000 7f40d2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.279+0000 7f40d2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:23.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40e7596700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T06:42:23.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40e7596700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T06:42:23.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40e7596700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40e7596700 1 -- 172.21.15.189:0/1621687762 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7f40c8005930 con 0x7f40e0135db0 2024-09-16T06:42:23.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40d2ffd700 1 -- 172.21.15.189:0/1621687762 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v112) v1 ==== 154+0+0 (unknown 703388148 0 0) 0x7f40d8012f80 con 0x7f40e0135db0 2024-09-16T06:42:23.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40d2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T06:42:23.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.455+0000 7f40d2ffd700 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T06:42:23.459 INFO:tasks.workunit.client.0.smithi189.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T06:42:23.463 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:42:23.464 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:23.464 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.12 2024-09-16T06:42:23.464 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 155 2024-09-16T06:42:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.559+0000 7eff1c8c9700 10 monclient: get_monmap_and_config 2024-09-16T06:42:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.559+0000 7eff1c8c9700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.559+0000 7eff1c8c9700 10 monclient: monmap: 2024-09-16T06:42:23.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:23.564241+0000 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:23.564241+0000 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:23.564 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 Processor -- start 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- start start 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient: init 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-a con 0x7eff1809ed40 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-c con 0x7eff18097860 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-b con 0x7eff1809f830 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18099b10 con 0x7eff1809ed40 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18098890 con 0x7eff18097860 2024-09-16T06:42:23.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18098b70 con 0x7eff1809f830 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient(hunting): authenticate will time out at 1096.077376s 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff0dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7eff18097860 0x7eff181386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46510/0 (socket says 172.21.15.189:46510) 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff0dd9b700 1 -- 172.21.15.189:0/3274359419 learned_addr learned my addr 172.21.15.189:0/3274359419 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415266569 0 0) 0x7eff18098890 con 0x7eff18097860 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): my global_id is 134707 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff040035d0 con 0x7eff18097860 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173750985 0 0) 0x7eff18099b10 con 0x7eff1809ed40 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): my global_id is 154134 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18098890 con 0x7eff1809ed40 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289907133 0 0) 0x7eff18098b70 con 0x7eff1809f830 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient(hunting): my global_id is 125000 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18099b10 con 0x7eff1809f830 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312153335 0 0) 0x7eff040035d0 con 0x7eff18097860 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 >> v1:172.21.15.76:6790/0 conn(0x7eff1809f830 legacy=0x7eff1813be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 >> v1:172.21.15.76:6789/0 conn(0x7eff1809ed40 legacy=0x7eff1809f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 monclient: found mon.noname-c 2024-09-16T06:42:23.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180973a0 con 0x7eff18097860 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 5 monclient: authenticate success, global_id 134707 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7eff00001610 con 0x7eff18097860 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: dump: 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.571 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff000018f0 con 0x7eff18097860 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:23.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 4 set_mon_vals no callback set 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 1 -- 172.21.15.189:0/3274359419 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7eff00001c80 con 0x7eff18097860 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff15d9b700 10 monclient: dump: 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 10 monclient: shutdown 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3274359419 >> v1:172.21.15.189:6789/0 conn(0x7eff18097860 legacy=0x7eff181386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3274359419 shutdown_connections 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3274359419 >> 172.21.15.189:0/3274359419 conn(0x7eff1809fe30 msgr2=0x7eff180a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3274359419 shutdown_connections 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.563+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3274359419 wait complete. 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: monmap: 2024-09-16T06:42:23.574 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:23.575 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 Processor -- start 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- start start 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: init 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-c con 0x7eff18097860 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-b con 0x7eff1809ed40 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient(hunting): picked mon.noname-a con 0x7eff1809f830 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1809a590 con 0x7eff1809f830 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1812c7a0 con 0x7eff1809ed40 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1812afa0 con 0x7eff18097860 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient(hunting): authenticate will time out at 1096.081639s 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7eff1809ed40 0x7eff181335c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46522/0 (socket says 172.21.15.189:46522) 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0dd9b700 1 -- 172.21.15.189:0/3765431150 learned_addr learned my addr 172.21.15.189:0/3765431150 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088379085 0 0) 0x7eff1812c7a0 con 0x7eff1809ed40 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): my global_id is 134710 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef80035d0 con 0x7eff1809ed40 2024-09-16T06:42:23.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048253499 0 0) 0x7eff1812afa0 con 0x7eff18097860 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): my global_id is 125003 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff1812c7a0 con 0x7eff18097860 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857488906 0 0) 0x7eff18099b10 con 0x7eff1809f830 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): my global_id is 154137 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff1812afa0 con 0x7eff1809f830 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1987061984 0 0) 0x7efef80035d0 con 0x7eff1809ed40 2024-09-16T06:42:23.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff18099b10 con 0x7eff1809ed40 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7eff00001c00 con 0x7eff1809ed40 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.581 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2184747640 0 0) 0x7eff18099b10 con 0x7eff1809ed40 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 >> v1:172.21.15.76:6790/0 conn(0x7eff18097860 legacy=0x7eff1812b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 >> v1:172.21.15.76:6789/0 conn(0x7eff1809f830 legacy=0x7eff181d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 monclient: found mon.b 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18133cd0 con 0x7eff1809ed40 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 5 monclient: authenticate success, global_id 134710 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _renew_subs 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff18129b40 con 0x7eff1809ed40 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _renew_subs 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff18129fd0 con 0x7eff1809ed40 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff00004ca0 con 0x7eff1809ed40 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7eff00004fb0 con 0x7eff1809ed40 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff1559a700 4 set_mon_vals no callback set 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.567+0000 7eff0f7fe700 10 monclient: dump: 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff1c8c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff1c8c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff1c8c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc005100 con 0x7eff1809ed40 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7eff0000fa20 con 0x7eff1809ed40 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7eff00043c70 con 0x7eff1809ed40 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.571+0000 7eff1659c700 10 monclient: get_auth_request con 0x7efef80362c0 auth_method 0 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.579+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7eff00014830 con 0x7eff1809ed40 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.579+0000 7eff0f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.579+0000 7eff0f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff1c8c9700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff1c8c9700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff1c8c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7efefc005bc0 con 0x7eff1809ed40 2024-09-16T06:42:23.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (unknown 3664589273 0 1335834520) 0x7eff00014650 con 0x7eff1809ed40 2024-09-16T06:42:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff0f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:23.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.747+0000 7eff0f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7efefc005680 con 0x7eff1809ed40 2024-09-16T06:42:23.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff0f7fe700 1 -- 172.21.15.189:0/3765431150 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (unknown 3664589273 0 1335834520) 0x7eff00014830 con 0x7eff1809ed40 2024-09-16T06:42:23.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff0f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:23.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff0f7fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 >> v1:172.21.15.189:6812/3098810674 conn(0x7efef80362c0 legacy=0x7efef8038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 10 monclient: shutdown 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 >> v1:172.21.15.189:6789/0 conn(0x7eff1809ed40 legacy=0x7eff181335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 shutdown_connections 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 >> 172.21.15.189:0/3765431150 conn(0x7eff1809fe30 msgr2=0x7eff181b93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 shutdown_connections 2024-09-16T06:42:23.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.847+0000 7eff1c8c9700 1 -- 172.21.15.189:0/3765431150 wait complete. 2024-09-16T06:42:23.860 INFO:tasks.workunit.client.0.smithi189.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-16T06:42:23.862 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd tree 2024-09-16T06:42:23.862 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep osd.14 2024-09-16T06:42:23.862 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 155 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: get_monmap_and_config 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: monmap: 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:23.962941+0000 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:23.962941+0000 2024-09-16T06:42:23.962 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.963 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:23.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:23.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 1 Processor -- start 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 1 -- start start 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: init 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-a con 0x7f0cf80984b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-c con 0x7f0cf8072140 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-b con 0x7f0cf81384d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.959+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf809b2a0 con 0x7f0cf80984b0 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8099940 con 0x7f0cf8072140 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8099d10 con 0x7f0cf81384d0 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient(hunting): authenticate will time out at 1096.476050s 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0cf8072140 0x7f0cf8098fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46532/0 (socket says 172.21.15.189:46532) 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf6ffd700 1 -- 172.21.15.189:0/3596621231 learned_addr learned my addr 172.21.15.189:0/3596621231 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809598442 0 0) 0x7f0cf8099940 con 0x7f0cf8072140 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): my global_id is 134713 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00035d0 con 0x7f0cf8072140 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189490964 0 0) 0x7f0cf809b2a0 con 0x7f0cf80984b0 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): my global_id is 154140 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf8099940 con 0x7f0cf80984b0 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665922925 0 0) 0x7f0cf8099d10 con 0x7f0cf81384d0 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient(hunting): my global_id is 125006 2024-09-16T06:42:23.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf809b2a0 con 0x7f0cf81384d0 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539333169 0 0) 0x7f0ce00035d0 con 0x7f0cf8072140 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 >> v1:172.21.15.76:6790/0 conn(0x7f0cf81384d0 legacy=0x7f0cf813a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 >> v1:172.21.15.76:6789/0 conn(0x7f0cf80984b0 legacy=0x7f0cf8098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 monclient: found mon.noname-c 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80996b0 con 0x7f0cf8072140 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 5 monclient: authenticate success, global_id 134713 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0cec001610 con 0x7f0cf8072140 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: dump: 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.970 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cec0018f0 con 0x7f0cf8072140 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 4 set_mon_vals no callback set 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 1 -- 172.21.15.189:0/3596621231 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0cec001c80 con 0x7f0cf8072140 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cf67fc700 10 monclient: dump: 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient: shutdown 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/3596621231 >> v1:172.21.15.189:6789/0 conn(0x7f0cf8072140 legacy=0x7f0cf8098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/3596621231 shutdown_connections 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/3596621231 >> 172.21.15.189:0/3596621231 conn(0x7f0cf809fe30 msgr2=0x7f0cf80a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/3596621231 shutdown_connections 2024-09-16T06:42:23.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/3596621231 wait complete. 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient: build_initial_monmap 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.963+0000 7f0cfe0f7700 10 monclient: monmap: 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:23.976 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 Processor -- start 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- start start 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: init 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-b con 0x7f0cf8072140 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-a con 0x7f0cf80984b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient(hunting): picked mon.noname-c con 0x7f0cf81384d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf809a590 con 0x7f0cf80984b0 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8096c30 con 0x7f0cf8072140 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf808ec90 con 0x7f0cf81384d0 2024-09-16T06:42:23.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient(hunting): authenticate will time out at 1096.480982s 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cf77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0cf8072140 0x7f0cf81377a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46540/0 (socket says 172.21.15.189:46540) 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cf77fe700 1 -- 172.21.15.189:0/1041051795 learned_addr learned my addr 172.21.15.189:0/1041051795 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497339144 0 0) 0x7f0cf8096c30 con 0x7f0cf8072140 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): my global_id is 134716 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce40035d0 con 0x7f0cf8072140 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253391686 0 0) 0x7f0cf80996b0 con 0x7f0cf80984b0 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): my global_id is 154143 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf8096c30 con 0x7f0cf80984b0 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893565791 0 0) 0x7f0cf808ec90 con 0x7f0cf81384d0 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): my global_id is 125009 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf80996b0 con 0x7f0cf81384d0 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037547994 0 0) 0x7f0ce40035d0 con 0x7f0cf8072140 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf808ec90 con 0x7f0cf8072140 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ce80018f0 con 0x7f0cf8072140 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): dump: 2024-09-16T06:42:23.979 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3610818930 0 0) 0x7f0cf808ec90 con 0x7f0cf8072140 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 >> v1:172.21.15.76:6790/0 conn(0x7f0cf81384d0 legacy=0x7f0cf81d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 >> v1:172.21.15.76:6789/0 conn(0x7f0cf80984b0 legacy=0x7f0cf81cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 monclient: found mon.b 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf808f7f0 con 0x7f0cf8072140 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: _finish_auth 0 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 5 monclient: authenticate success, global_id 134716 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _renew_subs 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf808ef50 con 0x7f0cf8072140 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _renew_subs 2024-09-16T06:42:23.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf808dff0 con 0x7f0cf8072140 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce80036a0 con 0x7f0cf8072140 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ce80039b0 con 0x7f0cf8072140 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cf57fa700 4 set_mon_vals no callback set 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cdffff700 10 monclient: dump: 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:23.982 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.967+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8005100 con 0x7f0cf8072140 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.971+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ce8003b10 con 0x7f0cf8072140 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.971+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7f0ce8043a20 con 0x7f0cf8072140 2024-09-16T06:42:23.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.971+0000 7f0cf6ffd700 10 monclient: get_auth_request con 0x7f0ce4036390 auth_method 0 2024-09-16T06:42:23.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.975+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ce8004330 con 0x7f0cf8072140 2024-09-16T06:42:23.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.975+0000 7f0cdffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:23.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:23.975+0000 7f0cdffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:24.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cfe0f7700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cfe0f7700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cfe0f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0cd8005bc0 con 0x7f0cf8072140 2024-09-16T06:42:24.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (unknown 3664589273 0 1335834520) 0x7f0ce8012ca0 con 0x7f0cf8072140 2024-09-16T06:42:24.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cdffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T06:42:24.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.143+0000 7f0cdffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:24.202 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:42:24.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:42:24.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cfe0f7700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T06:42:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cfe0f7700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cfe0f7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f0cd8005650 con 0x7f0cf8072140 2024-09-16T06:42:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cdffff700 1 -- 172.21.15.189:0/1041051795 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v112) v1 ==== 56+0+1449 (unknown 3664589273 0 1335834520) 0x7f0ce8014420 con 0x7f0cf8072140 2024-09-16T06:42:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cdffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T06:42:24.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.251+0000 7f0cdffff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:24.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ce4036390 legacy=0x7f0ce4038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 10 monclient: shutdown 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 >> v1:172.21.15.189:6789/0 conn(0x7f0cf8072140 legacy=0x7f0cf81377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 shutdown_connections 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 >> 172.21.15.189:0/1041051795 conn(0x7f0cf809fe30 msgr2=0x7f0cf813a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 shutdown_connections 2024-09-16T06:42:24.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.255+0000 7f0cfe0f7700 1 -- 172.21.15.189:0/1041051795 wait complete. 2024-09-16T06:42:24.273 INFO:tasks.workunit.client.0.smithi189.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-16T06:42:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.12 2024-09-16T06:42:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.317+0000 7f88e6aea700 10 monclient: get_monmap_and_config 2024-09-16T06:42:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.317+0000 7f88e6aea700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.317+0000 7f88e6aea700 10 monclient: monmap: 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:24.323882+0000 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:24.323882+0000 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:24.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 1 Processor -- start 2024-09-16T06:42:24.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 1 -- start start 2024-09-16T06:42:24.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient: init 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-a con 0x7f88e00a4840 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-c con 0x7f88e0071d80 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-b con 0x7f88e0138610 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00a11a0 con 0x7f88e00a4840 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00a4c20 con 0x7f88e0071d80 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00a06e0 con 0x7f88e0138610 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e6aea700 10 monclient(hunting): authenticate will time out at 1091.458939s 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e5ae8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f88e00a4840 0x7f88e009b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35336/0 (socket says 172.21.15.76:35336) 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e5ae8700 1 -- 172.21.15.76:0/96828271 learned_addr learned my addr 172.21.15.76:0/96828271 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893166231 0 0) 0x7f88e00a06e0 con 0x7f88e0138610 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): my global_id is 125012 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88cc0035d0 con 0x7f88e0138610 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676072455 0 0) 0x7f88e00a11a0 con 0x7f88e00a4840 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): my global_id is 154146 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e00a06e0 con 0x7f88e00a4840 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664278725 0 0) 0x7f88e00a4c20 con 0x7f88e0071d80 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 10 monclient(hunting): my global_id is 134719 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e00a11a0 con 0x7f88e0071d80 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606943745 0 0) 0x7f88cc0035d0 con 0x7f88e0138610 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 >> v1:172.21.15.189:6789/0 conn(0x7f88e0071d80 legacy=0x7f88e009ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.321+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 >> v1:172.21.15.76:6789/0 conn(0x7f88e00a4840 legacy=0x7f88e009b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 1 monclient: found mon.noname-b 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e009c120 con 0x7f88e0138610 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88d8001990 con 0x7f88e0138610 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 5 monclient: authenticate success, global_id 125012 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: dump: 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88d8001bf0 con 0x7f88e0138610 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 1 -- 172.21.15.76:0/96828271 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88d8001fc0 con 0x7f88e0138610 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 4 set_mon_vals no callback set 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e4ae6700 10 monclient: dump: 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: shutdown 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- 172.21.15.76:0/96828271 >> v1:172.21.15.76:6790/0 conn(0x7f88e0138610 legacy=0x7f88e013aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- 172.21.15.76:0/96828271 shutdown_connections 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- 172.21.15.76:0/96828271 >> 172.21.15.76:0/96828271 conn(0x7f88e00a4430 msgr2=0x7f88e00a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- 172.21.15.76:0/96828271 shutdown_connections 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- 172.21.15.76:0/96828271 wait complete. 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: monmap: 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:24.334 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:24.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 Processor -- start 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- start start 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: init 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-a con 0x7f88e0071d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-b con 0x7f88e00a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient(hunting): picked mon.noname-c con 0x7f88e0138610 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e0121cd0 con 0x7f88e0071d80 2024-09-16T06:42:24.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e0070e00 con 0x7f88e00a4840 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00a26a0 con 0x7f88e0138610 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e6aea700 10 monclient(hunting): authenticate will time out at 1091.462785s 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e62e9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f88e0138610 0x7f88e01ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60422/0 (socket says 172.21.15.76:60422) 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e5ae8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f88e0071d80 0x7f88e01c59f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35340/0 (socket says 172.21.15.76:35340) 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88e62e9700 1 -- 172.21.15.76:0/2375321126 learned_addr learned my addr 172.21.15.76:0/2375321126 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856982700 0 0) 0x7f88e0121cd0 con 0x7f88e0071d80 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): my global_id is 154149 2024-09-16T06:42:24.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c40035d0 con 0x7f88e0071d80 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791377407 0 0) 0x7f88e00a26a0 con 0x7f88e0138610 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): my global_id is 125015 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e0121cd0 con 0x7f88e0138610 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739930145 0 0) 0x7f88e0070e00 con 0x7f88e00a4840 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): my global_id is 134722 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e00a26a0 con 0x7f88e00a4840 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86807444 0 0) 0x7f88c40035d0 con 0x7f88e0071d80 2024-09-16T06:42:24.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88e0070e00 con 0x7f88e0071d80 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88dc001900 con 0x7f88e0071d80 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828424942 0 0) 0x7f88e0121cd0 con 0x7f88e0138610 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88c40035d0 con 0x7f88e0138610 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88d8001890 con 0x7f88e0138610 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712351727 0 0) 0x7f88e00a26a0 con 0x7f88e00a4840 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88e0121cd0 con 0x7f88e00a4840 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88d0001830 con 0x7f88e00a4840 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.325+0000 7f88d67fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.344 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1409066971 0 0) 0x7f88e0070e00 con 0x7f88e0071d80 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 >> v1:172.21.15.76:6790/0 conn(0x7f88e0138610 legacy=0x7f88e01ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 >> v1:172.21.15.189:6789/0 conn(0x7f88e00a4840 legacy=0x7f88e01caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 monclient: found mon.a 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e01219f0 con 0x7f88e0071d80 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 5 monclient: authenticate success, global_id 154149 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _renew_subs 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e01202c0 con 0x7f88e0071d80 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _renew_subs 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e011ed30 con 0x7f88e0071d80 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88dc001c70 con 0x7f88e0071d80 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f88dc004ce0 con 0x7f88e0071d80 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 10 monclient: dump: 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d77fe700 4 set_mon_vals no callback set 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f88dc00f750 con 0x7f88e0071d80 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7f88dc0439c0 con 0x7f88e0071d80 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8005100 con 0x7f88e0071d80 2024-09-16T06:42:24.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.329+0000 7f88e52e7700 10 monclient: get_auth_request con 0x7f88c4036560 auth_method 0 2024-09-16T06:42:24.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.337+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f88dc014750 con 0x7f88e0071d80 2024-09-16T06:42:24.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.337+0000 7f88d67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.337+0000 7f88d67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:24.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: get_monmap_and_config 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: monmap: 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:24.382618+0000 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:24.382618+0000 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:24.382 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 1 Processor -- start 2024-09-16T06:42:24.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 1 -- start start 2024-09-16T06:42:24.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: init 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-c con 0x7fdf980a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-b con 0x7fdf9809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-a con 0x7fdf980a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9809a040 con 0x7fdf980a3ca0 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf980a33d0 con 0x7fdf980a2350 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf98099d60 con 0x7fdf9809aa20 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdfa05f9700 10 monclient(hunting): authenticate will time out at 1096.895726s 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdf9e395700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf980a2350 0x7fdf980a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46554/0 (socket says 172.21.15.189:46554) 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.379+0000 7fdf9e395700 1 -- 172.21.15.189:0/3339559818 learned_addr learned my addr 172.21.15.189:0/3339559818 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:24.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563229697 0 0) 0x7fdf980a33d0 con 0x7fdf980a2350 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): my global_id is 134725 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf840035d0 con 0x7fdf980a2350 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767462907 0 0) 0x7fdf9809a040 con 0x7fdf980a3ca0 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient(hunting): my global_id is 154152 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf980a33d0 con 0x7fdf980a3ca0 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 125538625 0 0) 0x7fdf840035d0 con 0x7fdf980a2350 2024-09-16T06:42:24.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 >> v1:172.21.15.76:6790/0 conn(0x7fdf9809aa20 legacy=0x7fdf9813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a3ca0 legacy=0x7fdf9813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 monclient: found mon.noname-c 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980a3920 con 0x7fdf980a2350 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 5 monclient: authenticate success, global_id 134725 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88001990 con 0x7fdf980a2350 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: dump: 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf88001bb0 con 0x7fdf980a2350 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 4 set_mon_vals no callback set 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 1 -- 172.21.15.189:0/3339559818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88001f40 con 0x7fdf980a2350 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9d393700 10 monclient: dump: 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.390 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.390 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.390 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: shutdown 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- 172.21.15.189:0/3339559818 >> v1:172.21.15.189:6789/0 conn(0x7fdf980a2350 legacy=0x7fdf980a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- 172.21.15.189:0/3339559818 shutdown_connections 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- 172.21.15.189:0/3339559818 >> 172.21.15.189:0/3339559818 conn(0x7fdf980a7d70 msgr2=0x7fdf980a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- 172.21.15.189:0/3339559818 shutdown_connections 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- 172.21.15.189:0/3339559818 wait complete. 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: monmap: 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:24.395 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 Processor -- start 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- start start 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: init 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:24.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-b con 0x7fdf9809aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-a con 0x7fdf980a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient(hunting): picked mon.noname-c con 0x7fdf980a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf981cab80 con 0x7fdf980a2350 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9809b7c0 con 0x7fdf9809aa20 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9806bed0 con 0x7fdf980a3ca0 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdfa05f9700 10 monclient(hunting): authenticate will time out at 1096.899893s 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9e395700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 0x7fdf98135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46556/0 (socket says 172.21.15.189:46556) 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.383+0000 7fdf9e395700 1 -- 172.21.15.189:0/4156232749 learned_addr learned my addr 172.21.15.189:0/4156232749 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912810769 0 0) 0x7fdf9809b7c0 con 0x7fdf9809aa20 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): my global_id is 134728 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf780035d0 con 0x7fdf9809aa20 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578022923 0 0) 0x7fdf981cab80 con 0x7fdf980a2350 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): my global_id is 154155 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9809b7c0 con 0x7fdf980a2350 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339212673 0 0) 0x7fdf9806bed0 con 0x7fdf980a3ca0 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): my global_id is 125021 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf981cab80 con 0x7fdf980a3ca0 2024-09-16T06:42:24.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392249087 0 0) 0x7fdf780035d0 con 0x7fdf9809aa20 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf9806bed0 con 0x7fdf9809aa20 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88001890 con 0x7fdf9809aa20 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587422421 0 0) 0x7fdf9809b7c0 con 0x7fdf980a2350 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf780035d0 con 0x7fdf980a2350 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf90001870 con 0x7fdf980a2350 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 445075847 0 0) 0x7fdf9806bed0 con 0x7fdf9809aa20 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6790/0 conn(0x7fdf980a3ca0 legacy=0x7fdf981d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 legacy=0x7fdf981cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 monclient: found mon.b 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf98136170 con 0x7fdf9809aa20 2024-09-16T06:42:24.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 5 monclient: authenticate success, global_id 134728 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _renew_subs 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf98122c30 con 0x7fdf9809aa20 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _renew_subs 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf98121500 con 0x7fdf9809aa20 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf88004ec0 con 0x7fdf9809aa20 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88005250 con 0x7fdf9809aa20 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 10 monclient: dump: 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf9cb92700 4 set_mon_vals no callback set 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdf880053b0 con 0x7fdf9809aa20 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7fdf88043e00 con 0x7fdf9809aa20 2024-09-16T06:42:24.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdf9db94700 10 monclient: get_auth_request con 0x7fdf78036230 auth_method 0 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.387+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c005100 con 0x7fdf9809aa20 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.395+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdf88014db0 con 0x7fdf9809aa20 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.395+0000 7fdf96ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.395+0000 7fdf96ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:24.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.567+0000 7fdfa05f9700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:24.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.567+0000 7fdfa05f9700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:24.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.567+0000 7fdfa05f9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:42:24.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:24.567+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fdf7c0053b0 con 0x7fdf9809aa20 2024-09-16T06:42:24.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:42:24.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:24.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f88c8005930 con 0x7f88e0071d80 2024-09-16T06:42:24.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88d67fc700 1 -- 172.21.15.76:0/2375321126 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 105398375) 0x7f88dc012bc0 con 0x7f88e0071d80 2024-09-16T06:42:24.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88d67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:24.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88d67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:24.572 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:42:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 >> v1:172.21.15.189:6812/3098810674 conn(0x7f88c4036560 legacy=0x7f88c4038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 10 monclient: shutdown 2024-09-16T06:42:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 >> v1:172.21.15.76:6789/0 conn(0x7f88e0071d80 legacy=0x7f88e01c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 shutdown_connections 2024-09-16T06:42:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 >> 172.21.15.76:0/2375321126 conn(0x7f88e00a4430 msgr2=0x7f88e009b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 shutdown_connections 2024-09-16T06:42:24.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.569+0000 7f88e6aea700 1 -- 172.21.15.76:0/2375321126 wait complete. 2024-09-16T06:42:24.586 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:42:24.587 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:42:24.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:42:24.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:42:24.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: get_monmap_and_config 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: monmap: 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:24.704064+0000 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:24.704064+0000 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:24.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 Processor -- start 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- start start 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: init 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-a con 0x7f13ec09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-b con 0x7f13ec0709d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-c con 0x7f13ec070de0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec09f140 con 0x7f13ec09ed60 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec072530 con 0x7f13ec070de0 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec097f00 con 0x7f13ec0709d0 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient(hunting): authenticate will time out at 1091.837410s 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13eaffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f13ec09ed60 0x7f13ec0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35342/0 (socket says 172.21.15.76:35342) 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13eaffd700 1 -- 172.21.15.76:0/138475767 learned_addr learned my addr 172.21.15.76:0/138475767 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13e27fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13ec0709d0 0x7f13ec13a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60426/0 (socket says 172.21.15.76:60426) 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269040927 0 0) 0x7f13ec09f140 con 0x7f13ec09ed60 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): my global_id is 154158 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d80035d0 con 0x7f13ec09ed60 2024-09-16T06:42:24.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118560900 0 0) 0x7f13ec097f00 con 0x7f13ec0709d0 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): my global_id is 125024 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec09f140 con 0x7f13ec0709d0 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961346686 0 0) 0x7f13ec072530 con 0x7f13ec070de0 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient(hunting): my global_id is 134731 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec097f00 con 0x7f13ec070de0 2024-09-16T06:42:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939416249 0 0) 0x7f13d80035d0 con 0x7f13ec09ed60 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 >> v1:172.21.15.76:6790/0 conn(0x7f13ec0709d0 legacy=0x7f13ec13a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 >> v1:172.21.15.189:6789/0 conn(0x7f13ec070de0 legacy=0x7f13ec13de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 monclient: found mon.noname-a 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec09a320 con 0x7f13ec09ed60 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 5 monclient: authenticate success, global_id 154158 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc001610 con 0x7f13ec09ed60 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: dump: 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13dc0018f0 con 0x7f13ec09ed60 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 1 -- 172.21.15.76:0/138475767 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc001c80 con 0x7f13ec09ed60 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 4 set_mon_vals no callback set 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ea7fc700 10 monclient: dump: 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.711 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 10 monclient: shutdown 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- 172.21.15.76:0/138475767 >> v1:172.21.15.76:6789/0 conn(0x7f13ec09ed60 legacy=0x7f13ec0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- 172.21.15.76:0/138475767 shutdown_connections 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- 172.21.15.76:0/138475767 >> 172.21.15.76:0/138475767 conn(0x7f13ec0a0030 msgr2=0x7f13ec0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- 172.21.15.76:0/138475767 shutdown_connections 2024-09-16T06:42:24.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.701+0000 7f13ebfff700 1 -- 172.21.15.76:0/138475767 wait complete. 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: monmap: 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.718 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 Processor -- start 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- start start 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: init 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-b con 0x7f13ec0709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-a con 0x7f13ec070de0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient(hunting): picked mon.noname-c con 0x7f13ec09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec090260 con 0x7f13ec070de0 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0a28c0 con 0x7f13ec0709d0 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec09c9f0 con 0x7f13ec09ed60 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient(hunting): authenticate will time out at 1091.841150s 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e27fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f13ec070de0 0x7f13ec1cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35346/0 (socket says 172.21.15.76:35346) 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e27fc700 1 -- 172.21.15.76:0/3832472405 learned_addr learned my addr 172.21.15.76:0/3832472405 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810275795 0 0) 0x7f13ec090260 con 0x7f13ec070de0 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): my global_id is 154161 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13cc0035d0 con 0x7f13ec070de0 2024-09-16T06:42:24.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292789446 0 0) 0x7f13ec09c9f0 con 0x7f13ec09ed60 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): my global_id is 125027 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec090260 con 0x7f13ec09ed60 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120820934 0 0) 0x7f13ec0a28c0 con 0x7f13ec0709d0 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): my global_id is 134734 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec09c9f0 con 0x7f13ec0709d0 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881405789 0 0) 0x7f13cc0035d0 con 0x7f13ec070de0 2024-09-16T06:42:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13ec0a28c0 con 0x7f13ec070de0 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13d40017f0 con 0x7f13ec070de0 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): dump: 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.722 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4173970507 0 0) 0x7f13ec090260 con 0x7f13ec09ed60 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13cc0035d0 con 0x7f13ec09ed60 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13e4001c30 con 0x7f13ec09ed60 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient(hunting): dump: 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3513769966 0 0) 0x7f13ec0a28c0 con 0x7f13ec070de0 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6790/0 conn(0x7f13ec09ed60 legacy=0x7f13ec1d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.189:6789/0 conn(0x7f13ec0709d0 legacy=0x7f13ec0965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 monclient: found mon.a 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec09a540 con 0x7f13ec070de0 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 5 monclient: authenticate success, global_id 154161 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _renew_subs 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec091da0 con 0x7f13ec070de0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _renew_subs 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec08fd20 con 0x7f13ec070de0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d4004890 con 0x7f13ec070de0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13d4004ba0 con 0x7f13ec070de0 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e9ffb700 4 set_mon_vals no callback set 2024-09-16T06:42:24.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.705+0000 7f13e3fff700 10 monclient: dump: 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13ebfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13d0006590 con 0x7f13d0002830 2024-09-16T06:42:24.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13d400f610 con 0x7f13ec070de0 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7f13d4043860 con 0x7f13ec070de0 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989009326 0 0) 0x7f13d0006590 con 0x7f13d0002830 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13eb7fe700 10 monclient: get_auth_request con 0x7f13cc036230 auth_method 0 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient(hunting): my global_id is 154164 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec0a28c0 con 0x7f13d0002830 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 302366088 0 0) 0x7f13ec0a28c0 con 0x7f13d0002830 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d0006590 con 0x7f13d0002830 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc001d60 con 0x7f13d0002830 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient: dump: 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1670460069 0 0) 0x7f13d0006590 con 0x7f13d0002830 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13d000a190 con 0x7f13d0002830 2024-09-16T06:42:24.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f13dc004de0 con 0x7f13d0002830 2024-09-16T06:42:24.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:24.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6789/0 conn(0x7f13d0002830 legacy=0x7f13d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.709+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6789/0 conn(0x7f13d0002830 legacy=0x7f13d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13d0009e30 con 0x7f13d0009150 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193187032 0 0) 0x7f13d0009e30 con 0x7f13d0009150 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient(hunting): my global_id is 154167 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0006590 con 0x7f13d0009150 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 545754246 0 0) 0x7f13d0006590 con 0x7f13d0009150 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d0009e30 con 0x7f13d0009150 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc0053f0 con 0x7f13d0009150 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient: dump: 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.731 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1614790070 0 0) 0x7f13d0009e30 con 0x7f13d0009150 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13d000c340 con 0x7f13d0009150 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2690986200) 0x7f13dc006770 con 0x7f13d0009150 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6789/0 conn(0x7f13d0009150 legacy=0x7f13d00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13e3fff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6789/0 conn(0x7f13d0009150 legacy=0x7f13d00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:24.744 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13cc036230 legacy=0x7f13cc0386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 10 monclient: shutdown 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 >> v1:172.21.15.76:6789/0 conn(0x7f13ec070de0 legacy=0x7f13ec1cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13eb7fe700 1 -- 172.21.15.76:0/3832472405 reap_dead start 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13eb7fe700 1 -- 172.21.15.76:0/3832472405 reap_dead start 2024-09-16T06:42:24.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 shutdown_connections 2024-09-16T06:42:24.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 >> 172.21.15.76:0/3832472405 conn(0x7f13ec0a0030 msgr2=0x7f13ec0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 shutdown_connections 2024-09-16T06:42:24.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.721+0000 7f13ebfff700 1 -- 172.21.15.76:0/3832472405 wait complete. 2024-09-16T06:42:24.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:24.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:42:24.746 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:42:24.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.861+0000 7fecce9b7700 10 monclient: get_monmap_and_config 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.861+0000 7fecce9b7700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.861+0000 7fecce9b7700 10 monclient: monmap: 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:24.867045+0000 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:24.867045+0000 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:24.866 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 Processor -- start 2024-09-16T06:42:24.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- start start 2024-09-16T06:42:24.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient: init 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-b con 0x7fecc8097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-c con 0x7fecc8099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-a con 0x7fecc8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8098990 con 0x7fecc8139500 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8097f40 con 0x7fecc8099090 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc809d610 con 0x7fecc8097b60 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient(hunting): authenticate will time out at 1092.000956s 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccd9b5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fecc8097b60 0x7fecc809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60454/0 (socket says 172.21.15.76:60454) 2024-09-16T06:42:24.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccd9b5700 1 -- 172.21.15.76:0/915769824 learned_addr learned my addr 172.21.15.76:0/915769824 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245810626 0 0) 0x7fecc809d610 con 0x7fecc8097b60 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): my global_id is 125030 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb40035d0 con 0x7fecc8097b60 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445491116 0 0) 0x7fecc8098990 con 0x7fecc8139500 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): my global_id is 154170 2024-09-16T06:42:24.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc809d610 con 0x7fecc8139500 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741042211 0 0) 0x7fecc8097f40 con 0x7fecc8099090 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient(hunting): my global_id is 134737 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc8098990 con 0x7fecc8099090 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041807063 0 0) 0x7fecb40035d0 con 0x7fecc8097b60 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 >> v1:172.21.15.189:6789/0 conn(0x7fecc8099090 legacy=0x7fecc809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 >> v1:172.21.15.76:6789/0 conn(0x7fecc8139500 legacy=0x7fecc813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 monclient: found mon.noname-b 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80986b0 con 0x7fecc8097b60 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc4001610 con 0x7fecc8097b60 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 5 monclient: authenticate success, global_id 125030 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: dump: 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.874 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc40018f0 con 0x7fecc8097b60 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 4 set_mon_vals no callback set 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 1 -- 172.21.15.76:0/915769824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc4001c80 con 0x7fecc8097b60 2024-09-16T06:42:24.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7feccc9b3700 10 monclient: dump: 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.876 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 10 monclient: shutdown 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- 172.21.15.76:0/915769824 >> v1:172.21.15.76:6790/0 conn(0x7fecc8097b60 legacy=0x7fecc809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- 172.21.15.76:0/915769824 shutdown_connections 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- 172.21.15.76:0/915769824 >> 172.21.15.76:0/915769824 conn(0x7fecc80a0030 msgr2=0x7fecc80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- 172.21.15.76:0/915769824 shutdown_connections 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.865+0000 7fecce9b7700 1 -- 172.21.15.76:0/915769824 wait complete. 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: build_initial_monmap 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: monmap: 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:24.878 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 Processor -- start 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- start start 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: init 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-a con 0x7fecc8097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-b con 0x7fecc8099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient(hunting): picked mon.noname-c con 0x7fecc8139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:24.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8090820 con 0x7fecc8097b60 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80a2c50 con 0x7fecc8099090 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc809c3e0 con 0x7fecc8139500 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient(hunting): authenticate will time out at 1092.005468s 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7feccd9b5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fecc8097b60 0x7fecc80962a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35392/0 (socket says 172.21.15.76:35392) 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7feccd9b5700 1 -- 172.21.15.76:0/2356089544 learned_addr learned my addr 172.21.15.76:0/2356089544 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68714551 0 0) 0x7fecc809c3e0 con 0x7fecc8139500 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): my global_id is 125033 2024-09-16T06:42:24.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecac0035d0 con 0x7fecc8139500 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854334672 0 0) 0x7fecc8090820 con 0x7fecc8097b60 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): my global_id is 154173 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc809c3e0 con 0x7fecc8097b60 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777714926 0 0) 0x7fecc80a2c50 con 0x7fecc8099090 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): my global_id is 134740 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc8090820 con 0x7fecc8099090 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460182313 0 0) 0x7fecac0035d0 con 0x7fecc8139500 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc80a2c50 con 0x7fecc8139500 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058673876 0 0) 0x7fecc809c3e0 con 0x7fecc8097b60 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecac0035d0 con 0x7fecc8097b60 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc0001c30 con 0x7fecc8139500 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.887 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc40015d0 con 0x7fecc8097b60 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314200362 0 0) 0x7fecc8090820 con 0x7fecc8099090 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc809c3e0 con 0x7fecc8099090 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecb80018b0 con 0x7fecc8099090 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3491557834 0 0) 0x7fecc80a2c50 con 0x7fecc8139500 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.189:6789/0 conn(0x7fecc8099090 legacy=0x7fecc81ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6789/0 conn(0x7fecc8097b60 legacy=0x7fecc80962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 monclient: found mon.c 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8096f10 con 0x7fecc8139500 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 5 monclient: authenticate success, global_id 125033 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _renew_subs 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc80969b0 con 0x7fecc8139500 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _renew_subs 2024-09-16T06:42:24.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8090540 con 0x7fecc8139500 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc0003a20 con 0x7fecc8139500 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc0003d30 con 0x7fecc8139500 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbffff700 4 set_mon_vals no callback set 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.869+0000 7fecbe7fc700 10 monclient: dump: 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.892 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fecc000e7a0 con 0x7fecc8139500 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecce9b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fecb0006590 con 0x7fecb0002830 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7fecc0042b00 con 0x7fecc8139500 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588410455 0 0) 0x7fecb0006590 con 0x7fecb0002830 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient(hunting): my global_id is 125036 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc80a2c50 con 0x7fecb0002830 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7feccd1b4700 10 monclient: get_auth_request con 0x7fecac0364b0 auth_method 0 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3769337443 0 0) 0x7fecc80a2c50 con 0x7fecb0002830 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb0006590 con 0x7fecb0002830 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc4004e80 con 0x7fecb0002830 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient: dump: 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 575377442 0 0) 0x7fecb0006590 con 0x7fecb0002830 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecb000a190 con 0x7fecb0002830 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fecc4007270 con 0x7fecb0002830 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6790/0 conn(0x7fecb0002830 legacy=0x7fecb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.873+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6790/0 conn(0x7fecb0002830 legacy=0x7fecb0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fecb0009e30 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382744520 0 0) 0x7fecb0009e30 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient(hunting): my global_id is 125039 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0006590 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2984250110 0 0) 0x7fecb0006590 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb0009e30 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fecc4005680 con 0x7fecb0009150 2024-09-16T06:42:24.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient: dump: 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1122555180 0 0) 0x7fecb0009e30 con 0x7fecb0009150 2024-09-16T06:42:24.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fecb000c340 con 0x7fecb0009150 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 451890064) 0x7fecc4006af0 con 0x7fecb0009150 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6790/0 conn(0x7fecb0009150 legacy=0x7fecb00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecbe7fc700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6790/0 conn(0x7fecb0009150 legacy=0x7fecb00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:24.900 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.901 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.902 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:24.903 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:24.904 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:24.905 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:24.906 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:24.906 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.906 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:24.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.909 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:24.910 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:24.911 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.913 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:24.914 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.189:6812/3098810674 conn(0x7fecac0364b0 legacy=0x7fecac038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 10 monclient: shutdown 2024-09-16T06:42:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 >> v1:172.21.15.76:6790/0 conn(0x7fecc8139500 legacy=0x7fecc81cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:24.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce1b6700 1 -- 172.21.15.76:0/2356089544 reap_dead start 2024-09-16T06:42:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce1b6700 1 -- 172.21.15.76:0/2356089544 reap_dead start 2024-09-16T06:42:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 shutdown_connections 2024-09-16T06:42:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 >> 172.21.15.76:0/2356089544 conn(0x7fecc80a0030 msgr2=0x7fecc80a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 shutdown_connections 2024-09-16T06:42:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:24.885+0000 7fecce9b7700 1 -- 172.21.15.76:0/2356089544 wait complete. 2024-09-16T06:42:24.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:24.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:42:24.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:42:25.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: get_monmap_and_config 2024-09-16T06:42:25.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: build_initial_monmap 2024-09-16T06:42:25.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: monmap: 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:25.038206+0000 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:25.038206+0000 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:25.038 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:25.039 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:25.039 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 1 Processor -- start 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 1 -- start start 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: init 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-c con 0x7ff07c09b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:25.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-a con 0x7ff07c09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-b con 0x7ff07c138840 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c09bfb0 con 0x7ff07c09f230 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c09d3c0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c09c220 con 0x7ff07c138840 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.033+0000 7ff080e6f700 10 monclient(hunting): authenticate will time out at 1092.171966s 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff079d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff07c09f230 0x7ff07c137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35394/0 (socket says 172.21.15.76:35394) 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07ad9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff07c138840 0x7ff07c13ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60484/0 (socket says 172.21.15.76:60484) 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff079d9b700 1 -- 172.21.15.76:0/2262629699 learned_addr learned my addr 172.21.15.76:0/2262629699 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173040270 0 0) 0x7ff07c09bfb0 con 0x7ff07c09f230 2024-09-16T06:42:25.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): my global_id is 154176 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680035d0 con 0x7ff07c09f230 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288704775 0 0) 0x7ff07c09c220 con 0x7ff07c138840 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): my global_id is 125042 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c09bfb0 con 0x7ff07c138840 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988077445 0 0) 0x7ff07c09d3c0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient(hunting): my global_id is 134743 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c09c220 con 0x7ff07c09b6a0 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2424025426 0 0) 0x7ff07c09bfb0 con 0x7ff07c138840 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 >> v1:172.21.15.189:6789/0 conn(0x7ff07c09b6a0 legacy=0x7ff07c0a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 >> v1:172.21.15.76:6789/0 conn(0x7ff07c09f230 legacy=0x7ff07c137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: _finish_hunting 0 2024-09-16T06:42:25.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 monclient: found mon.noname-b 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c09c500 con 0x7ff07c138840 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: _finish_auth 0 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: _check_auth_tickets 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 5 monclient: authenticate success, global_id 125042 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff074001970 con 0x7ff07c138840 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: dump: 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff074001c10 con 0x7ff07c138840 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 1 -- 172.21.15.76:0/2262629699 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff074001fe0 con 0x7ff07c138840 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 4 set_mon_vals no callback set 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff07959a700 10 monclient: dump: 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 10 monclient: shutdown 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 1 -- 172.21.15.76:0/2262629699 >> v1:172.21.15.76:6790/0 conn(0x7ff07c138840 legacy=0x7ff07c13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 1 -- 172.21.15.76:0/2262629699 shutdown_connections 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 1 -- 172.21.15.76:0/2262629699 >> 172.21.15.76:0/2262629699 conn(0x7ff07c0a3e50 msgr2=0x7ff07c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 1 -- 172.21.15.76:0/2262629699 shutdown_connections 2024-09-16T06:42:25.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 1 -- 172.21.15.76:0/2262629699 wait complete. 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 10 monclient: build_initial_monmap 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.037+0000 7ff080e6f700 10 monclient: monmap: 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:25.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 Processor -- start 2024-09-16T06:42:25.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- start start 2024-09-16T06:42:25.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: init 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-a con 0x7ff07c09b6a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-b con 0x7ff07c09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient(hunting): picked mon.noname-c con 0x7ff07c138840 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c072450 con 0x7ff07c09b6a0 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c0a2d40 con 0x7ff07c09f230 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07c0a27f0 con 0x7ff07c138840 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient(hunting): authenticate will time out at 1092.177726s 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff07a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff07c09b6a0 0x7ff07c132b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35410/0 (socket says 172.21.15.76:35410) 2024-09-16T06:42:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff07a59c700 1 -- 172.21.15.76:0/3640598127 learned_addr learned my addr 172.21.15.76:0/3640598127 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567955852 0 0) 0x7ff07c072450 con 0x7ff07c09b6a0 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): my global_id is 154179 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c0035d0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995772426 0 0) 0x7ff07c0a27f0 con 0x7ff07c138840 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): my global_id is 125045 2024-09-16T06:42:25.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c072450 con 0x7ff07c138840 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451887209 0 0) 0x7ff07c0a2d40 con 0x7ff07c09f230 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): my global_id is 134746 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c0a27f0 con 0x7ff07c09f230 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242296777 0 0) 0x7ff05c0035d0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff07c0a2d40 con 0x7ff07c09b6a0 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff06c001930 con 0x7ff07c09b6a0 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383437049 0 0) 0x7ff07c072450 con 0x7ff07c138840 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff05c0035d0 con 0x7ff07c138840 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff074001f20 con 0x7ff07c138840 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650107699 0 0) 0x7ff07c0a27f0 con 0x7ff07c09f230 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff07c072450 con 0x7ff07c09f230 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff064001190 con 0x7ff07c09f230 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:25.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.063 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2536925691 0 0) 0x7ff07c0a2d40 con 0x7ff07c09b6a0 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.76:6790/0 conn(0x7ff07c138840 legacy=0x7ff07c1ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6789/0 conn(0x7ff07c09f230 legacy=0x7ff07c1cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 monclient: found mon.a 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c11da40 con 0x7ff07c09b6a0 2024-09-16T06:42:25.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 5 monclient: authenticate success, global_id 154179 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _renew_subs 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c11d1a0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _renew_subs 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c11c240 con 0x7ff07c09b6a0 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff06c0036c0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff06c0039d0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff078d99700 4 set_mon_vals no callback set 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.041+0000 7ff072ffd700 10 monclient: dump: 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff06c003bb0 con 0x7ff07c09b6a0 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff080e6f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff060006590 con 0x7ff060002830 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7ff06c043b30 con 0x7ff07c09b6a0 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff079d9b700 10 monclient: get_auth_request con 0x7ff05c03ab20 auth_method 0 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119155956 0 0) 0x7ff060006590 con 0x7ff060002830 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient(hunting): my global_id is 134749 2024-09-16T06:42:25.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c0a2d40 con 0x7ff060002830 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2044088922 0 0) 0x7ff07c0a2d40 con 0x7ff060002830 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff060006590 con 0x7ff060002830 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff074004fa0 con 0x7ff060002830 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient: dump: 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.069 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3315840845 0 0) 0x7ff060006590 con 0x7ff060002830 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff06000a190 con 0x7ff060002830 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff074007280 con 0x7ff060002830 2024-09-16T06:42:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:25.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:25.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6789/0 conn(0x7ff060002830 legacy=0x7ff060004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.045+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6789/0 conn(0x7ff060002830 legacy=0x7ff060004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:25.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff080e6f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:25.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff060009e30 con 0x7ff060009150 2024-09-16T06:42:25.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607772610 0 0) 0x7ff060009e30 con 0x7ff060009150 2024-09-16T06:42:25.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient(hunting): my global_id is 134752 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff060006590 con 0x7ff060009150 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4257978487 0 0) 0x7ff060006590 con 0x7ff060009150 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff060009e30 con 0x7ff060009150 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff074005690 con 0x7ff060009150 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 10 monclient: dump: 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.074 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2437413648 0 0) 0x7ff060009e30 con 0x7ff060009150 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.057+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff06000c340 con 0x7ff060009150 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 973451007) 0x7ff074006a20 con 0x7ff060009150 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff072ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff072ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6789/0 conn(0x7ff060009150 legacy=0x7ff0600095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff072ffd700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6789/0 conn(0x7ff060009150 legacy=0x7ff0600095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:25.075 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 44, 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:25.076 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:25.077 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:25.078 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:25.079 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:25.080 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:25.081 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:25.082 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:25.083 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:25.084 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:25.085 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:25.086 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.087 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:25.088 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:25.089 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:25.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:25.089 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:25.089 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff05c03ab20 legacy=0x7ff05c03cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 10 monclient: shutdown 2024-09-16T06:42:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 >> v1:172.21.15.76:6789/0 conn(0x7ff07c09b6a0 legacy=0x7ff07c132b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff07ad9d700 1 -- 172.21.15.76:0/3640598127 reap_dead start 2024-09-16T06:42:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff07ad9d700 1 -- 172.21.15.76:0/3640598127 reap_dead start 2024-09-16T06:42:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 shutdown_connections 2024-09-16T06:42:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 >> 172.21.15.76:0/3640598127 conn(0x7ff07c0a3e50 msgr2=0x7ff07c09a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 shutdown_connections 2024-09-16T06:42:25.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:25.061+0000 7ff080e6f700 1 -- 172.21.15.76:0/3640598127 wait complete. 2024-09-16T06:42:25.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:42:25.091 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:42:25.091 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T06:42:25.091 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:42:25.092 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:42:25.092 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:42:25.092 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:42:25.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.095+0000 7fdf9e395700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 legacy=0x7fdf98135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:42:25.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.095+0000 7fdf9e395700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 legacy=0x7fdf98135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.095+0000 7fdf9e395700 1 --1- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 0x7fdf98135550 :-1 s=OPENED pgs=489 cs=1 l=1).handle_message read tag failed 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.095+0000 7fdf9e395700 1 --1- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 0x7fdf98135550 :-1 s=OPENED pgs=489 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.095+0000 7fdf96ffd700 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf9809aa20 legacy=0x7fdf98135550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): picked mon.c con 0x7fdf980a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): picked mon.b con 0x7fdf780427a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:25.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): picked mon.a con 0x7fdf980a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf9806bed0 con 0x7fdf980a2350 2024-09-16T06:42:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf78044c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf981cab80 con 0x7fdf780427a0 2024-09-16T06:42:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf9809b7c0 con 0x7fdf980a3ca0 2024-09-16T06:42:25.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996350637 0 0) 0x7fdf7c0053b0 con 0x7fdf980a2350 2024-09-16T06:42:25.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdf7804a330 con 0x7fdf980a2350 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489385177 0 0) 0x7fdf9809b7c0 con 0x7fdf980a3ca0 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdf7c0053b0 con 0x7fdf980a3ca0 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1493159098 0 0) 0x7fdf7c0053b0 con 0x7fdf980a2350 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6790/0 conn(0x7fdf980a3ca0 legacy=0x7fdf78040e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf78044c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 monclient: found mon.a 2024-09-16T06:42:25.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fdf7804d480 con 0x7fdf980a2350 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fdf981cab80 con 0x7fdf980a2350 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88015240 con 0x7fdf980a2350 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.099+0000 7fdf96ffd700 10 monclient: dump: 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:25.106 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:25.107 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.103+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf880153e0 con 0x7fdf980a2350 2024-09-16T06:42:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.103+0000 7fdf96ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.103+0000 7fdf9cb92700 4 set_mon_vals no callback set 2024-09-16T06:42:25.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:25.103+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdf880115c0 con 0x7fdf980a2350 2024-09-16T06:42:25.131 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:42:25.131 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T06:42:25.132 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T06:42:25.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:42:27.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:27.383+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:27.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:27.383+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:27.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:27.383+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:27.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:27.383+0000 7fdf95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:28.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:28.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:28.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:28.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:29.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:29.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:29.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:29.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:29.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:30.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:30.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:30.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:30.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:30.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:30.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:31.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:31.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:31.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:31.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:31.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:31.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:32.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:32.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:32.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:32.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:32.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:32.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:33.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:33.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:33.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:33.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:33.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:33.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:34.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:34.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:34.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:34.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:35.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:35.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:35.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:35.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:35.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:35.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:36.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:36.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:36.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:37.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:37.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:37.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:37.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:37.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:37.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:37.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:37.387+0000 7fdf95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:38.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:38.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:38.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:38.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:38.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:38.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:39.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:39.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:39.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:40.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:40.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:40.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:40.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:40.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:40.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:41.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:41.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:41.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:41.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:41.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:41.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:42.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:42.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:42.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:42.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:42.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:42.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:43.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:43.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:43.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:43.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:43.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:43.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:44.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:44.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:44.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:44.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:44.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:44.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:45.133 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T06:42:45.133 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T06:42:45.134 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:42:45.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:42:45.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.245+0000 7fc3f7403700 10 monclient: get_monmap_and_config 2024-09-16T06:42:45.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.245+0000 7fc3f7403700 10 monclient: build_initial_monmap 2024-09-16T06:42:45.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.245+0000 7fc3f7403700 10 monclient: monmap: 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:45.251825+0000 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:45.251825+0000 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:45.251 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:45.252 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 1 Processor -- start 2024-09-16T06:42:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 1 -- start start 2024-09-16T06:42:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient: init 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-a con 0x7fc3f012fbf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-b con 0x7fc3f012f7e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-c con 0x7fc3f0139720 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f009db90 con 0x7fc3f012fbf0 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f009d0f0 con 0x7fc3f0139720 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f6401700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 0x7fc3f012ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50888/0 (socket says 172.21.15.76:50888) 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f6401700 1 -- 172.21.15.76:0/3866246657 learned_addr learned my addr 172.21.15.76:0/3866246657 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f01306e0 con 0x7fc3f012f7e0 2024-09-16T06:42:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f6c02700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3f013bb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.249+0000 7fc3f7403700 10 monclient(hunting): authenticate will time out at 1112.387319s 2024-09-16T06:42:45.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:45.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:45.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:45.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:45.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:45.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:45.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.453+0000 7fc3f6c02700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3f013bb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:45.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:45.857+0000 7fc3f6c02700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3f013bb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:46.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.387+0000 7fdf95ffb700 10 monclient: tick 2024-09-16T06:42:46.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.387+0000 7fdf95ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:46.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.387+0000 7fdf95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:46.661+0000 7fc3f6c02700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3f013bb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf9e395700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 legacy=0x7fdf780487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf9e395700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 legacy=0x7fdf780487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf9e395700 1 --1- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 0x7fdf780487d0 :-1 s=OPENED pgs=468 cs=1 l=1).handle_message read tag failed 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf9e395700 1 --1- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 0x7fdf780487d0 :-1 s=OPENED pgs=468 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6789/0 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf980a2350 legacy=0x7fdf780487d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:47.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient(hunting): picked mon.a con 0x7fdf9809aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient(hunting): picked mon.b con 0x7fdf780427a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient(hunting): picked mon.c con 0x7fdf980a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf7c0053b0 con 0x7fdf9809aa20 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf9809b7c0 con 0x7fdf780427a0 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf7804d0b0 con 0x7fdf980a3ca0 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf96ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:47.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:46.999+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf78044bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:47.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:47.001+0000 7fc3f6401700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 legacy=0x7fc3f012ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:42:47.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:47.001+0000 7fc3f6401700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 legacy=0x7fc3f012ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:42:47.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:47.001+0000 7fc3f6401700 1 --1- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 0x7fc3f012ffd0 :-1 s=OPENED pgs=486 cs=1 l=1).handle_message read tag failed 2024-09-16T06:42:47.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:47.001+0000 7fc3f6401700 1 --1- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 0x7fc3f012ffd0 :-1 s=OPENED pgs=486 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:42:47.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:47.001+0000 7fc3f53ff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:42:47.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.203+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf78044bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): tick 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6790/0 conn(0x7fdf980a3ca0 legacy=0x7fdf78040e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf78044bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf9809aa20 legacy=0x7fdf7804c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): picked mon.c con 0x7fdf700036c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): picked mon.b con 0x7fdf780427a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:47.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): picked mon.a con 0x7fdf9809aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf780516b0 con 0x7fdf9809aa20 2024-09-16T06:42:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf9809b7c0 con 0x7fdf780427a0 2024-09-16T06:42:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf70009360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdf70010770 con 0x7fdf700036c0 2024-09-16T06:42:47.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.387+0000 7fdf95ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:47.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.591+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf70009360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:47.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:47.995+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf70009360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): tick 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 monclient(hunting): continuing hunt 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6790/0 conn(0x7fc3f012f7e0 legacy=0x7fc3f0138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3f013bb10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 legacy=0x7fc3f012ffd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc3d40036a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc3f0139720 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc3f012fbf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f009cc70 con 0x7fc3f012fbf0 2024-09-16T06:42:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f009d0f0 con 0x7fc3f0139720 2024-09-16T06:42:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d4010340 con 0x7fc3d40036a0 2024-09-16T06:42:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3e7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.253+0000 7fc3f5c00700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3d4009370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:48.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.457+0000 7fc3f5c00700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3d4009370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:48.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:48.799+0000 7fdf9eb96700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf70009360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:48.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:48.861+0000 7fc3f5c00700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3d4009370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:49.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:49.665+0000 7fc3f5c00700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3d4009370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.150+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676581212 0 0) 0x7fdf7804d480 con 0x7fdf9809aa20 2024-09-16T06:42:50.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.150+0000 7fdf96ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.150+0000 7fdf96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.150+0000 7fdf96ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:42:50.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.150+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fdf7804d0b0 con 0x7fdf9809aa20 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1199798777 0 0) 0x7fdf7804d0b0 con 0x7fdf9809aa20 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6790/0 conn(0x7fdf700036c0 legacy=0x7fdf70005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6789/0 conn(0x7fdf780427a0 legacy=0x7fdf70009360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 monclient: found mon.a 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fdf70011980 con 0x7fdf9809aa20 2024-09-16T06:42:50.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fdf9809b7c0 con 0x7fdf9809aa20 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdf88017ec0 con 0x7fdf9809aa20 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: dump: 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.159 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.160 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:50.160 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.160 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.160 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.160 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639164238 0 0) 0x7fc3f009cc70 con 0x7fc3f012fbf0 2024-09-16T06:42:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient(hunting): my global_id is 154182 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc0035d0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580823279 0 0) 0x7fc3dc0035d0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6790/0 conn(0x7fc3d40036a0 legacy=0x7fc3d4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.189:6789/0 conn(0x7fc3f0139720 legacy=0x7fc3d4009370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 monclient: found mon.noname-a 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4011550 con 0x7fc3f012fbf0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3e8004200 con 0x7fc3f012fbf0 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f53ff700 10 monclient: dump: 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f7403700 5 monclient: authenticate success, global_id 154182 2024-09-16T06:42:50.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.153+0000 7fc3f7403700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:50.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf88013600 con 0x7fdf9809aa20 2024-09-16T06:42:50.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf97fff700 4 set_mon_vals no callback set 2024-09-16T06:42:50.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:50.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdf88013240 con 0x7fdf9809aa20 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e8004420 con 0x7fc3f012fbf0 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f7403700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 1 -- 172.21.15.76:0/3866246657 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3e8004830 con 0x7fc3f012fbf0 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f7403700 4 set_mon_vals no callback set 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.157+0000 7fc3f53ff700 10 monclient: dump: 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: shutdown 2024-09-16T06:42:50.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 legacy=0x7fc3d400cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 shutdown_connections 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 >> 172.21.15.76:0/3866246657 conn(0x7fc3f00a5870 msgr2=0x7fc3f00a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 shutdown_connections 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- 172.21.15.76:0/3866246657 wait complete. 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: build_initial_monmap 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: monmap: 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 Processor -- start 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 1 -- start start 2024-09-16T06:42:50.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: init 2024-09-16T06:42:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:50.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-b con 0x7fc3d40036a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-a con 0x7fc3f012fbf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.161+0000 7fc3f7403700 10 monclient(hunting): picked mon.noname-c con 0x7fc3f0139720 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f01391e0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f009c640 con 0x7fc3d40036a0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f6401700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc3d40036a0 legacy=0x7fc3f01c7980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f6c02700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc3f0139720 0x7fc3f01cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46050/0 (socket says 172.21.15.76:46050) 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f01c81c0 con 0x7fc3f0139720 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f6c02700 1 -- 172.21.15.76:0/1939419422 learned_addr learned my addr 172.21.15.76:0/1939419422 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient(hunting): authenticate will time out at 1117.300203s 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89602337 0 0) 0x7fc3f01391e0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): my global_id is 154185 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4009e40 con 0x7fc3f012fbf0 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 217405669 0 0) 0x7fc3d4009e40 con 0x7fc3f012fbf0 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3f01391e0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3e0001820 con 0x7fc3f012fbf0 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): dump: 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763473304 0 0) 0x7fc3f01391e0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 >> v1:172.21.15.76:6790/0 conn(0x7fc3f0139720 legacy=0x7fc3f01cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 >> v1:172.21.15.189:6789/0 conn(0x7fc3d40036a0 legacy=0x7fc3f01c7980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 monclient: found mon.a 2024-09-16T06:42:50.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f012bca0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 5 monclient: authenticate success, global_id 154185 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _renew_subs 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f0072600 con 0x7fc3f012fbf0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _renew_subs 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f012e360 con 0x7fc3f012fbf0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e00036c0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3e00039d0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f4bfe700 4 set_mon_vals no callback set 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 10 monclient: dump: 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.177 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc3e0003bb0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8005100 con 0x7fc3f012fbf0 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7fc3e0043930 con 0x7fc3f012fbf0 2024-09-16T06:42:50.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.165+0000 7fc3f6401700 10 monclient: get_auth_request con 0x7fc3d4036be0 auth_method 0 2024-09-16T06:42:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.173+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc3e0014660 con 0x7fc3f012fbf0 2024-09-16T06:42:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.173+0000 7fc3e6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.173+0000 7fc3e6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3f7403700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:42:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3f7403700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3f7403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3d8005930 con 0x7fc3f012fbf0 2024-09-16T06:42:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3e6ffd700 1 -- 172.21.15.76:0/1939419422 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3208401593) 0x7fc3e0012ad0 con 0x7fc3f012fbf0 2024-09-16T06:42:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3e6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:42:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.405+0000 7fc3e6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:50.409 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc3d4036be0 legacy=0x7fc3d40390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 10 monclient: shutdown 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 >> v1:172.21.15.76:6789/0 conn(0x7fc3f012fbf0 legacy=0x7fc3f012da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 shutdown_connections 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 >> 172.21.15.76:0/1939419422 conn(0x7fc3f00a5870 msgr2=0x7fc3f01328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 shutdown_connections 2024-09-16T06:42:50.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.409+0000 7fc3f7403700 1 -- 172.21.15.76:0/1939419422 wait complete. 2024-09-16T06:42:50.426 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:42:50.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:42:50.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: monmap: 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:50.540344+0000 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:50.540344+0000 2024-09-16T06:42:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:50.540 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 Processor -- start 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- start start 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: init 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c09d560 con 0x7f7f4c099030 2024-09-16T06:42:50.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c09f140 con 0x7f7f4c098be0 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c09d790 con 0x7f7f4c09ed60 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient(hunting): authenticate will time out at 1117.673549s 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f43fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7f4c098be0 legacy=0x7f7f4c1374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7f4c099030 0x7f7f4c13ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50948/0 (socket says 172.21.15.76:50948) 2024-09-16T06:42:50.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7f4c09ed60 0x7f7f4c0984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46060/0 (socket says 172.21.15.76:46060) 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4b7fe700 1 -- 172.21.15.76:0/1614923107 learned_addr learned my addr 172.21.15.76:0/1614923107 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578929489 0 0) 0x7f7f4c09d790 con 0x7f7f4c09ed60 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): my global_id is 125048 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f380035d0 con 0x7f7f4c09ed60 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153797342 0 0) 0x7f7f4c09d560 con 0x7f7f4c099030 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient(hunting): my global_id is 154188 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c09d790 con 0x7f7f4c099030 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575252236 0 0) 0x7f7f380035d0 con 0x7f7f4c09ed60 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 >> v1:172.21.15.189:6789/0 conn(0x7f7f4c098be0 legacy=0x7f7f4c1374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 >> v1:172.21.15.76:6789/0 conn(0x7f7f4c099030 legacy=0x7f7f4c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 monclient: found mon.noname-b 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c09f620 con 0x7f7f4c09ed60 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 5 monclient: authenticate success, global_id 125048 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f3c001610 con 0x7f7f4c09ed60 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: dump: 2024-09-16T06:42:50.554 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c001830 con 0x7f7f4c09ed60 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 1 -- 172.21.15.76:0/1614923107 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f3c001bc0 con 0x7f7f4c09ed60 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 4 set_mon_vals no callback set 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4a7fc700 10 monclient: dump: 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 10 monclient: shutdown 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- 172.21.15.76:0/1614923107 >> v1:172.21.15.76:6790/0 conn(0x7f7f4c09ed60 legacy=0x7f7f4c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- 172.21.15.76:0/1614923107 shutdown_connections 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.537+0000 7f7f4bfff700 1 -- 172.21.15.76:0/1614923107 >> 172.21.15.76:0/1614923107 conn(0x7f7f4c0a0030 msgr2=0x7f7f4c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- 172.21.15.76:0/1614923107 shutdown_connections 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- 172.21.15.76:0/1614923107 wait complete. 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: monmap: 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 Processor -- start 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- start start 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: init 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c06bf30 con 0x7f7f4c099030 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c08ee10 con 0x7f7f4c09ed60 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c098610 con 0x7f7f4c098be0 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient(hunting): authenticate will time out at 1117.678531s 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7f4c098be0 0x7f7f4c0714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46070/0 (socket says 172.21.15.76:46070) 2024-09-16T06:42:50.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4affd700 1 -- 172.21.15.76:0/2209515649 learned_addr learned my addr 172.21.15.76:0/2209515649 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4b7fe700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.189:6789/0 conn(0x7f7f4c09ed60 legacy=0x7f7f4c1cdfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617301968 0 0) 0x7f7f4c06bf30 con 0x7f7f4c099030 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): my global_id is 154191 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c0035d0 con 0x7f7f4c099030 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372784372 0 0) 0x7f7f4c098610 con 0x7f7f4c098be0 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): my global_id is 125051 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c06bf30 con 0x7f7f4c098be0 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436430956 0 0) 0x7f7f4c06bf30 con 0x7f7f4c098be0 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c098610 con 0x7f7f4c098be0 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f3c0015d0 con 0x7f7f4c098be0 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.561 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1195462069 0 0) 0x7f7f2c0035d0 con 0x7f7f4c099030 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c06bf30 con 0x7f7f4c099030 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f340041a0 con 0x7f7f4c099030 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient(hunting): dump: 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.562 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2553699885 0 0) 0x7f7f4c098610 con 0x7f7f4c098be0 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.189:6789/0 conn(0x7f7f4c09ed60 legacy=0x7f7f4c1cdfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6789/0 conn(0x7f7f4c099030 legacy=0x7f7f4c1ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 monclient: found mon.c 2024-09-16T06:42:50.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c09a540 con 0x7f7f4c098be0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 5 monclient: authenticate success, global_id 125051 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _renew_subs 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c0a2e50 con 0x7f7f4c098be0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _renew_subs 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c071bb0 con 0x7f7f4c098be0 2024-09-16T06:42:50.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c001230 con 0x7f7f4c098be0 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f3c004db0 con 0x7f7f4c098be0 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: dump: 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.565 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.541+0000 7f7f497fa700 4 set_mon_vals no callback set 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f4bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f30006590 con 0x7f7f30002830 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793581427 0 0) 0x7f7f30006590 con 0x7f7f30002830 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient(hunting): my global_id is 154194 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c098610 con 0x7f7f30002830 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7f3c004f90 con 0x7f7f4c098be0 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7f7f3c043cc0 con 0x7f7f4c098be0 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3823270774 0 0) 0x7f7f4c098610 con 0x7f7f30002830 2024-09-16T06:42:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f30006590 con 0x7f7f30002830 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f34004820 con 0x7f7f30002830 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: dump: 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.567 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1390359959 0 0) 0x7f7f30006590 con 0x7f7f30002830 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f3000a190 con 0x7f7f30002830 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f4b7fe700 10 monclient: get_auth_request con 0x7f7f2c036370 auth_method 0 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7f34006ac0 con 0x7f7f30002830 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6789/0 conn(0x7f7f30002830 legacy=0x7f7f30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.545+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6789/0 conn(0x7f7f30002830 legacy=0x7f7f30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f30009e30 con 0x7f7f30009150 2024-09-16T06:42:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424962089 0 0) 0x7f7f30009e30 con 0x7f7f30009150 2024-09-16T06:42:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient(hunting): my global_id is 154197 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30006590 con 0x7f7f30009150 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1252440749 0 0) 0x7f7f30006590 con 0x7f7f30009150 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f30009e30 con 0x7f7f30009150 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f34005050 con 0x7f7f30009150 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient: dump: 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.571 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2051660926 0 0) 0x7f7f30009e30 con 0x7f7f30009150 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f3000c340 con 0x7f7f30009150 2024-09-16T06:42:50.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 855396066) 0x7f7f340063a0 con 0x7f7f30009150 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6789/0 conn(0x7f7f30009150 legacy=0x7f7f300095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f48ff9700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6789/0 conn(0x7f7f30009150 legacy=0x7f7f300095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 48, 2024-09-16T06:42:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.585 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:50.585 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7f2c036370 legacy=0x7f7f2c038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 10 monclient: shutdown 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 >> v1:172.21.15.76:6790/0 conn(0x7f7f4c098be0 legacy=0x7f7f4c0714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4b7fe700 1 -- 172.21.15.76:0/2209515649 reap_dead start 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4b7fe700 1 -- 172.21.15.76:0/2209515649 reap_dead start 2024-09-16T06:42:50.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 shutdown_connections 2024-09-16T06:42:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 >> 172.21.15.76:0/2209515649 conn(0x7f7f4c0a0030 msgr2=0x7f7f4c0972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 shutdown_connections 2024-09-16T06:42:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.553+0000 7f7f4bfff700 1 -- 172.21.15.76:0/2209515649 wait complete. 2024-09-16T06:42:50.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:42:50.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:42:50.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:42:50.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: get_monmap_and_config 2024-09-16T06:42:50.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: build_initial_monmap 2024-09-16T06:42:50.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: monmap: 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:42:50.701772+0000 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:42:50.701772+0000 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:50.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 1 Processor -- start 2024-09-16T06:42:50.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 1 -- start start 2024-09-16T06:42:50.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: init 2024-09-16T06:42:50.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:50.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-c con 0x7fd2c409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c4098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-a con 0x7fd2c4099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409d560 con 0x7fd2c4099030 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409f140 con 0x7fd2c409ed60 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409d790 con 0x7fd2c4098be0 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2caabb700 10 monclient(hunting): authenticate will time out at 1117.835586s 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2ca2ba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2c4099030 0x7fd2c413ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50984/0 (socket says 172.21.15.76:50984) 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2ca2ba700 1 -- 172.21.15.76:0/3187835354 learned_addr learned my addr 172.21.15.76:0/3187835354 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.697+0000 7fd2c9ab9700 1 -- 172.21.15.76:0/3187835354 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409ed60 legacy=0x7fd2c40984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826787877 0 0) 0x7fd2c409d560 con 0x7fd2c4099030 2024-09-16T06:42:50.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): my global_id is 154200 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b00035d0 con 0x7fd2c4099030 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198767901 0 0) 0x7fd2c409d790 con 0x7fd2c4098be0 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient(hunting): my global_id is 125054 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c409d560 con 0x7fd2c4098be0 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485060990 0 0) 0x7fd2b00035d0 con 0x7fd2c4099030 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 >> v1:172.21.15.76:6790/0 conn(0x7fd2c4098be0 legacy=0x7fd2c41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409ed60 legacy=0x7fd2c40984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 monclient: found mon.noname-a 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c409f620 con 0x7fd2c4099030 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2bc001990 con 0x7fd2c4099030 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: dump: 2024-09-16T06:42:50.706 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 5 monclient: authenticate success, global_id 154200 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2bc001c70 con 0x7fd2c4099030 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 1 -- 172.21.15.76:0/3187835354 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2bc002000 con 0x7fd2c4099030 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2c8ab7700 10 monclient: dump: 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.708 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:50.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 4 set_mon_vals no callback set 2024-09-16T06:42:50.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 10 monclient: shutdown 2024-09-16T06:42:50.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 1 -- 172.21.15.76:0/3187835354 >> v1:172.21.15.76:6789/0 conn(0x7fd2c4099030 legacy=0x7fd2c413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 1 -- 172.21.15.76:0/3187835354 shutdown_connections 2024-09-16T06:42:50.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 1 -- 172.21.15.76:0/3187835354 >> 172.21.15.76:0/3187835354 conn(0x7fd2c40a0030 msgr2=0x7fd2c40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 1 -- 172.21.15.76:0/3187835354 shutdown_connections 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 1 -- 172.21.15.76:0/3187835354 wait complete. 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 10 monclient: build_initial_monmap 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.701+0000 7fd2caabb700 10 monclient: monmap: 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:50.715 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 Processor -- start 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- start start 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: init 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-a con 0x7fd2c4098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-c con 0x7fd2c4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4090e00 con 0x7fd2c4098be0 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40986f0 con 0x7fd2c409ed60 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40a12e0 con 0x7fd2c4099030 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2c92b8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd2c4099030 0x7fd2c41ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46080/0 (socket says 172.21.15.76:46080) 2024-09-16T06:42:50.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2c92b8700 1 -- 172.21.15.76:0/446321381 learned_addr learned my addr 172.21.15.76:0/446321381 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2c9ab9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2c4098be0 0x7fd2c4096370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50992/0 (socket says 172.21.15.76:50992) 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient(hunting): authenticate will time out at 1117.841839s 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2ca2ba700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409ed60 legacy=0x7fd2c41cded0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218474450 0 0) 0x7fd2c40a12e0 con 0x7fd2c4099030 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): my global_id is 125057 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a80035d0 con 0x7fd2c4099030 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166542030 0 0) 0x7fd2c4090e00 con 0x7fd2c4098be0 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): my global_id is 154203 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40a12e0 con 0x7fd2c4098be0 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201372732 0 0) 0x7fd2a80035d0 con 0x7fd2c4099030 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c4090e00 con 0x7fd2c4099030 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b8001850 con 0x7fd2c4099030 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:50.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): dump: 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.722 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286517509 0 0) 0x7fd2c40a12e0 con 0x7fd2c4098be0 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2a80035d0 con 0x7fd2c4098be0 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2c0004170 con 0x7fd2c4098be0 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient(hunting): dump: 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193561484 0 0) 0x7fd2c4090e00 con 0x7fd2c4099030 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.189:6789/0 conn(0x7fd2c409ed60 legacy=0x7fd2c41cded0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6789/0 conn(0x7fd2c4098be0 legacy=0x7fd2c4096370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 monclient: found mon.c 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c409a540 con 0x7fd2c4099030 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 5 monclient: authenticate success, global_id 125057 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _renew_subs 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c4090b20 con 0x7fd2c4099030 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _renew_subs 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4096a80 con 0x7fd2c4099030 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b80036c0 con 0x7fd2c4099030 2024-09-16T06:42:50.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2b8003a50 con 0x7fd2c4099030 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b7fff700 4 set_mon_vals no callback set 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2b67fc700 10 monclient: dump: 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.705+0000 7fd2caabb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd2b8003c30 con 0x7fd2c4099030 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2ac006590 con 0x7fd2ac002830 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(112..112 src has 66..112) v4 ==== 5731+0+0 (unknown 2084348820 0 0) 0x7fd2b8043b10 con 0x7fd2c4099030 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2ca2ba700 10 monclient: get_auth_request con 0x7fd2a80362a0 auth_method 0 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805336877 0 0) 0x7fd2ac006590 con 0x7fd2ac002830 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient(hunting): my global_id is 125060 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4090e00 con 0x7fd2ac002830 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 266465014 0 0) 0x7fd2c4090e00 con 0x7fd2ac002830 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ac006590 con 0x7fd2ac002830 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2c00047f0 con 0x7fd2ac002830 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient: dump: 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.728 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4167503462 0 0) 0x7fd2ac006590 con 0x7fd2ac002830 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2ac00a190 con 0x7fd2ac002830 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd2c0006ad0 con 0x7fd2ac002830 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6790/0 conn(0x7fd2ac002830 legacy=0x7fd2ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.709+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6790/0 conn(0x7fd2ac002830 legacy=0x7fd2ac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:50.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.717+0000 7fd2caabb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:42:50.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.717+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2ac009e50 con 0x7fd2ac009170 2024-09-16T06:42:50.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427032547 0 0) 0x7fd2ac009e50 con 0x7fd2ac009170 2024-09-16T06:42:50.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:50.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient(hunting): my global_id is 125063 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac006590 con 0x7fd2ac009170 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 314556236 0 0) 0x7fd2ac006590 con 0x7fd2ac009170 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ac009e50 con 0x7fd2ac009170 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2c0004fc0 con 0x7fd2ac009170 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient: dump: 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2982618455 0 0) 0x7fd2ac009e50 con 0x7fd2ac009170 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd2ac00c340 con 0x7fd2ac009170 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 3132693850) 0x7fd2c0006470 con 0x7fd2ac009170 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:50.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6790/0 conn(0x7fd2ac009170 legacy=0x7fd2ac009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2b67fc700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6790/0 conn(0x7fd2ac009170 legacy=0x7fd2ac009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 48, 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:42:50.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.735 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.736 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:42:50.737 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:42:50.738 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:42:50.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:42:50.740 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:42:50.741 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:42:50.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:42:50.745 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:42:50.746 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:42:50.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd2a80362a0 legacy=0x7fd2a8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 10 monclient: shutdown 2024-09-16T06:42:50.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:50.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 >> v1:172.21.15.76:6790/0 conn(0x7fd2c4099030 legacy=0x7fd2c41ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2ca2ba700 1 -- 172.21.15.76:0/446321381 reap_dead start 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2ca2ba700 1 -- 172.21.15.76:0/446321381 reap_dead start 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 shutdown_connections 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 >> 172.21.15.76:0/446321381 conn(0x7fd2c40a0030 msgr2=0x7fd2c406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 shutdown_connections 2024-09-16T06:42:50.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:42:50.721+0000 7fd2caabb700 1 -- 172.21.15.76:0/446321381 wait complete. 2024-09-16T06:42:50.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:42:50.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:42:51.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.154+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v113) v1 ==== 125+0+0 (unknown 674883614 0 0) 0x7fdf88011870 con 0x7fdf9809aa20 2024-09-16T06:42:51.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.154+0000 7fdf96ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:51.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.154+0000 7fdf96ffd700 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-16T06:42:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.266+0000 7fdfa05f9700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.266+0000 7fdfa05f9700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.266+0000 7fdfa05f9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:51.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.266+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7fdf7c0023c0 con 0x7fdf9809aa20 2024-09-16T06:42:51.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdf96ffd700 1 -- 172.21.15.189:0/4156232749 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v113) v1 ==== 126+0+0 (unknown 2389322427 0 0) 0x7fdf88044f90 con 0x7fdf9809aa20 2024-09-16T06:42:51.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdf96ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T06:42:51.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdf96ffd700 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-16T06:42:51.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdf78036230 legacy=0x7fdf780386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 10 monclient: shutdown 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 >> v1:172.21.15.76:6789/0 conn(0x7fdf9809aa20 legacy=0x7fdf7000cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 shutdown_connections 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 >> 172.21.15.189:0/4156232749 conn(0x7fdf980a7d70 msgr2=0x7fdf981bd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 shutdown_connections 2024-09-16T06:42:51.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.270+0000 7fdfa05f9700 1 -- 172.21.15.189:0/4156232749 wait complete. 2024-09-16T06:42:51.276 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.12' does not appear in the crush map 2024-09-16T06:42:51.285 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm osd.14 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: get_monmap_and_config 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: build_initial_monmap 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: monmap: 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:51.384978+0000 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:51.384978+0000 2024-09-16T06:42:51.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 1 Processor -- start 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 1 -- start start 2024-09-16T06:42:51.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: init 2024-09-16T06:42:51.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-c con 0x7fade809d830 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-a con 0x7fade809dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-b con 0x7fade8137dd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadedb97700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fade809d830 legacy=0x7fade8131930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade809a790 con 0x7fade809dc10 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade8071510 con 0x7fade809d830 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade8132040 con 0x7fade8137dd0 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadeeb99700 10 monclient(hunting): authenticate will time out at 1123.899504s 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadee398700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fade8137dd0 0x7fade813a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48828/0 (socket says 172.21.15.189:48828) 2024-09-16T06:42:51.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7fadee398700 1 -- 172.21.15.189:0/4127854604 learned_addr learned my addr 172.21.15.189:0/4127854604 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.382+0000 7faded396700 1 --1- 172.21.15.189:0/4127854604 >> v1:172.21.15.76:6789/0 conn(0x7fade809dc10 0x7fade81366b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46458/0 (socket says 172.21.15.189:46458) 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926063411 0 0) 0x7fade8132040 con 0x7fade8137dd0 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _negotiate 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): my global_id is 125066 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd40035d0 con 0x7fade8137dd0 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373344830 0 0) 0x7fade809a790 con 0x7fade809dc10 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _negotiate 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient(hunting): my global_id is 154206 2024-09-16T06:42:51.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8132040 con 0x7fade809dc10 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641986821 0 0) 0x7fadd40035d0 con 0x7fade8137dd0 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 >> v1:172.21.15.189:6789/0 conn(0x7fade809d830 legacy=0x7fade8131930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 >> v1:172.21.15.76:6789/0 conn(0x7fade809dc10 legacy=0x7fade81366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: _finish_hunting 0 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 monclient: found mon.noname-b 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:51.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade809e060 con 0x7fade8137dd0 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: _finish_auth 0 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: _check_auth_tickets 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fade0001990 con 0x7fade8137dd0 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: dump: 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.392 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 5 monclient: authenticate success, global_id 125066 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fade0001c70 con 0x7fade8137dd0 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 1 -- 172.21.15.189:0/4127854604 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fade0002000 con 0x7fade8137dd0 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:51.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadecb95700 10 monclient: dump: 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:51.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 4 set_mon_vals no callback set 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 10 monclient: shutdown 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 1 -- 172.21.15.189:0/4127854604 >> v1:172.21.15.76:6790/0 conn(0x7fade8137dd0 legacy=0x7fade813a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 1 -- 172.21.15.189:0/4127854604 shutdown_connections 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 1 -- 172.21.15.189:0/4127854604 >> 172.21.15.189:0/4127854604 conn(0x7fade80a4320 msgr2=0x7fade80a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 1 -- 172.21.15.189:0/4127854604 shutdown_connections 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 1 -- 172.21.15.189:0/4127854604 wait complete. 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 10 monclient: build_initial_monmap 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.386+0000 7fadeeb99700 10 monclient: monmap: 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:51.396 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 Processor -- start 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- start start 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: init 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-b con 0x7fade809d830 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-c con 0x7fade809dc10 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadedb97700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fade809d830 legacy=0x7fade811a1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient(hunting): picked mon.noname-a con 0x7fade8137dd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:51.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade811c0e0 con 0x7fade8137dd0 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade811d650 con 0x7fade809d830 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade811afd0 con 0x7fade809dc10 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient(hunting): authenticate will time out at 1123.906245s 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7faded396700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fade809dc10 0x7fade811a8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48834/0 (socket says 172.21.15.189:48834) 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7faded396700 1 -- 172.21.15.189:0/459308213 learned_addr learned my addr 172.21.15.189:0/459308213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882798266 0 0) 0x7fade811afd0 con 0x7fade809dc10 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): my global_id is 125069 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc0035d0 con 0x7fade809dc10 2024-09-16T06:42:51.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200587416 0 0) 0x7fade811c0e0 con 0x7fade8137dd0 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): my global_id is 154209 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade811afd0 con 0x7fade8137dd0 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 171047113 0 0) 0x7fadcc0035d0 con 0x7fade809dc10 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade811c0e0 con 0x7fade809dc10 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd8001910 con 0x7fade809dc10 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): dump: 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:51.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2129992606 0 0) 0x7fade811c0e0 con 0x7fade809dc10 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 >> v1:172.21.15.189:6789/0 conn(0x7fade809d830 legacy=0x7fade811a1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 >> v1:172.21.15.76:6789/0 conn(0x7fade8137dd0 legacy=0x7fade81d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:51.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 monclient: found mon.c 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80a2e00 con 0x7fade809dc10 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 5 monclient: authenticate success, global_id 125069 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _renew_subs 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade80a2a40 con 0x7fade809dc10 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _renew_subs 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade811d370 con 0x7fade809dc10 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd8001d00 con 0x7fade809dc10 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7faddffff700 4 set_mon_vals no callback set 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fadd8004ce0 con 0x7fade809dc10 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.390+0000 7fadde7fc700 10 monclient: dump: 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fadd800f750 con 0x7fade809dc10 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadeeb99700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadeeb99700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadeeb99700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0005100 con 0x7fade809dc10 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(113..113 src has 66..113) v4 ==== 5709+0+0 (unknown 719690718 0 0) 0x7fadd8043a30 con 0x7fade809dc10 2024-09-16T06:42:51.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.394+0000 7fadedb97700 10 monclient: get_auth_request con 0x7fadcc035fd0 auth_method 0 2024-09-16T06:42:51.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.406+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fadd80146b0 con 0x7fade809dc10 2024-09-16T06:42:51.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.406+0000 7fadde7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:51.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.406+0000 7fadde7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:51.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.578+0000 7fadeeb99700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:51.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.578+0000 7fadeeb99700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:51.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.578+0000 7fadeeb99700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:51.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:51.578+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fadd00053b0 con 0x7fade809dc10 2024-09-16T06:42:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.174+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v114) v1 ==== 125+0+0 (unknown 3924400599 0 0) 0x7fadd80197a0 con 0x7fade809dc10 2024-09-16T06:42:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.174+0000 7fadde7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.174+0000 7fadde7fc700 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-16T06:42:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fadd00023c0 con 0x7fade809dc10 2024-09-16T06:42:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadde7fc700 1 -- 172.21.15.189:0/459308213 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v114) v1 ==== 126+0+0 (unknown 4096000897 0 0) 0x7fadd8012d50 con 0x7fade809dc10 2024-09-16T06:42:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadde7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T06:42:52.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadde7fc700 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 >> v1:172.21.15.189:6812/3098810674 conn(0x7fadcc035fd0 legacy=0x7fadcc038490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 10 monclient: shutdown 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 >> v1:172.21.15.76:6790/0 conn(0x7fade809dc10 legacy=0x7fade811a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 shutdown_connections 2024-09-16T06:42:52.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 >> 172.21.15.189:0/459308213 conn(0x7fade80a4320 msgr2=0x7fade8130840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 shutdown_connections 2024-09-16T06:42:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.290+0000 7fadeeb99700 1 -- 172.21.15.189:0/459308213 wait complete. 2024-09-16T06:42:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:device 'osd.14' does not appear in the crush map 2024-09-16T06:42:52.306 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.12 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: get_monmap_and_config 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: build_initial_monmap 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: monmap: 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:52.405882+0000 2024-09-16T06:42:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:52.405882+0000 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:52.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 1 Processor -- start 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 1 -- start start 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: init 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-b con 0x7f13b009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-c con 0x7f13b009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-a con 0x7f13b0138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:52.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b009cca0 con 0x7f13b0138da0 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b009f750 con 0x7f13b009c5f0 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0097620 con 0x7f13b009bb00 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b686c700 10 monclient(hunting): authenticate will time out at 1124.919064s 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13af7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f13b009c5f0 legacy=0x7f13b0137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13affff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13b009bb00 0x7f13b009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48842/0 (socket says 172.21.15.189:48842) 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13b4e09700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f13b0138da0 0x7f13b013b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46476/0 (socket says 172.21.15.189:46476) 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13affff700 1 -- 172.21.15.189:0/2473721128 learned_addr learned my addr 172.21.15.189:0/2473721128 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.402+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173140622 0 0) 0x7f13b009cca0 con 0x7f13b0138da0 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:52.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): my global_id is 154212 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c0035d0 con 0x7f13b0138da0 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877530586 0 0) 0x7f13b0097620 con 0x7f13b009bb00 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient(hunting): my global_id is 125072 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b009cca0 con 0x7f13b009bb00 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113604485 0 0) 0x7f139c0035d0 con 0x7f13b0138da0 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 >> v1:172.21.15.76:6790/0 conn(0x7f13b009bb00 legacy=0x7f13b009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 >> v1:172.21.15.189:6789/0 conn(0x7f13b009c5f0 legacy=0x7f13b0137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 monclient: found mon.noname-a 2024-09-16T06:42:52.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0097900 con 0x7f13b0138da0 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: _finish_auth 0 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 5 monclient: authenticate success, global_id 154212 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a8001990 con 0x7f13b0138da0 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: dump: 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.418 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a8001c70 con 0x7f13b0138da0 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 4 set_mon_vals no callback set 2024-09-16T06:42:52.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 1 -- 172.21.15.189:0/2473721128 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a8002000 con 0x7f13b0138da0 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13aeffd700 10 monclient: dump: 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: shutdown 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- 172.21.15.189:0/2473721128 >> v1:172.21.15.76:6789/0 conn(0x7f13b0138da0 legacy=0x7f13b013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- 172.21.15.189:0/2473721128 shutdown_connections 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- 172.21.15.189:0/2473721128 >> 172.21.15.189:0/2473721128 conn(0x7f13b00a03e0 msgr2=0x7f13b00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- 172.21.15.189:0/2473721128 shutdown_connections 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- 172.21.15.189:0/2473721128 wait complete. 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: build_initial_monmap 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: monmap: 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:52.421 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 Processor -- start 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- start start 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: init 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-a con 0x7f13b009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-c con 0x7f13b009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient(hunting): picked mon.noname-b con 0x7f13b0138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:52.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0090e60 con 0x7f13b009bb00 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b008d8b0 con 0x7f13b0138da0 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b00a2d00 con 0x7f13b009c5f0 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b4e09700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f13b0138da0 legacy=0x7f13b01cdf60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13b686c700 10 monclient(hunting): authenticate will time out at 1124.923349s 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13af7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13b009c5f0 0x7f13b008d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48856/0 (socket says 172.21.15.189:48856) 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.406+0000 7f13af7fe700 1 -- 172.21.15.189:0/2930513151 learned_addr learned my addr 172.21.15.189:0/2930513151 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823686822 0 0) 0x7f13b00a2d00 con 0x7f13b009c5f0 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): my global_id is 125075 2024-09-16T06:42:52.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13900035d0 con 0x7f13b009c5f0 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264558704 0 0) 0x7f13b0090e60 con 0x7f13b009bb00 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): my global_id is 154215 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b00a2d00 con 0x7f13b009bb00 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844984755 0 0) 0x7f13900035d0 con 0x7f13b009c5f0 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13b0090e60 con 0x7f13b009c5f0 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1398004200 con 0x7f13b009c5f0 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): dump: 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536597092 0 0) 0x7f13b00a2d00 con 0x7f13b009bb00 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13900035d0 con 0x7f13b009bb00 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a0001830 con 0x7f13b009bb00 2024-09-16T06:42:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient(hunting): dump: 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1072429810 0 0) 0x7f13b0090e60 con 0x7f13b009c5f0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 >> v1:172.21.15.189:6789/0 conn(0x7f13b0138da0 legacy=0x7f13b01cdf60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 >> v1:172.21.15.76:6789/0 conn(0x7f13b009bb00 legacy=0x7f13b008ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 monclient: found mon.c 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b009a6e0 con 0x7f13b009c5f0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 5 monclient: authenticate success, global_id 125075 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _renew_subs 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b008f030 con 0x7f13b009c5f0 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _renew_subs 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b008e6e0 con 0x7f13b009c5f0 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1398004470 con 0x7f13b009c5f0 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13ae7fc700 4 set_mon_vals no callback set 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1398004800 con 0x7f13b009c5f0 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 10 monclient: dump: 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:52.428 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1394005100 con 0x7f13b009c5f0 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13980049e0 con 0x7f13b009c5f0 2024-09-16T06:42:52.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.410+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(114..114 src has 66..114) v4 ==== 5687+0+0 (unknown 379076143 0 0) 0x7f13980439b0 con 0x7f13b009c5f0 2024-09-16T06:42:52.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.414+0000 7f13affff700 10 monclient: get_auth_request con 0x7f13900362a0 auth_method 0 2024-09-16T06:42:52.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.422+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1398005230 con 0x7f13b009c5f0 2024-09-16T06:42:52.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.422+0000 7f13acff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:52.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.422+0000 7f13acff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:52.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.598+0000 7f13b686c700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:52.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.598+0000 7f13b686c700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:52.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.598+0000 7f13b686c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:52.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:52.598+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f1394005930 con 0x7f13b009c5f0 2024-09-16T06:42:52.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:52.605+0000 7f2941141700 -1 mon.c@2(leader).osd e114 definitely_dead 0 2024-09-16T06:42:53.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.178+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v115) v1 ==== 87+0+0 (unknown 3523029245 0 0) 0x7f1398012c80 con 0x7f13b009c5f0 2024-09-16T06:42:53.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.178+0000 7f13acff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:53.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.178+0000 7f13acff9700 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T06:42:53.303 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:53.301+0000 7f2941141700 -1 mon.c@2(leader).osd e115 definitely_dead 0 2024-09-16T06:42:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13b686c700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13b686c700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13b686c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f1394005420 con 0x7f13b009c5f0 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13acff9700 1 -- 172.21.15.189:0/2930513151 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v115) v1 ==== 96+0+0 (unknown 783064281 0 0) 0x7f13980145e0 con 0x7f13b009c5f0 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13acff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T06:42:53.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.298+0000 7f13acff9700 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T06:42:53.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13900362a0 legacy=0x7f1390038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 10 monclient: shutdown 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 >> v1:172.21.15.76:6790/0 conn(0x7f13b009c5f0 legacy=0x7f13b008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 shutdown_connections 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 >> 172.21.15.189:0/2930513151 conn(0x7f13b00a03e0 msgr2=0x7f13b009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 shutdown_connections 2024-09-16T06:42:53.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.302+0000 7f13b686c700 1 -- 172.21.15.189:0/2930513151 wait complete. 2024-09-16T06:42:53.308 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T06:42:53.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm osd.14 2024-09-16T06:42:53.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: get_monmap_and_config 2024-09-16T06:42:53.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: build_initial_monmap 2024-09-16T06:42:53.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: monmap: 2024-09-16T06:42:53.422 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:53.422893+0000 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:53.422893+0000 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:53.423 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 1 Processor -- start 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 1 -- start start 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: init 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-a con 0x7ff00c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-c con 0x7ff00c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-b con 0x7ff00c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c09cca0 con 0x7ff00c09bb00 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c09f750 con 0x7ff00c09c5f0 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c097620 con 0x7ff00c138da0 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:53.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff011e45700 10 monclient(hunting): authenticate will time out at 1125.935900s 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.418+0000 7ff00affd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff00c09c5f0 legacy=0x7ff00c137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff00c138da0 0x7ff00c13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48868/0 (socket says 172.21.15.189:48868) 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00bfff700 1 -- 172.21.15.189:0/4040173637 learned_addr learned my addr 172.21.15.189:0/4040173637 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440720495 0 0) 0x7ff00c09cca0 con 0x7ff00c09bb00 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): my global_id is 154218 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff40035d0 con 0x7ff00c09bb00 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245918711 0 0) 0x7ff00c097620 con 0x7ff00c138da0 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:53.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient(hunting): my global_id is 125078 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c09cca0 con 0x7ff00c138da0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170089020 0 0) 0x7ff00c09cca0 con 0x7ff00c138da0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 >> v1:172.21.15.189:6789/0 conn(0x7ff00c09c5f0 legacy=0x7ff00c137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 >> v1:172.21.15.76:6789/0 conn(0x7ff00c09bb00 legacy=0x7ff00c09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 monclient: found mon.noname-b 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c097900 con 0x7ff00c138da0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:53.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff0040018f0 con 0x7ff00c138da0 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 5 monclient: authenticate success, global_id 125078 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: dump: 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:53.429 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff004001b10 con 0x7ff00c138da0 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 1 -- 172.21.15.189:0/4040173637 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff004001ee0 con 0x7ff00c138da0 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 4 set_mon_vals no callback set 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff00a7fc700 10 monclient: dump: 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.430 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:53.431 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 10 monclient: shutdown 2024-09-16T06:42:53.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:53.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- 172.21.15.189:0/4040173637 >> v1:172.21.15.76:6790/0 conn(0x7ff00c138da0 legacy=0x7ff00c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:53.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- 172.21.15.189:0/4040173637 shutdown_connections 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- 172.21.15.189:0/4040173637 >> 172.21.15.189:0/4040173637 conn(0x7ff00c0a03e0 msgr2=0x7ff00c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- 172.21.15.189:0/4040173637 shutdown_connections 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- 172.21.15.189:0/4040173637 wait complete. 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 10 monclient: build_initial_monmap 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 10 monclient: monmap: 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:53.437 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 Processor -- start 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.422+0000 7ff011e45700 1 -- start start 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: init 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-b con 0x7ff00c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-c con 0x7ff00c09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient(hunting): picked mon.noname-a con 0x7ff00c138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c1cad70 con 0x7ff00c138da0 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c0940a0 con 0x7ff00c09bb00 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c09a6e0 con 0x7ff00c09c5f0 2024-09-16T06:42:53.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient(hunting): authenticate will time out at 1125.940605s 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff00b7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff00c09bb00 legacy=0x7ff00c1c9f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff00affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff00c09c5f0 0x7ff00c1ca660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48882/0 (socket says 172.21.15.189:48882) 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff00bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff00c138da0 0x7ff00c1d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46500/0 (socket says 172.21.15.189:46500) 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff00affd700 1 -- 172.21.15.189:0/1839876672 learned_addr learned my addr 172.21.15.189:0/1839876672 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795849336 0 0) 0x7ff00c1cad70 con 0x7ff00c138da0 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): my global_id is 154221 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff80035d0 con 0x7ff00c138da0 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223341983 0 0) 0x7ff00c09a6e0 con 0x7ff00c09c5f0 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:53.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): my global_id is 125081 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c1cad70 con 0x7ff00c09c5f0 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155871027 0 0) 0x7feff80035d0 con 0x7ff00c138da0 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff00c09a6e0 con 0x7ff00c138da0 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff004001e20 con 0x7ff00c138da0 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): dump: 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.440 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973468962 0 0) 0x7ff00c1cad70 con 0x7ff00c09c5f0 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feff80035d0 con 0x7ff00c09c5f0 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feffc004200 con 0x7ff00c09c5f0 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient(hunting): dump: 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:53.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2047297099 0 0) 0x7ff00c09a6e0 con 0x7ff00c138da0 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 >> v1:172.21.15.76:6790/0 conn(0x7ff00c09c5f0 legacy=0x7ff00c1ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 >> v1:172.21.15.189:6789/0 conn(0x7ff00c09bb00 legacy=0x7ff00c1c9f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 monclient: found mon.a 2024-09-16T06:42:53.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c092940 con 0x7ff00c138da0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 5 monclient: authenticate success, global_id 154221 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _renew_subs 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c094400 con 0x7ff00c138da0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _renew_subs 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c093510 con 0x7ff00c138da0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff004004ec0 con 0x7ff00c138da0 2024-09-16T06:42:53.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7ff0097fa700 4 set_mon_vals no callback set 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff0040051d0 con 0x7ff00c138da0 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 10 monclient: dump: 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:53.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.426+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff0040053b0 con 0x7ff00c138da0 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7ff011e45700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7ff011e45700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7ff011e45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec005100 con 0x7ff00c138da0 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(115..115 src has 66..115) v4 ==== 5687+0+0 (unknown 2439491067 0 0) 0x7ff004043f30 con 0x7ff00c138da0 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.430+0000 7ff00b7fe700 10 monclient: get_auth_request con 0x7feff80362a0 auth_method 0 2024-09-16T06:42:53.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.438+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff004014cf0 con 0x7ff00c138da0 2024-09-16T06:42:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.438+0000 7feff3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:53.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.438+0000 7feff3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:53.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.622+0000 7ff011e45700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:53.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.622+0000 7ff011e45700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:53.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.622+0000 7ff011e45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:53.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:53.622+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fefec005930 con 0x7ff00c138da0 2024-09-16T06:42:53.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:53.621+0000 7f2941141700 -1 mon.c@2(leader).osd e115 definitely_dead 0 2024-09-16T06:42:54.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.190+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v116) v1 ==== 87+0+0 (unknown 2025787314 0 0) 0x7ff004013160 con 0x7ff00c138da0 2024-09-16T06:42:54.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.190+0000 7feff3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:54.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.190+0000 7feff3fff700 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T06:42:54.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7ff011e45700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:54.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7ff011e45700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:54.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7ff011e45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:54.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fefec005420 con 0x7ff00c138da0 2024-09-16T06:42:54.310 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:42:54.309+0000 7f2941141700 -1 mon.c@2(leader).osd e116 definitely_dead 0 2024-09-16T06:42:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7feff3fff700 1 -- 172.21.15.189:0/1839876672 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v116) v1 ==== 96+0+0 (unknown 4069942990 0 0) 0x7ff0040148e0 con 0x7ff00c138da0 2024-09-16T06:42:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7feff3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T06:42:54.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.306+0000 7feff3fff700 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T06:42:54.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 >> v1:172.21.15.189:6812/3098810674 conn(0x7feff80362a0 legacy=0x7feff8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 10 monclient: shutdown 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 >> v1:172.21.15.76:6789/0 conn(0x7ff00c138da0 legacy=0x7ff00c1d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 shutdown_connections 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 >> 172.21.15.189:0/1839876672 conn(0x7ff00c0a03e0 msgr2=0x7ff00c09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 shutdown_connections 2024-09-16T06:42:54.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.310+0000 7ff011e45700 1 -- 172.21.15.189:0/1839876672 wait complete. 2024-09-16T06:42:54.315 INFO:tasks.workunit.client.0.smithi189.stderr:osd.14 does not exist. 2024-09-16T06:42:54.324 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-16T06:42:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: get_monmap_and_config 2024-09-16T06:42:54.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: build_initial_monmap 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: monmap: 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:54.426403+0000 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:54.426403+0000 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:54.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 1 Processor -- start 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 1 -- start start 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: init 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-b con 0x7f00881352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-c con 0x7f0088135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-a con 0x7f00880a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008809b050 con 0x7f00880a2030 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008806ab20 con 0x7f0088135db0 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008806a840 con 0x7f00881352c0 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008de3e700 10 monclient(hunting): authenticate will time out at 1126.939494s 2024-09-16T06:42:54.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f0087fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0088135db0 legacy=0x7f008813a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008ce3c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f00881352c0 0x7f00881356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48890/0 (socket says 172.21.15.189:48890) 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.422+0000 7f008d63d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f00880a2030 0x7f008813df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46516/0 (socket says 172.21.15.189:46516) 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008ce3c700 1 -- 172.21.15.189:0/3329847080 learned_addr learned my addr 172.21.15.189:0/3329847080 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255414451 0 0) 0x7f008806a840 con 0x7f00881352c0 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): my global_id is 125084 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00700035d0 con 0x7f00881352c0 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925301280 0 0) 0x7f008809b050 con 0x7f00880a2030 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:54.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient(hunting): my global_id is 154224 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f008806a840 con 0x7f00880a2030 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 99648066 0 0) 0x7f00700035d0 con 0x7f00881352c0 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 >> v1:172.21.15.189:6789/0 conn(0x7f0088135db0 legacy=0x7f008813a860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 >> v1:172.21.15.76:6789/0 conn(0x7f00880a2030 legacy=0x7f008813df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 monclient: found mon.noname-b 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008806c1b0 con 0x7f00881352c0 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:54.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f007c001610 con 0x7f00881352c0 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 5 monclient: authenticate success, global_id 125084 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: dump: 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:54.433 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f007c001830 con 0x7f00881352c0 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 1 -- 172.21.15.189:0/3329847080 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f007c001c00 con 0x7f00881352c0 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 4 set_mon_vals no callback set 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:54.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f00877fe700 10 monclient: dump: 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: shutdown 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- 172.21.15.189:0/3329847080 >> v1:172.21.15.76:6790/0 conn(0x7f00881352c0 legacy=0x7f00881356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:54.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- 172.21.15.189:0/3329847080 shutdown_connections 2024-09-16T06:42:54.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- 172.21.15.189:0/3329847080 >> 172.21.15.189:0/3329847080 conn(0x7f00880a4550 msgr2=0x7f00880a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:54.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- 172.21.15.189:0/3329847080 shutdown_connections 2024-09-16T06:42:54.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- 172.21.15.189:0/3329847080 wait complete. 2024-09-16T06:42:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: build_initial_monmap 2024-09-16T06:42:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: monmap: 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 Processor -- start 2024-09-16T06:42:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- start start 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: init 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-c con 0x7f00880a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-b con 0x7f00881352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient(hunting): picked mon.noname-a con 0x7f0088135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0088138500 con 0x7f0088135db0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0088139fc0 con 0x7f00881352c0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0088138f00 con 0x7f00880a2030 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f008de3e700 10 monclient(hunting): authenticate will time out at 1126.943856s 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.426+0000 7f0087fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f00881352c0 legacy=0x7f008813dd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008d63d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0088135db0 0x7f00881d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46522/0 (socket says 172.21.15.189:46522) 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008ce3c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f00880a2030 0x7f00881c9770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48894/0 (socket says 172.21.15.189:48894) 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008d63d700 1 -- 172.21.15.189:0/759692913 learned_addr learned my addr 172.21.15.189:0/759692913 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734883041 0 0) 0x7f0088138f00 con 0x7f00880a2030 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): my global_id is 125087 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00740035d0 con 0x7f00880a2030 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113963259 0 0) 0x7f0088138500 con 0x7f0088135db0 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _negotiate 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): my global_id is 154227 2024-09-16T06:42:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0088138f00 con 0x7f0088135db0 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486132766 0 0) 0x7f00740035d0 con 0x7f00880a2030 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0088138500 con 0x7f00880a2030 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f007c001b40 con 0x7f00880a2030 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): dump: 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2319973756 0 0) 0x7f0088138f00 con 0x7f0088135db0 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00740035d0 con 0x7f0088135db0 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0080001b50 con 0x7f0088135db0 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient(hunting): dump: 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.444 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1888728400 0 0) 0x7f0088138500 con 0x7f00880a2030 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 >> v1:172.21.15.189:6789/0 conn(0x7f00881352c0 legacy=0x7f008813dd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 >> v1:172.21.15.76:6789/0 conn(0x7f0088135db0 legacy=0x7f00881d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: _finish_hunting 0 2024-09-16T06:42:54.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 monclient: found mon.c 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008809e820 con 0x7f00880a2030 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: _finish_auth 0 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: _check_auth_tickets 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 5 monclient: authenticate success, global_id 125087 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _renew_subs 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00880a66b0 con 0x7f00880a2030 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _renew_subs 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f008813a580 con 0x7f00880a2030 2024-09-16T06:42:54.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f007c004b40 con 0x7f00880a2030 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f007c004ed0 con 0x7f00880a2030 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 10 monclient: dump: 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:54.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f0086ffd700 4 set_mon_vals no callback set 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f007c0050b0 con 0x7f00880a2030 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0068005100 con 0x7f00880a2030 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(116..116 src has 66..116) v4 ==== 5687+0+0 (unknown 145608132 0 0) 0x7f007c043be0 con 0x7f00880a2030 2024-09-16T06:42:54.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.430+0000 7f0087fff700 10 monclient: get_auth_request con 0x7f00740361e0 auth_method 0 2024-09-16T06:42:54.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.442+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f007c014ac0 con 0x7f00880a2030 2024-09-16T06:42:54.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.442+0000 7f00857fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:54.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.442+0000 7f00857fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.622+0000 7f008de3e700 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.622+0000 7f008de3e700 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.622+0000 7f008de3e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:54.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:54.622+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f0068005930 con 0x7f00880a2030 2024-09-16T06:42:55.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.190+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v117) v1 ==== 145+0+0 (unknown 2204521362 0 0) 0x7f007c012f30 con 0x7f00880a2030 2024-09-16T06:42:55.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.190+0000 7f00857fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:55.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.190+0000 7f00857fa700 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-16T06:42:55.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.306+0000 7f008de3e700 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:55.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.306+0000 7f008de3e700 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:55.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.306+0000 7f008de3e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:42:55.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:55.306+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f00680053b0 con 0x7f00880a2030 2024-09-16T06:42:56.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.206+0000 7f00857fa700 1 -- 172.21.15.189:0/759692913 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v118) v1 ==== 145+0+0 (unknown 3673119606 0 0) 0x7f007c0146b0 con 0x7f00880a2030 2024-09-16T06:42:56.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.206+0000 7f00857fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T06:42:56.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.206+0000 7f00857fa700 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00740361e0 legacy=0x7f00740386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 10 monclient: shutdown 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 >> v1:172.21.15.76:6790/0 conn(0x7f00880a2030 legacy=0x7f00881c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 shutdown_connections 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 >> 172.21.15.189:0/759692913 conn(0x7f00880a4550 msgr2=0x7f00880a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 shutdown_connections 2024-09-16T06:42:56.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.210+0000 7f008de3e700 1 -- 172.21.15.189:0/759692913 wait complete. 2024-09-16T06:42:56.217 INFO:tasks.workunit.client.0.smithi189.stderr:set require_min_compat_client to luminous 2024-09-16T06:42:56.231 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set dump 2024-09-16T06:42:56.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: get_monmap_and_config 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: build_initial_monmap 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: monmap: 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:56.346501+0000 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:56.346501+0000 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:56.346 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:56.347 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:56.347 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 1 Processor -- start 2024-09-16T06:42:56.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 1 -- start start 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: init 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient(hunting): picked mon.noname-c con 0x7f616009eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient(hunting): picked mon.noname-b con 0x7f616009f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient(hunting): picked mon.noname-a con 0x7f6160139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:56.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616009ab80 con 0x7f6160139a50 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616009a690 con 0x7f616009eba0 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6160072230 con 0x7f616009f690 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6165637700 10 monclient(hunting): authenticate will time out at 1128.859400s 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f615ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f616009eba0 legacy=0x7f616009ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6164e36700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6160139a50 0x7f616013be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46530/0 (socket says 172.21.15.189:46530) 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f6164e36700 1 -- 172.21.15.189:0/1800924997 learned_addr learned my addr 172.21.15.189:0/1800924997 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.342+0000 7f615f7fe700 1 --1- 172.21.15.189:0/1800924997 >> v1:172.21.15.76:6790/0 conn(0x7f616009f690 0x7f6160138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48898/0 (socket says 172.21.15.189:48898) 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329159112 0 0) 0x7f6160072230 con 0x7f616009f690 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): my global_id is 125090 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61480035d0 con 0x7f616009f690 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910549016 0 0) 0x7f616009ab80 con 0x7f6160139a50 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient(hunting): my global_id is 154230 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6160072230 con 0x7f6160139a50 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081186232 0 0) 0x7f61480035d0 con 0x7f616009f690 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 >> v1:172.21.15.189:6789/0 conn(0x7f616009eba0 legacy=0x7f616009ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 >> v1:172.21.15.76:6789/0 conn(0x7f6160139a50 legacy=0x7f616013be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 monclient: found mon.noname-b 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616009be90 con 0x7f616009f690 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: _finish_auth 0 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 5 monclient: authenticate success, global_id 125090 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6150001540 con 0x7f616009f690 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: dump: 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:56.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6150001820 con 0x7f616009f690 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 1 -- 172.21.15.189:0/1800924997 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6150001bf0 con 0x7f616009f690 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 4 set_mon_vals no callback set 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f615effd700 10 monclient: dump: 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: shutdown 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- 172.21.15.189:0/1800924997 >> v1:172.21.15.76:6790/0 conn(0x7f616009f690 legacy=0x7f6160138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- 172.21.15.189:0/1800924997 shutdown_connections 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- 172.21.15.189:0/1800924997 >> 172.21.15.189:0/1800924997 conn(0x7f616009fe70 msgr2=0x7f61600a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- 172.21.15.189:0/1800924997 shutdown_connections 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- 172.21.15.189:0/1800924997 wait complete. 2024-09-16T06:42:56.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: build_initial_monmap 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: monmap: 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:56.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 Processor -- start 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 1 -- start start 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: init 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.346+0000 7f6165637700 10 monclient(hunting): picked mon.noname-c con 0x7f616009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient(hunting): picked mon.noname-a con 0x7f616009f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient(hunting): picked mon.noname-b con 0x7f6160139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6160092b80 con 0x7f616009f690 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6160090820 con 0x7f6160139a50 2024-09-16T06:42:56.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616009a380 con 0x7f616009eba0 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient(hunting): authenticate will time out at 1128.864104s 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6164e36700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6160139a50 legacy=0x7f616008f9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f616009eba0 0x7f616008ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48908/0 (socket says 172.21.15.189:48908) 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615ffff700 1 -- 172.21.15.189:0/2305629325 learned_addr learned my addr 172.21.15.189:0/2305629325 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489419773 0 0) 0x7f6160092b80 con 0x7f616009f690 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): my global_id is 154233 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f614c0035d0 con 0x7f616009f690 2024-09-16T06:42:56.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692467743 0 0) 0x7f616009a380 con 0x7f616009eba0 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): my global_id is 125093 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6160092b80 con 0x7f616009eba0 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979107711 0 0) 0x7f614c0035d0 con 0x7f616009f690 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f616009a380 con 0x7f616009f690 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6150001b30 con 0x7f616009f690 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:56.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): dump: 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562438874 0 0) 0x7f6160092b80 con 0x7f616009eba0 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f614c0035d0 con 0x7f616009eba0 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6154004180 con 0x7f616009eba0 2024-09-16T06:42:56.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient(hunting): dump: 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.368 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1277112363 0 0) 0x7f616009a380 con 0x7f616009f690 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 >> v1:172.21.15.76:6790/0 conn(0x7f616009eba0 legacy=0x7f616008ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 >> v1:172.21.15.189:6789/0 conn(0x7f6160139a50 legacy=0x7f616008f9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:56.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 monclient: found mon.a 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160071150 con 0x7f616009f690 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: _finish_auth 0 2024-09-16T06:42:56.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 5 monclient: authenticate success, global_id 154233 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _renew_subs 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616009e4e0 con 0x7f616009f690 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _renew_subs 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160092fa0 con 0x7f616009f690 2024-09-16T06:42:56.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6150001120 con 0x7f616009f690 2024-09-16T06:42:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:56.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6150004ce0 con 0x7f616009f690 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615e7fc700 4 set_mon_vals no callback set 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 10 monclient: dump: 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f615000f750 con 0x7f616009f690 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(118..118 src has 66..118) v4 ==== 5687+0+0 (unknown 46046365 0 0) 0x7f6150043a70 con 0x7f616009f690 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6140005100 con 0x7f616009f690 2024-09-16T06:42:56.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.350+0000 7f615ffff700 10 monclient: get_auth_request con 0x7f614c036360 auth_method 0 2024-09-16T06:42:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.362+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6150011210 con 0x7f616009f690 2024-09-16T06:42:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.362+0000 7f615cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.362+0000 7f615cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f6165637700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f6165637700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f6165637700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f6140005930 con 0x7f616009f690 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v118) v1 ==== 73+0+3 (unknown 1887001698 0 2423995997) 0x7f6150012d60 con 0x7f616009f690 2024-09-16T06:42:56.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f615cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.538+0000 7f615cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:56.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f6165637700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f6165637700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f6165637700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f6140001140 con 0x7f616009f690 2024-09-16T06:42:56.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f615cff9700 1 -- 172.21.15.189:0/2305629325 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v118) v1 ==== 73+0+3 (unknown 1887001698 0 2423995997) 0x7f61500144e0 con 0x7f616009f690 2024-09-16T06:42:56.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f615cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:42:56.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.642+0000 7f615cff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:56.646 INFO:tasks.workunit.client.0.smithi189.stdout:{} 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 >> v1:172.21.15.189:6812/3098810674 conn(0x7f614c036360 legacy=0x7f614c038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 10 monclient: shutdown 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 >> v1:172.21.15.76:6789/0 conn(0x7f616009f690 legacy=0x7f616008f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 shutdown_connections 2024-09-16T06:42:56.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 >> 172.21.15.189:0/2305629325 conn(0x7f616009fe70 msgr2=0x7f61600a2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:56.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 shutdown_connections 2024-09-16T06:42:56.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.646+0000 7f6165637700 1 -- 172.21.15.189:0/2305629325 wait complete. 2024-09-16T06:42:56.658 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:42:56.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.754+0000 7fb624e68700 10 monclient: get_monmap_and_config 2024-09-16T06:42:56.759 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.754+0000 7fb624e68700 10 monclient: build_initial_monmap 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.754+0000 7fb624e68700 10 monclient: monmap: 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:56.760343+0000 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:56.760343+0000 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:56.760 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 1 Processor -- start 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 1 -- start start 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient: init 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-a con 0x7fb62009f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-c con 0x7fb620099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-b con 0x7fb6201396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62009f520 con 0x7fb62009f140 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62009bc10 con 0x7fb620099700 2024-09-16T06:42:56.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62009b810 con 0x7fb6201396a0 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient(hunting): authenticate will time out at 1129.273178s 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61dd9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb620099700 legacy=0x7fb62009b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb62009f140 0x7fb62009a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46558/0 (socket says 172.21.15.189:46558) 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61e59c700 1 -- 172.21.15.189:0/2392848397 learned_addr learned my addr 172.21.15.189:0/2392848397 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61ed9d700 1 --1- 172.21.15.189:0/2392848397 >> v1:172.21.15.76:6790/0 conn(0x7fb6201396a0 0x7fb62013ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48920/0 (socket says 172.21.15.189:48920) 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570332262 0 0) 0x7fb62009f520 con 0x7fb62009f140 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): my global_id is 154236 2024-09-16T06:42:56.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6080035d0 con 0x7fb62009f140 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320661312 0 0) 0x7fb62009b810 con 0x7fb6201396a0 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient(hunting): my global_id is 125096 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62009f520 con 0x7fb6201396a0 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750156284 0 0) 0x7fb6080035d0 con 0x7fb62009f140 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 >> v1:172.21.15.76:6790/0 conn(0x7fb6201396a0 legacy=0x7fb62013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 >> v1:172.21.15.189:6789/0 conn(0x7fb620099700 legacy=0x7fb62009b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 monclient: found mon.noname-a 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62009ceb0 con 0x7fb62009f140 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: _finish_auth 0 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb618001990 con 0x7fb62009f140 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: dump: 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.767 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 5 monclient: authenticate success, global_id 154236 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb618001bf0 con 0x7fb62009f140 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 1 -- 172.21.15.189:0/2392848397 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb618001fc0 con 0x7fb62009f140 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb61d59a700 10 monclient: dump: 2024-09-16T06:42:56.768 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.758+0000 7fb624e68700 4 set_mon_vals no callback set 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: shutdown 2024-09-16T06:42:56.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:56.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- 172.21.15.189:0/2392848397 >> v1:172.21.15.76:6789/0 conn(0x7fb62009f140 legacy=0x7fb62009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- 172.21.15.189:0/2392848397 shutdown_connections 2024-09-16T06:42:56.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- 172.21.15.189:0/2392848397 >> 172.21.15.189:0/2392848397 conn(0x7fb6200a03e0 msgr2=0x7fb6200a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:56.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- 172.21.15.189:0/2392848397 shutdown_connections 2024-09-16T06:42:56.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- 172.21.15.189:0/2392848397 wait complete. 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: build_initial_monmap 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: monmap: 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:56.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 Processor -- start 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- start start 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: init 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-c con 0x7fb620099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-b con 0x7fb62009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:56.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient(hunting): picked mon.noname-a con 0x7fb6201396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb620139510 con 0x7fb6201396a0 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb61dd9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb62009f140 legacy=0x7fb6201ca830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb620090960 con 0x7fb62009f140 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62008e600 con 0x7fb620099700 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.762+0000 7fb624e68700 10 monclient(hunting): authenticate will time out at 1129.279903s 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb61e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb620099700 0x7fb620138e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48928/0 (socket says 172.21.15.189:48928) 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb61e59c700 1 -- 172.21.15.189:0/3152169136 learned_addr learned my addr 172.21.15.189:0/3152169136 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283498274 0 0) 0x7fb62008e600 con 0x7fb620099700 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): my global_id is 125099 2024-09-16T06:42:56.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6000035d0 con 0x7fb620099700 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859141015 0 0) 0x7fb620139510 con 0x7fb6201396a0 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): my global_id is 154239 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62008e600 con 0x7fb6201396a0 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497546233 0 0) 0x7fb62008e600 con 0x7fb6201396a0 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb620139510 con 0x7fb6201396a0 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 463344377 0 0) 0x7fb6000035d0 con 0x7fb620099700 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb62008e600 con 0x7fb620099700 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6100011d0 con 0x7fb6201396a0 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): dump: 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb618001f00 con 0x7fb620099700 2024-09-16T06:42:56.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient(hunting): dump: 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1120366102 0 0) 0x7fb620139510 con 0x7fb6201396a0 2024-09-16T06:42:56.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 >> v1:172.21.15.76:6790/0 conn(0x7fb620099700 legacy=0x7fb620138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 >> v1:172.21.15.189:6789/0 conn(0x7fb62009f140 legacy=0x7fb6201ca830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 monclient: found mon.a 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62009a6e0 con 0x7fb6201396a0 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: _finish_auth 0 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 5 monclient: authenticate success, global_id 154239 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _renew_subs 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6200990b0 con 0x7fb6201396a0 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _renew_subs 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6200900c0 con 0x7fb6201396a0 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb610004890 con 0x7fb6201396a0 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60ffff700 4 set_mon_vals no callback set 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb610004ba0 con 0x7fb6201396a0 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 10 monclient: dump: 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.782 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb61000f610 con 0x7fb6201396a0 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(118..118 src has 66..118) v4 ==== 5687+0+0 (unknown 46046365 0 0) 0x7fb6100439b0 con 0x7fb6201396a0 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604005100 con 0x7fb6201396a0 2024-09-16T06:42:56.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.766+0000 7fb61e59c700 10 monclient: get_auth_request con 0x7fb6000362f0 auth_method 0 2024-09-16T06:42:56.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.778+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb610014710 con 0x7fb6201396a0 2024-09-16T06:42:56.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.778+0000 7fb60effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:56.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.778+0000 7fb60effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb624e68700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb624e68700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb624e68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb604005930 con 0x7fb6201396a0 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v118) v1 ==== 71+0+0 (unknown 1001918175 0 0) 0x7fb610012b80 con 0x7fb6201396a0 2024-09-16T06:42:56.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb60effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:56.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:56.958+0000 7fb60effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:42:57.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.058+0000 7fb624e68700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:57.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.058+0000 7fb624e68700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:57.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.058+0000 7fb624e68700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.058+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb604005650 con 0x7fb6201396a0 2024-09-16T06:42:57.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb60effd700 1 -- 172.21.15.189:0/3152169136 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v118) v1 ==== 71+0+0 (unknown 1001918175 0 0) 0x7fb610014710 con 0x7fb6201396a0 2024-09-16T06:42:57.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb60effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:42:57.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb60effd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:42:57.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb6000362f0 legacy=0x7fb6000387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 10 monclient: shutdown 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 >> v1:172.21.15.76:6789/0 conn(0x7fb6201396a0 legacy=0x7fb6201cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 shutdown_connections 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 >> 172.21.15.189:0/3152169136 conn(0x7fb6200a03e0 msgr2=0x7fb620071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 shutdown_connections 2024-09-16T06:42:57.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.062+0000 7fb624e68700 1 -- 172.21.15.189:0/3152169136 wait complete. 2024-09-16T06:42:57.076 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T06:42:57.076 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:42:57.076 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.178+0000 7f3607688700 10 monclient: get_monmap_and_config 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.178+0000 7f3607688700 10 monclient: build_initial_monmap 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.178+0000 7f3607688700 10 monclient: monmap: 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:57.183919+0000 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:57.183919+0000 2024-09-16T06:42:57.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:57.184 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:57.184 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 Processor -- start 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- start start 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient: init 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient(hunting): picked mon.noname-b con 0x7f3600135510 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient(hunting): picked mon.noname-a con 0x7f36000a3bc0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient(hunting): picked mon.noname-c con 0x7f360013bbb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36000a3330 con 0x7f36000a3bc0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36000a16e0 con 0x7f360013bbb0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3605c25700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f360013bbb0 legacy=0x7f360013dfa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3600099dd0 con 0x7f3600135510 2024-09-16T06:42:57.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient(hunting): authenticate will time out at 1129.697086s 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3605424700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3600135510 0x7f36001358f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48936/0 (socket says 172.21.15.189:48936) 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3605424700 1 -- 172.21.15.189:0/3479547549 learned_addr learned my addr 172.21.15.189:0/3479547549 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058540846 0 0) 0x7f36000a3330 con 0x7f36000a3bc0 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): my global_id is 154242 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ec0035d0 con 0x7f36000a3bc0 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889199949 0 0) 0x7f3600099dd0 con 0x7f3600135510 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient(hunting): my global_id is 125102 2024-09-16T06:42:57.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36000a3330 con 0x7f3600135510 2024-09-16T06:42:57.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58979878 0 0) 0x7f35ec0035d0 con 0x7f36000a3bc0 2024-09-16T06:42:57.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 >> v1:172.21.15.76:6790/0 conn(0x7f3600135510 legacy=0x7f36001358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 >> v1:172.21.15.189:6789/0 conn(0x7f360013bbb0 legacy=0x7f360013dfa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:57.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 monclient: found mon.noname-a 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360009a0b0 con 0x7f36000a3bc0 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: _finish_auth 0 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35fc001610 con 0x7f36000a3bc0 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 5 monclient: authenticate success, global_id 154242 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:57.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: dump: 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35fc0018f0 con 0x7f36000a3bc0 2024-09-16T06:42:57.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 1 -- 172.21.15.189:0/3479547549 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35fc001c80 con 0x7f36000a3bc0 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 4 set_mon_vals no callback set 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f35f7fff700 10 monclient: dump: 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.195 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 10 monclient: shutdown 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- 172.21.15.189:0/3479547549 >> v1:172.21.15.76:6789/0 conn(0x7f36000a3bc0 legacy=0x7f360013a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- 172.21.15.189:0/3479547549 shutdown_connections 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- 172.21.15.189:0/3479547549 >> 172.21.15.189:0/3479547549 conn(0x7f36000a7e70 msgr2=0x7f36000a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- 172.21.15.189:0/3479547549 shutdown_connections 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.182+0000 7f3607688700 1 -- 172.21.15.189:0/3479547549 wait complete. 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient: build_initial_monmap 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient: monmap: 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:57.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 1 Processor -- start 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 1 -- start start 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient: init 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient(hunting): picked mon.noname-c con 0x7f36000a3bc0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:57.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient(hunting): picked mon.noname-b con 0x7f3600135510 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient(hunting): picked mon.noname-a con 0x7f360013bbb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360006bdb0 con 0x7f360013bbb0 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360011f6d0 con 0x7f3600135510 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3604c23700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3600135510 legacy=0x7f36001cefe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36000a0680 con 0x7f36000a3bc0 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3607688700 10 monclient(hunting): authenticate will time out at 1129.702368s 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3605424700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f36000a3bc0 0x7f36001c9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48944/0 (socket says 172.21.15.189:48944) 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f3605424700 1 -- 172.21.15.189:0/3935611415 learned_addr learned my addr 172.21.15.189:0/3935611415 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:57.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769470297 0 0) 0x7f360006bdb0 con 0x7f360013bbb0 2024-09-16T06:42:57.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): my global_id is 154245 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e40035d0 con 0x7f360013bbb0 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188654915 0 0) 0x7f36000a0680 con 0x7f36000a3bc0 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): my global_id is 125105 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f360006bdb0 con 0x7f36000a3bc0 2024-09-16T06:42:57.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414640449 0 0) 0x7f35e40035d0 con 0x7f360013bbb0 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36000a0680 con 0x7f360013bbb0 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35f8004360 con 0x7f360013bbb0 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): dump: 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.201 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584179467 0 0) 0x7f360006bdb0 con 0x7f36000a3bc0 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e40035d0 con 0x7f36000a3bc0 2024-09-16T06:42:57.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35f00018f0 con 0x7f36000a3bc0 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 10 monclient(hunting): dump: 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.203 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.186+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2860092233 0 0) 0x7f36000a0680 con 0x7f360013bbb0 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 >> v1:172.21.15.76:6790/0 conn(0x7f36000a3bc0 legacy=0x7f36001c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 >> v1:172.21.15.189:6789/0 conn(0x7f3600135510 legacy=0x7f36001cefe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 monclient: found mon.a 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360013b0b0 con 0x7f360013bbb0 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:57.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 5 monclient: authenticate success, global_id 154245 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _renew_subs 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36000a6aa0 con 0x7f360013bbb0 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _renew_subs 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36000a0200 con 0x7f360013bbb0 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35f8004880 con 0x7f360013bbb0 2024-09-16T06:42:57.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f35f8004b90 con 0x7f360013bbb0 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f77fe700 4 set_mon_vals no callback set 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 10 monclient: dump: 2024-09-16T06:42:57.207 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.208 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3607688700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0005100 con 0x7f360013bbb0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f35f8004cf0 con 0x7f360013bbb0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(118..118 src has 66..118) v4 ==== 5687+0+0 (unknown 46046365 0 0) 0x7f35f8043bd0 con 0x7f360013bbb0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.190+0000 7f3605424700 10 monclient: get_auth_request con 0x7f35e40362f0 auth_method 0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.198+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f35f80146a0 con 0x7f360013bbb0 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.198+0000 7f35f5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.198+0000 7f35f5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:57.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.378+0000 7f3607688700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:42:57.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.378+0000 7f3607688700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:42:57.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.378+0000 7f3607688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.378+0000 7f3607688700 1 -- 172.21.15.189:0/3935611415 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f35e0005930 con 0x7f360013bbb0 2024-09-16T06:42:57.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.382+0000 7f35f5ffb700 1 -- 172.21.15.189:0/3935611415 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v118) v1 ==== 152+0+0 (unknown 2159622029 0 0) 0x7f35f801cf20 con 0x7f360013bbb0 2024-09-16T06:42:57.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.382+0000 7f35f5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:42:57.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.382+0000 7f35f5ffb700 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-16T06:42:57.386 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: pool 'fooset' not found 2024-09-16T06:42:57.390 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:42:57.390 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create fooset 8 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: get_monmap_and_config 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: build_initial_monmap 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: monmap: 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:57.493021+0000 2024-09-16T06:42:57.492 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:57.493021+0000 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 1 Processor -- start 2024-09-16T06:42:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 1 -- start start 2024-09-16T06:42:57.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: init 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-a con 0x7faba809f570 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-c con 0x7faba80a0060 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-b con 0x7faba8138a80 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba809d310 con 0x7faba809f570 2024-09-16T06:42:57.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba809c370 con 0x7faba80a0060 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabad929700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faba80a0060 legacy=0x7faba8137360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba809ade0 con 0x7faba8138a80 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabaf12c700 10 monclient(hunting): authenticate will time out at 1130.007292s 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabae12a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faba809f570 0x7faba809f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46604/0 (socket says 172.21.15.189:46604) 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabae92b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faba8138a80 0x7faba813ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48948/0 (socket says 172.21.15.189:48948) 2024-09-16T06:42:57.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.490+0000 7fabae12a700 1 -- 172.21.15.189:0/1694645028 learned_addr learned my addr 172.21.15.189:0/1694645028 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61985885 0 0) 0x7faba4001040 con 0x7faba809f570 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): my global_id is 154248 2024-09-16T06:42:57.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab940035d0 con 0x7faba809f570 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113765413 0 0) 0x7faba809ade0 con 0x7faba8138a80 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient(hunting): my global_id is 125108 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba4001040 con 0x7faba8138a80 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425825979 0 0) 0x7fab940035d0 con 0x7faba809f570 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 >> v1:172.21.15.76:6790/0 conn(0x7faba8138a80 legacy=0x7faba813ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 >> v1:172.21.15.189:6789/0 conn(0x7faba80a0060 legacy=0x7faba8137360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: _finish_hunting 0 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 monclient: found mon.noname-a 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:57.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba809c090 con 0x7faba809f570 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: _finish_auth 0 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: _check_auth_tickets 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba4001530 con 0x7faba809f570 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 5 monclient: authenticate success, global_id 154248 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: dump: 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.501 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faba40017c0 con 0x7faba809f570 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 4 set_mon_vals no callback set 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 1 -- 172.21.15.189:0/1694645028 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba4001b50 con 0x7faba809f570 2024-09-16T06:42:57.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabad128700 10 monclient: dump: 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 10 monclient: shutdown 2024-09-16T06:42:57.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 1 -- 172.21.15.189:0/1694645028 >> v1:172.21.15.76:6789/0 conn(0x7faba809f570 legacy=0x7faba809f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 1 -- 172.21.15.189:0/1694645028 shutdown_connections 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 1 -- 172.21.15.189:0/1694645028 >> 172.21.15.189:0/1694645028 conn(0x7faba80a4320 msgr2=0x7faba80a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 1 -- 172.21.15.189:0/1694645028 shutdown_connections 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 1 -- 172.21.15.189:0/1694645028 wait complete. 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 10 monclient: build_initial_monmap 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.494+0000 7fabaf12c700 10 monclient: monmap: 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:57.504 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:57.505 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:57.505 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:57.505 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:57.505 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 Processor -- start 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- start start 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: init 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-c con 0x7faba809f570 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:57.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-a con 0x7faba80a0060 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient(hunting): picked mon.noname-b con 0x7faba8138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba80a32f0 con 0x7faba80a0060 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8072210 con 0x7faba8138a80 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba8070d90 con 0x7faba809f570 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient(hunting): authenticate will time out at 1130.013881s 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabae92b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faba8138a80 legacy=0x7faba81d1c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabae12a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faba809f570 0x7faba8122050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48958/0 (socket says 172.21.15.189:48958) 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabad929700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faba80a0060 0x7faba8122760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46618/0 (socket says 172.21.15.189:46618) 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabae12a700 1 -- 172.21.15.189:0/260565239 learned_addr learned my addr 172.21.15.189:0/260565239 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880421642 0 0) 0x7faba8070d90 con 0x7faba809f570 2024-09-16T06:42:57.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): my global_id is 125111 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c0035d0 con 0x7faba809f570 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393463382 0 0) 0x7fab98003af0 con 0x7faba80a0060 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _negotiate 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): my global_id is 154251 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8070d90 con 0x7faba80a0060 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139073377 0 0) 0x7faba8070d90 con 0x7faba80a0060 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab98003af0 con 0x7faba80a0060 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658213856 0 0) 0x7fab8c0035d0 con 0x7faba809f570 2024-09-16T06:42:57.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faba8070d90 con 0x7faba809f570 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab98004160 con 0x7faba80a0060 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): dump: 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.512 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba4004890 con 0x7faba809f570 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient(hunting): dump: 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.513 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780135793 0 0) 0x7fab98003af0 con 0x7faba80a0060 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 >> v1:172.21.15.76:6790/0 conn(0x7faba809f570 legacy=0x7faba8122050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 >> v1:172.21.15.189:6789/0 conn(0x7faba8138a80 legacy=0x7faba81d1c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient: _finish_hunting 0 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 monclient: found mon.a 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba8124560 con 0x7faba80a0060 2024-09-16T06:42:57.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient: _finish_auth 0 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 10 monclient: _check_auth_tickets 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fab9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 5 monclient: authenticate success, global_id 154251 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _renew_subs 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba8125a80 con 0x7faba80a0060 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _renew_subs 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.498+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faba8125190 con 0x7faba80a0060 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab980046b0 con 0x7faba80a0060 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab98004a40 con 0x7faba80a0060 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:57.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabac927700 4 set_mon_vals no callback set 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 10 monclient: dump: 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabaf12c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabaf12c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabaf12c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90005100 con 0x7faba80a0060 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fab98004c20 con 0x7faba80a0060 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(118..118 src has 66..118) v4 ==== 5687+0+0 (unknown 46046365 0 0) 0x7fab980438c0 con 0x7faba80a0060 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.502+0000 7fabae12a700 10 monclient: get_auth_request con 0x7fab8c036210 auth_method 0 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.514+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fab98014780 con 0x7faba80a0060 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.514+0000 7fab9effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:57.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.514+0000 7fab9effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:57.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.682+0000 7fabaf12c700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:57.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.682+0000 7fabaf12c700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:57.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.682+0000 7fabaf12c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:57.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:57.682+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fab90005930 con 0x7faba80a0060 2024-09-16T06:42:58.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.218+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v119) v1 ==== 115+0+0 (unknown 2594174340 0 0) 0x7fab9801cd70 con 0x7faba80a0060 2024-09-16T06:42:58.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.218+0000 7fab9effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:58.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.218+0000 7fab9effd700 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-16T06:42:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fabaf12c700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fabaf12c700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fabaf12c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7fab900053b0 con 0x7faba80a0060 2024-09-16T06:42:58.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fab9effd700 1 -- 172.21.15.189:0/260565239 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v119) v1 ==== 122+0+0 (unknown 2847755681 0 0) 0x7fab98012bf0 con 0x7faba80a0060 2024-09-16T06:42:58.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fab9effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T06:42:58.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.326+0000 7fab9effd700 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-16T06:42:58.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 >> v1:172.21.15.189:6812/3098810674 conn(0x7fab8c036210 legacy=0x7fab8c0386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:58.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 10 monclient: shutdown 2024-09-16T06:42:58.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:58.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 >> v1:172.21.15.76:6789/0 conn(0x7faba80a0060 legacy=0x7faba8122760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:58.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 shutdown_connections 2024-09-16T06:42:58.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 >> 172.21.15.189:0/260565239 conn(0x7faba80a4320 msgr2=0x7faba80a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:58.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 shutdown_connections 2024-09-16T06:42:58.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.330+0000 7fabaf12c700 1 -- 172.21.15.189:0/260565239 wait complete. 2024-09-16T06:42:58.334 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'fooset' already exists 2024-09-16T06:42:58.343 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create barset 8 2024-09-16T06:42:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: get_monmap_and_config 2024-09-16T06:42:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: build_initial_monmap 2024-09-16T06:42:58.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: monmap: 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:58.446996+0000 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:58.446996+0000 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 1 Processor -- start 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 1 -- start start 2024-09-16T06:42:58.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: init 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-b con 0x7f91ec09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-a con 0x7f91ec09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-c con 0x7f91ec138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:58.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec09cca0 con 0x7f91ec09c5f0 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec09f750 con 0x7f91ec138da0 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec097620 con 0x7f91ec09bb00 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f3396700 10 monclient(hunting): authenticate will time out at 1130.959816s 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.442+0000 7f91f1933700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f91ec138da0 legacy=0x7f91ec13b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f0931700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f91ec09c5f0 0x7f91ec137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46624/0 (socket says 172.21.15.189:46624) 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f0931700 1 -- 172.21.15.189:0/2396966584 learned_addr learned my addr 172.21.15.189:0/2396966584 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645180464 0 0) 0x7f91ec097620 con 0x7f91ec09bb00 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): my global_id is 125114 2024-09-16T06:42:58.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d80035d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665068942 0 0) 0x7f91ec09cca0 con 0x7f91ec09c5f0 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _negotiate 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient(hunting): my global_id is 154254 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec097620 con 0x7f91ec09c5f0 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911735163 0 0) 0x7f91d80035d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 >> v1:172.21.15.189:6789/0 conn(0x7f91ec138da0 legacy=0x7f91ec13b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 >> v1:172.21.15.76:6789/0 conn(0x7f91ec09c5f0 legacy=0x7f91ec137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: _finish_hunting 0 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 monclient: found mon.noname-b 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:58.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec097900 con 0x7f91ec09bb00 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: _finish_auth 0 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: _check_auth_tickets 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e40018c0 con 0x7f91ec09bb00 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 5 monclient: authenticate success, global_id 125114 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: dump: 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:58.453 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91e4001ba0 con 0x7f91ec09bb00 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 1 -- 172.21.15.189:0/2396966584 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e4001f30 con 0x7f91ec09bb00 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 4 set_mon_vals no callback set 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:58.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91dffff700 10 monclient: dump: 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 10 monclient: shutdown 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- 172.21.15.189:0/2396966584 >> v1:172.21.15.76:6790/0 conn(0x7f91ec09bb00 legacy=0x7f91ec09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- 172.21.15.189:0/2396966584 shutdown_connections 2024-09-16T06:42:58.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- 172.21.15.189:0/2396966584 >> 172.21.15.189:0/2396966584 conn(0x7f91ec0a03e0 msgr2=0x7f91ec0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:58.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- 172.21.15.189:0/2396966584 shutdown_connections 2024-09-16T06:42:58.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- 172.21.15.189:0/2396966584 wait complete. 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 10 monclient: build_initial_monmap 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 10 monclient: monmap: 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:58.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 Processor -- start 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 1 -- start start 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.446+0000 7f91f3396700 10 monclient: init 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-a con 0x7f91ec09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-c con 0x7f91ec09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient(hunting): picked mon.noname-b con 0x7f91ec138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec094c50 con 0x7f91ec09bb00 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec0a2ee0 con 0x7f91ec138da0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec0943d0 con 0x7f91ec09c5f0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient(hunting): authenticate will time out at 1130.964380s 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f1933700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f91ec138da0 legacy=0x7f91ec1d1fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:58.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f1132700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f91ec09bb00 0x7f91ec0912c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:46640/0 (socket says 172.21.15.189:46640) 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f1132700 1 -- 172.21.15.189:0/940587009 learned_addr learned my addr 172.21.15.189:0/940587009 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876306039 0 0) 0x7f91ec094c50 con 0x7f91ec09bb00 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): my global_id is 154257 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d00035d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032967918 0 0) 0x7f91ec0943d0 con 0x7f91ec09c5f0 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): my global_id is 125117 2024-09-16T06:42:58.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec094c50 con 0x7f91ec09c5f0 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2567212269 0 0) 0x7f91d00035d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ec0943d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e4001f70 con 0x7f91ec09bb00 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): dump: 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:58.463 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896749664 0 0) 0x7f91ec094c50 con 0x7f91ec09c5f0 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d00035d0 con 0x7f91ec09c5f0 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e0001830 con 0x7f91ec09c5f0 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient(hunting): dump: 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 734996215 0 0) 0x7f91ec0943d0 con 0x7f91ec09bb00 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 >> v1:172.21.15.76:6790/0 conn(0x7f91ec09c5f0 legacy=0x7f91ec0919d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 >> v1:172.21.15.189:6789/0 conn(0x7f91ec138da0 legacy=0x7f91ec1d1fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 monclient: found mon.a 2024-09-16T06:42:58.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec094970 con 0x7f91ec09bb00 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 5 monclient: authenticate success, global_id 154257 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _renew_subs 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec094f30 con 0x7f91ec09bb00 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _renew_subs 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec092b60 con 0x7f91ec09bb00 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91e4003cd0 con 0x7f91ec09bb00 2024-09-16T06:42:58.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91df7fe700 4 set_mon_vals no callback set 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e4004060 con 0x7f91ec09bb00 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 10 monclient: dump: 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:58.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f91e4004240 con 0x7f91ec09bb00 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.450+0000 7f91f3396700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.454+0000 7f91f3396700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.454+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91cc005100 con 0x7f91ec09bb00 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.454+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(119..119 src has 66..119) v4 ==== 6032+0+0 (unknown 3810474262 0 0) 0x7f91e4044060 con 0x7f91ec09bb00 2024-09-16T06:42:58.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.454+0000 7f91f0931700 10 monclient: get_auth_request con 0x7f91d0036360 auth_method 0 2024-09-16T06:42:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.462+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f91e4014b00 con 0x7f91ec09bb00 2024-09-16T06:42:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.462+0000 7f91ddffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:58.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.462+0000 7f91ddffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:58.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.634+0000 7f91f3396700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:58.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.634+0000 7f91f3396700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:58.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.634+0000 7f91f3396700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:58.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:58.634+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f91cc005930 con 0x7f91ec09bb00 2024-09-16T06:42:59.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.222+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v120) v1 ==== 115+0+0 (unknown 1312594989 0 0) 0x7f91e4014920 con 0x7f91ec09bb00 2024-09-16T06:42:59.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.222+0000 7f91ddffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:59.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.222+0000 7f91ddffb700 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-16T06:42:59.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.330+0000 7f91f3396700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:59.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.330+0000 7f91f3396700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:59.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.330+0000 7f91f3396700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.330+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f91cc0053b0 con 0x7f91ec09bb00 2024-09-16T06:42:59.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91ddffb700 1 -- 172.21.15.189:0/940587009 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v120) v1 ==== 122+0+0 (unknown 2287243759 0 0) 0x7f91e4014920 con 0x7f91ec09bb00 2024-09-16T06:42:59.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91ddffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T06:42:59.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91ddffb700 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 >> v1:172.21.15.189:6812/3098810674 conn(0x7f91d0036360 legacy=0x7f91d0038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 10 monclient: shutdown 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 >> v1:172.21.15.76:6789/0 conn(0x7f91ec09bb00 legacy=0x7f91ec0912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 shutdown_connections 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 >> 172.21.15.189:0/940587009 conn(0x7f91ec0a03e0 msgr2=0x7f91ec09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 shutdown_connections 2024-09-16T06:42:59.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.334+0000 7f91f3396700 1 -- 172.21.15.189:0/940587009 wait complete. 2024-09-16T06:42:59.340 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'barset' already exists 2024-09-16T06:42:59.349 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set barset size 3 2024-09-16T06:42:59.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: get_monmap_and_config 2024-09-16T06:42:59.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: build_initial_monmap 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: monmap: 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:42:59.461225+0000 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:42:59.461225+0000 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:42:59.461 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:42:59.462 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 1 Processor -- start 2024-09-16T06:42:59.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 1 -- start start 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: init 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-b con 0x7f179c09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-c con 0x7f179c099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-a con 0x7f179c1396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c09f520 con 0x7f179c1396a0 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c09bc10 con 0x7f179c099700 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c09b810 con 0x7f179c09f140 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 10 monclient(hunting): authenticate will time out at 1131.974193s 2024-09-16T06:42:59.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f179a7fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f179c099700 legacy=0x7f179c09b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f179affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f179c09f140 0x7f179c09a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57716/0 (socket says 172.21.15.189:57716) 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f179affd700 1 -- 172.21.15.189:0/3013763944 learned_addr learned my addr 172.21.15.189:0/3013763944 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874149442 0 0) 0x7f179c09f520 con 0x7f179c1396a0 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): my global_id is 154260 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17840035d0 con 0x7f179c1396a0 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914730666 0 0) 0x7f179c09b810 con 0x7f179c09f140 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:42:59.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient(hunting): my global_id is 125120 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c09f520 con 0x7f179c09f140 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218537873 0 0) 0x7f179c09f520 con 0x7f179c09f140 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 >> v1:172.21.15.189:6789/0 conn(0x7f179c099700 legacy=0x7f179c09b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 >> v1:172.21.15.76:6789/0 conn(0x7f179c1396a0 legacy=0x7f179c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 monclient: found mon.noname-b 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c09ceb0 con 0x7f179c09f140 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: _finish_auth 0 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:42:59.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1790001570 con 0x7f179c09f140 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f1799ffb700 10 monclient: dump: 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:59.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 5 monclient: authenticate success, global_id 125120 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.458+0000 7f17a16ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1790001810 con 0x7f179c09f140 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: get_monmap_and_config success 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 4 set_mon_vals no callback set 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 1 -- 172.21.15.189:0/3013763944 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1790001ba0 con 0x7f179c09f140 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f1799ffb700 10 monclient: dump: 2024-09-16T06:42:59.469 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: shutdown 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- 172.21.15.189:0/3013763944 >> v1:172.21.15.76:6790/0 conn(0x7f179c09f140 legacy=0x7f179c09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- 172.21.15.189:0/3013763944 shutdown_connections 2024-09-16T06:42:59.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- 172.21.15.189:0/3013763944 >> 172.21.15.189:0/3013763944 conn(0x7f179c0a03e0 msgr2=0x7f179c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:42:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- 172.21.15.189:0/3013763944 shutdown_connections 2024-09-16T06:42:59.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- 172.21.15.189:0/3013763944 wait complete. 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: build_initial_monmap 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: monmap: 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:42:59.474 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 Processor -- start 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- start start 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: init 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-b con 0x7f179c099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-a con 0x7f179c09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:42:59.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient(hunting): picked mon.noname-c con 0x7f179c1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f179affd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f179c099700 legacy=0x7f179c138c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c1326d0 con 0x7f179c09f140 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c09a6e0 con 0x7f179c099700 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c098eb0 con 0x7f179c1396a0 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f17a16ea700 10 monclient(hunting): authenticate will time out at 1131.979366s 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f179b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f179c1396a0 0x7f179c1d2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57724/0 (socket says 172.21.15.189:57724) 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f179a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f179c09f140 0x7f179c1ceb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42370/0 (socket says 172.21.15.189:42370) 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.462+0000 7f179b7fe700 1 -- 172.21.15.189:0/1561032644 learned_addr learned my addr 172.21.15.189:0/1561032644 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506078777 0 0) 0x7f179c1326d0 con 0x7f179c09f140 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:59.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): my global_id is 154263 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17880035d0 con 0x7f179c09f140 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806520820 0 0) 0x7f179c098eb0 con 0x7f179c1396a0 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _negotiate 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): my global_id is 125123 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c1326d0 con 0x7f179c1396a0 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4157230812 0 0) 0x7f17880035d0 con 0x7f179c09f140 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f179c098eb0 con 0x7f179c09f140 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f178c0040a0 con 0x7f179c09f140 2024-09-16T06:42:59.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): dump: 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:42:59.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3225851677 0 0) 0x7f179c1326d0 con 0x7f179c1396a0 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17880035d0 con 0x7f179c1396a0 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1794001d20 con 0x7f179c1396a0 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient(hunting): dump: 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828458108 0 0) 0x7f179c098eb0 con 0x7f179c09f140 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 >> v1:172.21.15.76:6790/0 conn(0x7f179c1396a0 legacy=0x7f179c1d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 >> v1:172.21.15.189:6789/0 conn(0x7f179c099700 legacy=0x7f179c138c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: _finish_hunting 0 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 monclient: found mon.a 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c13cb20 con 0x7f179c09f140 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: _finish_auth 0 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: _check_auth_tickets 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 5 monclient: authenticate success, global_id 154263 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _renew_subs 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c1333f0 con 0x7f179c09f140 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _renew_subs 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c071590 con 0x7f179c09f140 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f178c0042c0 con 0x7f179c09f140 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:42:59.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f178c004650 con 0x7f179c09f140 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 10 monclient: dump: 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:42:59.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17997fa700 4 set_mon_vals no callback set 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f178c0048b0 con 0x7f179c09f140 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(120..120 src has 66..120) v4 ==== 6377+0+0 (unknown 2452908237 0 0) 0x7f178c043b60 con 0x7f179c09f140 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f179affd700 10 monclient: get_auth_request con 0x7f1788036360 auth_method 0 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.466+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179c132e30 con 0x7f179c09f140 2024-09-16T06:42:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.478+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f178c014670 con 0x7f179c09f140 2024-09-16T06:42:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.478+0000 7f17837fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:42:59.484 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.478+0000 7f17837fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:42:59.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.646+0000 7f17a16ea700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:42:59.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.646+0000 7f17a16ea700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:42:59.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.646+0000 7f17a16ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:42:59.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:42:59.646+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7f179c070f70 con 0x7f179c09f140 2024-09-16T06:43:00.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.250+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v121) v1 ==== 126+0+0 (unknown 3200291347 0 0) 0x7f178c012ae0 con 0x7f179c09f140 2024-09-16T06:43:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.250+0000 7f17837fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:43:00.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.250+0000 7f17837fe700 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-16T06:43:00.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.354+0000 7f17a16ea700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:43:00.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.354+0000 7f17a16ea700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:43:00.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.354+0000 7f17a16ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:00.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:00.354+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7f179c0008d0 con 0x7f179c09f140 2024-09-16T06:43:01.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.246+0000 7f17837fe700 1 -- 172.21.15.189:0/1561032644 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v122) v1 ==== 126+0+0 (unknown 1015358924 0 0) 0x7f178c014440 con 0x7f179c09f140 2024-09-16T06:43:01.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.246+0000 7f17837fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T06:43:01.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.246+0000 7f17837fe700 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-16T06:43:01.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1788036360 legacy=0x7f1788038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 10 monclient: shutdown 2024-09-16T06:43:01.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 >> v1:172.21.15.76:6789/0 conn(0x7f179c09f140 legacy=0x7f179c1ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 shutdown_connections 2024-09-16T06:43:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 >> 172.21.15.189:0/1561032644 conn(0x7f179c0a03e0 msgr2=0x7f179c13ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 shutdown_connections 2024-09-16T06:43:01.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.254+0000 7f17a16ea700 1 -- 172.21.15.189:0/1561032644 wait complete. 2024-09-16T06:43:01.261 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 10 size to 3 2024-09-16T06:43:01.273 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T06:43:01.273 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:43:01.273 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T06:43:01.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 10 monclient: get_monmap_and_config 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 10 monclient: build_initial_monmap 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 10 monclient: monmap: 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:01.387420+0000 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:01.387420+0000 2024-09-16T06:43:01.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:01.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:01.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 1 Processor -- start 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 1 -- start start 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.382+0000 7f1a242f3700 10 monclient: init 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a1c06a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:01.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a1c09d010 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a1c06ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0a16e0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c06c100 con 0x7f1a1c06a340 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c09cb40 con 0x7f1a1c09d010 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient(hunting): authenticate will time out at 1133.900531s 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2208f700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1a1c06a340 legacy=0x7f1a1c06a720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2188e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1a1c09d010 0x7f1a1c13a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57726/0 (socket says 172.21.15.189:57726) 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a22890700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1a1c06ae30 0x7f1a1c13df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42384/0 (socket says 172.21.15.189:42384) 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2188e700 1 -- 172.21.15.189:0/4101427803 learned_addr learned my addr 172.21.15.189:0/4101427803 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36481829 0 0) 0x7f1a1c0a16e0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): my global_id is 154266 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a040035d0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914249678 0 0) 0x7f1a1c09cb40 con 0x7f1a1c09d010 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient(hunting): my global_id is 125126 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c0a16e0 con 0x7f1a1c09d010 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589822233 0 0) 0x7f1a040035d0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 >> v1:172.21.15.76:6790/0 conn(0x7f1a1c09d010 legacy=0x7f1a1c13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 >> v1:172.21.15.189:6789/0 conn(0x7f1a1c06a340 legacy=0x7f1a1c06a720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:01.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: _finish_hunting 0 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 monclient: found mon.noname-a 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c06b260 con 0x7f1a1c06ae30 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: _finish_auth 0 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: _check_auth_tickets 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a140018c0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 5 monclient: authenticate success, global_id 154266 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:01.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: dump: 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a14001ae0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:01.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 1 -- 172.21.15.189:0/4101427803 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a14001eb0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 4 set_mon_vals no callback set 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a2108d700 10 monclient: dump: 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.395 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.396 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 10 monclient: shutdown 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- 172.21.15.189:0/4101427803 >> v1:172.21.15.76:6789/0 conn(0x7f1a1c06ae30 legacy=0x7f1a1c13df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- 172.21.15.189:0/4101427803 shutdown_connections 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- 172.21.15.189:0/4101427803 >> 172.21.15.189:0/4101427803 conn(0x7f1a1c0a2180 msgr2=0x7f1a1c0a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- 172.21.15.189:0/4101427803 shutdown_connections 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.386+0000 7f1a242f3700 1 -- 172.21.15.189:0/4101427803 wait complete. 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: build_initial_monmap 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: monmap: 2024-09-16T06:43:01.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 Processor -- start 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- start start 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: init 2024-09-16T06:43:01.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a1c06a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a1c06ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a1c09d010 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0a0ca0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c09c690 con 0x7f1a1c09d010 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c09f380 con 0x7f1a1c06a340 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient(hunting): authenticate will time out at 1133.905993s 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a22890700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1a1c09d010 legacy=0x7f1a1c1d1bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a2188e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1a1c06ae30 0x7f1a1c13ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42388/0 (socket says 172.21.15.189:42388) 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a2188e700 1 -- 172.21.15.189:0/2697765626 learned_addr learned my addr 172.21.15.189:0/2697765626 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a2208f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1a1c06a340 0x7f1a1c1c9870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57734/0 (socket says 172.21.15.189:57734) 2024-09-16T06:43:01.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326864335 0 0) 0x7f1a100011d0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): my global_id is 154269 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a080035d0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405270067 0 0) 0x7f1a1c09f380 con 0x7f1a1c06a340 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): my global_id is 125129 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a100011d0 con 0x7f1a1c06a340 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2185592779 0 0) 0x7f1a080035d0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a1c09f380 con 0x7f1a1c06ae30 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a10001c90 con 0x7f1a1c06ae30 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): dump: 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.404 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2255443359 0 0) 0x7f1a100011d0 con 0x7f1a1c06a340 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a080035d0 con 0x7f1a1c06a340 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a0c004170 con 0x7f1a1c06a340 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient(hunting): dump: 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 457337973 0 0) 0x7f1a1c09f380 con 0x7f1a1c06ae30 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 >> v1:172.21.15.76:6790/0 conn(0x7f1a1c06a340 legacy=0x7f1a1c1c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 >> v1:172.21.15.189:6789/0 conn(0x7f1a1c09d010 legacy=0x7f1a1c1d1bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 monclient: found mon.a 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c13b540 con 0x7f1a1c06ae30 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient: _finish_auth 0 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a1affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 5 monclient: authenticate success, global_id 154269 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _renew_subs 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c13aca0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _renew_subs 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.390+0000 7f1a242f3700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c13f1a0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a10004890 con 0x7f1a1c06ae30 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a10004ba0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a2088c700 4 set_mon_vals no callback set 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 10 monclient: dump: 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a242f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1a1000f690 con 0x7f1a1c06ae30 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a242f3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a242f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a242f3700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc005100 con 0x7f1a1c06ae30 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(122..122 src has 66..122) v4 ==== 6377+0+0 (unknown 2387294143 0 0) 0x7f1a10043de0 con 0x7f1a1c06ae30 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.394+0000 7f1a2208f700 10 monclient: get_auth_request con 0x7f1a080362f0 auth_method 0 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.406+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1a10010c80 con 0x7f1a1c06ae30 2024-09-16T06:43:01.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.406+0000 7f1a1affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.406+0000 7f1a1affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:01.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.582+0000 7f1a242f3700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:01.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.582+0000 7f1a242f3700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:01.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.582+0000 7f1a242f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.582+0000 7f1a242f3700 1 -- 172.21.15.189:0/2697765626 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f19fc005930 con 0x7f1a1c06ae30 2024-09-16T06:43:01.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.586+0000 7f1a1affd700 1 -- 172.21.15.189:0/2697765626 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v122) v1 ==== 160+0+0 (unknown 4173571234 0 0) 0x7f1a10012d00 con 0x7f1a1c06ae30 2024-09-16T06:43:01.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.586+0000 7f1a1affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:01.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.586+0000 7f1a1affd700 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-16T06:43:01.591 INFO:tasks.workunit.client.0.smithi189.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-16T06:43:01.596 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:43:01.596 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: get_monmap_and_config 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: build_initial_monmap 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: monmap: 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:01.705482+0000 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:01.705482+0000 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:01.705 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.706 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:01.706 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:01.706 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:01.706 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 1 Processor -- start 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 1 -- start start 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: init 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-c con 0x7fe16809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:01.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-b con 0x7fe168099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-a con 0x7fe1681396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16809f520 con 0x7fe1681396a0 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16809bc10 con 0x7fe16809f140 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16809b810 con 0x7fe168099700 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1677fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe16809f140 legacy=0x7fe16809a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 10 monclient(hunting): authenticate will time out at 1134.218390s 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe167fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe1681396a0 0x7fe16813ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42390/0 (socket says 172.21.15.189:42390) 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe167fff700 1 -- 172.21.15.189:0/3894683934 learned_addr learned my addr 172.21.15.189:0/3894683934 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462232232 0 0) 0x7fe16809f520 con 0x7fe1681396a0 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): my global_id is 154272 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1540035d0 con 0x7fe1681396a0 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106494481 0 0) 0x7fe16809b810 con 0x7fe168099700 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient(hunting): my global_id is 125132 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16809f520 con 0x7fe168099700 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033862211 0 0) 0x7fe1540035d0 con 0x7fe1681396a0 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 >> v1:172.21.15.76:6790/0 conn(0x7fe168099700 legacy=0x7fe16809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 >> v1:172.21.15.189:6789/0 conn(0x7fe16809f140 legacy=0x7fe16809a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:01.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 monclient: found mon.noname-a 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16809ceb0 con 0x7fe1681396a0 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe160001990 con 0x7fe1681396a0 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe1667fc700 10 monclient: dump: 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.712 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 5 monclient: authenticate success, global_id 154272 2024-09-16T06:43:01.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.702+0000 7fe16db36700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe160001c70 con 0x7fe1681396a0 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 1 -- 172.21.15.189:0/3894683934 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe160002000 con 0x7fe1681396a0 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1667fc700 10 monclient: dump: 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.714 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 4 set_mon_vals no callback set 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: shutdown 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- 172.21.15.189:0/3894683934 >> v1:172.21.15.76:6789/0 conn(0x7fe1681396a0 legacy=0x7fe16813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- 172.21.15.189:0/3894683934 shutdown_connections 2024-09-16T06:43:01.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- 172.21.15.189:0/3894683934 >> 172.21.15.189:0/3894683934 conn(0x7fe1680a03e0 msgr2=0x7fe1680a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- 172.21.15.189:0/3894683934 shutdown_connections 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- 172.21.15.189:0/3894683934 wait complete. 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: build_initial_monmap 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: monmap: 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:01.716 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:01.717 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 Processor -- start 2024-09-16T06:43:01.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- start start 2024-09-16T06:43:01.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: init 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-b con 0x7fe168099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-a con 0x7fe16809f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient(hunting): picked mon.noname-c con 0x7fe1681396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:01.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe168092910 con 0x7fe16809f140 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1680914d0 con 0x7fe168099700 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe168071480 con 0x7fe1681396a0 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe16db36700 10 monclient(hunting): authenticate will time out at 1134.222137s 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe1677fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe168099700 legacy=0x7fe168096140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe167fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe1681396a0 0x7fe1681cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57748/0 (socket says 172.21.15.189:57748) 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe167fff700 1 -- 172.21.15.189:0/3143507378 learned_addr learned my addr 172.21.15.189:0/3143507378 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661622070 0 0) 0x7fe168071480 con 0x7fe1681396a0 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): my global_id is 125135 2024-09-16T06:43:01.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1480035d0 con 0x7fe1681396a0 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175792882 0 0) 0x7fe168092910 con 0x7fe16809f140 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): my global_id is 154275 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe168071480 con 0x7fe16809f140 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277391909 0 0) 0x7fe1480035d0 con 0x7fe1681396a0 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe168092910 con 0x7fe1681396a0 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe160001950 con 0x7fe1681396a0 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:01.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946160495 0 0) 0x7fe168071480 con 0x7fe16809f140 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1480035d0 con 0x7fe16809f140 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1580018b0 con 0x7fe16809f140 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.706+0000 7fe15ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.723 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1755838297 0 0) 0x7fe168092910 con 0x7fe1681396a0 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 >> v1:172.21.15.189:6789/0 conn(0x7fe168099700 legacy=0x7fe168096140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 >> v1:172.21.15.76:6789/0 conn(0x7fe16809f140 legacy=0x7fe1681cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 monclient: found mon.c 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680a2ee0 con 0x7fe1681396a0 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: _finish_auth 0 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:01.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 5 monclient: authenticate success, global_id 125135 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _renew_subs 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe168092bf0 con 0x7fe1681396a0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _renew_subs 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe168090890 con 0x7fe1681396a0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1600013e0 con 0x7fe1681396a0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1600051c0 con 0x7fe1681396a0 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe165ffb700 4 set_mon_vals no callback set 2024-09-16T06:43:01.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 10 monclient: dump: 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:01.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c005100 con 0x7fe1681396a0 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe1600053a0 con 0x7fe1681396a0 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(122..122 src has 66..122) v4 ==== 6377+0+0 (unknown 2387294143 0 0) 0x7fe160044110 con 0x7fe1681396a0 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.710+0000 7fe1677fe700 10 monclient: get_auth_request con 0x7fe1480362c0 auth_method 0 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.718+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe160014c10 con 0x7fe1681396a0 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.718+0000 7fe15ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:01.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.718+0000 7fe15ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:01.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.898+0000 7fe16db36700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:01.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.898+0000 7fe16db36700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:01.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.898+0000 7fe16db36700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:01.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:01.898+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7fe14c005930 con 0x7fe1681396a0 2024-09-16T06:43:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.266+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v123) v1 ==== 109+0+0 (unknown 308570655 0 0) 0x7fe1600132b0 con 0x7fe1681396a0 2024-09-16T06:43:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.266+0000 7fe15ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:02.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.266+0000 7fe15ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.378+0000 7fe16db36700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.378+0000 7fe16db36700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.378+0000 7fe16db36700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:02.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.378+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7fe14c0053b0 con 0x7fe1681396a0 2024-09-16T06:43:02.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.382+0000 7fe15ffff700 1 -- 172.21.15.189:0/3143507378 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v123) v1 ==== 153+0+0 (unknown 3682873716 0 0) 0x7fe160014a30 con 0x7fe1681396a0 2024-09-16T06:43:02.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.382+0000 7fe15ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T06:43:02.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.382+0000 7fe15ffff700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-16T06:43:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe1480362c0 legacy=0x7fe148038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 10 monclient: shutdown 2024-09-16T06:43:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 >> v1:172.21.15.76:6790/0 conn(0x7fe1681396a0 legacy=0x7fe1681cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:02.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 shutdown_connections 2024-09-16T06:43:02.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 >> 172.21.15.189:0/3143507378 conn(0x7fe1680a03e0 msgr2=0x7fe1680a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:02.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 shutdown_connections 2024-09-16T06:43:02.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.386+0000 7fe16db36700 1 -- 172.21.15.189:0/3143507378 wait complete. 2024-09-16T06:43:02.390 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'fooset' already created 2024-09-16T06:43:02.404 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create barset positional 2024-09-16T06:43:02.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: get_monmap_and_config 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: build_initial_monmap 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: monmap: 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:02.514366+0000 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:02.514366+0000 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:02.514 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 1 Processor -- start 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 1 -- start start 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: init 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-a con 0x7f4e4009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-b con 0x7f4e4009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-c con 0x7f4e40138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4009cca0 con 0x7f4e4009bb00 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4009f750 con 0x7f4e40138da0 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e40097620 con 0x7f4e4009c5f0 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:02.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e46194700 10 monclient(hunting): authenticate will time out at 1135.027588s 2024-09-16T06:43:02.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.510+0000 7f4e3ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4e40138da0 legacy=0x7f4e4013b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:02.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4e4009bb00 0x7f4e4009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42396/0 (socket says 172.21.15.189:42396) 2024-09-16T06:43:02.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4e4009c5f0 0x7f4e40137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57752/0 (socket says 172.21.15.189:57752) 2024-09-16T06:43:02.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3f7fe700 1 -- 172.21.15.189:0/1973312299 learned_addr learned my addr 172.21.15.189:0/1973312299 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159309435 0 0) 0x7f4e4009cca0 con 0x7f4e4009bb00 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): my global_id is 154278 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c0035d0 con 0x7f4e4009bb00 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161920923 0 0) 0x7f4e40097620 con 0x7f4e4009c5f0 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient(hunting): my global_id is 125138 2024-09-16T06:43:02.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e4009cca0 con 0x7f4e4009c5f0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283537734 0 0) 0x7f4e4009cca0 con 0x7f4e4009c5f0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 >> v1:172.21.15.189:6789/0 conn(0x7f4e40138da0 legacy=0x7f4e4013b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 >> v1:172.21.15.76:6789/0 conn(0x7f4e4009bb00 legacy=0x7f4e4009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e40097900 con 0x7f4e4009c5f0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 5 monclient: authenticate success, global_id 125138 2024-09-16T06:43:02.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4e30001520 con 0x7f4e4009c5f0 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: dump: 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:02.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e300017c0 con 0x7f4e4009c5f0 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 4 set_mon_vals no callback set 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 1 -- 172.21.15.189:0/1973312299 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4e30001b90 con 0x7f4e4009c5f0 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e3e7fc700 10 monclient: dump: 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 10 monclient: shutdown 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 -- 172.21.15.189:0/1973312299 >> v1:172.21.15.76:6790/0 conn(0x7f4e4009c5f0 legacy=0x7f4e40137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:02.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 -- 172.21.15.189:0/1973312299 shutdown_connections 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 -- 172.21.15.189:0/1973312299 >> 172.21.15.189:0/1973312299 conn(0x7f4e400a03e0 msgr2=0x7f4e400a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 -- 172.21.15.189:0/1973312299 shutdown_connections 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 -- 172.21.15.189:0/1973312299 wait complete. 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 10 monclient: build_initial_monmap 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 10 monclient: monmap: 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:02.524 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:02.525 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.514+0000 7f4e46194700 1 Processor -- start 2024-09-16T06:43:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- start start 2024-09-16T06:43:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: init 2024-09-16T06:43:02.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-c con 0x7f4e4009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-b con 0x7f4e4009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient(hunting): picked mon.noname-a con 0x7f4e40138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e40090c10 con 0x7f4e40138da0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e400a2c70 con 0x7f4e4009c5f0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e40099380 con 0x7f4e4009bb00 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient(hunting): authenticate will time out at 1135.032931s 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e3effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4e4009c5f0 legacy=0x7f4e4008d6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e3ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4e40138da0 0x7f4e401cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42410/0 (socket says 172.21.15.189:42410) 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e3ffff700 1 -- 172.21.15.189:0/355748289 learned_addr learned my addr 172.21.15.189:0/355748289 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:02.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079048607 0 0) 0x7f4e40090c10 con 0x7f4e40138da0 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): my global_id is 154281 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e200035d0 con 0x7f4e40138da0 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668220807 0 0) 0x7f4e40099380 con 0x7f4e4009bb00 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): my global_id is 125141 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e40090c10 con 0x7f4e4009bb00 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300828733 0 0) 0x7f4e200035d0 con 0x7f4e40138da0 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e40099380 con 0x7f4e40138da0 2024-09-16T06:43:02.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4e38004430 con 0x7f4e40138da0 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): dump: 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:02.532 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3822901663 0 0) 0x7f4e40090c10 con 0x7f4e4009bb00 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e200035d0 con 0x7f4e4009bb00 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4e28001170 con 0x7f4e4009bb00 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient(hunting): dump: 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 593569179 0 0) 0x7f4e40099380 con 0x7f4e40138da0 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 >> v1:172.21.15.76:6790/0 conn(0x7f4e4009bb00 legacy=0x7f4e4008cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 >> v1:172.21.15.189:6789/0 conn(0x7f4e4009c5f0 legacy=0x7f4e4008d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 monclient: found mon.a 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4009a6e0 con 0x7f4e40138da0 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: _finish_auth 0 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:02.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 5 monclient: authenticate success, global_id 154281 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _renew_subs 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4008e8b0 con 0x7f4e40138da0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _renew_subs 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4008ddc0 con 0x7f4e40138da0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e38004710 con 0x7f4e40138da0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4e38004aa0 con 0x7f4e40138da0 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e37fff700 10 monclient: dump: 2024-09-16T06:43:02.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.518+0000 7f4e3dffb700 4 set_mon_vals no callback set 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e46194700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e46194700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e46194700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:02.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4e38004c80 con 0x7f4e40138da0 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e24005100 con 0x7f4e40138da0 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(123..123 src has 66..123) v4 ==== 6709+0+0 (unknown 3464448001 0 0) 0x7f4e38044120 con 0x7f4e40138da0 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.522+0000 7f4e3f7fe700 10 monclient: get_auth_request con 0x7f4e200362e0 auth_method 0 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.534+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4e38014ae0 con 0x7f4e40138da0 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.534+0000 7f4e37fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:02.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.534+0000 7f4e37fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:02.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.714+0000 7f4e46194700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:02.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.714+0000 7f4e46194700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:02.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.714+0000 7f4e46194700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:02.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:02.714+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f4e24005930 con 0x7f4e40138da0 2024-09-16T06:43:03.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.274+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v124) v1 ==== 115+0+0 (unknown 62422334 0 0) 0x7f4e38012f50 con 0x7f4e40138da0 2024-09-16T06:43:03.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.274+0000 7f4e37fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:03.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.274+0000 7f4e37fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.394+0000 7f4e46194700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.394+0000 7f4e46194700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.394+0000 7f4e46194700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:03.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.394+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f4e240053b0 con 0x7f4e40138da0 2024-09-16T06:43:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.398+0000 7f4e37fff700 1 -- 172.21.15.189:0/355748289 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v124) v1 ==== 159+0+0 (unknown 175948579 0 0) 0x7f4e380146d0 con 0x7f4e40138da0 2024-09-16T06:43:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.398+0000 7f4e37fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T06:43:03.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.398+0000 7f4e37fff700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-16T06:43:03.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4e200362e0 legacy=0x7f4e200387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 10 monclient: shutdown 2024-09-16T06:43:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 >> v1:172.21.15.76:6789/0 conn(0x7f4e40138da0 legacy=0x7f4e401cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 shutdown_connections 2024-09-16T06:43:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 >> 172.21.15.189:0/355748289 conn(0x7f4e400a03e0 msgr2=0x7f4e4009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:03.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 shutdown_connections 2024-09-16T06:43:03.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.402+0000 7f4e46194700 1 -- 172.21.15.189:0/355748289 wait complete. 2024-09-16T06:43:03.411 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'barset' already created 2024-09-16T06:43:03.415 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:03.416 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fooset 2024-09-16T06:43:03.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: get_monmap_and_config 2024-09-16T06:43:03.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: build_initial_monmap 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: monmap: 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:03.522548+0000 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:03.522548+0000 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:03.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:03.524 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 1 Processor -- start 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 1 -- start start 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: init 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-c con 0x7fa2340a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-b con 0x7fa23409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-a con 0x7fa2340a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23409a040 con 0x7fa2340a3ca0 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340a33d0 con 0x7fa2340a2350 2024-09-16T06:43:03.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234099d60 con 0x7fa23409aa20 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa23a52a700 10 monclient(hunting): authenticate will time out at 1136.035911s 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.518+0000 7fa233fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa2340a2350 legacy=0x7fa2340a2730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa2337fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa23409aa20 0x7fa23413a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57764/0 (socket says 172.21.15.189:57764) 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa238ac7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa2340a3ca0 0x7fa23413dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42414/0 (socket says 172.21.15.189:42414) 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa2337fe700 1 -- 172.21.15.189:0/3919187460 learned_addr learned my addr 172.21.15.189:0/3919187460 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404272858 0 0) 0x7fa234099d60 con 0x7fa23409aa20 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): my global_id is 125144 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c0035d0 con 0x7fa23409aa20 2024-09-16T06:43:03.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642597651 0 0) 0x7fa23409a040 con 0x7fa2340a3ca0 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient(hunting): my global_id is 154284 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa234099d60 con 0x7fa2340a3ca0 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582970356 0 0) 0x7fa21c0035d0 con 0x7fa23409aa20 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 >> v1:172.21.15.189:6789/0 conn(0x7fa2340a2350 legacy=0x7fa2340a2730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 >> v1:172.21.15.76:6789/0 conn(0x7fa2340a3ca0 legacy=0x7fa23413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:03.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 monclient: found mon.noname-b 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340a3920 con 0x7fa23409aa20 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa224001610 con 0x7fa23409aa20 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: dump: 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 5 monclient: authenticate success, global_id 125144 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2240018f0 con 0x7fa23409aa20 2024-09-16T06:43:03.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 4 set_mon_vals no callback set 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 1 -- 172.21.15.189:0/3919187460 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa224001c80 con 0x7fa23409aa20 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa232ffd700 10 monclient: dump: 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.534 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 10 monclient: shutdown 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 1 -- 172.21.15.189:0/3919187460 >> v1:172.21.15.76:6790/0 conn(0x7fa23409aa20 legacy=0x7fa23413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 1 -- 172.21.15.189:0/3919187460 shutdown_connections 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 1 -- 172.21.15.189:0/3919187460 >> 172.21.15.189:0/3919187460 conn(0x7fa2340a7d70 msgr2=0x7fa2340a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 1 -- 172.21.15.189:0/3919187460 shutdown_connections 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.522+0000 7fa23a52a700 1 -- 172.21.15.189:0/3919187460 wait complete. 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient: build_initial_monmap 2024-09-16T06:43:03.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient: monmap: 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:03.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 1 Processor -- start 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 1 -- start start 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient: init 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-b con 0x7fa23409aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-a con 0x7fa2340a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient(hunting): picked mon.noname-c con 0x7fa2340a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234124300 con 0x7fa2340a2350 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa233fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa23409aa20 legacy=0x7fa2341356f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23406bed0 con 0x7fa23409aa20 2024-09-16T06:43:03.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23409b240 con 0x7fa2340a3ca0 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa23a52a700 10 monclient(hunting): authenticate will time out at 1136.043109s 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa2337fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa2340a2350 0x7fa2341cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42424/0 (socket says 172.21.15.189:42424) 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa238ac7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa2340a3ca0 0x7fa2341d23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57768/0 (socket says 172.21.15.189:57768) 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa2337fe700 1 -- 172.21.15.189:0/310043722 learned_addr learned my addr 172.21.15.189:0/310043722 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586645736 0 0) 0x7fa234124300 con 0x7fa2340a2350 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa230ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa230ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa230ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.526+0000 7fa230ff9700 10 monclient(hunting): my global_id is 154287 2024-09-16T06:43:03.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2200035d0 con 0x7fa2340a2350 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905980203 0 0) 0x7fa23409b240 con 0x7fa2340a3ca0 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): my global_id is 125147 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa234124300 con 0x7fa2340a3ca0 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842362877 0 0) 0x7fa234124300 con 0x7fa2340a3ca0 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa23409b240 con 0x7fa2340a3ca0 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa22c001bb0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:03.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:03.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843384753 0 0) 0x7fa2200035d0 con 0x7fa2340a2350 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa234124300 con 0x7fa2340a2350 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2240015d0 con 0x7fa2340a2350 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3264642515 0 0) 0x7fa23409b240 con 0x7fa2340a3ca0 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 >> v1:172.21.15.189:6789/0 conn(0x7fa23409aa20 legacy=0x7fa2341356f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 >> v1:172.21.15.76:6789/0 conn(0x7fa2340a2350 legacy=0x7fa2341cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 monclient: found mon.c 2024-09-16T06:43:03.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23409ae80 con 0x7fa2340a3ca0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: _finish_auth 0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 5 monclient: authenticate success, global_id 125147 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _renew_subs 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa234124a70 con 0x7fa2340a3ca0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _renew_subs 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa234123d80 con 0x7fa2340a3ca0 2024-09-16T06:43:03.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa22c001fc0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa22c005060 con 0x7fa2340a3ca0 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa2327fc700 4 set_mon_vals no callback set 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 10 monclient: dump: 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.548 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2340a73b0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa22c00fad0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(124..124 src has 66..124) v4 ==== 7321+0+0 (unknown 76663028 0 0) 0x7fa22c0444e0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.530+0000 7fa233fff700 10 monclient: get_auth_request con 0x7fa2200362c0 auth_method 0 2024-09-16T06:43:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.542+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa22c014a10 con 0x7fa2340a3ca0 2024-09-16T06:43:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.542+0000 7fa230ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.542+0000 7fa230ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:03.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa23a52a700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa23a52a700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa23a52a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fa234125840 con 0x7fa2340a3ca0 2024-09-16T06:43:03.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (unknown 3707170821 0 3003798303) 0x7fa22c0130b0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa230ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.718+0000 7fa230ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.822+0000 7fa23a52a700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.822+0000 7fa23a52a700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.822+0000 7fa23a52a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.827 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.822+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fa2340008d0 con 0x7fa2340a3ca0 2024-09-16T06:43:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa230ff9700 1 -- 172.21.15.189:0/310043722 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (unknown 3707170821 0 3003798303) 0x7fa22c014830 con 0x7fa2340a3ca0 2024-09-16T06:43:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa230ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:03.828 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa230ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa2200362c0 legacy=0x7fa220038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 10 monclient: shutdown 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 >> v1:172.21.15.76:6790/0 conn(0x7fa2340a3ca0 legacy=0x7fa2341d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 shutdown_connections 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 >> 172.21.15.189:0/310043722 conn(0x7fa2340a7d70 msgr2=0x7fa23409b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 shutdown_connections 2024-09-16T06:43:03.831 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.826+0000 7fa23a52a700 1 -- 172.21.15.189:0/310043722 wait complete. 2024-09-16T06:43:03.841 INFO:tasks.workunit.client.0.smithi189.stdout:fooset 2024-09-16T06:43:03.842 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:03.842 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep barset 2024-09-16T06:43:03.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: get_monmap_and_config 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: build_initial_monmap 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: monmap: 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:03.948769+0000 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:03.948769+0000 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:03.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:03.950 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 1 Processor -- start 2024-09-16T06:43:03.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 1 -- start start 2024-09-16T06:43:03.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: init 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6adc09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6adc09f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6adc139a50 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc09ab80 con 0x7f6adc09f690 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc09a690 con 0x7f6adc09eba0 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc072230 con 0x7f6adc139a50 2024-09-16T06:43:03.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient(hunting): authenticate will time out at 1136.461626s 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adaffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6adc09eba0 legacy=0x7f6adc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6adc139a50 0x7f6adc13be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57780/0 (socket says 172.21.15.189:57780) 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ada7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6adc09f690 0x7f6adc138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42440/0 (socket says 172.21.15.189:42440) 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adb7fe700 1 -- 172.21.15.189:0/2305368428 learned_addr learned my addr 172.21.15.189:0/2305368428 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066775111 0 0) 0x7f6adc072230 con 0x7f6adc139a50 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): my global_id is 125150 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac40035d0 con 0x7f6adc139a50 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303238606 0 0) 0x7f6adc09ab80 con 0x7f6adc09f690 2024-09-16T06:43:03.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient(hunting): my global_id is 154290 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6adc072230 con 0x7f6adc09f690 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381351811 0 0) 0x7f6ac40035d0 con 0x7f6adc139a50 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 >> v1:172.21.15.189:6789/0 conn(0x7f6adc09eba0 legacy=0x7f6adc09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 >> v1:172.21.15.76:6789/0 conn(0x7f6adc09f690 legacy=0x7f6adc138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 monclient: found mon.noname-b 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:03.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc09be90 con 0x7f6adc139a50 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 5 monclient: authenticate success, global_id 125150 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ad40018c0 con 0x7f6adc139a50 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: dump: 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.961 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ad4001ae0 con 0x7f6adc139a50 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6adbfff700 4 set_mon_vals no callback set 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 1 -- 172.21.15.189:0/2305368428 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ad4001e70 con 0x7f6adc139a50 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.946+0000 7f6ad9ffb700 10 monclient: dump: 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: shutdown 2024-09-16T06:43:03.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- 172.21.15.189:0/2305368428 >> v1:172.21.15.76:6790/0 conn(0x7f6adc139a50 legacy=0x7f6adc13be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- 172.21.15.189:0/2305368428 shutdown_connections 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- 172.21.15.189:0/2305368428 >> 172.21.15.189:0/2305368428 conn(0x7f6adc09fe70 msgr2=0x7f6adc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- 172.21.15.189:0/2305368428 shutdown_connections 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- 172.21.15.189:0/2305368428 wait complete. 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: build_initial_monmap 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: monmap: 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:03.964 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 Processor -- start 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- start start 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: init 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6adc09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6adc09f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6adc139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:03.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc098f70 con 0x7f6adc139a50 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc093920 con 0x7f6adc09f690 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc093c00 con 0x7f6adc09eba0 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adbfff700 10 monclient(hunting): authenticate will time out at 1136.466423s 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ada7fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6adc09f690 legacy=0x7f6adc090db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adaffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6adc09eba0 0x7f6adc0906a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57794/0 (socket says 172.21.15.189:57794) 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adb7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6adc139a50 0x7f6adc1d1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42454/0 (socket says 172.21.15.189:42454) 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6adaffd700 1 -- 172.21.15.189:0/1085636136 learned_addr learned my addr 172.21.15.189:0/1085636136 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409324814 0 0) 0x7f6adc093c00 con 0x7f6adc09eba0 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): my global_id is 125153 2024-09-16T06:43:03.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac80035d0 con 0x7f6adc09eba0 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871491668 0 0) 0x7f6adc098f70 con 0x7f6adc139a50 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 10 monclient(hunting): my global_id is 154293 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6adc093c00 con 0x7f6adc139a50 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.950+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3770548954 0 0) 0x7f6adc093c00 con 0x7f6adc139a50 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6adc098f70 con 0x7f6adc139a50 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ad40017c0 con 0x7f6adc139a50 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): dump: 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.967 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274580670 0 0) 0x7f6ac80035d0 con 0x7f6adc09eba0 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6adc093c00 con 0x7f6adc09eba0 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ad0004180 con 0x7f6adc09eba0 2024-09-16T06:43:03.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient(hunting): dump: 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190062062 0 0) 0x7f6adc098f70 con 0x7f6adc139a50 2024-09-16T06:43:03.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 >> v1:172.21.15.76:6790/0 conn(0x7f6adc09eba0 legacy=0x7f6adc0906a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 >> v1:172.21.15.189:6789/0 conn(0x7f6adc09f690 legacy=0x7f6adc090db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 monclient: found mon.a 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc06b950 con 0x7f6adc139a50 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: _finish_auth 0 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 5 monclient: authenticate success, global_id 154293 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _renew_subs 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc071530 con 0x7f6adc139a50 2024-09-16T06:43:03.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _renew_subs 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc093640 con 0x7f6adc139a50 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ad4001480 con 0x7f6adc139a50 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ad4005060 con 0x7f6adc139a50 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ad97fa700 4 set_mon_vals no callback set 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 10 monclient: dump: 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc005100 con 0x7f6adc139a50 2024-09-16T06:43:03.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6ad40051c0 con 0x7f6adc139a50 2024-09-16T06:43:03.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(124..124 src has 66..124) v4 ==== 7321+0+0 (unknown 76663028 0 0) 0x7f6ad40444f0 con 0x7f6adc139a50 2024-09-16T06:43:03.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.954+0000 7f6adaffd700 10 monclient: get_auth_request con 0x7f6ac8036340 auth_method 0 2024-09-16T06:43:03.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.962+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6ad4014ae0 con 0x7f6adc139a50 2024-09-16T06:43:03.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.962+0000 7f6ac3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:03.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:03.962+0000 7f6ac3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.142+0000 7f6adbfff700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.142+0000 7f6adbfff700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.142+0000 7f6adbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:04.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.142+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f6abc005930 con 0x7f6adc139a50 2024-09-16T06:43:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.146+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (unknown 3707170821 0 3003798303) 0x7f6ad4013180 con 0x7f6adc139a50 2024-09-16T06:43:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.146+0000 7f6ac3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.146+0000 7f6ac3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:04.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6adbfff700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6adbfff700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6adbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:04.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f6abc005420 con 0x7f6adc139a50 2024-09-16T06:43:04.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6ac3fff700 1 -- 172.21.15.189:0/1085636136 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+14 (unknown 3707170821 0 3003798303) 0x7f6ad4014900 con 0x7f6adc139a50 2024-09-16T06:43:04.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6ac3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:04.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.246+0000 7f6ac3fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:04.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6ac8036340 legacy=0x7f6ac8038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 10 monclient: shutdown 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 >> v1:172.21.15.76:6789/0 conn(0x7f6adc139a50 legacy=0x7f6adc1d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 shutdown_connections 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 >> 172.21.15.189:0/1085636136 conn(0x7f6adc09fe70 msgr2=0x7f6adc0a2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 shutdown_connections 2024-09-16T06:43:04.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.250+0000 7f6adbfff700 1 -- 172.21.15.189:0/1085636136 wait complete. 2024-09-16T06:43:04.262 INFO:tasks.workunit.client.0.smithi189.stdout:barset 2024-09-16T06:43:04.263 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set dump 2024-09-16T06:43:04.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: get_monmap_and_config 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: build_initial_monmap 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: monmap: 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:04.365441+0000 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:04.365441+0000 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:04.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:04.366 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 1 Processor -- start 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 1 -- start start 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: init 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-a con 0x7fd28c09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:04.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-c con 0x7fd28c099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-b con 0x7fd28c1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c09f520 con 0x7fd28c09f140 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c09bc10 con 0x7fd28c099700 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c09b810 con 0x7fd28c1396a0 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 10 monclient(hunting): authenticate will time out at 1136.878358s 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28bfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd28c099700 legacy=0x7fd28c09b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd29116d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd28c1396a0 0x7fd28c13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57810/0 (socket says 172.21.15.189:57810) 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd29116d700 1 -- 172.21.15.189:0/2633124809 learned_addr learned my addr 172.21.15.189:0/2633124809 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:04.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401644171 0 0) 0x7fd28c09b810 con 0x7fd28c1396a0 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): my global_id is 125156 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2740035d0 con 0x7fd28c1396a0 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548962746 0 0) 0x7fd28c09f520 con 0x7fd28c09f140 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient(hunting): my global_id is 154296 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28c09b810 con 0x7fd28c09f140 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3321570666 0 0) 0x7fd2740035d0 con 0x7fd28c1396a0 2024-09-16T06:43:04.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 >> v1:172.21.15.189:6789/0 conn(0x7fd28c099700 legacy=0x7fd28c09b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 >> v1:172.21.15.76:6789/0 conn(0x7fd28c09f140 legacy=0x7fd28c09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c09ceb0 con 0x7fd28c1396a0 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 5 monclient: authenticate success, global_id 125156 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:04.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2840019b0 con 0x7fd28c1396a0 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd28b7fe700 10 monclient: dump: 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.362+0000 7fd292bd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd284001c90 con 0x7fd28c1396a0 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 4 set_mon_vals no callback set 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 1 -- 172.21.15.189:0/2633124809 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd284002020 con 0x7fd28c1396a0 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd28b7fe700 10 monclient: dump: 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: shutdown 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- 172.21.15.189:0/2633124809 >> v1:172.21.15.76:6790/0 conn(0x7fd28c1396a0 legacy=0x7fd28c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- 172.21.15.189:0/2633124809 shutdown_connections 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- 172.21.15.189:0/2633124809 >> 172.21.15.189:0/2633124809 conn(0x7fd28c0a03e0 msgr2=0x7fd28c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- 172.21.15.189:0/2633124809 shutdown_connections 2024-09-16T06:43:04.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- 172.21.15.189:0/2633124809 wait complete. 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: build_initial_monmap 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: monmap: 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 Processor -- start 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- start start 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: init 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-b con 0x7fd28c099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-a con 0x7fd28c09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient(hunting): picked mon.noname-c con 0x7fd28c1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c092100 con 0x7fd28c09f140 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0918f0 con 0x7fd28c099700 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c1c7e00 con 0x7fd28c1396a0 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient(hunting): authenticate will time out at 1136.882171s 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd29096c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd28c099700 legacy=0x7fd28c1c9fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:04.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd29116d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd28c1396a0 0x7fd28c1d23b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57826/0 (socket says 172.21.15.189:57826) 2024-09-16T06:43:04.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd29116d700 1 -- 172.21.15.189:0/2374485174 learned_addr learned my addr 172.21.15.189:0/2374485174 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996524108 0 0) 0x7fd28c1c7e00 con 0x7fd28c1396a0 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): my global_id is 125159 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2780035d0 con 0x7fd28c1396a0 2024-09-16T06:43:04.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101446344 0 0) 0x7fd28c092100 con 0x7fd28c09f140 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): my global_id is 154299 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28c1c7e00 con 0x7fd28c09f140 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 634837901 0 0) 0x7fd2780035d0 con 0x7fd28c1396a0 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd28c092100 con 0x7fd28c1396a0 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd284001970 con 0x7fd28c1396a0 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): dump: 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484436323 0 0) 0x7fd28c1c7e00 con 0x7fd28c09f140 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2780035d0 con 0x7fd28c09f140 2024-09-16T06:43:04.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd27c004170 con 0x7fd28c09f140 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient(hunting): dump: 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 352913630 0 0) 0x7fd28c092100 con 0x7fd28c1396a0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 >> v1:172.21.15.189:6789/0 conn(0x7fd28c099700 legacy=0x7fd28c1c9fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 >> v1:172.21.15.76:6789/0 conn(0x7fd28c09f140 legacy=0x7fd28c1cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 monclient: found mon.c 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c091e20 con 0x7fd28c1396a0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd2897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 5 monclient: authenticate success, global_id 125159 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: _renew_subs 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.366+0000 7fd292bd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c091b40 con 0x7fd28c1396a0 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 10 monclient: _renew_subs 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c0993b0 con 0x7fd28c1396a0 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd284001400 con 0x7fd28c1396a0 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2840051c0 con 0x7fd28c1396a0 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd28a7fc700 4 set_mon_vals no callback set 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 10 monclient: dump: 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd284005320 con 0x7fd28c1396a0 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c005100 con 0x7fd28c1396a0 2024-09-16T06:43:04.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(124..124 src has 66..124) v4 ==== 7321+0+0 (unknown 76663028 0 0) 0x7fd284044550 con 0x7fd28c1396a0 2024-09-16T06:43:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.370+0000 7fd29096c700 10 monclient: get_auth_request con 0x7fd2780362a0 auth_method 0 2024-09-16T06:43:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.378+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd284014ba0 con 0x7fd28c1396a0 2024-09-16T06:43:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.378+0000 7fd2897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.378+0000 7fd2897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd292bd0700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd292bd0700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd292bd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fd26c005930 con 0x7fd28c1396a0 2024-09-16T06:43:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v124) v1 ==== 73+0+6748 (unknown 2922337620 0 1281724412) 0x7fd284013240 con 0x7fd28c1396a0 2024-09-16T06:43:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd2897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.542+0000 7fd2897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:04.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.642+0000 7fd292bd0700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.642+0000 7fd292bd0700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.642+0000 7fd292bd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.642+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7fd26c001140 con 0x7fd28c1396a0 2024-09-16T06:43:04.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd2897fa700 1 -- 172.21.15.189:0/2374485174 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v124) v1 ==== 73+0+6748 (unknown 2922337620 0 1281724412) 0x7fd2840149c0 con 0x7fd28c1396a0 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd2897fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd2897fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout:{ 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: "9": [ 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -1, 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.648 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -2, 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.649 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -3, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -4, 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.650 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -5, 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.651 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -6, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.652 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -7, 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -8, 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.653 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -9, 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.654 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -10, 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -11, 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -12, 2024-09-16T06:43:04.655 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -13, 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -14, 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.656 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -15, 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: "10": [ 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.657 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -1, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.658 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: 0.263671875, 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -2, 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.659 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -3, 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -4, 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.661 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -5, 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.662 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.663 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -6, 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.664 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.665 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -7, 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -8, 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.666 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -9, 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.667 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: ], 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: [ 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0.087890625, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0, 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: 0 2024-09-16T06:43:04.668 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -10, 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.669 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -11, 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -12, 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.670 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -13, 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -14, 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.671 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: }, 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: { 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: "bucket_id": -15, 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: "weight_set": [ 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: [], 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: [] 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: } 2024-09-16T06:43:04.672 INFO:tasks.workunit.client.0.smithi189.stdout: ] 2024-09-16T06:43:04.673 INFO:tasks.workunit.client.0.smithi189.stdout:} 2024-09-16T06:43:04.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd2780362a0 legacy=0x7fd278038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 10 monclient: shutdown 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 >> v1:172.21.15.76:6790/0 conn(0x7fd28c1396a0 legacy=0x7fd28c1d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 shutdown_connections 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 >> 172.21.15.189:0/2374485174 conn(0x7fd28c0a03e0 msgr2=0x7fd28c0a07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 shutdown_connections 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.646+0000 7fd292bd0700 1 -- 172.21.15.189:0/2374485174 wait complete. 2024-09-16T06:43:04.674 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.762+0000 7f43fe5ce700 10 monclient: get_monmap_and_config 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.762+0000 7f43fe5ce700 10 monclient: build_initial_monmap 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.762+0000 7f43fe5ce700 10 monclient: monmap: 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:04.767783+0000 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:04.767783+0000 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:04.767 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.768 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:04.768 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:04.768 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:04.768 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 Processor -- start 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- start start 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient: init 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-a con 0x7f43f81352c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-b con 0x7f43f8135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-c con 0x7f43f80a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809b050 con 0x7f43f81352c0 2024-09-16T06:43:04.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f806ab20 con 0x7f43f80a2030 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f806a840 con 0x7f43f8135db0 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient(hunting): authenticate will time out at 1137.281343s 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fddcd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f43f80a2030 legacy=0x7f43f813df90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fcdcb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f43f8135db0 0x7f43f813a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57830/0 (socket says 172.21.15.189:57830) 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fcdcb700 1 -- 172.21.15.189:0/525804430 learned_addr learned my addr 172.21.15.189:0/525804430 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fd5cc700 1 --1- 172.21.15.189:0/525804430 >> v1:172.21.15.76:6789/0 conn(0x7f43f81352c0 0x7f43f81356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42476/0 (socket says 172.21.15.189:42476) 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254177767 0 0) 0x7f43f806a840 con 0x7f43f8135db0 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): my global_id is 125162 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e40035d0 con 0x7f43f8135db0 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589651257 0 0) 0x7f43f809b050 con 0x7f43f81352c0 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient(hunting): my global_id is 154302 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f806a840 con 0x7f43f81352c0 2024-09-16T06:43:04.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160973367 0 0) 0x7f43e40035d0 con 0x7f43f8135db0 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 >> v1:172.21.15.189:6789/0 conn(0x7f43f80a2030 legacy=0x7f43f813df90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 >> v1:172.21.15.76:6789/0 conn(0x7f43f81352c0 legacy=0x7f43f81356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 monclient: found mon.noname-b 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f806c1b0 con 0x7f43f8135db0 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: _finish_auth 0 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 5 monclient: authenticate success, global_id 125162 2024-09-16T06:43:04.775 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e8001610 con 0x7f43f8135db0 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: dump: 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.776 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e80018f0 con 0x7f43f8135db0 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 1 -- 172.21.15.189:0/525804430 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e8001c80 con 0x7f43f8135db0 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 4 set_mon_vals no callback set 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43effff700 10 monclient: dump: 2024-09-16T06:43:04.777 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 10 monclient: shutdown 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- 172.21.15.189:0/525804430 >> v1:172.21.15.76:6790/0 conn(0x7f43f8135db0 legacy=0x7f43f813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- 172.21.15.189:0/525804430 shutdown_connections 2024-09-16T06:43:04.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- 172.21.15.189:0/525804430 >> 172.21.15.189:0/525804430 conn(0x7f43f80a4550 msgr2=0x7f43f80a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- 172.21.15.189:0/525804430 shutdown_connections 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.766+0000 7f43fe5ce700 1 -- 172.21.15.189:0/525804430 wait complete. 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: build_initial_monmap 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: monmap: 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:04.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:04.780 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 Processor -- start 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- start start 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: init 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-a con 0x7f43f80a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-b con 0x7f43f81352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient(hunting): picked mon.noname-c con 0x7f43f8135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8132030 con 0x7f43f80a2030 2024-09-16T06:43:04.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809e890 con 0x7f43f81352c0 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8130e00 con 0x7f43f8135db0 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient(hunting): authenticate will time out at 1137.285630s 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fcdcb700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f43f81352c0 legacy=0x7f43f812dc20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fd5cc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f43f80a2030 0x7f43f812d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42490/0 (socket says 172.21.15.189:42490) 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fd5cc700 1 -- 172.21.15.189:0/2642511179 learned_addr learned my addr 172.21.15.189:0/2642511179 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918717729 0 0) 0x7f43f8132030 con 0x7f43f80a2030 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): my global_id is 154305 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43dc0035d0 con 0x7f43f80a2030 2024-09-16T06:43:04.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183359808 0 0) 0x7f43f8130e00 con 0x7f43f8135db0 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): my global_id is 125165 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f8132030 con 0x7f43f8135db0 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540988327 0 0) 0x7f43f8132030 con 0x7f43f8135db0 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43f8130e00 con 0x7f43f8135db0 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43f00043e0 con 0x7f43f8135db0 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): dump: 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.785 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553780743 0 0) 0x7f43dc0035d0 con 0x7f43f80a2030 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43f8132030 con 0x7f43f80a2030 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43f40018b0 con 0x7f43f80a2030 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient(hunting): dump: 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1721767905 0 0) 0x7f43f8130e00 con 0x7f43f8135db0 2024-09-16T06:43:04.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 >> v1:172.21.15.189:6789/0 conn(0x7f43f81352c0 legacy=0x7f43f812dc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 >> v1:172.21.15.76:6789/0 conn(0x7f43f80a2030 legacy=0x7f43f812d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 monclient: found mon.c 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f81310e0 con 0x7f43f8135db0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: _finish_auth 0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 5 monclient: authenticate success, global_id 125165 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _renew_subs 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f80a6780 con 0x7f43f8135db0 2024-09-16T06:43:04.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _renew_subs 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f81313c0 con 0x7f43f8135db0 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43f00046c0 con 0x7f43f8135db0 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43f0004a50 con 0x7f43f8135db0 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43eeffd700 4 set_mon_vals no callback set 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.770+0000 7f43edffb700 10 monclient: dump: 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.789 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43fe5ce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43fe5ce700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43fe5ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d8005100 con 0x7f43f8135db0 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f43f0004bb0 con 0x7f43f8135db0 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(124..124 src has 66..124) v4 ==== 7321+0+0 (unknown 76663028 0 0) 0x7f43f0044280 con 0x7f43f8135db0 2024-09-16T06:43:04.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.774+0000 7f43fd5cc700 10 monclient: get_auth_request con 0x7f43dc036120 auth_method 0 2024-09-16T06:43:04.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.786+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f43f0005380 con 0x7f43f8135db0 2024-09-16T06:43:04.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.786+0000 7f43edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:04.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.786+0000 7f43edffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.958+0000 7f43fe5ce700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.958+0000 7f43fe5ce700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.958+0000 7f43fe5ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:04.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:04.958+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f43d8005930 con 0x7f43f8135db0 2024-09-16T06:43:05.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.302+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v125) v1 ==== 129+0+0 (unknown 903083362 0 0) 0x7f43f001cf70 con 0x7f43f8135db0 2024-09-16T06:43:05.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.302+0000 7f43edffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:05.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.302+0000 7f43edffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.418+0000 7f43fe5ce700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.418+0000 7f43fe5ce700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.418+0000 7f43fe5ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:05.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:05.418+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f43d8005420 con 0x7f43f8135db0 2024-09-16T06:43:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.318+0000 7f43edffb700 1 -- 172.21.15.189:0/2642511179 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v126) v1 ==== 129+0+0 (unknown 4042788291 0 0) 0x7f43f0012df0 con 0x7f43f8135db0 2024-09-16T06:43:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.318+0000 7f43edffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:06.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.318+0000 7f43edffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 >> v1:172.21.15.189:6812/3098810674 conn(0x7f43dc036120 legacy=0x7f43dc0385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 10 monclient: shutdown 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 >> v1:172.21.15.76:6790/0 conn(0x7f43f8135db0 legacy=0x7f43f81d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 shutdown_connections 2024-09-16T06:43:06.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 >> 172.21.15.189:0/2642511179 conn(0x7f43f80a4550 msgr2=0x7f43f8133ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:06.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 shutdown_connections 2024-09-16T06:43:06.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.322+0000 7f43fe5ce700 1 -- 172.21.15.189:0/2642511179 wait complete. 2024-09-16T06:43:06.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T06:43:06.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:43:06.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T06:43:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 10 monclient: get_monmap_and_config 2024-09-16T06:43:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 10 monclient: build_initial_monmap 2024-09-16T06:43:06.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 10 monclient: monmap: 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:06.459305+0000 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:06.459305+0000 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:06.459 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 1 Processor -- start 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 1 -- start start 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.454+0000 7f90b8dd5700 10 monclient: init 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:06.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-b con 0x7f90b409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-a con 0x7f90b409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-c con 0x7f90b4138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b409cca0 con 0x7f90b409c5f0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b409f750 con 0x7f90b4138da0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4097620 con 0x7f90b409bb00 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient(hunting): authenticate will time out at 1138.972490s 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b2d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f90b4138da0 legacy=0x7f90b413b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b1d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f90b409c5f0 0x7f90b4137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42494/0 (socket says 172.21.15.189:42494) 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b1d9b700 1 -- 172.21.15.189:0/3822272171 learned_addr learned my addr 172.21.15.189:0/3822272171 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690237790 0 0) 0x7f90b409cca0 con 0x7f90b409c5f0 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): my global_id is 154308 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f909c0035d0 con 0x7f90b409c5f0 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330381208 0 0) 0x7f90b4097620 con 0x7f90b409bb00 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient(hunting): my global_id is 125168 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b409cca0 con 0x7f90b409bb00 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008585075 0 0) 0x7f90b409cca0 con 0x7f90b409bb00 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 >> v1:172.21.15.189:6789/0 conn(0x7f90b4138da0 legacy=0x7f90b413b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 >> v1:172.21.15.76:6789/0 conn(0x7f90b409c5f0 legacy=0x7f90b4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: _finish_hunting 0 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 monclient: found mon.noname-b 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4097900 con 0x7f90b409bb00 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: _finish_auth 0 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: _check_auth_tickets 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 5 monclient: authenticate success, global_id 125168 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f90a80015f0 con 0x7f90b409bb00 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:06.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: dump: 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a8001890 con 0x7f90b409bb00 2024-09-16T06:43:06.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 4 set_mon_vals no callback set 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 1 -- 172.21.15.189:0/3822272171 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f90a8001c60 con 0x7f90b409bb00 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b159a700 10 monclient: dump: 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 10 monclient: shutdown 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- 172.21.15.189:0/3822272171 >> v1:172.21.15.76:6790/0 conn(0x7f90b409bb00 legacy=0x7f90b409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- 172.21.15.189:0/3822272171 shutdown_connections 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- 172.21.15.189:0/3822272171 >> 172.21.15.189:0/3822272171 conn(0x7f90b40a03e0 msgr2=0x7f90b40a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- 172.21.15.189:0/3822272171 shutdown_connections 2024-09-16T06:43:06.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.458+0000 7f90b8dd5700 1 -- 172.21.15.189:0/3822272171 wait complete. 2024-09-16T06:43:06.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: build_initial_monmap 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: monmap: 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 Processor -- start 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- start start 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: init 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-b con 0x7f90b409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-a con 0x7f90b409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient(hunting): picked mon.noname-c con 0x7f90b4138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4132cc0 con 0x7f90b409c5f0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b41329e0 con 0x7f90b409bb00 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4070c40 con 0x7f90b4138da0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient(hunting): authenticate will time out at 1138.977825s 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b259c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f90b409bb00 legacy=0x7f90b41c9d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:06.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b1d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f90b409c5f0 0x7f90b4138230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42498/0 (socket says 172.21.15.189:42498) 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b1d9b700 1 -- 172.21.15.189:0/4238930000 learned_addr learned my addr 172.21.15.189:0/4238930000 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682664996 0 0) 0x7f90b4132cc0 con 0x7f90b409c5f0 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): my global_id is 154311 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90940035d0 con 0x7f90b409c5f0 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859866354 0 0) 0x7f90b4070c40 con 0x7f90b4138da0 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): my global_id is 125171 2024-09-16T06:43:06.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4132cc0 con 0x7f90b4138da0 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3266369692 0 0) 0x7f90940035d0 con 0x7f90b409c5f0 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b4070c40 con 0x7f90b409c5f0 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f90a4001a70 con 0x7f90b409c5f0 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352284066 0 0) 0x7f90b4132cc0 con 0x7f90b4138da0 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90940035d0 con 0x7f90b4138da0 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f90ac004310 con 0x7f90b4138da0 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:06.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4277936128 0 0) 0x7f90b4070c40 con 0x7f90b409c5f0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 >> v1:172.21.15.76:6790/0 conn(0x7f90b4138da0 legacy=0x7f90b41d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 >> v1:172.21.15.189:6789/0 conn(0x7f90b409bb00 legacy=0x7f90b41c9d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 monclient: found mon.a 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4070f20 con 0x7f90b409c5f0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 5 monclient: authenticate success, global_id 154311 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _renew_subs 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b4138940 con 0x7f90b409c5f0 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _renew_subs 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.462+0000 7f90b8dd5700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b4071740 con 0x7f90b409c5f0 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a4001ce0 con 0x7f90b409c5f0 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f90a4004ce0 con 0x7f90b409c5f0 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b0d99700 4 set_mon_vals no callback set 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 10 monclient: dump: 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b8dd5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b8dd5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b8dd5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b8dd5700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098005100 con 0x7f90b409c5f0 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f90a400f750 con 0x7f90b409c5f0 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(126..126 src has 66..126) v4 ==== 7321+0+0 (unknown 1215027322 0 0) 0x7f90a40441a0 con 0x7f90b409c5f0 2024-09-16T06:43:06.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.466+0000 7f90b259c700 10 monclient: get_auth_request con 0x7f9094036280 auth_method 0 2024-09-16T06:43:06.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.478+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f90a4010c50 con 0x7f90b409c5f0 2024-09-16T06:43:06.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.478+0000 7f90a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.478+0000 7f90a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:06.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.658+0000 7f90b8dd5700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T06:43:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90b8dd5700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T06:43:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90b8dd5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90b8dd5700 1 -- 172.21.15.189:0/4238930000 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7f9098005930 con 0x7f90b409c5f0 2024-09-16T06:43:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90a2ffd700 1 -- 172.21.15.189:0/4238930000 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v126) v1 ==== 168+0+0 (unknown 1213370565 0 0) 0x7f90a4012d10 con 0x7f90b409c5f0 2024-09-16T06:43:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T06:43:06.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.662+0000 7f90a2ffd700 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-16T06:43:06.669 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-16T06:43:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:43:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T06:43:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -x 2024-09-16T06:43:06.673 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 10 monclient: get_monmap_and_config 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 10 monclient: build_initial_monmap 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 10 monclient: monmap: 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.778 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:06.779086+0000 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:06.779086+0000 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.779 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 1 Processor -- start 2024-09-16T06:43:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 1 -- start start 2024-09-16T06:43:06.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.774+0000 7fd2289c6700 10 monclient: init 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-b con 0x7fd224132110 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-c con 0x7fd22406b8b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-a con 0x7fd22413ba10 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224132c00 con 0x7fd22413ba10 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22406bd30 con 0x7fd22406b8b0 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240a3240 con 0x7fd224132110 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:06.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient(hunting): authenticate will time out at 1139.292849s 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd222ffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd22406b8b0 legacy=0x7fd22413a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd223fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd22413ba10 0x7fd22413de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42502/0 (socket says 172.21.15.189:42502) 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd223fff700 1 -- 172.21.15.189:0/2082605427 learned_addr learned my addr 172.21.15.189:0/2082605427 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2237fe700 1 --1- 172.21.15.189:0/2082605427 >> v1:172.21.15.76:6790/0 conn(0x7fd224132110 0x7fd2241324f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57854/0 (socket says 172.21.15.189:57854) 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438250393 0 0) 0x7fd224132c00 con 0x7fd22413ba10 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): my global_id is 154314 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2100035d0 con 0x7fd22413ba10 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042744012 0 0) 0x7fd2240a3240 con 0x7fd224132110 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient(hunting): my global_id is 125174 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224132c00 con 0x7fd224132110 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 144063905 0 0) 0x7fd2100035d0 con 0x7fd22413ba10 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 >> v1:172.21.15.76:6790/0 conn(0x7fd224132110 legacy=0x7fd2241324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 >> v1:172.21.15.189:6789/0 conn(0x7fd22406b8b0 legacy=0x7fd22413a2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 monclient: found mon.noname-a 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22406a910 con 0x7fd22413ba10 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:06.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd21c001990 con 0x7fd22413ba10 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: dump: 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 5 monclient: authenticate success, global_id 154314 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd21c001bf0 con 0x7fd22413ba10 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 1 -- 172.21.15.189:0/2082605427 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd21c001fc0 con 0x7fd22413ba10 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2227fc700 10 monclient: dump: 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.788 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 4 set_mon_vals no callback set 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 10 monclient: shutdown 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- 172.21.15.189:0/2082605427 >> v1:172.21.15.76:6789/0 conn(0x7fd22413ba10 legacy=0x7fd22413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- 172.21.15.189:0/2082605427 shutdown_connections 2024-09-16T06:43:06.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- 172.21.15.189:0/2082605427 >> 172.21.15.189:0/2082605427 conn(0x7fd2240a4130 msgr2=0x7fd2240a4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- 172.21.15.189:0/2082605427 shutdown_connections 2024-09-16T06:43:06.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.778+0000 7fd2289c6700 1 -- 172.21.15.189:0/2082605427 wait complete. 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient: build_initial_monmap 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient: monmap: 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:06.795 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 1 Processor -- start 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 1 -- start start 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient: init 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-c con 0x7fd22406b8b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-b con 0x7fd224132110 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient(hunting): picked mon.noname-a con 0x7fd22413ba10 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224137dc0 con 0x7fd22413ba10 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22409e820 con 0x7fd224132110 2024-09-16T06:43:06.796 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224136720 con 0x7fd22406b8b0 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd222ffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd224132110 legacy=0x7fd224136010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2289c6700 10 monclient(hunting): authenticate will time out at 1139.298677s 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2237fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd22406b8b0 0x7fd224135900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57864/0 (socket says 172.21.15.189:57864) 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd2237fe700 1 -- 172.21.15.189:0/728812839 learned_addr learned my addr 172.21.15.189:0/728812839 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413175338 0 0) 0x7fd224137dc0 con 0x7fd22413ba10 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): my global_id is 154317 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2040035d0 con 0x7fd22413ba10 2024-09-16T06:43:06.797 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706011570 0 0) 0x7fd224136720 con 0x7fd22406b8b0 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 10 monclient(hunting): my global_id is 125177 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.782+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224137dc0 con 0x7fd22406b8b0 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028154800 0 0) 0x7fd2040035d0 con 0x7fd22413ba10 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd224136720 con 0x7fd22413ba10 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd21c001d60 con 0x7fd22413ba10 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): dump: 2024-09-16T06:43:06.798 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804684183 0 0) 0x7fd224137dc0 con 0x7fd22406b8b0 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2040035d0 con 0x7fd22406b8b0 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd20c0011d0 con 0x7fd22406b8b0 2024-09-16T06:43:06.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient(hunting): dump: 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.800 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149095282 0 0) 0x7fd224136720 con 0x7fd22413ba10 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 >> v1:172.21.15.76:6790/0 conn(0x7fd22406b8b0 legacy=0x7fd224135900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 >> v1:172.21.15.189:6789/0 conn(0x7fd224132110 legacy=0x7fd224136010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 monclient: found mon.a 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240a7050 con 0x7fd22413ba10 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: _finish_auth 0 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 5 monclient: authenticate success, global_id 154317 2024-09-16T06:43:06.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _renew_subs 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2240a6b00 con 0x7fd22413ba10 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _renew_subs 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd224139790 con 0x7fd22413ba10 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd21c005020 con 0x7fd22413ba10 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd21c005330 con 0x7fd22413ba10 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd221ffb700 4 set_mon_vals no callback set 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 10 monclient: dump: 2024-09-16T06:43:06.802 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd21c00fda0 con 0x7fd22413ba10 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2289c6700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd208005100 con 0x7fd22413ba10 2024-09-16T06:43:06.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(126..126 src has 66..126) v4 ==== 7321+0+0 (unknown 1215027322 0 0) 0x7fd21c044570 con 0x7fd22413ba10 2024-09-16T06:43:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.786+0000 7fd2237fe700 10 monclient: get_auth_request con 0x7fd204036280 auth_method 0 2024-09-16T06:43:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.794+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd21c014b00 con 0x7fd22413ba10 2024-09-16T06:43:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.794+0000 7fd21bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:06.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.794+0000 7fd21bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd2289c6700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd2289c6700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd2289c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:06.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd2289c6700 1 -- 172.21.15.189:0/728812839 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fd208005930 con 0x7fd22413ba10 2024-09-16T06:43:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd21bfff700 1 -- 172.21.15.189:0/728812839 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v126) v1 ==== 163+0+0 (unknown 2709714329 0 0) 0x7fd21c0131a0 con 0x7fd22413ba10 2024-09-16T06:43:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd21bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T06:43:06.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:06.982+0000 7fd21bfff700 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-16T06:43:06.988 INFO:tasks.workunit.client.0.smithi189.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-16T06:43:06.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:43:06.992 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-16T06:43:07.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: get_monmap_and_config 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: build_initial_monmap 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: monmap: 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:07.101524+0000 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:07.101524+0000 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:07.101 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 1 Processor -- start 2024-09-16T06:43:07.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 1 -- start start 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: init 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-b con 0x7f72fc09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-c con 0x7f72fc09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-a con 0x7f72fc138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc09cca0 con 0x7f72fc138da0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc09f750 con 0x7f72fc09c5f0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc097620 con 0x7f72fc09bb00 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f73042c1700 10 monclient(hunting): authenticate will time out at 1139.614723s 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730185c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f72fc09c5f0 legacy=0x7f72fc137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:07.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730285e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f72fc138da0 0x7f72fc13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42520/0 (socket says 172.21.15.189:42520) 2024-09-16T06:43:07.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730205d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f72fc09bb00 0x7f72fc09bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57874/0 (socket says 172.21.15.189:57874) 2024-09-16T06:43:07.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730285e700 1 -- 172.21.15.189:0/1298446239 learned_addr learned my addr 172.21.15.189:0/1298446239 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235441467 0 0) 0x7f72fc09cca0 con 0x7f72fc138da0 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): my global_id is 154320 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e40035d0 con 0x7f72fc138da0 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529942415 0 0) 0x7f72fc097620 con 0x7f72fc09bb00 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 10 monclient(hunting): my global_id is 125180 2024-09-16T06:43:07.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.098+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc09cca0 con 0x7f72fc09bb00 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533520073 0 0) 0x7f72e40035d0 con 0x7f72fc138da0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 >> v1:172.21.15.76:6790/0 conn(0x7f72fc09bb00 legacy=0x7f72fc09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 >> v1:172.21.15.189:6789/0 conn(0x7f72fc09c5f0 legacy=0x7f72fc137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: _finish_hunting 0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 monclient: found mon.noname-a 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc097900 con 0x7f72fc138da0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: _finish_auth 0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: _check_auth_tickets 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72ec0018c0 con 0x7f72fc138da0 2024-09-16T06:43:07.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 5 monclient: authenticate success, global_id 154320 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: dump: 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:07.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72ec001ae0 con 0x7f72fc138da0 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 1 -- 172.21.15.189:0/1298446239 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72ec001eb0 con 0x7f72fc138da0 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 4 set_mon_vals no callback set 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f730105b700 10 monclient: dump: 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.109 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 10 monclient: shutdown 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- 172.21.15.189:0/1298446239 >> v1:172.21.15.76:6789/0 conn(0x7f72fc138da0 legacy=0x7f72fc13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- 172.21.15.189:0/1298446239 shutdown_connections 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- 172.21.15.189:0/1298446239 >> 172.21.15.189:0/1298446239 conn(0x7f72fc0a03e0 msgr2=0x7f72fc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- 172.21.15.189:0/1298446239 shutdown_connections 2024-09-16T06:43:07.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- 172.21.15.189:0/1298446239 wait complete. 2024-09-16T06:43:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 10 monclient: build_initial_monmap 2024-09-16T06:43:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 10 monclient: monmap: 2024-09-16T06:43:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:07.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 Processor -- start 2024-09-16T06:43:07.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 1 -- start start 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.102+0000 7f73042c1700 10 monclient: init 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-c con 0x7f72fc09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-b con 0x7f72fc09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient(hunting): picked mon.noname-a con 0x7f72fc138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc090cd0 con 0x7f72fc138da0 2024-09-16T06:43:07.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc099380 con 0x7f72fc09c5f0 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f730185c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f72fc09c5f0 legacy=0x7f72fc08d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc09a6e0 con 0x7f72fc09bb00 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient(hunting): authenticate will time out at 1139.620457s 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f730205d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f72fc09bb00 0x7f72fc08cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57882/0 (socket says 172.21.15.189:57882) 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f730285e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f72fc138da0 0x7f72fc1cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:42530/0 (socket says 172.21.15.189:42530) 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f730205d700 1 -- 172.21.15.189:0/4186852083 learned_addr learned my addr 172.21.15.189:0/4186852083 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224037587 0 0) 0x7f72fc090cd0 con 0x7f72fc138da0 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): my global_id is 154323 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e80035d0 con 0x7f72fc138da0 2024-09-16T06:43:07.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156378144 0 0) 0x7f72fc09a6e0 con 0x7f72fc09bb00 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): my global_id is 125183 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc090cd0 con 0x7f72fc09bb00 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840717965 0 0) 0x7f72fc090cd0 con 0x7f72fc09bb00 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72fc09a6e0 con 0x7f72fc09bb00 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477242840 0 0) 0x7f72e80035d0 con 0x7f72fc138da0 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72fc090cd0 con 0x7f72fc138da0 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72f40019d0 con 0x7f72fc09bb00 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): dump: 2024-09-16T06:43:07.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72ec001df0 con 0x7f72fc138da0 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:07.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient(hunting): dump: 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 44439667 0 0) 0x7f72fc09a6e0 con 0x7f72fc09bb00 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 >> v1:172.21.15.189:6789/0 conn(0x7f72fc09c5f0 legacy=0x7f72fc08d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:07.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 >> v1:172.21.15.76:6789/0 conn(0x7f72fc138da0 legacy=0x7f72fc1cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 monclient: found mon.c 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc09cf50 con 0x7f72fc09bb00 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 5 monclient: authenticate success, global_id 125183 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _renew_subs 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc08e970 con 0x7f72fc09bb00 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _renew_subs 2024-09-16T06:43:07.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc08dd60 con 0x7f72fc09bb00 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72f4001d00 con 0x7f72fc09bb00 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72f4004ce0 con 0x7f72fc09bb00 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72fbfff700 4 set_mon_vals no callback set 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.106+0000 7f72faffd700 10 monclient: dump: 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:07.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f73042c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f73042c1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f73042c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc005100 con 0x7f72fc09bb00 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f72f400f750 con 0x7f72fc09bb00 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(126..126 src has 66..126) v4 ==== 7321+0+0 (unknown 1215027322 0 0) 0x7f72f4044170 con 0x7f72fc09bb00 2024-09-16T06:43:07.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.110+0000 7f730185c700 10 monclient: get_auth_request con 0x7f72e80362f0 auth_method 0 2024-09-16T06:43:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.122+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f72f4010ca0 con 0x7f72fc09bb00 2024-09-16T06:43:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.122+0000 7f72faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:07.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.122+0000 7f72faffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:07.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.294+0000 7f73042c1700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.294+0000 7f73042c1700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.294+0000 7f73042c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.294+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f72dc0053b0 con 0x7f72fc09bb00 2024-09-16T06:43:07.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.322+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v127) v1 ==== 139+0+0 (unknown 4202094069 0 0) 0x7f72f4012d40 con 0x7f72fc09bb00 2024-09-16T06:43:07.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.322+0000 7f72faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:07.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.322+0000 7f72faffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:07.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.434+0000 7f73042c1700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:07.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.434+0000 7f73042c1700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:07.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.434+0000 7f73042c1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:07.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:07.434+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f72dc0010a0 con 0x7f72fc09bb00 2024-09-16T06:43:08.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.322+0000 7f72faffd700 1 -- 172.21.15.189:0/4186852083 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v128) v1 ==== 139+0+0 (unknown 2653420188 0 0) 0x7f72f40144c0 con 0x7f72fc09bb00 2024-09-16T06:43:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.326+0000 7f72faffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T06:43:08.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.326+0000 7f72faffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 >> v1:172.21.15.189:6812/3098810674 conn(0x7f72e80362f0 legacy=0x7f72e80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 10 monclient: shutdown 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 >> v1:172.21.15.76:6790/0 conn(0x7f72fc09bb00 legacy=0x7f72fc08cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 shutdown_connections 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 >> 172.21.15.189:0/4186852083 conn(0x7f72fc0a03e0 msgr2=0x7f72fc09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 shutdown_connections 2024-09-16T06:43:08.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.330+0000 7f73042c1700 1 -- 172.21.15.189:0/4186852083 wait complete. 2024-09-16T06:43:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c fooset 2024-09-16T06:43:08.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 1 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: get_monmap_and_config 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: build_initial_monmap 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: monmap: 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:08.456972+0000 2024-09-16T06:43:08.456 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:08.456972+0000 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:08.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 1 Processor -- start 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 1 -- start start 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: init 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:08.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient(hunting): picked mon.noname-b con 0x7f31840707a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient(hunting): picked mon.noname-c con 0x7f3184070b80 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient(hunting): picked mon.noname-a con 0x7f318413b5b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840a1330 con 0x7f318413b5b0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318409f110 con 0x7f3184070b80 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840a0670 con 0x7f31840707a0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient(hunting): authenticate will time out at 1140.970030s 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318259c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3184070b80 legacy=0x7f3184071010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3182d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f31840707a0 0x7f318409ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57884/0 (socket says 172.21.15.189:57884) 2024-09-16T06:43:08.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3182d9d700 1 -- 172.21.15.189:0/3255205388 learned_addr learned my addr 172.21.15.189:0/3255205388 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:08.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702086209 0 0) 0x7f31840a1330 con 0x7f318413b5b0 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): my global_id is 154326 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31700035d0 con 0x7f318413b5b0 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323853323 0 0) 0x7f31840a0670 con 0x7f31840707a0 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient(hunting): my global_id is 125186 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31840a1330 con 0x7f31840707a0 2024-09-16T06:43:08.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108305274 0 0) 0x7f31700035d0 con 0x7f318413b5b0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 >> v1:172.21.15.76:6790/0 conn(0x7f31840707a0 legacy=0x7f318409ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 >> v1:172.21.15.189:6789/0 conn(0x7f3184070b80 legacy=0x7f3184071010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 monclient: found mon.noname-a 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318409f380 con 0x7f318413b5b0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: _finish_auth 0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f317c001990 con 0x7f318413b5b0 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 5 monclient: authenticate success, global_id 154326 2024-09-16T06:43:08.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: dump: 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f317c001c00 con 0x7f318413b5b0 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 1 -- 172.21.15.189:0/3255205388 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f317c001f90 con 0x7f318413b5b0 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f318922d700 4 set_mon_vals no callback set 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.454+0000 7f3181d9b700 10 monclient: dump: 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.465 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: shutdown 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- 172.21.15.189:0/3255205388 >> v1:172.21.15.76:6789/0 conn(0x7f318413b5b0 legacy=0x7f318413d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- 172.21.15.189:0/3255205388 shutdown_connections 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- 172.21.15.189:0/3255205388 >> 172.21.15.189:0/3255205388 conn(0x7f31840a44a0 msgr2=0x7f31840a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- 172.21.15.189:0/3255205388 shutdown_connections 2024-09-16T06:43:08.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- 172.21.15.189:0/3255205388 wait complete. 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: build_initial_monmap 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: monmap: 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 Processor -- start 2024-09-16T06:43:08.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- start start 2024-09-16T06:43:08.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: init 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient(hunting): picked mon.noname-c con 0x7f31840707a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient(hunting): picked mon.noname-a con 0x7f3184070b80 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient(hunting): picked mon.noname-b con 0x7f318413b5b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:08.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31841372e0 con 0x7f3184070b80 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840a3620 con 0x7f318413b5b0 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318409b0d0 con 0x7f31840707a0 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318922d700 10 monclient(hunting): authenticate will time out at 1140.974657s 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f318359e700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f318413b5b0 legacy=0x7f31841d2070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f3182d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f31840707a0 0x7f31841337f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:57892/0 (socket says 172.21.15.189:57892) 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f3182d9d700 1 -- 172.21.15.189:0/1024599282 learned_addr learned my addr 172.21.15.189:0/1024599282 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083909957 0 0) 0x7f31841372e0 con 0x7f3184070b80 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): my global_id is 154329 2024-09-16T06:43:08.472 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31640035d0 con 0x7f3184070b80 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882942934 0 0) 0x7f318409b0d0 con 0x7f31840707a0 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 10 monclient(hunting): my global_id is 125189 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.458+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31841372e0 con 0x7f31840707a0 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759346449 0 0) 0x7f31640035d0 con 0x7f3184070b80 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f318409b0d0 con 0x7f3184070b80 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f316c004170 con 0x7f3184070b80 2024-09-16T06:43:08.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): dump: 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.474 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800604244 0 0) 0x7f31841372e0 con 0x7f31840707a0 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31640035d0 con 0x7f31840707a0 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31740018b0 con 0x7f31840707a0 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient(hunting): dump: 2024-09-16T06:43:08.475 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1621846556 0 0) 0x7f318409b0d0 con 0x7f3184070b80 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 >> v1:172.21.15.76:6790/0 conn(0x7f31840707a0 legacy=0x7f31841337f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 >> v1:172.21.15.189:6789/0 conn(0x7f318413b5b0 legacy=0x7f31841d2070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 monclient: found mon.a 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318413ebc0 con 0x7f3184070b80 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 5 monclient: authenticate success, global_id 154329 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _renew_subs 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31841351d0 con 0x7f3184070b80 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _renew_subs 2024-09-16T06:43:08.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f318413b290 con 0x7f3184070b80 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f316c0043e0 con 0x7f3184070b80 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f316c004770 con 0x7f3184070b80 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f3180d99700 4 set_mon_vals no callback set 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 10 monclient: dump: 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3168005100 con 0x7f3184070b80 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f316c0048d0 con 0x7f3184070b80 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(128..128 src has 66..128) v4 ==== 7321+0+0 (unknown 3301138605 0 0) 0x7f316c043f80 con 0x7f3184070b80 2024-09-16T06:43:08.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.462+0000 7f3182d9d700 10 monclient: get_auth_request con 0x7f31640362c0 auth_method 0 2024-09-16T06:43:08.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.470+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f316c0050a0 con 0x7f3184070b80 2024-09-16T06:43:08.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.470+0000 7f317b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.470+0000 7f317b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f318922d700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f318922d700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f318922d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f3168005930 con 0x7f3184070b80 2024-09-16T06:43:08.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v128) v1 ==== 71+0+14 (unknown 43352416 0 3003798303) 0x7f316c012a90 con 0x7f3184070b80 2024-09-16T06:43:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f317b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.654+0000 7f317b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:08.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.758+0000 7f318922d700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.758+0000 7f318922d700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.758+0000 7f318922d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.758+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f3168005ce0 con 0x7f3184070b80 2024-09-16T06:43:08.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.758+0000 7f317b7fe700 1 -- 172.21.15.189:0/1024599282 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v128) v1 ==== 71+0+14 (unknown 43352416 0 3003798303) 0x7f316c014210 con 0x7f3184070b80 2024-09-16T06:43:08.763 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.762+0000 7f317b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:08.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.762+0000 7f317b7fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:08.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 >> v1:172.21.15.189:6812/3098810674 conn(0x7f31640362c0 legacy=0x7f3164038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.768 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 10 monclient: shutdown 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 >> v1:172.21.15.76:6789/0 conn(0x7f3184070b80 legacy=0x7f3184133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 shutdown_connections 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 >> 172.21.15.189:0/1024599282 conn(0x7f31840a44a0 msgr2=0x7f3184130240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 shutdown_connections 2024-09-16T06:43:08.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.766+0000 7f318922d700 1 -- 172.21.15.189:0/1024599282 wait complete. 2024-09-16T06:43:08.783 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm fooset 2024-09-16T06:43:08.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: get_monmap_and_config 2024-09-16T06:43:08.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: build_initial_monmap 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: monmap: 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:08.886325+0000 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:08.886325+0000 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:08.886 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 1 Processor -- start 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 1 -- start start 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: init 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-b con 0x7f0fbc06a8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-a con 0x7f0fbc0a6090 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:08.890 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-c con 0x7f0fbc13bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc06acb0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc099d60 con 0x7f0fbc13bab0 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc09a040 con 0x7f0fbc06a8d0 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc462b700 10 monclient(hunting): authenticate will time out at 1141.399708s 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.882+0000 7f0fc2bc8700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0fbc13bab0 legacy=0x7f0fbc13dea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc23c7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0fbc06a8d0 0x7f0fbc0a5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:53464/0 (socket says 172.21.15.189:53464) 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc23c7700 1 -- 172.21.15.189:0/1333341564 learned_addr learned my addr 172.21.15.189:0/1333341564 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc1bc6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0fbc0a6090 0x7f0fbc13a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:58100/0 (socket says 172.21.15.189:58100) 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984551311 0 0) 0x7f0fbc06acb0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.891 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): my global_id is 154332 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa80035d0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042624681 0 0) 0x7f0fbc09a040 con 0x7f0fbc06a8d0 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient(hunting): my global_id is 125192 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fbc06acb0 con 0x7f0fbc06a8d0 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020870091 0 0) 0x7f0fa80035d0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 >> v1:172.21.15.76:6790/0 conn(0x7f0fbc06a8d0 legacy=0x7f0fbc0a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 >> v1:172.21.15.189:6789/0 conn(0x7f0fbc13bab0 legacy=0x7f0fbc13dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:08.892 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: _finish_hunting 0 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 monclient: found mon.noname-a 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc09a320 con 0x7f0fbc0a6090 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: _finish_auth 0 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: _check_auth_tickets 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 5 monclient: authenticate success, global_id 154332 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0fb0001540 con 0x7f0fbc0a6090 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:08.893 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: dump: 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb0001760 con 0x7f0fbc0a6090 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:08.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 4 set_mon_vals no callback set 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 1 -- 172.21.15.189:0/1333341564 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0fb0001af0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc13c5700 10 monclient: dump: 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.895 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 10 monclient: shutdown 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 1 -- 172.21.15.189:0/1333341564 >> v1:172.21.15.76:6789/0 conn(0x7f0fbc0a6090 legacy=0x7f0fbc13a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 1 -- 172.21.15.189:0/1333341564 shutdown_connections 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 1 -- 172.21.15.189:0/1333341564 >> 172.21.15.189:0/1333341564 conn(0x7f0fbc0a7bf0 msgr2=0x7f0fbc0a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 1 -- 172.21.15.189:0/1333341564 shutdown_connections 2024-09-16T06:43:08.896 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 1 -- 172.21.15.189:0/1333341564 wait complete. 2024-09-16T06:43:08.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 10 monclient: build_initial_monmap 2024-09-16T06:43:08.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.886+0000 7f0fc462b700 10 monclient: monmap: 2024-09-16T06:43:08.900 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:08.900 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 Processor -- start 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- start start 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: init 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:08.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-b con 0x7f0fbc06a8d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-c con 0x7f0fbc0a6090 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient(hunting): picked mon.noname-a con 0x7f0fbc13bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc11ec40 con 0x7f0fbc13bab0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc23c7700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f0fbc06a8d0 legacy=0x7f0fbc11de20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc0a07d0 con 0x7f0fbc06a8d0 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc13b7f0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient(hunting): authenticate will time out at 1141.405424s 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc2bc8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0fbc13bab0 0x7f0fbc1d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:58116/0 (socket says 172.21.15.189:58116) 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc1bc6700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0fbc0a6090 0x7f0fbc11e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:53476/0 (socket says 172.21.15.189:53476) 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc2bc8700 1 -- 172.21.15.189:0/3330830615 learned_addr learned my addr 172.21.15.189:0/3330830615 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340640196 0 0) 0x7f0fbc13b7f0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): my global_id is 125195 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa00035d0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397329213 0 0) 0x7f0fbc11ec40 con 0x7f0fbc13bab0 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): my global_id is 154335 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fbc13b7f0 con 0x7f0fbc13bab0 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474084411 0 0) 0x7f0fbc13b7f0 con 0x7f0fbc13bab0 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fbc11ec40 con 0x7f0fbc13bab0 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0fb4004460 con 0x7f0fbc13bab0 2024-09-16T06:43:08.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): dump: 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.904 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857128070 0 0) 0x7f0fa00035d0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fbc13b7f0 con 0x7f0fbc0a6090 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0fb0001440 con 0x7f0fbc0a6090 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient(hunting): dump: 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.905 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662693172 0 0) 0x7f0fbc11ec40 con 0x7f0fbc13bab0 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 >> v1:172.21.15.76:6790/0 conn(0x7f0fbc0a6090 legacy=0x7f0fbc11e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 >> v1:172.21.15.189:6789/0 conn(0x7f0fbc06a8d0 legacy=0x7f0fbc11de20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 monclient: found mon.a 2024-09-16T06:43:08.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc0a33c0 con 0x7f0fbc13bab0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 5 monclient: authenticate success, global_id 154335 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _renew_subs 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc120620 con 0x7f0fbc13bab0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _renew_subs 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc09f8f0 con 0x7f0fbc13bab0 2024-09-16T06:43:08.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb4004930 con 0x7f0fbc13bab0 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0fb4004c40 con 0x7f0fbc13bab0 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0fc0bc4700 4 set_mon_vals no callback set 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.890+0000 7f0faeffd700 10 monclient: dump: 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:08.908 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0fc462b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0fc462b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0fc462b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa4005100 con 0x7f0fbc13bab0 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0fb4004e20 con 0x7f0fbc13bab0 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(128..128 src has 66..128) v4 ==== 7321+0+0 (unknown 3301138605 0 0) 0x7f0fb4044300 con 0x7f0fbc13bab0 2024-09-16T06:43:08.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.894+0000 7f0fc23c7700 10 monclient: get_auth_request con 0x7f0fa0036360 auth_method 0 2024-09-16T06:43:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.902+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0fb40149b0 con 0x7f0fbc13bab0 2024-09-16T06:43:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.902+0000 7f0faeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:08.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:08.902+0000 7f0faeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.078+0000 7f0fc462b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.078+0000 7f0fc462b700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.078+0000 7f0fc462b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.078+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f0fa40053b0 con 0x7f0fbc13bab0 2024-09-16T06:43:09.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.346+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v129) v1 ==== 89+0+0 (unknown 37901553 0 0) 0x7f0fb4012e20 con 0x7f0fbc13bab0 2024-09-16T06:43:09.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.346+0000 7f0faeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:09.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.346+0000 7f0faeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:09.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.462+0000 7f0fc462b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:09.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.462+0000 7f0fc462b700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:09.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.462+0000 7f0fc462b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:09.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:09.462+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f0fa4005900 con 0x7f0fbc13bab0 2024-09-16T06:43:10.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.154+0000 7f0faeffd700 1 -- 172.21.15.189:0/3330830615 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v130) v1 ==== 89+0+0 (unknown 3302232522 0 0) 0x7f0fb40145a0 con 0x7f0fbc13bab0 2024-09-16T06:43:10.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.154+0000 7f0faeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T06:43:10.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.154+0000 7f0faeffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0fa0036360 legacy=0x7f0fa0038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 10 monclient: shutdown 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 >> v1:172.21.15.76:6789/0 conn(0x7f0fbc13bab0 legacy=0x7f0fbc1d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 shutdown_connections 2024-09-16T06:43:10.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 >> 172.21.15.189:0/3330830615 conn(0x7f0fbc0a7bf0 msgr2=0x7f0fbc13a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:10.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 shutdown_connections 2024-09-16T06:43:10.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.162+0000 7f0fc462b700 1 -- 172.21.15.189:0/3330830615 wait complete. 2024-09-16T06:43:10.178 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:10.178 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c fooset 2024-09-16T06:43:10.179 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T06:43:10.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.278+0000 7fa6731af700 10 monclient: get_monmap_and_config 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.278+0000 7fa6731af700 10 monclient: build_initial_monmap 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient: monmap: 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:10.284554+0000 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:10.284554+0000 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:10.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 1 Processor -- start 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 1 -- start start 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient: init 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-c con 0x7fa66c09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-a con 0x7fa66c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:10.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-b con 0x7fa66c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c09cca0 con 0x7fa66c09c5f0 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c09f750 con 0x7fa66c09bb00 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c097620 con 0x7fa66c138da0 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient(hunting): authenticate will time out at 1142.797452s 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa670f4b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa66c09bb00 legacy=0x7fa66c09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa66c09c5f0 0x7fa66c137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:58130/0 (socket says 172.21.15.189:58130) 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66bfff700 1 -- 172.21.15.189:0/3716575384 learned_addr learned my addr 172.21.15.189:0/3716575384 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201430928 0 0) 0x7fa66c097620 con 0x7fa66c138da0 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): my global_id is 125198 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6540035d0 con 0x7fa66c138da0 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715596628 0 0) 0x7fa66c09cca0 con 0x7fa66c09c5f0 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient(hunting): my global_id is 154338 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c097620 con 0x7fa66c09c5f0 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 369820072 0 0) 0x7fa6540035d0 con 0x7fa66c138da0 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 >> v1:172.21.15.189:6789/0 conn(0x7fa66c09bb00 legacy=0x7fa66c09bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 >> v1:172.21.15.76:6789/0 conn(0x7fa66c09c5f0 legacy=0x7fa66c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:43:10.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c097900 con 0x7fa66c138da0 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 5 monclient: authenticate success, global_id 125198 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa664001990 con 0x7fa66c138da0 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: dump: 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.291 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa664001c70 con 0x7fa66c138da0 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 1 -- 172.21.15.189:0/3716575384 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa664002040 con 0x7fa66c138da0 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa66b7fe700 10 monclient: dump: 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.282+0000 7fa6731af700 4 set_mon_vals no callback set 2024-09-16T06:43:10.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: shutdown 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- 172.21.15.189:0/3716575384 >> v1:172.21.15.76:6790/0 conn(0x7fa66c138da0 legacy=0x7fa66c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- 172.21.15.189:0/3716575384 shutdown_connections 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- 172.21.15.189:0/3716575384 >> 172.21.15.189:0/3716575384 conn(0x7fa66c0a03e0 msgr2=0x7fa66c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- 172.21.15.189:0/3716575384 shutdown_connections 2024-09-16T06:43:10.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- 172.21.15.189:0/3716575384 wait complete. 2024-09-16T06:43:10.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: build_initial_monmap 2024-09-16T06:43:10.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: monmap: 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 Processor -- start 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- start start 2024-09-16T06:43:10.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: init 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-a con 0x7fa66c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-b con 0x7fa66c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient(hunting): picked mon.noname-c con 0x7fa66c138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0a2c10 con 0x7fa66c09bb00 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c09a6e0 con 0x7fa66c09c5f0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c08ea00 con 0x7fa66c138da0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa66bfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa66c09c5f0 legacy=0x7fa66c08d730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6731af700 10 monclient(hunting): authenticate will time out at 1142.802198s 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa67174c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa66c138da0 0x7fa66c1cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:53510/0 (socket says 172.21.15.189:53510) 2024-09-16T06:43:10.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa67174c700 1 -- 172.21.15.189:0/478101529 learned_addr learned my addr 172.21.15.189:0/478101529 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079357239 0 0) 0x7fa66c08ea00 con 0x7fa66c138da0 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): my global_id is 125201 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6580035d0 con 0x7fa66c138da0 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850341890 0 0) 0x7fa66c0a2c10 con 0x7fa66c09bb00 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): my global_id is 154341 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c08ea00 con 0x7fa66c09bb00 2024-09-16T06:43:10.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454323364 0 0) 0x7fa6580035d0 con 0x7fa66c138da0 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa66c0a2c10 con 0x7fa66c138da0 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa664001f80 con 0x7fa66c138da0 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): dump: 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.303 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236933546 0 0) 0x7fa66c08ea00 con 0x7fa66c09bb00 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6580035d0 con 0x7fa66c09bb00 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa660004170 con 0x7fa66c09bb00 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.286+0000 7fa6697fa700 10 monclient(hunting): dump: 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.304 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389216151 0 0) 0x7fa66c0a2c10 con 0x7fa66c138da0 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 >> v1:172.21.15.189:6789/0 conn(0x7fa66c09c5f0 legacy=0x7fa66c08d730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 >> v1:172.21.15.76:6789/0 conn(0x7fa66c09bb00 legacy=0x7fa66c08d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 monclient: found mon.c 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c090d60 con 0x7fa66c138da0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 5 monclient: authenticate success, global_id 125201 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _renew_subs 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c08f350 con 0x7fa66c138da0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _renew_subs 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c08de40 con 0x7fa66c138da0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa664003d90 con 0x7fa66c138da0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa664004120 con 0x7fa66c138da0 2024-09-16T06:43:10.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa66affd700 4 set_mon_vals no callback set 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 10 monclient: dump: 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa664004280 con 0x7fa66c138da0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c005100 con 0x7fa66c138da0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7fa664044380 con 0x7fa66c138da0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.290+0000 7fa670f4b700 10 monclient: get_auth_request con 0x7fa6580362f0 auth_method 0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.302+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa664014ac0 con 0x7fa66c138da0 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.302+0000 7fa6697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.302+0000 7fa6697fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6731af700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.485 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6731af700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6731af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fa64c005930 con 0x7fa66c138da0 2024-09-16T06:43:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+7 (unknown 914116082 0 1352219128) 0x7fa664013160 con 0x7fa66c138da0 2024-09-16T06:43:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.486 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.482+0000 7fa6697fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:10.603 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.598+0000 7fa6731af700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.598+0000 7fa6731af700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.598+0000 7fa6731af700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.598+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fa64c005420 con 0x7fa66c138da0 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6697fa700 1 -- 172.21.15.189:0/478101529 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+7 (unknown 914116082 0 1352219128) 0x7fa6640148e0 con 0x7fa66c138da0 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6697fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6697fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa6580362f0 legacy=0x7fa6580387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 10 monclient: shutdown 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 >> v1:172.21.15.76:6790/0 conn(0x7fa66c138da0 legacy=0x7fa66c1cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 shutdown_connections 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 >> 172.21.15.189:0/478101529 conn(0x7fa66c0a03e0 msgr2=0x7fa66c09a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 shutdown_connections 2024-09-16T06:43:10.607 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.602+0000 7fa6731af700 1 -- 172.21.15.189:0/478101529 wait complete. 2024-09-16T06:43:10.617 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:10.618 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep barset 2024-09-16T06:43:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.718+0000 7fbe02119700 10 monclient: get_monmap_and_config 2024-09-16T06:43:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.718+0000 7fbe02119700 10 monclient: build_initial_monmap 2024-09-16T06:43:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.718+0000 7fbe02119700 10 monclient: monmap: 2024-09-16T06:43:10.723 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:10.724168+0000 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:10.724168+0000 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:10.724 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 1 Processor -- start 2024-09-16T06:43:10.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 1 -- start start 2024-09-16T06:43:10.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient: init 2024-09-16T06:43:10.726 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc12ea00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc09e5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc1385e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09dc90 con 0x7fbdfc1385e0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09d360 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09d640 con 0x7fbdfc12ea00 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient(hunting): authenticate will time out at 1143.237472s 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe00916700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbdfc09e5f0 legacy=0x7fbdfc136ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:10.727 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe01117700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fbdfc12ea00 0x7fbdfc12ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:53516/0 (socket says 172.21.15.189:53516) 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe01918700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbdfc1385e0 0x7fbdfc13a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:58144/0 (socket says 172.21.15.189:58144) 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe01117700 1 -- 172.21.15.189:0/404989491 learned_addr learned my addr 172.21.15.189:0/404989491 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761328851 0 0) 0x7fbdfc09dc90 con 0x7fbdfc1385e0 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): my global_id is 154344 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde80035d0 con 0x7fbdfc1385e0 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469398530 0 0) 0x7fbdfc09d640 con 0x7fbdfc12ea00 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.728 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient(hunting): my global_id is 125204 2024-09-16T06:43:10.729 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc09dc90 con 0x7fbdfc12ea00 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392133515 0 0) 0x7fbde80035d0 con 0x7fbdfc1385e0 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 >> v1:172.21.15.76:6790/0 conn(0x7fbdfc12ea00 legacy=0x7fbdfc12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 >> v1:172.21.15.189:6789/0 conn(0x7fbdfc09e5f0 legacy=0x7fbdfc136ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 monclient: found mon.noname-a 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc09d920 con 0x7fbdfc1385e0 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: _finish_auth 0 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:10.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 5 monclient: authenticate success, global_id 154344 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001610 con 0x7fbdfc1385e0 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: dump: 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.736 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.737 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf4001880 con 0x7fbdfc1385e0 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 1 -- 172.21.15.189:0/404989491 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001c50 con 0x7fbdfc1385e0 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbe02119700 4 set_mon_vals no callback set 2024-09-16T06:43:10.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.722+0000 7fbdeffff700 10 monclient: dump: 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.739 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: shutdown 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- 172.21.15.189:0/404989491 >> v1:172.21.15.76:6789/0 conn(0x7fbdfc1385e0 legacy=0x7fbdfc13a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- 172.21.15.189:0/404989491 shutdown_connections 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- 172.21.15.189:0/404989491 >> 172.21.15.189:0/404989491 conn(0x7fbdfc0a0eb0 msgr2=0x7fbdfc0a12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- 172.21.15.189:0/404989491 shutdown_connections 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- 172.21.15.189:0/404989491 wait complete. 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: build_initial_monmap 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: monmap: 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:10.740 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 Processor -- start 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- start start 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: init 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:10.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc09e5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc12ea00 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc1385e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc071620 con 0x7fbdfc1385e0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09b5a0 con 0x7fbdfc12ea00 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc1382c0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe00916700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbdfc12ea00 legacy=0x7fbdfc070640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe02119700 10 monclient(hunting): authenticate will time out at 1143.242364s 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe01117700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fbdfc09e5f0 0x7fbdfc06ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:53532/0 (socket says 172.21.15.189:53532) 2024-09-16T06:43:10.742 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe01918700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbdfc1385e0 0x7fbdfc1cf7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:58160/0 (socket says 172.21.15.189:58160) 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbe01117700 1 -- 172.21.15.189:0/3218171069 learned_addr learned my addr 172.21.15.189:0/3218171069 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008527819 0 0) 0x7fbdfc1382c0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): my global_id is 125207 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde00035d0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327297010 0 0) 0x7fbdfc071620 con 0x7fbdfc1385e0 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:10.744 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): my global_id is 154347 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc1382c0 con 0x7fbdfc1385e0 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310740835 0 0) 0x7fbde00035d0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdfc071620 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf00019d0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): dump: 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.745 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129814272 0 0) 0x7fbdfc1382c0 con 0x7fbdfc1385e0 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde00035d0 con 0x7fbdfc1385e0 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001b90 con 0x7fbdfc1385e0 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:10.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.726+0000 7fbdedffb700 10 monclient(hunting): dump: 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.747 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935539031 0 0) 0x7fbdfc071620 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 >> v1:172.21.15.189:6789/0 conn(0x7fbdfc12ea00 legacy=0x7fbdfc070640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 >> v1:172.21.15.76:6789/0 conn(0x7fbdfc1385e0 legacy=0x7fbdfc1cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: _finish_hunting 0 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 monclient: found mon.c 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc137d30 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: _finish_auth 0 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: _check_auth_tickets 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:10.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 5 monclient: authenticate success, global_id 125207 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _renew_subs 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc070d50 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _renew_subs 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc072150 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf0001ca0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf0004ce0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:10.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdef7fe700 4 set_mon_vals no callback set 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 10 monclient: dump: 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:10.750 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc005100 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbdf000f750 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7fbdf0043f70 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.751 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.730+0000 7fbe00916700 10 monclient: get_auth_request con 0x7fbde00362c0 auth_method 0 2024-09-16T06:43:10.751 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:43:10.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:43:10.752 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:43:10.752 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:43:10.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.738+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbdf00146d0 con 0x7fbdfc09e5f0 2024-09-16T06:43:10.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.738+0000 7fbdedffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:10.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.738+0000 7fbdedffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:10.755 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:43:10.755 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T06:43:10.755 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T06:43:10.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:43:10.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.918+0000 7fbe02119700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.918+0000 7fbe02119700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:10.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.918+0000 7fbe02119700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:10.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:10.918+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fbddc005930 con 0x7fbdfc09e5f0 2024-09-16T06:43:13.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:13.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:13.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:13.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:13.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:13.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:13.783 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:13.726+0000 7fbdecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:14.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:14.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:14.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:14.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:14.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:15.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:15.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:15.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:15.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:15.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:15.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:16.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:16.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:16.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:16.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:16.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:16.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:17.730 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:17.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:17.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:17.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:17.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:17.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:18.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:18.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:18.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:18.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:19.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:19.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:19.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:19.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:20.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:20.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:20.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:20.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:21.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:21.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:21.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:21.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:22.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:22.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:22.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:22.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:22.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:22.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:23.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:23.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:23.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:23.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:23.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:23.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:23.726+0000 7fbdecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:24.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:24.726+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:24.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:24.726+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:24.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:24.726+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:25.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:25.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:25.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:25.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:26.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:26.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:26.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:26.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:26.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:26.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:27.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:27.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:27.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:27.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:27.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:27.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:28.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:28.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:28.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:28.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:29.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:29.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:29.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:29.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:30.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:30.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:30.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:30.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:30.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:30.756 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T06:43:30.756 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T06:43:30.757 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:43:30.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: get_monmap_and_config 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: build_initial_monmap 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: monmap: 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:30.872853+0000 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:30.872853+0000 2024-09-16T06:43:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:30.872 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 1 Processor -- start 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 1 -- start start 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: init 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-a con 0x7f0ad80973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:30.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-b con 0x7f0ad8097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-c con 0x7f0ad81400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad8099350 con 0x7f0ad80973c0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad809f690 con 0x7f0ad81400d0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad8099bf0 con 0x7f0ad8097eb0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient(hunting): authenticate will time out at 1158.006134s 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0addeac700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0ad80973c0 0x7f0ad80977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51478/0 (socket says 172.21.15.76:51478) 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0add6ab700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ad8097eb0 0x7f0ad813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50732/0 (socket says 172.21.15.76:50732) 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0addeac700 1 -- 172.21.15.76:0/1542843048 learned_addr learned my addr 172.21.15.76:0/1542843048 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959711904 0 0) 0x7f0ad8099350 con 0x7f0ad80973c0 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:30.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): my global_id is 154350 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac40035d0 con 0x7f0ad80973c0 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295388330 0 0) 0x7f0ad809f690 con 0x7f0ad81400d0 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient(hunting): my global_id is 164098 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad8099350 con 0x7f0ad81400d0 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520379367 0 0) 0x7f0ac40035d0 con 0x7f0ad80973c0 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 >> v1:172.21.15.76:6790/0 conn(0x7f0ad8097eb0 legacy=0x7f0ad813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 legacy=0x7f0ad81424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:30.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 monclient: found mon.noname-a 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad8098360 con 0x7f0ad80973c0 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: _finish_auth 0 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 5 monclient: authenticate success, global_id 154350 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ac8001610 con 0x7f0ad80973c0 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:30.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: dump: 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ac80018f0 con 0x7f0ad80973c0 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:30.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adeeae700 4 set_mon_vals no callback set 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 1 -- 172.21.15.76:0/1542843048 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ac8001c80 con 0x7f0ad80973c0 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.868+0000 7f0adceaa700 10 monclient: dump: 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: shutdown 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- 172.21.15.76:0/1542843048 >> v1:172.21.15.76:6789/0 conn(0x7f0ad80973c0 legacy=0x7f0ad80977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- 172.21.15.76:0/1542843048 shutdown_connections 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- 172.21.15.76:0/1542843048 >> 172.21.15.76:0/1542843048 conn(0x7f0ad80a0030 msgr2=0x7f0ad80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- 172.21.15.76:0/1542843048 shutdown_connections 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- 172.21.15.76:0/1542843048 wait complete. 2024-09-16T06:43:30.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: build_initial_monmap 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: monmap: 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 Processor -- start 2024-09-16T06:43:30.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- start start 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: init 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-c con 0x7f0ad80973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-a con 0x7f0ad8097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient(hunting): picked mon.noname-b con 0x7f0ad81400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad8072660 con 0x7f0ad8097eb0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad813fb70 con 0x7f0ad81400d0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad806bdb0 con 0x7f0ad80973c0 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0adeeae700 10 monclient(hunting): authenticate will time out at 1158.010836s 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0addeac700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ad80973c0 0x7f0ad81ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:50746/0 (socket says 172.21.15.76:50746) 2024-09-16T06:43:30.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0addeac700 1 -- 172.21.15.76:0/4014068868 learned_addr learned my addr 172.21.15.76:0/4014068868 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842495412 0 0) 0x7f0ad8072660 con 0x7f0ad8097eb0 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): my global_id is 154353 2024-09-16T06:43:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0abc0035d0 con 0x7f0ad8097eb0 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675353364 0 0) 0x7f0abc0035d0 con 0x7f0ad8097eb0 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ade6ad700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 legacy=0x7f0ad81d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ade6ad700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 legacy=0x7f0ad81d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ade6ad700 1 --1- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 0x7f0ad81d6bc0 :-1 s=OPENED pgs=483 cs=1 l=1).handle_message read tag failed 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ade6ad700 1 --1- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 0x7f0ad81d6bc0 :-1 s=OPENED pgs=483 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ad8072660 con 0x7f0ad8097eb0 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad4001b60 con 0x7f0ad8097eb0 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): dump: 2024-09-16T06:43:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.872+0000 7f0ace7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902849424 0 0) 0x7f0ad8072660 con 0x7f0ad8097eb0 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.76:6790/0 conn(0x7f0ad80973c0 legacy=0x7f0ad81ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:30.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6789/0 conn(0x7f0ad81400d0 legacy=0x7f0ad81d6bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 monclient: found mon.a 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad813eb40 con 0x7f0ad8097eb0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 5 monclient: authenticate success, global_id 154353 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _renew_subs 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad809a540 con 0x7f0ad8097eb0 2024-09-16T06:43:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _renew_subs 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad806c630 con 0x7f0ad8097eb0 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ad4004c10 con 0x7f0ad8097eb0 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad4004fa0 con 0x7f0ad8097eb0 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0acffff700 4 set_mon_vals no callback set 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 10 monclient: dump: 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0005100 con 0x7f0ad8097eb0 2024-09-16T06:43:30.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ad4005100 con 0x7f0ad8097eb0 2024-09-16T06:43:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7f0ad4044260 con 0x7f0ad8097eb0 2024-09-16T06:43:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.876+0000 7f0addeac700 10 monclient: get_auth_request con 0x7f0abc0360a0 auth_method 0 2024-09-16T06:43:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.884+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ad4014a50 con 0x7f0ad8097eb0 2024-09-16T06:43:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.884+0000 7f0ace7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:30.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:30.884+0000 7f0ace7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0adeeae700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0adeeae700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0adeeae700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ac0005930 con 0x7f0ad8097eb0 2024-09-16T06:43:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0ace7fc700 1 -- 172.21.15.76:0/4014068868 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1261213218) 0x7f0ad401d0a0 con 0x7f0ad8097eb0 2024-09-16T06:43:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0ace7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.116+0000 7f0ace7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:31.122 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0abc0360a0 legacy=0x7f0abc038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 10 monclient: shutdown 2024-09-16T06:43:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 >> v1:172.21.15.76:6789/0 conn(0x7f0ad8097eb0 legacy=0x7f0ad81d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 shutdown_connections 2024-09-16T06:43:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 >> 172.21.15.76:0/4014068868 conn(0x7f0ad80a0030 msgr2=0x7f0ad809c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 shutdown_connections 2024-09-16T06:43:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:31.120+0000 7f0adeeae700 1 -- 172.21.15.76:0/4014068868 wait complete. 2024-09-16T06:43:31.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:31.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:31.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:31.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:31.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:32.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:32.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:32.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:32.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:32.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:32.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:33.730+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:33.730+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:33.730+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:33.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:33.730+0000 7fbdecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:34.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: get_monmap_and_config 2024-09-16T06:43:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: build_initial_monmap 2024-09-16T06:43:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: monmap: 2024-09-16T06:43:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:34.253667+0000 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:34.253667+0000 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:34.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 1 Processor -- start 2024-09-16T06:43:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 1 -- start start 2024-09-16T06:43:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: init 2024-09-16T06:43:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-a con 0x7f6d8c09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-c con 0x7f6d8c09d520 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c09fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c09b520 con 0x7f6d8c09f230 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c071f30 con 0x7f6d8c09d520 2024-09-16T06:43:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c09b0e0 con 0x7f6d8c09fd20 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d9222f700 10 monclient(hunting): authenticate will time out at 1161.387031s 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6d8c09f230 0x7f6d8c09f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35072/0 (socket says 172.21.15.76:35072) 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8b7fe700 1 -- 172.21.15.76:0/2619958135 learned_addr learned my addr 172.21.15.76:0/2619958135 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991983471 0 0) 0x7f6d8c09b520 con 0x7f6d8c09f230 2024-09-16T06:43:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): my global_id is 154356 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d780035d0 con 0x7f6d8c09f230 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523441859 0 0) 0x7f6d8c071f30 con 0x7f6d8c09d520 2024-09-16T06:43:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient(hunting): my global_id is 164104 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c09b520 con 0x7f6d8c09d520 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186708040 0 0) 0x7f6d780035d0 con 0x7f6d8c09f230 2024-09-16T06:43:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 >> v1:172.21.15.76:6790/0 conn(0x7f6d8c09fd20 legacy=0x7f6d8c13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 >> v1:172.21.15.189:6789/0 conn(0x7f6d8c09d520 legacy=0x7f6d8c137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:34.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c09c7d0 con 0x7f6d8c09f230 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d84001990 con 0x7f6d8c09f230 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.248+0000 7f6d8a7fc700 10 monclient: dump: 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:34.276 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 5 monclient: authenticate success, global_id 154356 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d84001c00 con 0x7f6d8c09f230 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 1 -- 172.21.15.76:0/2619958135 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d84001f90 con 0x7f6d8c09f230 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d8a7fc700 10 monclient: dump: 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.277 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 4 set_mon_vals no callback set 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: shutdown 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- 172.21.15.76:0/2619958135 >> v1:172.21.15.76:6789/0 conn(0x7f6d8c09f230 legacy=0x7f6d8c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- 172.21.15.76:0/2619958135 shutdown_connections 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- 172.21.15.76:0/2619958135 >> 172.21.15.76:0/2619958135 conn(0x7f6d8c0a3e50 msgr2=0x7f6d8c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:34.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- 172.21.15.76:0/2619958135 shutdown_connections 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- 172.21.15.76:0/2619958135 wait complete. 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: build_initial_monmap 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: monmap: 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 Processor -- start 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- start start 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: init 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-c con 0x7f6d8c09fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient(hunting): picked mon.noname-a con 0x7f6d8c132af0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c1c6720 con 0x7f6d8c132af0 2024-09-16T06:43:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c072450 con 0x7f6d8c09f230 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c11fdc0 con 0x7f6d8c09fd20 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.252+0000 7f6d9222f700 10 monclient(hunting): authenticate will time out at 1161.391880s 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d8affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6d8c09fd20 0x7f6d8c1cab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55522/0 (socket says 172.21.15.76:55522) 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d8affd700 1 -- 172.21.15.76:0/3006677956 learned_addr learned my addr 172.21.15.76:0/3006677956 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92669769 0 0) 0x7f6d8c072450 con 0x7f6d8c09f230 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): my global_id is 164107 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c0035d0 con 0x7f6d8c09f230 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849964344 0 0) 0x7f6d8c1c6720 con 0x7f6d8c132af0 2024-09-16T06:43:34.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): my global_id is 154359 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c072450 con 0x7f6d8c132af0 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872265730 0 0) 0x7f6d8c072450 con 0x7f6d8c132af0 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c1c6720 con 0x7f6d8c132af0 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d7c001720 con 0x7f6d8c132af0 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): dump: 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:34.282 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1175896275 0 0) 0x7f6d6c0035d0 con 0x7f6d8c09f230 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c072450 con 0x7f6d8c09f230 2024-09-16T06:43:34.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d84001950 con 0x7f6d8c09f230 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient(hunting): dump: 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:34.284 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2198608920 0 0) 0x7f6d8c1c6720 con 0x7f6d8c132af0 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 >> v1:172.21.15.76:6790/0 conn(0x7f6d8c09fd20 legacy=0x7f6d8c1cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 >> v1:172.21.15.189:6789/0 conn(0x7f6d8c09f230 legacy=0x7f6d8c1323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 monclient: found mon.a 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c070550 con 0x7f6d8c132af0 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: _finish_auth 0 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 5 monclient: authenticate success, global_id 154359 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _renew_subs 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c0a2d40 con 0x7f6d8c132af0 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _renew_subs 2024-09-16T06:43:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c11e3b0 con 0x7f6d8c132af0 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d7c0036c0 con 0x7f6d8c132af0 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d7c003a50 con 0x7f6d8c132af0 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d89ffb700 4 set_mon_vals no callback set 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 10 monclient: dump: 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:34.288 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6d7c003c70 con 0x7f6d8c132af0 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d70005100 con 0x7f6d8c132af0 2024-09-16T06:43:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7f6d7c042b90 con 0x7f6d8c132af0 2024-09-16T06:43:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.256+0000 7f6d8b7fe700 10 monclient: get_auth_request con 0x7f6d6c0362c0 auth_method 0 2024-09-16T06:43:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.264+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6d7c013540 con 0x7f6d8c132af0 2024-09-16T06:43:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.264+0000 7f6d83fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.264+0000 7f6d83fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:34.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.496+0000 7f6d9222f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:34.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.496+0000 7f6d9222f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:34.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.496+0000 7f6d9222f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:34.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.496+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6d70005930 con 0x7f6d8c132af0 2024-09-16T06:43:34.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d83fff700 1 -- 172.21.15.76:0/3006677956 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1994806257) 0x7f6d7c01b490 con 0x7f6d8c132af0 2024-09-16T06:43:34.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d83fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:34.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d83fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:34.504 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6d6c0362c0 legacy=0x7f6d6c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 10 monclient: shutdown 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 >> v1:172.21.15.76:6789/0 conn(0x7f6d8c132af0 legacy=0x7f6d8c1ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 shutdown_connections 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 >> 172.21.15.76:0/3006677956 conn(0x7f6d8c0a3e50 msgr2=0x7f6d8c070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:34.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 shutdown_connections 2024-09-16T06:43:34.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:34.500+0000 7f6d9222f700 1 -- 172.21.15.76:0/3006677956 wait complete. 2024-09-16T06:43:34.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:34.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:34.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:34.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:34.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:34.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:35.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:35.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:35.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:35.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:35.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:35.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:36.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:36.733+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:36.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:36.733+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:36.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:36.733+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:37.518 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:37.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: get_monmap_and_config 2024-09-16T06:43:37.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: build_initial_monmap 2024-09-16T06:43:37.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: monmap: 2024-09-16T06:43:37.640 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:37.642077+0000 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:37.642077+0000 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:37.641 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 1 Processor -- start 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 1 -- start start 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: init 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-c con 0x7f3ff0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-a con 0x7f3ff0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:37.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-b con 0x7f3ff0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff0098990 con 0x7f3ff0099090 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff0097f40 con 0x7f3ff0097b60 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff009d610 con 0x7f3ff0139500 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff5760700 10 monclient(hunting): authenticate will time out at 1164.775310s 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3fef7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3ff0099090 0x7f3ff009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35088/0 (socket says 172.21.15.76:35088) 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3ff4f5f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3ff0139500 0x7f3ff013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55538/0 (socket says 172.21.15.76:55538) 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.635+0000 7f3fef7fe700 1 -- 172.21.15.76:0/3056844745 learned_addr learned my addr 172.21.15.76:0/3056844745 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:37.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748083900 0 0) 0x7f3ff0097f40 con 0x7f3ff0097b60 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient(hunting): my global_id is 164110 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fdc0035d0 con 0x7f3ff0097b60 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877298179 0 0) 0x7f3fdc0035d0 con 0x7f3ff0097b60 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 >> v1:172.21.15.76:6790/0 conn(0x7f3ff0139500 legacy=0x7f3ff013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 >> v1:172.21.15.76:6789/0 conn(0x7f3ff0099090 legacy=0x7f3ff009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 monclient: found mon.noname-c 2024-09-16T06:43:37.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff00986b0 con 0x7f3ff0097b60 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 5 monclient: authenticate success, global_id 164110 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:37.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fe0001610 con 0x7f3ff0097b60 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: dump: 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:37.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fe0001830 con 0x7f3ff0097b60 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 4 set_mon_vals no callback set 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 1 -- 172.21.15.76:0/3056844745 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fe0001bc0 con 0x7f3ff0097b60 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:37.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3feeffd700 10 monclient: dump: 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:37.651 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: shutdown 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- 172.21.15.76:0/3056844745 >> v1:172.21.15.189:6789/0 conn(0x7f3ff0097b60 legacy=0x7f3ff009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- 172.21.15.76:0/3056844745 shutdown_connections 2024-09-16T06:43:37.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- 172.21.15.76:0/3056844745 >> 172.21.15.76:0/3056844745 conn(0x7f3ff00a0030 msgr2=0x7f3ff00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- 172.21.15.76:0/3056844745 shutdown_connections 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- 172.21.15.76:0/3056844745 wait complete. 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: build_initial_monmap 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: monmap: 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:37.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 Processor -- start 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- start start 2024-09-16T06:43:37.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: init 2024-09-16T06:43:37.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:37.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:37.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-b con 0x7f3ff0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:37.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-a con 0x7f3ff0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient(hunting): picked mon.noname-c con 0x7f3ff0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff009c140 con 0x7f3ff0099090 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff013c320 con 0x7f3ff0097b60 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff009b250 con 0x7f3ff0139500 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff5760700 10 monclient(hunting): authenticate will time out at 1164.779659s 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff4f5f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3ff0139500 0x7f3ff01d2a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55548/0 (socket says 172.21.15.76:55548) 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.639+0000 7f3ff4f5f700 1 -- 172.21.15.76:0/1862823257 learned_addr learned my addr 172.21.15.76:0/1862823257 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:37.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557620646 0 0) 0x7f3ff013c320 con 0x7f3ff0097b60 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): my global_id is 164113 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd00035d0 con 0x7f3ff0097b60 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696605660 0 0) 0x7f3fd00035d0 con 0x7f3ff0097b60 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff013c320 con 0x7f3ff0097b60 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fe0001610 con 0x7f3ff0097b60 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): dump: 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:37.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 636457612 0 0) 0x7f3ff013c320 con 0x7f3ff0097b60 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 >> v1:172.21.15.76:6790/0 conn(0x7f3ff0139500 legacy=0x7f3ff01d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 >> v1:172.21.15.76:6789/0 conn(0x7f3ff0099090 legacy=0x7f3ff01cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: _finish_hunting 0 2024-09-16T06:43:37.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 monclient: found mon.b 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff009c700 con 0x7f3ff0097b60 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: _finish_auth 0 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 5 monclient: authenticate success, global_id 164113 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _renew_subs 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff009c420 con 0x7f3ff0097b60 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _renew_subs 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff009afb0 con 0x7f3ff0097b60 2024-09-16T06:43:37.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fe0001230 con 0x7f3ff0097b60 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3fe0003c10 con 0x7f3ff0097b60 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fee7fc700 4 set_mon_vals no callback set 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 10 monclient: dump: 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:37.663 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4005100 con 0x7f3ff0097b60 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3fe0003d70 con 0x7f3ff0097b60 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.643+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7f3fe0042fa0 con 0x7f3ff0097b60 2024-09-16T06:43:37.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.647+0000 7f3fef7fe700 10 monclient: get_auth_request con 0x7f3fd0036100 auth_method 0 2024-09-16T06:43:37.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.655+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3fe0005310 con 0x7f3ff0097b60 2024-09-16T06:43:37.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.655+0000 7f3fecff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:37.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.655+0000 7f3fecff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:37.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:37.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:37.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:37.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:37.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:37.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:37.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:37.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3fd4005930 con 0x7f3ff0097b60 2024-09-16T06:43:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3fecff9700 1 -- 172.21.15.76:0/1862823257 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 646003319) 0x7f3fe0011d20 con 0x7f3ff0097b60 2024-09-16T06:43:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3fecff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3fecff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:37.892 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3fd0036100 legacy=0x7f3fd00385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 10 monclient: shutdown 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 >> v1:172.21.15.189:6789/0 conn(0x7f3ff0097b60 legacy=0x7f3ff0096310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 shutdown_connections 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 >> 172.21.15.76:0/1862823257 conn(0x7f3ff00a0030 msgr2=0x7f3ff00a11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 shutdown_connections 2024-09-16T06:43:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:37.887+0000 7f3ff5760700 1 -- 172.21.15.76:0/1862823257 wait complete. 2024-09-16T06:43:38.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:38.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:38.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:38.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:38.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:38.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:39.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:39.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:39.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:39.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:39.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:39.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:40.733+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:40.733+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:40.733+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:40.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: get_monmap_and_config 2024-09-16T06:43:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: build_initial_monmap 2024-09-16T06:43:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: monmap: 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:41.025209+0000 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:41.025209+0000 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:41.024 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 1 Processor -- start 2024-09-16T06:43:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 1 -- start start 2024-09-16T06:43:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: init 2024-09-16T06:43:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6324097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6324099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6324139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6324098990 con 0x7f6324099090 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6324097f40 con 0x7f6324139500 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632409d610 con 0x7f6324097b60 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f6323fff700 10 monclient(hunting): authenticate will time out at 1168.158475s 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f63227fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6324099090 0x7f632409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35104/0 (socket says 172.21.15.76:35104) 2024-09-16T06:43:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.023+0000 7f63227fc700 1 -- 172.21.15.76:0/3346992976 learned_addr learned my addr 172.21.15.76:0/3346992976 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:41.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.729+0000 7fbdecff9700 10 monclient: tick 2024-09-16T06:43:41.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.729+0000 7fbdecff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:41.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.729+0000 7fbdecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:41.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.813+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+7 (unknown 914116082 0 1352219128) 0x7fbdf0012d70 con 0x7fbdfc09e5f0 2024-09-16T06:43:41.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.813+0000 7fbdedffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:41.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.813+0000 7fbdedffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:41.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058916730 0 0) 0x7f6324098990 con 0x7f6324099090 2024-09-16T06:43:41.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:41.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:41.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:41.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): my global_id is 154362 2024-09-16T06:43:41.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63100035d0 con 0x7f6324099090 2024-09-16T06:43:41.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314059405 0 0) 0x7f632409d610 con 0x7f6324097b60 2024-09-16T06:43:41.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:41.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:41.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:41.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient(hunting): my global_id is 125210 2024-09-16T06:43:41.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324098990 con 0x7f6324097b60 2024-09-16T06:43:41.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428607559 0 0) 0x7f6324098990 con 0x7f6324097b60 2024-09-16T06:43:41.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 >> v1:172.21.15.189:6789/0 conn(0x7f6324139500 legacy=0x7f632413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 >> v1:172.21.15.76:6789/0 conn(0x7f6324099090 legacy=0x7f632409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:43:41.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 monclient: found mon.noname-b 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240986b0 con 0x7f6324097b60 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: _finish_auth 0 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6323fff700 5 monclient: authenticate success, global_id 125210 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f630c0015f0 con 0x7f6324097b60 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6323fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:41.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6321ffb700 10 monclient: dump: 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:41.828 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:41.829 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.823+0000 7f6323fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f630c001810 con 0x7f6324097b60 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 1 -- 172.21.15.76:0/3346992976 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f630c001ba0 con 0x7f6324097b60 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 4 set_mon_vals no callback set 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:41.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6321ffb700 10 monclient: dump: 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:41.835 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:41.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 10 monclient: shutdown 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 -- 172.21.15.76:0/3346992976 >> v1:172.21.15.76:6790/0 conn(0x7f6324097b60 legacy=0x7f632409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 -- 172.21.15.76:0/3346992976 shutdown_connections 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 -- 172.21.15.76:0/3346992976 >> 172.21.15.76:0/3346992976 conn(0x7f63240a0030 msgr2=0x7f63240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 -- 172.21.15.76:0/3346992976 shutdown_connections 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 -- 172.21.15.76:0/3346992976 wait complete. 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 10 monclient: build_initial_monmap 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 10 monclient: monmap: 2024-09-16T06:43:41.837 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.827+0000 7f6323fff700 1 Processor -- start 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- start start 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: init 2024-09-16T06:43:41.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6324097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6324099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6324139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406ed00 con 0x7f6324139500 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406f930 con 0x7f6324099090 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406ccb0 con 0x7f6324097b60 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient(hunting): authenticate will time out at 1168.964689s 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f63237fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6324139500 0x7f63241cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35108/0 (socket says 172.21.15.76:35108) 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f63237fe700 1 -- 172.21.15.76:0/3609733794 learned_addr learned my addr 172.21.15.76:0/3609733794 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:41.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121091664 0 0) 0x7f632406ed00 con 0x7f6324139500 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): my global_id is 154365 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63040035d0 con 0x7f6324139500 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624009599 0 0) 0x7f632406ccb0 con 0x7f6324097b60 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): my global_id is 125213 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632406ed00 con 0x7f6324097b60 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3316042128 0 0) 0x7f63040035d0 con 0x7f6324139500 2024-09-16T06:43:41.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632406ccb0 con 0x7f6324139500 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6314001890 con 0x7f6324139500 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): dump: 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:41.841 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026974123 0 0) 0x7f632406ed00 con 0x7f6324097b60 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63040035d0 con 0x7f6324097b60 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f630c0014f0 con 0x7f6324097b60 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient(hunting): dump: 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.842 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167288332 0 0) 0x7f632406ccb0 con 0x7f6324139500 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 >> v1:172.21.15.76:6790/0 conn(0x7f6324097b60 legacy=0x7f632406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 >> v1:172.21.15.189:6789/0 conn(0x7f6324099090 legacy=0x7f632406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 monclient: found mon.a 2024-09-16T06:43:41.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632406f230 con 0x7f6324139500 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 5 monclient: authenticate success, global_id 154365 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _renew_subs 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632406f510 con 0x7f6324139500 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _renew_subs 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63140036c0 con 0x7f6324139500 2024-09-16T06:43:41.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324071ba0 con 0x7f6324139500 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6314003a50 con 0x7f6324139500 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 10 monclient: dump: 2024-09-16T06:43:41.845 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f63217fa700 4 set_mon_vals no callback set 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6314003bb0 con 0x7f6324139500 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:41.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7f6314043eb0 con 0x7f6324139500 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.831+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63240008d0 con 0x7f6324139500 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.835+0000 7f6322ffd700 10 monclient: get_auth_request con 0x7f6304036280 auth_method 0 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.839+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6314014700 con 0x7f6324139500 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.839+0000 7f631b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:41.839+0000 7f631b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:41.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fbddc005420 con 0x7fbdfc09e5f0 2024-09-16T06:43:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbdedffb700 1 -- 172.21.15.189:0/3218171069 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+7 (unknown 914116082 0 1352219128) 0x7fbdf00144f0 con 0x7fbdfc09e5f0 2024-09-16T06:43:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbdedffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:41.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbdedffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbde00362c0 legacy=0x7fbde0038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 10 monclient: shutdown 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 >> v1:172.21.15.76:6790/0 conn(0x7fbdfc09e5f0 legacy=0x7fbdfc06ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 shutdown_connections 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 >> 172.21.15.189:0/3218171069 conn(0x7fbdfc0a0eb0 msgr2=0x7fbdfc12d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.925+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 shutdown_connections 2024-09-16T06:43:41.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:41.929+0000 7fbe02119700 1 -- 172.21.15.189:0/3218171069 wait complete. 2024-09-16T06:43:41.942 INFO:tasks.workunit.client.0.smithi189.stdout:barset 2024-09-16T06:43:41.942 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm barset 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 10 monclient: get_monmap_and_config 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 10 monclient: build_initial_monmap 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 10 monclient: monmap: 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:42.054782+0000 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:42.054782+0000 2024-09-16T06:43:42.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:42.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.055 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:42.055 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:42.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:42.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 1 Processor -- start 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 1 -- start start 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.049+0000 7f99c23f3700 10 monclient: init 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-b con 0x7f99bc0a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:42.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-c con 0x7f99bc09aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-a con 0x7f99bc0a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc09a040 con 0x7f99bc0a3ca0 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc0a33d0 con 0x7f99bc09aa20 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc099d60 con 0x7f99bc0a2350 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient(hunting): authenticate will time out at 1174.568380s 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99bb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99bc09aa20 0x7f99bc13a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38430/0 (socket says 172.21.15.189:38430) 2024-09-16T06:43:42.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99bb7fe700 1 -- 172.21.15.189:0/3480666013 learned_addr learned my addr 172.21.15.189:0/3480666013 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:42.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929355093 0 0) 0x7f99bc09a040 con 0x7f99bc0a3ca0 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): my global_id is 154368 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a40035d0 con 0x7f99bc0a3ca0 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261740477 0 0) 0x7f99bc099d60 con 0x7f99bc0a2350 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient(hunting): my global_id is 125216 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc09a040 con 0x7f99bc0a2350 2024-09-16T06:43:42.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054386981 0 0) 0x7f99bc09a040 con 0x7f99bc0a2350 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 >> v1:172.21.15.189:6789/0 conn(0x7f99bc09aa20 legacy=0x7f99bc13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 >> v1:172.21.15.76:6789/0 conn(0x7f99bc0a3ca0 legacy=0x7f99bc13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 monclient: found mon.noname-b 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc0a3920 con 0x7f99bc0a2350 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f99ac0015f0 con 0x7f99bc0a2350 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 5 monclient: authenticate success, global_id 125216 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:42.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: dump: 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99ac001820 con 0x7f99bc0a2350 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 1 -- 172.21.15.189:0/3480666013 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f99ac001bf0 con 0x7f99bc0a2350 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 4 set_mon_vals no callback set 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99baffd700 10 monclient: dump: 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.065 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: shutdown 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- 172.21.15.189:0/3480666013 >> v1:172.21.15.76:6790/0 conn(0x7f99bc0a2350 legacy=0x7f99bc0a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- 172.21.15.189:0/3480666013 shutdown_connections 2024-09-16T06:43:42.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- 172.21.15.189:0/3480666013 >> 172.21.15.189:0/3480666013 conn(0x7f99bc0a7d70 msgr2=0x7f99bc0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- 172.21.15.189:0/3480666013 shutdown_connections 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 1 -- 172.21.15.189:0/3480666013 wait complete. 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: build_initial_monmap 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.053+0000 7f99c23f3700 10 monclient: monmap: 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:42.067 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 Processor -- start 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- start start 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: init 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-a con 0x7f99bc09aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-c con 0x7f99bc0a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient(hunting): picked mon.noname-b con 0x7f99bc0a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc121a70 con 0x7f99bc09aa20 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc121fa0 con 0x7f99bc0a3ca0 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99bc06bed0 con 0x7f99bc0a2350 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:42.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient(hunting): authenticate will time out at 1174.573023s 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c0990700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99bc0a3ca0 0x7f99bc1d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38436/0 (socket says 172.21.15.189:38436) 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c0990700 1 -- 172.21.15.189:0/1969091018 learned_addr learned my addr 172.21.15.189:0/1969091018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952184342 0 0) 0x7f99bc06bed0 con 0x7f99bc0a2350 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): my global_id is 125219 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a80035d0 con 0x7f99bc0a2350 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323952648 0 0) 0x7f99bc121a70 con 0x7f99bc09aa20 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:42.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:42.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): my global_id is 154371 2024-09-16T06:43:42.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc06bed0 con 0x7f99bc09aa20 2024-09-16T06:43:42.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218387375 0 0) 0x7f99bc06bed0 con 0x7f99bc09aa20 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99bc121a70 con 0x7f99bc09aa20 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3837099078 0 0) 0x7f99a80035d0 con 0x7f99bc0a2350 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99bc06bed0 con 0x7f99bc0a2350 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f99ac001c30 con 0x7f99bc09aa20 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.073 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f99b0001760 con 0x7f99bc0a2350 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:42.074 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2408076260 0 0) 0x7f99bc121a70 con 0x7f99bc09aa20 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 >> v1:172.21.15.76:6790/0 conn(0x7f99bc0a2350 legacy=0x7f99bc1cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 >> v1:172.21.15.189:6789/0 conn(0x7f99bc0a3ca0 legacy=0x7f99bc1d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:43:42.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 monclient: found mon.a 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc11fbd0 con 0x7f99bc09aa20 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 5 monclient: authenticate success, global_id 154371 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _renew_subs 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc121700 con 0x7f99bc09aa20 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _renew_subs 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc09b5b0 con 0x7f99bc09aa20 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99ac0039d0 con 0x7f99bc09aa20 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f99ac003d60 con 0x7f99bc09aa20 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99b8ff9700 10 monclient: dump: 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:42.077 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.057+0000 7f99ba7fc700 4 set_mon_vals no callback set 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99c23f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99c23f3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99c23f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c005100 con 0x7f99bc09aa20 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f99ac003ec0 con 0x7f99bc09aa20 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(130..130 src has 66..130) v4 ==== 6989+0+0 (unknown 1510088577 0 0) 0x7f99ac042f30 con 0x7f99bc09aa20 2024-09-16T06:43:42.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.061+0000 7f99bb7fe700 10 monclient: get_auth_request con 0x7f99a8036300 auth_method 0 2024-09-16T06:43:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.069+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f99ac0054b0 con 0x7f99bc09aa20 2024-09-16T06:43:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.069+0000 7f99b8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:42.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.069+0000 7f99b8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:42.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.071+0000 7f6323fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.071+0000 7f6323fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.071+0000 7f6323fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.071+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f632406d390 con 0x7f6324139500 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f631b7fe700 1 -- 172.21.15.76:0/3609733794 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 3667937097) 0x7f6314012b70 con 0x7f6324139500 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f631b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f631b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:42.080 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:42.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6304036280 legacy=0x7f6304038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 10 monclient: shutdown 2024-09-16T06:43:42.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:42.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 >> v1:172.21.15.76:6789/0 conn(0x7f6324139500 legacy=0x7f63241cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:42.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 shutdown_connections 2024-09-16T06:43:42.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 >> 172.21.15.76:0/3609733794 conn(0x7f63240a0030 msgr2=0x7f63240a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:42.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 shutdown_connections 2024-09-16T06:43:42.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:42.075+0000 7f6323fff700 1 -- 172.21.15.76:0/3609733794 wait complete. 2024-09-16T06:43:42.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.245+0000 7f99c23f3700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:42.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.245+0000 7f99c23f3700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:42.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.245+0000 7f99c23f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:42.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.245+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f999c0053b0 con 0x7f99bc09aa20 2024-09-16T06:43:42.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.889+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v131) v1 ==== 89+0+0 (unknown 1639954963 0 0) 0x7f99ac011cf0 con 0x7f99bc09aa20 2024-09-16T06:43:42.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.889+0000 7f99b8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:42.894 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:42.889+0000 7f99b8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:43.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.009+0000 7f99c23f3700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:43.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.009+0000 7f99c23f3700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:43.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.009+0000 7f99c23f3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:43.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.009+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f999c005900 con 0x7f99bc09aa20 2024-09-16T06:43:43.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.897+0000 7f99b8ff9700 1 -- 172.21.15.189:0/1969091018 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v132) v1 ==== 89+0+0 (unknown 2869787261 0 0) 0x7f99ac013880 con 0x7f99bc09aa20 2024-09-16T06:43:43.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.897+0000 7f99b8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T06:43:43.900 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.897+0000 7f99b8ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 >> v1:172.21.15.189:6812/3098810674 conn(0x7f99a8036300 legacy=0x7f99a80387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 10 monclient: shutdown 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 >> v1:172.21.15.76:6789/0 conn(0x7f99bc09aa20 legacy=0x7f99bc1345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 shutdown_connections 2024-09-16T06:43:43.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 >> 172.21.15.189:0/1969091018 conn(0x7f99bc0a7d70 msgr2=0x7f99bc0a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 shutdown_connections 2024-09-16T06:43:43.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:43.905+0000 7f99c23f3700 1 -- 172.21.15.189:0/1969091018 wait complete. 2024-09-16T06:43:43.922 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:43.922 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -c barset 2024-09-16T06:43:43.922 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep -q 0 2024-09-16T06:43:44.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: get_monmap_and_config 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: build_initial_monmap 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: monmap: 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:44.029397+0000 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:44.029397+0000 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:44.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 1 Processor -- start 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 1 -- start start 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: init 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-c con 0x7fdc4c06a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-b con 0x7fdc4c09d5b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-a con 0x7fdc4c13bb20 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c06b880 con 0x7fdc4c13bb20 2024-09-16T06:43:44.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c06be90 con 0x7fdc4c06a340 2024-09-16T06:43:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c0a1a40 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc53652700 10 monclient(hunting): authenticate will time out at 1176.542675s 2024-09-16T06:43:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc513ee700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc4c06a340 0x7fdc4c06a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38442/0 (socket says 172.21.15.189:38442) 2024-09-16T06:43:44.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc513ee700 1 -- 172.21.15.189:0/421432481 learned_addr learned my addr 172.21.15.189:0/421432481 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817819370 0 0) 0x7fdc4c0a1a40 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): my global_id is 125222 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc380035d0 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967532291 0 0) 0x7fdc4c06b880 con 0x7fdc4c13bb20 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 10 monclient(hunting): my global_id is 154374 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.025+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc4c0a1a40 con 0x7fdc4c13bb20 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724974584 0 0) 0x7fdc380035d0 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 >> v1:172.21.15.189:6789/0 conn(0x7fdc4c06a340 legacy=0x7fdc4c06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 >> v1:172.21.15.76:6789/0 conn(0x7fdc4c13bb20 legacy=0x7fdc4c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 monclient: found mon.noname-b 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c0a16f0 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: _finish_auth 0 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:44.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 5 monclient: authenticate success, global_id 125222 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc3c001610 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: dump: 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:44.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc3c0018f0 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 1 -- 172.21.15.189:0/421432481 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc3c001c80 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 4 set_mon_vals no callback set 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc43fff700 10 monclient: dump: 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: shutdown 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- 172.21.15.189:0/421432481 >> v1:172.21.15.76:6790/0 conn(0x7fdc4c09d5b0 legacy=0x7fdc4c06ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- 172.21.15.189:0/421432481 shutdown_connections 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- 172.21.15.189:0/421432481 >> 172.21.15.189:0/421432481 conn(0x7fdc4c0a21f0 msgr2=0x7fdc4c0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- 172.21.15.189:0/421432481 shutdown_connections 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- 172.21.15.189:0/421432481 wait complete. 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: build_initial_monmap 2024-09-16T06:43:44.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: monmap: 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 Processor -- start 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- start start 2024-09-16T06:43:44.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: init 2024-09-16T06:43:44.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:44.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-a con 0x7fdc4c06a340 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-b con 0x7fdc4c09d5b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient(hunting): picked mon.noname-c con 0x7fdc4c13bb20 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c134460 con 0x7fdc4c06a340 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c137320 con 0x7fdc4c09d5b0 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c09f900 con 0x7fdc4c13bb20 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.029+0000 7fdc53652700 10 monclient(hunting): authenticate will time out at 1176.547917s 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc50bed700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc4c09d5b0 0x7fdc4c1cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38452/0 (socket says 172.21.15.189:38452) 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc50bed700 1 -- 172.21.15.189:0/119883893 learned_addr learned my addr 172.21.15.189:0/119883893 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783107492 0 0) 0x7fdc4c09f900 con 0x7fdc4c13bb20 2024-09-16T06:43:44.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): my global_id is 125225 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc300035d0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111482287 0 0) 0x7fdc4c06be90 con 0x7fdc4c06a340 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): my global_id is 154377 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc4c09f900 con 0x7fdc4c06a340 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56839793 0 0) 0x7fdc300035d0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc4c06be90 con 0x7fdc4c13bb20 2024-09-16T06:43:44.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc440019b0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): dump: 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270851471 0 0) 0x7fdc4c09f900 con 0x7fdc4c06a340 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc300035d0 con 0x7fdc4c06a340 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc48001640 con 0x7fdc4c06a340 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient(hunting): dump: 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 777140375 0 0) 0x7fdc4c06be90 con 0x7fdc4c13bb20 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 >> v1:172.21.15.189:6789/0 conn(0x7fdc4c09d5b0 legacy=0x7fdc4c1cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 >> v1:172.21.15.76:6789/0 conn(0x7fdc4c06a340 legacy=0x7fdc4c13aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 monclient: found mon.c 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c09c700 con 0x7fdc4c13bb20 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: _finish_auth 0 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 5 monclient: authenticate success, global_id 125225 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 10 monclient: _renew_subs 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c138ab0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 10 monclient: _renew_subs 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c13b700 con 0x7fdc4c13bb20 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc44003a20 con 0x7fdc4c13bb20 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc44003d30 con 0x7fdc4c13bb20 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc437fe700 4 set_mon_vals no callback set 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 10 monclient: dump: 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdc4400e7a0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(132..132 src has 66..132) v4 ==== 6377+0+0 (unknown 2102999204 0 0) 0x7fdc44042e40 con 0x7fdc4c13bb20 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.033+0000 7fdc513ee700 10 monclient: get_auth_request con 0x7fdc30036310 auth_method 0 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.037+0000 7fdc53652700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.037+0000 7fdc53652700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.037+0000 7fdc53652700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.037+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c005100 con 0x7fdc4c13bb20 2024-09-16T06:43:44.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.045+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdc440136c0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.045+0000 7fdc41ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.045+0000 7fdc41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:44.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc53652700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc53652700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc53652700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fdc2c005930 con 0x7fdc4c13bb20 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v132) v1 ==== 71+0+0 (unknown 1799755844 0 0) 0x7fdc44011d60 con 0x7fdc4c13bb20 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc41ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.225+0000 7fdc41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.329+0000 7fdc53652700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.329+0000 7fdc53652700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.329+0000 7fdc53652700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.329+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fdc2c005650 con 0x7fdc4c13bb20 2024-09-16T06:43:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc41ffb700 1 -- 172.21.15.189:0/119883893 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v132) v1 ==== 71+0+0 (unknown 1799755844 0 0) 0x7fdc440134e0 con 0x7fdc4c13bb20 2024-09-16T06:43:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc41ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:44.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc41ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdc30036310 legacy=0x7fdc300387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 10 monclient: shutdown 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 >> v1:172.21.15.76:6790/0 conn(0x7fdc4c13bb20 legacy=0x7fdc4c1d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 shutdown_connections 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 >> 172.21.15.189:0/119883893 conn(0x7fdc4c0a21f0 msgr2=0x7fdc4c09e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 shutdown_connections 2024-09-16T06:43:44.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.333+0000 7fdc53652700 1 -- 172.21.15.189:0/119883893 wait complete. 2024-09-16T06:43:44.355 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T06:43:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: get_monmap_and_config 2024-09-16T06:43:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: build_initial_monmap 2024-09-16T06:43:44.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: monmap: 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:44.457241+0000 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:44.457241+0000 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:44.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 1 Processor -- start 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 1 -- start start 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: init 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-c con 0x7fed68132080 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-a con 0x7fed68138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-b con 0x7fed6813bf70 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6806abd0 con 0x7fed68138460 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6806b170 con 0x7fed68132080 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed68132b70 con 0x7fed6813bf70 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient(hunting): authenticate will time out at 1176.970269s 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed677fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fed68132080 0x7fed68132460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38456/0 (socket says 172.21.15.189:38456) 2024-09-16T06:43:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed677fe700 1 -- 172.21.15.189:0/1876837494 learned_addr learned my addr 172.21.15.189:0/1876837494 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244397924 0 0) 0x7fed6806abd0 con 0x7fed68138460 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): my global_id is 154380 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed500035d0 con 0x7fed68138460 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874287487 0 0) 0x7fed68132b70 con 0x7fed6813bf70 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient(hunting): my global_id is 125228 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6806abd0 con 0x7fed6813bf70 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219223367 0 0) 0x7fed6806abd0 con 0x7fed6813bf70 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 >> v1:172.21.15.189:6789/0 conn(0x7fed68132080 legacy=0x7fed68132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 >> v1:172.21.15.76:6789/0 conn(0x7fed68138460 legacy=0x7fed6813a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 monclient: found mon.noname-b 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680a38b0 con 0x7fed6813bf70 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 5 monclient: authenticate success, global_id 125228 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed60001970 con 0x7fed6813bf70 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: dump: 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed60001c90 con 0x7fed6813bf70 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 1 -- 172.21.15.189:0/1876837494 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed60002020 con 0x7fed6813bf70 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed667fc700 10 monclient: dump: 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.453+0000 7fed6cc63700 4 set_mon_vals no callback set 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 10 monclient: shutdown 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- 172.21.15.189:0/1876837494 >> v1:172.21.15.76:6790/0 conn(0x7fed6813bf70 legacy=0x7fed6813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- 172.21.15.189:0/1876837494 shutdown_connections 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- 172.21.15.189:0/1876837494 >> 172.21.15.189:0/1876837494 conn(0x7fed680a4660 msgr2=0x7fed68099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- 172.21.15.189:0/1876837494 shutdown_connections 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- 172.21.15.189:0/1876837494 wait complete. 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 10 monclient: build_initial_monmap 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 10 monclient: monmap: 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:44.467 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:44.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 Processor -- start 2024-09-16T06:43:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 1 -- start start 2024-09-16T06:43:44.473 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.457+0000 7fed6cc63700 10 monclient: init 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-c con 0x7fed68132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-a con 0x7fed68138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient(hunting): picked mon.noname-b con 0x7fed6813bf70 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed680a1870 con 0x7fed68138460 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6813bb90 con 0x7fed6813bf70 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed68135c30 con 0x7fed68132080 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient(hunting): authenticate will time out at 1176.976853s 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed67fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fed6813bf70 0x7fed681d2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38460/0 (socket says 172.21.15.189:38460) 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed67fff700 1 -- 172.21.15.189:0/1729874358 learned_addr learned my addr 172.21.15.189:0/1729874358 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:44.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806834281 0 0) 0x7fed68135c30 con 0x7fed68132080 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): my global_id is 125231 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed540035d0 con 0x7fed68132080 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729997807 0 0) 0x7fed680a1870 con 0x7fed68138460 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): my global_id is 154383 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed68135c30 con 0x7fed68138460 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2461927934 0 0) 0x7fed540035d0 con 0x7fed68132080 2024-09-16T06:43:44.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed680a1870 con 0x7fed68132080 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed5c0017e0 con 0x7fed68132080 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.476 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873009363 0 0) 0x7fed68135c30 con 0x7fed68138460 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed540035d0 con 0x7fed68138460 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed58001850 con 0x7fed68138460 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.477 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46073046 0 0) 0x7fed680a1870 con 0x7fed68132080 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 >> v1:172.21.15.189:6789/0 conn(0x7fed6813bf70 legacy=0x7fed681d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 >> v1:172.21.15.76:6789/0 conn(0x7fed68138460 legacy=0x7fed68133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 monclient: found mon.c 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680a0080 con 0x7fed68132080 2024-09-16T06:43:44.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: _finish_auth 0 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 5 monclient: authenticate success, global_id 125231 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _renew_subs 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed68134590 con 0x7fed68132080 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _renew_subs 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed681382b0 con 0x7fed68132080 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed5c001cb0 con 0x7fed68132080 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed5c004a30 con 0x7fed68132080 2024-09-16T06:43:44.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed657fa700 4 set_mon_vals no callback set 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.461+0000 7fed4ffff700 10 monclient: dump: 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:44.480 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed6cc63700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed6cc63700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed6cc63700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48005100 con 0x7fed68132080 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fed5c004c10 con 0x7fed68132080 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(132..132 src has 66..132) v4 ==== 6377+0+0 (unknown 2102999204 0 0) 0x7fed5c043c40 con 0x7fed68132080 2024-09-16T06:43:44.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.465+0000 7fed66ffd700 10 monclient: get_auth_request con 0x7fed540362f0 auth_method 0 2024-09-16T06:43:44.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.473+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fed5c012b80 con 0x7fed68132080 2024-09-16T06:43:44.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.473+0000 7fed4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:44.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.473+0000 7fed4ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.649+0000 7fed6cc63700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.649+0000 7fed6cc63700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.649+0000 7fed6cc63700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:44.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.649+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fed480053b0 con 0x7fed68132080 2024-09-16T06:43:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.897+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v133) v1 ==== 82+0+0 (unknown 1623075820 0 0) 0x7fed5c012b80 con 0x7fed68132080 2024-09-16T06:43:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.897+0000 7fed4ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:44.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:44.897+0000 7fed4ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed6cc63700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:45.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed6cc63700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:45.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed6cc63700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fed48005b90 con 0x7fed68132080 2024-09-16T06:43:45.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed4ffff700 1 -- 172.21.15.189:0/1729874358 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v133) v1 ==== 115+0+0 (unknown 348221919 0 0) 0x7fed5c015040 con 0x7fed68132080 2024-09-16T06:43:45.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed4ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:43:45.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.005+0000 7fed4ffff700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T06:43:45.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 >> v1:172.21.15.189:6812/3098810674 conn(0x7fed540362f0 legacy=0x7fed540387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 10 monclient: shutdown 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 >> v1:172.21.15.76:6790/0 conn(0x7fed68132080 legacy=0x7fed68133770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 shutdown_connections 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 >> 172.21.15.189:0/1729874358 conn(0x7fed680a4660 msgr2=0x7fed68136dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 shutdown_connections 2024-09-16T06:43:45.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.009+0000 7fed6cc63700 1 -- 172.21.15.189:0/1729874358 wait complete. 2024-09-16T06:43:45.017 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T06:43:45.030 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set ls 2024-09-16T06:43:45.030 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '(compat)' 2024-09-16T06:43:45.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.129+0000 7f566db63700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.129+0000 7f566db63700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.129+0000 7f566db63700 10 monclient: monmap: 2024-09-16T06:43:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:45.135 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:45.136088+0000 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:45.136088+0000 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.136 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 Processor -- start 2024-09-16T06:43:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- start start 2024-09-16T06:43:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient: init 2024-09-16T06:43:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient(hunting): picked mon.noname-b con 0x7f566809bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient(hunting): picked mon.noname-a con 0x7f566809c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient(hunting): picked mon.noname-c con 0x7f5668138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566809cca0 con 0x7f566809c5f0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566809f750 con 0x7f5668138da0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5668097620 con 0x7f566809bb00 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient(hunting): authenticate will time out at 1177.649258s 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f5667fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5668138da0 0x7f566813b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38466/0 (socket says 172.21.15.189:38466) 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f5667fff700 1 -- 172.21.15.189:0/3417519312 learned_addr learned my addr 172.21.15.189:0/3417519312 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548172002 0 0) 0x7f566809f750 con 0x7f5668138da0 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient(hunting): my global_id is 164116 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56500035d0 con 0x7f5668138da0 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536901855 0 0) 0x7f56500035d0 con 0x7f5668138da0 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 >> v1:172.21.15.76:6790/0 conn(0x7f566809bb00 legacy=0x7f566809bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 >> v1:172.21.15.76:6789/0 conn(0x7f566809c5f0 legacy=0x7f5668137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 monclient: found mon.noname-c 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668097900 con 0x7f5668138da0 2024-09-16T06:43:45.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5660001990 con 0x7f5668138da0 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: dump: 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.141 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 5 monclient: authenticate success, global_id 164116 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5660001cb0 con 0x7f5668138da0 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 1 -- 172.21.15.189:0/3417519312 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5660002040 con 0x7f5668138da0 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 4 set_mon_vals no callback set 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f56667fc700 10 monclient: dump: 2024-09-16T06:43:45.142 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.143 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 10 monclient: shutdown 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- 172.21.15.189:0/3417519312 >> v1:172.21.15.189:6789/0 conn(0x7f5668138da0 legacy=0x7f566813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- 172.21.15.189:0/3417519312 shutdown_connections 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- 172.21.15.189:0/3417519312 >> 172.21.15.189:0/3417519312 conn(0x7f56680a03e0 msgr2=0x7f56680a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- 172.21.15.189:0/3417519312 shutdown_connections 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.133+0000 7f566db63700 1 -- 172.21.15.189:0/3417519312 wait complete. 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient: monmap: 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:45.148 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 1 Processor -- start 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 1 -- start start 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient: init 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:45.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient(hunting): picked mon.noname-c con 0x7f566809bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient(hunting): picked mon.noname-a con 0x7f566809c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient(hunting): picked mon.noname-b con 0x7f5668138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5668090d70 con 0x7f566809c5f0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566809cf50 con 0x7f5668138da0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f5667fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5668138da0 0x7f56681ce880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38468/0 (socket says 172.21.15.189:38468) 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566809a6e0 con 0x7f566809bb00 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f5667fff700 1 -- 172.21.15.189:0/1255617197 learned_addr learned my addr 172.21.15.189:0/1255617197 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f566db63700 10 monclient(hunting): authenticate will time out at 1177.655378s 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320114839 0 0) 0x7f566809cf50 con 0x7f5668138da0 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): my global_id is 164119 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56540035d0 con 0x7f5668138da0 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211425011 0 0) 0x7f566809a6e0 con 0x7f566809bb00 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.137+0000 7f564ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): my global_id is 125237 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566809cf50 con 0x7f566809bb00 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195233949 0 0) 0x7f5668090d70 con 0x7f566809c5f0 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): my global_id is 154389 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566809a6e0 con 0x7f566809c5f0 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702541996 0 0) 0x7f56540035d0 con 0x7f5668138da0 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5668090d70 con 0x7f5668138da0 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f56600017b0 con 0x7f5668138da0 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.152 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740016567 0 0) 0x7f566809a6e0 con 0x7f566809c5f0 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56540035d0 con 0x7f566809c5f0 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f565c0017a0 con 0x7f566809c5f0 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6529234 0 0) 0x7f566809cf50 con 0x7f566809bb00 2024-09-16T06:43:45.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f566809a6e0 con 0x7f566809bb00 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237341751 0 0) 0x7f5668090d70 con 0x7f5668138da0 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 >> v1:172.21.15.76:6790/0 conn(0x7f566809bb00 legacy=0x7f56681c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 >> v1:172.21.15.76:6789/0 conn(0x7f566809c5f0 legacy=0x7f56681cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 monclient: found mon.b 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56680904d0 con 0x7f5668138da0 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 5 monclient: authenticate success, global_id 164119 2024-09-16T06:43:45.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _renew_subs 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56680907b0 con 0x7f5668138da0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _renew_subs 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5668090080 con 0x7f5668138da0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5660005070 con 0x7f5668138da0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5660005380 con 0x7f5668138da0 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f5665ffb700 4 set_mon_vals no callback set 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 10 monclient: dump: 2024-09-16T06:43:45.156 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f566000fdf0 con 0x7f5668138da0 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5648005100 con 0x7f5668138da0 2024-09-16T06:43:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(133..133 src has 66..133) v4 ==== 6709+0+0 (unknown 3610394018 0 0) 0x7f56600443b0 con 0x7f5668138da0 2024-09-16T06:43:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.141+0000 7f56677fe700 10 monclient: get_auth_request con 0x7f56540364e0 auth_method 0 2024-09-16T06:43:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.149+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5660014ba0 con 0x7f5668138da0 2024-09-16T06:43:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.149+0000 7f564ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.149+0000 7f564ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: monmap: 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:45.217490+0000 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:45.217490+0000 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.216 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 1 Processor -- start 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 1 -- start start 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: init 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f4540132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f454006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f454006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454006c280 con 0x7f4540132330 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454006bfa0 con 0x7f454006cf50 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454009ace0 con 0x7f454006d480 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f453ffff700 10 monclient(hunting): authenticate will time out at 1172.351724s 2024-09-16T06:43:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f4537fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f454006d480 0x7f454013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44294/0 (socket says 172.21.15.76:44294) 2024-09-16T06:43:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.215+0000 7f4537fff700 1 -- 172.21.15.76:0/593954840 learned_addr learned my addr 172.21.15.76:0/593954840 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946344190 0 0) 0x7f454009ace0 con 0x7f454006d480 2024-09-16T06:43:45.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): my global_id is 125240 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45200035d0 con 0x7f454006d480 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780929022 0 0) 0x7f454006c280 con 0x7f4540132330 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): my global_id is 154392 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454009ace0 con 0x7f4540132330 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457754099 0 0) 0x7f454006bfa0 con 0x7f454006cf50 2024-09-16T06:43:45.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient(hunting): my global_id is 164122 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454006c280 con 0x7f454006cf50 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 921309430 0 0) 0x7f45200035d0 con 0x7f454006d480 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 >> v1:172.21.15.189:6789/0 conn(0x7f454006cf50 legacy=0x7f454013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 >> v1:172.21.15.76:6789/0 conn(0x7f4540132330 legacy=0x7f4540132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:43:45.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454009ae70 con 0x7f454006d480 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 5 monclient: authenticate success, global_id 125240 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4528001610 con 0x7f454006d480 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: dump: 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.231 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45280018f0 con 0x7f454006d480 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 4 set_mon_vals no callback set 2024-09-16T06:43:45.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 1 -- 172.21.15.76:0/593954840 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4528001ae0 con 0x7f454006d480 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453e7fc700 10 monclient: dump: 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: shutdown 2024-09-16T06:43:45.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- 172.21.15.76:0/593954840 >> v1:172.21.15.76:6790/0 conn(0x7f454006d480 legacy=0x7f454013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- 172.21.15.76:0/593954840 shutdown_connections 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- 172.21.15.76:0/593954840 >> 172.21.15.76:0/593954840 conn(0x7f45400a41c0 msgr2=0x7f45400a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- 172.21.15.76:0/593954840 shutdown_connections 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- 172.21.15.76:0/593954840 wait complete. 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: monmap: 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:45.234 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 Processor -- start 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- start start 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: init 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f454006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f454006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f4540132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4540139330 con 0x7f454006d480 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45401361f0 con 0x7f454006cf50 2024-09-16T06:43:45.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45401312d0 con 0x7f4540132330 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453ffff700 10 monclient(hunting): authenticate will time out at 1172.355633s 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4540132330 0x7f45401d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44304/0 (socket says 172.21.15.76:44304) 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.219+0000 7f453f7fe700 1 -- 172.21.15.76:0/4157186604 learned_addr learned my addr 172.21.15.76:0/4157186604 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359542062 0 0) 0x7f45401312d0 con 0x7f4540132330 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): my global_id is 125243 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45240035d0 con 0x7f4540132330 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546421132 0 0) 0x7f4540139330 con 0x7f454006d480 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): my global_id is 154395 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45401312d0 con 0x7f454006d480 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233724468 0 0) 0x7f45401361f0 con 0x7f454006cf50 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): my global_id is 164125 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4540139330 con 0x7f454006cf50 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1430754274 0 0) 0x7f45240035d0 con 0x7f4540132330 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45401361f0 con 0x7f4540132330 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4538001c50 con 0x7f4540132330 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): dump: 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857987319 0 0) 0x7f45401312d0 con 0x7f454006d480 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45240035d0 con 0x7f454006d480 2024-09-16T06:43:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1538148990 0 0) 0x7f4540139330 con 0x7f454006cf50 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45401312d0 con 0x7f454006cf50 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4528001020 con 0x7f454006d480 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): dump: 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4530001830 con 0x7f454006cf50 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient(hunting): dump: 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 934405825 0 0) 0x7f45401361f0 con 0x7f4540132330 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 >> v1:172.21.15.189:6789/0 conn(0x7f454006cf50 legacy=0x7f45401353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 >> v1:172.21.15.76:6789/0 conn(0x7f454006d480 legacy=0x7f4540135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 monclient: found mon.c 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45401317e0 con 0x7f4540132330 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 5 monclient: authenticate success, global_id 125243 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _renew_subs 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4540139050 con 0x7f4540132330 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _renew_subs 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f454013dfe0 con 0x7f4540132330 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4538001fe0 con 0x7f4540132330 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4538003e90 con 0x7f4540132330 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453dffb700 4 set_mon_vals no callback set 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 10 monclient: dump: 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.243 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4538003ff0 con 0x7f4540132330 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4518005100 con 0x7f4540132330 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(133..133 src has 66..133) v4 ==== 6709+0+0 (unknown 3610394018 0 0) 0x7f4538044200 con 0x7f4540132330 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.223+0000 7f453effd700 10 monclient: get_auth_request con 0x7f4524036630 auth_method 0 2024-09-16T06:43:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.231+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4538014b60 con 0x7f4540132330 2024-09-16T06:43:45.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.231+0000 7f453cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.231+0000 7f453cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f566db63700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f566db63700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f566db63700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5648005930 con 0x7f5668138da0 2024-09-16T06:43:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+9 (unknown 1899167245 0 1822970295) 0x7f5660013240 con 0x7f5668138da0 2024-09-16T06:43:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f564ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.329+0000 7f564ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f566db63700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f566db63700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f566db63700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5648005420 con 0x7f5668138da0 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f564ffff700 1 -- 172.21.15.189:0/1255617197 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+9 (unknown 1899167245 0 1822970295) 0x7f56600149c0 con 0x7f5668138da0 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f564ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T06:43:45.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.429+0000 7f564ffff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 >> v1:172.21.15.189:6812/3098810674 conn(0x7f56540364e0 legacy=0x7f56540389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 10 monclient: shutdown 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 >> v1:172.21.15.189:6789/0 conn(0x7f5668138da0 legacy=0x7f56681ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 shutdown_connections 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 >> 172.21.15.189:0/1255617197 conn(0x7f56680a03e0 msgr2=0x7f566809a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 shutdown_connections 2024-09-16T06:43:45.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.437+0000 7f566db63700 1 -- 172.21.15.189:0/1255617197 wait complete. 2024-09-16T06:43:45.453 INFO:tasks.workunit.client.0.smithi189.stdout:(compat) 2024-09-16T06:43:45.454 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T06:43:45.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:45.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:45.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4518005930 con 0x7f4540132330 2024-09-16T06:43:45.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453cff9700 1 -- 172.21.15.76:0/4157186604 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4268743197) 0x7f4538014750 con 0x7f4540132330 2024-09-16T06:43:45.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:45.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.463+0000 7f453cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.464 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:45.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4524036630 legacy=0x7f4524038af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 10 monclient: shutdown 2024-09-16T06:43:45.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 >> v1:172.21.15.76:6790/0 conn(0x7f4540132330 legacy=0x7f45401d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 shutdown_connections 2024-09-16T06:43:45.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 >> 172.21.15.76:0/4157186604 conn(0x7f45400a41c0 msgr2=0x7f45400a6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 shutdown_connections 2024-09-16T06:43:45.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.467+0000 7f453ffff700 1 -- 172.21.15.76:0/4157186604 wait complete. 2024-09-16T06:43:45.477 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:43:45.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:43:45.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: monmap: 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:45.553614+0000 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:45.553614+0000 2024-09-16T06:43:45.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 1 Processor -- start 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 1 -- start start 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: init 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0a5c70 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0a7100 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-c con 0x7fd16c13bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0a5a90 con 0x7fd16c0a5c70 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0a6a90 con 0x7fd16c13bab0 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c099e40 con 0x7fd16c0a7100 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient(hunting): authenticate will time out at 1178.066869s 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd172662700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd16c13bab0 0x7fd16c13dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38470/0 (socket says 172.21.15.189:38470) 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd172662700 1 -- 172.21.15.189:0/3320386437 learned_addr learned my addr 172.21.15.189:0/3320386437 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407348937 0 0) 0x7fd16c0a6a90 con 0x7fd16c13bab0 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): my global_id is 164128 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1580035d0 con 0x7fd16c13bab0 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712125196 0 0) 0x7fd16c099e40 con 0x7fd16c0a7100 2024-09-16T06:43:45.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): my global_id is 125246 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c0a6a90 con 0x7fd16c0a7100 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872775252 0 0) 0x7fd16c0a5a90 con 0x7fd16c0a5c70 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient(hunting): my global_id is 154398 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c099e40 con 0x7fd16c0a5c70 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762968035 0 0) 0x7fd1580035d0 con 0x7fd16c13bab0 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 >> v1:172.21.15.76:6790/0 conn(0x7fd16c0a7100 legacy=0x7fd16c13a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 >> v1:172.21.15.76:6789/0 conn(0x7fd16c0a5c70 legacy=0x7fd16c0a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 monclient: found mon.noname-c 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c09a120 con 0x7fd16c13bab0 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 5 monclient: authenticate success, global_id 164128 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd164001990 con 0x7fd16c13bab0 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: dump: 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd164001c70 con 0x7fd16c13bab0 2024-09-16T06:43:45.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 1 -- 172.21.15.189:0/3320386437 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd164002000 con 0x7fd16c13bab0 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd1740c5700 4 set_mon_vals no callback set 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.549+0000 7fd170e5f700 10 monclient: dump: 2024-09-16T06:43:45.564 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.565 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: shutdown 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/3320386437 >> v1:172.21.15.189:6789/0 conn(0x7fd16c13bab0 legacy=0x7fd16c13dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/3320386437 shutdown_connections 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/3320386437 >> 172.21.15.189:0/3320386437 conn(0x7fd16c0a79e0 msgr2=0x7fd16c0a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/3320386437 shutdown_connections 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/3320386437 wait complete. 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: monmap: 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 Processor -- start 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- start start 2024-09-16T06:43:45.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: init 2024-09-16T06:43:45.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0a5c70 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0a7100 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient(hunting): picked mon.noname-c con 0x7fd16c13bab0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0a6f50 con 0x7fd16c0a7100 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0a40e0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd171e61700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd16c0a5c70 0x7fd16c1c9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38478/0 (socket says 172.21.15.189:38478) 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd171e61700 1 -- 172.21.15.189:0/541522665 learned_addr learned my addr 172.21.15.189:0/541522665 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c13b520 con 0x7fd16c13bab0 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1740c5700 10 monclient(hunting): authenticate will time out at 1178.071481s 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087112100 0 0) 0x7fd16c0a40e0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): my global_id is 164131 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1500035d0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649153693 0 0) 0x7fd16c13b520 con 0x7fd16c13bab0 2024-09-16T06:43:45.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 10 monclient(hunting): my global_id is 125249 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.553+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c0a40e0 con 0x7fd16c13bab0 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272084323 0 0) 0x7fd16c0a6f50 con 0x7fd16c0a7100 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): my global_id is 154401 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c13b520 con 0x7fd16c0a7100 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763212026 0 0) 0x7fd1500035d0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd16c0a6f50 con 0x7fd16c0a5c70 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd15c001930 con 0x7fd16c0a5c70 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): dump: 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.578 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767599585 0 0) 0x7fd16c0a6f50 con 0x7fd16c0a5c70 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 >> v1:172.21.15.76:6790/0 conn(0x7fd16c13bab0 legacy=0x7fd16c1d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 >> v1:172.21.15.76:6789/0 conn(0x7fd16c0a7100 legacy=0x7fd16c1cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 monclient: found mon.b 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0a0730 con 0x7fd16c0a5c70 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 5 monclient: authenticate success, global_id 164131 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _renew_subs 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c11f4c0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd15c0036c0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _renew_subs 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd15c0039d0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1637fe700 4 set_mon_vals no callback set 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c0a0220 con 0x7fd16c0a5c70 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 10 monclient: dump: 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd15c003b30 con 0x7fd16c0a5c70 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(133..133 src has 66..133) v4 ==== 6709+0+0 (unknown 3610394018 0 0) 0x7fd15c043e20 con 0x7fd16c0a5c70 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c0008d0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.557+0000 7fd171660700 10 monclient: get_auth_request con 0x7fd150036150 auth_method 0 2024-09-16T06:43:45.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.565+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd15c014780 con 0x7fd16c0a5c70 2024-09-16T06:43:45.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.565+0000 7fd1627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.565+0000 7fd1627fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.587+0000 7fc76def4700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.587+0000 7fc76def4700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.587+0000 7fc76def4700 10 monclient: monmap: 2024-09-16T06:43:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:45.591328+0000 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:45.591328+0000 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.590 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.587+0000 7fc76def4700 1 Processor -- start 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- start start 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient: init 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-a con 0x7fc768097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-b con 0x7fc768099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-c con 0x7fc768139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc768098990 con 0x7fc768097b60 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc768097f40 con 0x7fc768139500 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76809d610 con 0x7fc768099090 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient(hunting): authenticate will time out at 1172.724623s 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76cef2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc768097b60 0x7fc76809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60634/0 (socket says 172.21.15.76:60634) 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76cef2700 1 -- 172.21.15.76:0/620065489 learned_addr learned my addr 172.21.15.76:0/620065489 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52030997 0 0) 0x7fc768098990 con 0x7fc768097b60 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): my global_id is 154404 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7500035d0 con 0x7fc768097b60 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424887940 0 0) 0x7fc76809d610 con 0x7fc768099090 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): my global_id is 125252 2024-09-16T06:43:45.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc768098990 con 0x7fc768099090 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708907982 0 0) 0x7fc768097f40 con 0x7fc768139500 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient(hunting): my global_id is 164134 2024-09-16T06:43:45.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76809d610 con 0x7fc768139500 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650413164 0 0) 0x7fc768098990 con 0x7fc768099090 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 >> v1:172.21.15.189:6789/0 conn(0x7fc768139500 legacy=0x7fc76813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 >> v1:172.21.15.76:6789/0 conn(0x7fc768097b60 legacy=0x7fc76809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680986b0 con 0x7fc768099090 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 5 monclient: authenticate success, global_id 125252 2024-09-16T06:43:45.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7580015f0 con 0x7fc768099090 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: dump: 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.599 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc758001910 con 0x7fc768099090 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 1 -- 172.21.15.76:0/620065489 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001ce0 con 0x7fc768099090 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 4 set_mon_vals no callback set 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc75f7fe700 10 monclient: dump: 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 10 monclient: shutdown 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- 172.21.15.76:0/620065489 >> v1:172.21.15.76:6790/0 conn(0x7fc768099090 legacy=0x7fc76809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- 172.21.15.76:0/620065489 shutdown_connections 2024-09-16T06:43:45.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- 172.21.15.76:0/620065489 >> 172.21.15.76:0/620065489 conn(0x7fc7680a0030 msgr2=0x7fc7680a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- 172.21.15.76:0/620065489 shutdown_connections 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.591+0000 7fc76def4700 1 -- 172.21.15.76:0/620065489 wait complete. 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: monmap: 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.602 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 Processor -- start 2024-09-16T06:43:45.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- start start 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: init 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-c con 0x7fc768097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-a con 0x7fc768099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient(hunting): picked mon.noname-b con 0x7fc768139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76809a5b0 con 0x7fc768099090 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76809b1e0 con 0x7fc768139500 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76809be80 con 0x7fc768097b60 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient(hunting): authenticate will time out at 1172.729437s 2024-09-16T06:43:45.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 0x7fc7681ce170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44324/0 (socket says 172.21.15.76:44324) 2024-09-16T06:43:45.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 -- 172.21.15.76:0/2709015036 learned_addr learned my addr 172.21.15.76:0/2709015036 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 legacy=0x7fc7681ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:43:45.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 legacy=0x7fc7681ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:43:45.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 --1- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 0x7fc7681ce170 :-1 s=OPENED pgs=766 cs=1 l=1).handle_message read tag failed 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76cef2700 1 --1- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 0x7fc7681ce170 :-1 s=OPENED pgs=766 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727558245 0 0) 0x7fc7680986b0 con 0x7fc768099090 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): my global_id is 154407 2024-09-16T06:43:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7540035d0 con 0x7fc768099090 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025357912 0 0) 0x7fc76809b1e0 con 0x7fc768139500 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): my global_id is 164137 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7680986b0 con 0x7fc768139500 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570581096 0 0) 0x7fc7540035d0 con 0x7fc768099090 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc76809b1e0 con 0x7fc768099090 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14710118 0 0) 0x7fc7680986b0 con 0x7fc768139500 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7540035d0 con 0x7fc768139500 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001b40 con 0x7fc768099090 2024-09-16T06:43:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7600013f0 con 0x7fc768139500 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.612 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271747445 0 0) 0x7fc76809b1e0 con 0x7fc768099090 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6790/0 conn(0x7fc768097b60 legacy=0x7fc7681ce170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.189:6789/0 conn(0x7fc768139500 legacy=0x7fc7681d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 monclient: found mon.a 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768132c40 con 0x7fc768099090 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 5 monclient: authenticate success, global_id 154407 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _renew_subs 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76809c140 con 0x7fc768099090 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _renew_subs 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc758003a10 con 0x7fc768099090 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76809a850 con 0x7fc768099090 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc758001d30 con 0x7fc768099090 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75d7fa700 10 monclient: dump: 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.595+0000 7fc75effd700 4 set_mon_vals no callback set 2024-09-16T06:43:45.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc758003d20 con 0x7fc768099090 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc76def4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc768138af0 con 0x7fc768131c30 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(133..133 src has 66..133) v4 ==== 6709+0+0 (unknown 3610394018 0 0) 0x7fc758042ab0 con 0x7fc768099090 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc76d6f3700 10 monclient: get_auth_request con 0x7fc7540362b0 auth_method 0 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169746224 0 0) 0x7fc76809be80 con 0x7fc768131c30 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient(hunting): my global_id is 154410 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76809b1e0 con 0x7fc768131c30 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3595259785 0 0) 0x7fc76809b1e0 con 0x7fc768131c30 2024-09-16T06:43:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc76809be80 con 0x7fc768131c30 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc764001850 con 0x7fc768131c30 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient: dump: 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.617 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2318166636 0 0) 0x7fc76809be80 con 0x7fc768131c30 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc76809fb80 con 0x7fc768131c30 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc764001cb0 con 0x7fc768131c30 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6789/0 conn(0x7fc768131c30 legacy=0x7fc768061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.599+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6789/0 conn(0x7fc768131c30 legacy=0x7fc768061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc768138af0 con 0x7fc7681307f0 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181019257 0 0) 0x7fc764004890 con 0x7fc7681307f0 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient(hunting): my global_id is 154413 2024-09-16T06:43:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc76809be80 con 0x7fc7681307f0 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3161857036 0 0) 0x7fc76809be80 con 0x7fc7681307f0 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc764004890 con 0x7fc7681307f0 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc764004c10 con 0x7fc7681307f0 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient: dump: 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.621 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3313554433 0 0) 0x7fc764004890 con 0x7fc7681307f0 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc7680973c0 con 0x7fc7681307f0 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1171601819) 0x7fc764004f90 con 0x7fc7681307f0 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6789/0 conn(0x7fc7681307f0 legacy=0x7fc76802b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc75d7fa700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6789/0 conn(0x7fc7681307f0 legacy=0x7fc76802b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.622 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:45.644 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7540362b0 legacy=0x7fc754038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 10 monclient: shutdown 2024-09-16T06:43:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 >> v1:172.21.15.76:6789/0 conn(0x7fc768099090 legacy=0x7fc7681d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76d6f3700 1 -- 172.21.15.76:0/2709015036 reap_dead start 2024-09-16T06:43:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76d6f3700 1 -- 172.21.15.76:0/2709015036 reap_dead start 2024-09-16T06:43:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 shutdown_connections 2024-09-16T06:43:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 >> 172.21.15.76:0/2709015036 conn(0x7fc7680a0030 msgr2=0x7fc76806be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 shutdown_connections 2024-09-16T06:43:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.611+0000 7fc76def4700 1 -- 172.21.15.76:0/2709015036 wait complete. 2024-09-16T06:43:45.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:45.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:43:45.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:43:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.749+0000 7fd1740c5700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.749+0000 7fd1740c5700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:45.752 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.749+0000 7fd1740c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.749+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fd16c11f090 con 0x7fd16c0a5c70 2024-09-16T06:43:45.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 10 monclient: monmap: 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:45.766299+0000 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:45.766299+0000 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.765 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.766 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.766 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 1 Processor -- start 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 1 -- start start 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.763+0000 7f4d9185f700 10 monclient: init 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-b con 0x7f4d8c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-c con 0x7f4d8c06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-a con 0x7f4d8c06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c06c280 con 0x7f4d8c06cf50 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c06bfa0 con 0x7f4d8c06d480 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c09ace0 con 0x7f4d8c132330 2024-09-16T06:43:45.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient(hunting): authenticate will time out at 1172.900292s 2024-09-16T06:43:45.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9105e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4d8c06cf50 0x7f4d8c13da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60670/0 (socket says 172.21.15.76:60670) 2024-09-16T06:43:45.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9105e700 1 -- 172.21.15.76:0/3163485463 learned_addr learned my addr 172.21.15.76:0/3163485463 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9085d700 1 --1- 172.21.15.76:0/3163485463 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c132330 0x7f4d8c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44332/0 (socket says 172.21.15.76:44332) 2024-09-16T06:43:45.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.765+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v134) v1 ==== 78+0+0 (unknown 1300440148 0 0) 0x7fd15c0150b0 con 0x7fd16c0a5c70 2024-09-16T06:43:45.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.765+0000 7fd1627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:45.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.765+0000 7fd1627fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088731237 0 0) 0x7f4d8c06bfa0 con 0x7f4d8c06d480 2024-09-16T06:43:45.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): my global_id is 164140 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d780035d0 con 0x7f4d8c06d480 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644811058 0 0) 0x7f4d8c06c280 con 0x7f4d8c06cf50 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient(hunting): my global_id is 154416 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c06bfa0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4152552827 0 0) 0x7f4d780035d0 con 0x7f4d8c06d480 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c132330 legacy=0x7f4d8c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 >> v1:172.21.15.76:6789/0 conn(0x7f4d8c06cf50 legacy=0x7f4d8c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c09ae70 con 0x7f4d8c06d480 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d7c001610 con 0x7f4d8c06d480 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 5 monclient: authenticate success, global_id 164140 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:45.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: dump: 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.776 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d7c0018f0 con 0x7f4d8c06d480 2024-09-16T06:43:45.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 1 -- 172.21.15.76:0/3163485463 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d7c001ae0 con 0x7f4d8c06d480 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d9185f700 4 set_mon_vals no callback set 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.767+0000 7f4d8b7fe700 10 monclient: dump: 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 10 monclient: shutdown 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 1 -- 172.21.15.76:0/3163485463 >> v1:172.21.15.189:6789/0 conn(0x7f4d8c06d480 legacy=0x7f4d8c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 1 -- 172.21.15.76:0/3163485463 shutdown_connections 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 1 -- 172.21.15.76:0/3163485463 >> 172.21.15.76:0/3163485463 conn(0x7f4d8c0a41c0 msgr2=0x7f4d8c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 1 -- 172.21.15.76:0/3163485463 shutdown_connections 2024-09-16T06:43:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.771+0000 7f4d9185f700 1 -- 172.21.15.76:0/3163485463 wait complete. 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient: monmap: 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.784 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 1 Processor -- start 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 1 -- start start 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient: init 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-a con 0x7f4d8c06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-b con 0x7f4d8c06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient(hunting): picked mon.noname-c con 0x7f4d8c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c1391c0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c135fe0 con 0x7f4d8c06d480 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c09e8b0 con 0x7f4d8c132330 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9185f700 10 monclient(hunting): authenticate will time out at 1172.909699s 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9085d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4d8c06cf50 0x7f4d8c1351c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60686/0 (socket says 172.21.15.76:60686) 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.775+0000 7f4d9085d700 1 -- 172.21.15.76:0/2118914401 learned_addr learned my addr 172.21.15.76:0/2118914401 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897671666 0 0) 0x7f4d8c1391c0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): my global_id is 154419 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6c0035d0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993455118 0 0) 0x7f4d8c135fe0 con 0x7f4d8c06d480 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): my global_id is 164143 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c1391c0 con 0x7f4d8c06d480 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063056957 0 0) 0x7f4d8c09e8b0 con 0x7f4d8c132330 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): my global_id is 125258 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c135fe0 con 0x7f4d8c132330 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594202086 0 0) 0x7f4d6c0035d0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8c09e8b0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d74001860 con 0x7f4d8c06cf50 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): dump: 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.788 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 845112434 0 0) 0x7f4d8c135fe0 con 0x7f4d8c132330 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d6c0035d0 con 0x7f4d8c132330 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d84001ad0 con 0x7f4d8c132330 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient(hunting): dump: 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 12187345 0 0) 0x7f4d8c1391c0 con 0x7f4d8c06d480 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8c135fe0 con 0x7f4d8c06d480 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3768115829 0 0) 0x7f4d8c09e8b0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c132330 legacy=0x7f4d8c1d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.189:6789/0 conn(0x7f4d8c06d480 legacy=0x7f4d8c1358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 monclient: found mon.a 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c131890 con 0x7f4d8c06cf50 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 5 monclient: authenticate success, global_id 154419 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 10 monclient: _renew_subs 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c131360 con 0x7f4d8c06cf50 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 10 monclient: _renew_subs 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c136e60 con 0x7f4d8c06cf50 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d74001b40 con 0x7f4d8c06cf50 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d74004b80 con 0x7f4d8c06cf50 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 10 monclient: dump: 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d8a7fc700 4 set_mon_vals no callback set 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4d74004ce0 con 0x7f4d8c06cf50 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.779+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(134..134 src has 66..134) v4 ==== 6377+0+0 (unknown 1629906902 0 0) 0x7f4d74043c30 con 0x7f4d8c06cf50 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d8bfff700 10 monclient: get_auth_request con 0x7f4d6c036550 auth_method 0 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d9185f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d8c0a3430 con 0x7f4d8c13ea40 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839236123 0 0) 0x7f4d6c0035d0 con 0x7f4d8c13ea40 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient(hunting): my global_id is 125261 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c09e8b0 con 0x7f4d8c13ea40 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2445221333 0 0) 0x7f4d8c09e8b0 con 0x7f4d8c13ea40 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d6c0035d0 con 0x7f4d8c13ea40 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d84001f20 con 0x7f4d8c13ea40 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient: dump: 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1455280307 0 0) 0x7f4d6c0035d0 con 0x7f4d8c13ea40 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d8c13ee20 con 0x7f4d8c13ea40 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4d84007480 con 0x7f4d8c13ea40 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c13ea40 legacy=0x7f4d8c09c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.783+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c13ea40 legacy=0x7f4d8c09c510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d9185f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4d8c0a3430 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709734686 0 0) 0x7f4d8c06bfa0 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient(hunting): my global_id is 125264 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6c0035d0 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 139534683 0 0) 0x7f4d6c0035d0 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8c06bfa0 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4d840052f0 con 0x7f4d8c13da70 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 10 monclient: dump: 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3279308376 0 0) 0x7f4d8c06bfa0 con 0x7f4d8c13da70 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.791+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d8c13de80 con 0x7f4d8c13da70 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 586870596) 0x7f4d840075f0 con 0x7f4d8c13da70 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d897fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c13da70 legacy=0x7f4d8c02c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d897fa700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6790/0 conn(0x7f4d8c13da70 legacy=0x7f4d8c02c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.799 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4d6c036550 legacy=0x7f4d6c038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 10 monclient: shutdown 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9105e700 1 -- 172.21.15.76:0/2118914401 reap_dead start 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 >> v1:172.21.15.76:6789/0 conn(0x7f4d8c06cf50 legacy=0x7f4d8c1351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 shutdown_connections 2024-09-16T06:43:45.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 >> 172.21.15.76:0/2118914401 conn(0x7f4d8c0a41c0 msgr2=0x7f4d8c0a6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 shutdown_connections 2024-09-16T06:43:45.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.795+0000 7f4d9185f700 1 -- 172.21.15.76:0/2118914401 wait complete. 2024-09-16T06:43:45.823 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:45.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:43:45.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:43:45.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.881+0000 7fd1740c5700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:45.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.881+0000 7fd1740c5700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:45.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.881+0000 7fd1740c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:45.885 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:45.881+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fd16c11f220 con 0x7fd16c0a5c70 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: get_monmap_and_config 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: monmap: 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:45.929934+0000 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:45.929934+0000 2024-09-16T06:43:45.928 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:45.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:45.929 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 1 Processor -- start 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 1 -- start start 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: init 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc7e809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc7e8098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc7e8099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e809d560 con 0x7fc7e809ed60 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e809f140 con 0x7fc7e8098be0 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e809d790 con 0x7fc7e8099030 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e759e700 10 monclient(hunting): authenticate will time out at 1173.063532s 2024-09-16T06:43:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e6d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc7e8099030 0x7fc7e813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44382/0 (socket says 172.21.15.76:44382) 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e6d9d700 1 -- 172.21.15.76:0/2452604966 learned_addr learned my addr 172.21.15.76:0/2452604966 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.927+0000 7fc7e659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc7e809ed60 0x7fc7e80984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60688/0 (socket says 172.21.15.76:60688) 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204939284 0 0) 0x7fc7e809d790 con 0x7fc7e8099030 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): my global_id is 125267 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d40057e0 con 0x7fc7e8099030 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760153284 0 0) 0x7fc7e809d560 con 0x7fc7e809ed60 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): my global_id is 154422 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e809d790 con 0x7fc7e809ed60 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205550340 0 0) 0x7fc7e809f140 con 0x7fc7e8098be0 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient(hunting): my global_id is 164146 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e809d560 con 0x7fc7e8098be0 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956257657 0 0) 0x7fc7e809d790 con 0x7fc7e809ed60 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 >> v1:172.21.15.76:6790/0 conn(0x7fc7e8099030 legacy=0x7fc7e813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 >> v1:172.21.15.189:6789/0 conn(0x7fc7e8098be0 legacy=0x7fc7e81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 monclient: found mon.noname-a 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e809f620 con 0x7fc7e809ed60 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d8001970 con 0x7fc7e809ed60 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 5 monclient: authenticate success, global_id 154422 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: dump: 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7d8001c10 con 0x7fc7e809ed60 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 4 set_mon_vals no callback set 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 1 -- 172.21.15.76:0/2452604966 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d8001fe0 con 0x7fc7e809ed60 2024-09-16T06:43:45.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e559a700 10 monclient: dump: 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: shutdown 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- 172.21.15.76:0/2452604966 >> v1:172.21.15.76:6789/0 conn(0x7fc7e809ed60 legacy=0x7fc7e80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- 172.21.15.76:0/2452604966 shutdown_connections 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- 172.21.15.76:0/2452604966 >> 172.21.15.76:0/2452604966 conn(0x7fc7e80a0030 msgr2=0x7fc7e80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- 172.21.15.76:0/2452604966 shutdown_connections 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- 172.21.15.76:0/2452604966 wait complete. 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: build_initial_monmap 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: monmap: 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:45.939 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 Processor -- start 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- start start 2024-09-16T06:43:45.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: init 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc7e8098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc7e8099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc7e809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:45.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e80985d0 con 0x7fc7e809ed60 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e80a2e70 con 0x7fc7e8098be0 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e8070070 con 0x7fc7e8099030 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e759e700 10 monclient(hunting): authenticate will time out at 1173.067509s 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e6d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc7e809ed60 0x7fc7e81d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60700/0 (socket says 172.21.15.76:60700) 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7e6d9d700 1 -- 172.21.15.76:0/611501449 learned_addr learned my addr 172.21.15.76:0/611501449 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797431251 0 0) 0x7fc7e80985d0 con 0x7fc7e809ed60 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 10 monclient(hunting): my global_id is 154425 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.931+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c80035d0 con 0x7fc7e809ed60 2024-09-16T06:43:45.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213416198 0 0) 0x7fc7e8070070 con 0x7fc7e8099030 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): my global_id is 125270 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e80985d0 con 0x7fc7e8099030 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181031531 0 0) 0x7fc7e80a2e70 con 0x7fc7e8098be0 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): my global_id is 164149 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8070070 con 0x7fc7e8098be0 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3315323513 0 0) 0x7fc7c80035d0 con 0x7fc7e809ed60 2024-09-16T06:43:45.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e80a2e70 con 0x7fc7e809ed60 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7e00011a0 con 0x7fc7e809ed60 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): dump: 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.947 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227315452 0 0) 0x7fc7e80985d0 con 0x7fc7e8099030 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7c80035d0 con 0x7fc7e8099030 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d00017d0 con 0x7fc7e8099030 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): dump: 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4173001394 0 0) 0x7fc7e8070070 con 0x7fc7e8098be0 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e80985d0 con 0x7fc7e8098be0 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d8001f20 con 0x7fc7e8098be0 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): dump: 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 393024350 0 0) 0x7fc7e80a2e70 con 0x7fc7e809ed60 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.76:6790/0 conn(0x7fc7e8099030 legacy=0x7fc7e81cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6789/0 conn(0x7fc7e8098be0 legacy=0x7fc7e80715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:45.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 monclient: found mon.a 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e806fd90 con 0x7fc7e809ed60 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 5 monclient: authenticate success, global_id 154425 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 10 monclient: _renew_subs 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e8071cb0 con 0x7fc7e809ed60 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 10 monclient: _renew_subs 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e806eb00 con 0x7fc7e809ed60 2024-09-16T06:43:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e0004890 con 0x7fc7e809ed60 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7e0004ba0 con 0x7fc7e809ed60 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e4d99700 4 set_mon_vals no callback set 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: dump: 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.952 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc7e000f610 con 0x7fc7e809ed60 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7cc006590 con 0x7fc7cc002830 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(134..134 src has 66..134) v4 ==== 6377+0+0 (unknown 1629906902 0 0) 0x7fc7e0043c20 con 0x7fc7e809ed60 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7e5d9b700 10 monclient: get_auth_request con 0x7fc7c8036560 auth_method 0 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984580224 0 0) 0x7fc7cc006590 con 0x7fc7cc002830 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient(hunting): my global_id is 164152 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e80a2e70 con 0x7fc7cc002830 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3419906189 0 0) 0x7fc7e80a2e70 con 0x7fc7cc002830 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7cc006590 con 0x7fc7cc002830 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d8005020 con 0x7fc7cc002830 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 10 monclient: dump: 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2535890223 0 0) 0x7fc7cc006590 con 0x7fc7cc002830 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.935+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7cc00a190 con 0x7fc7cc002830 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.939+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc7d80073f0 con 0x7fc7cc002830 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.939+0000 7fc7deffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.939+0000 7fc7deffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.939+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6789/0 conn(0x7fc7cc002830 legacy=0x7fc7cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.939+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6789/0 conn(0x7fc7cc002830 legacy=0x7fc7cc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7e759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7cc009e30 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246529521 0 0) 0x7fc7cc009e30 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient(hunting): my global_id is 164155 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7cc006590 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4246089961 0 0) 0x7fc7cc006590 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7cc009e30 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7d80057e0 con 0x7fc7cc009150 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 10 monclient: dump: 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3100759039 0 0) 0x7fc7cc009e30 con 0x7fc7cc009150 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.947+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc7cc00c340 con 0x7fc7cc009150 2024-09-16T06:43:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1365233145) 0x7fc7d8006bf0 con 0x7fc7cc009150 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7deffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7deffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6789/0 conn(0x7fc7cc009150 legacy=0x7fc7cc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7deffd700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6789/0 conn(0x7fc7cc009150 legacy=0x7fc7cc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:43:45.959 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.960 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:45.962 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:45.963 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.965 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:45.966 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:45.967 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:45.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.969 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:45.970 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7c8036560 legacy=0x7fc7c8038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 10 monclient: shutdown 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 >> v1:172.21.15.76:6789/0 conn(0x7fc7e809ed60 legacy=0x7fc7e81d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e6d9d700 1 -- 172.21.15.76:0/611501449 reap_dead start 2024-09-16T06:43:45.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e6d9d700 1 -- 172.21.15.76:0/611501449 reap_dead start 2024-09-16T06:43:45.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 shutdown_connections 2024-09-16T06:43:45.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 >> 172.21.15.76:0/611501449 conn(0x7fc7e80a0030 msgr2=0x7fc7e80935c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:45.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 shutdown_connections 2024-09-16T06:43:45.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:45.951+0000 7fc7e759e700 1 -- 172.21.15.76:0/611501449 wait complete. 2024-09-16T06:43:45.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:45.973 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:43:45.973 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:43:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.083+0000 7f13d5efa700 10 monclient: get_monmap_and_config 2024-09-16T06:43:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.083+0000 7f13d5efa700 10 monclient: build_initial_monmap 2024-09-16T06:43:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.083+0000 7f13d5efa700 10 monclient: monmap: 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:46.087271+0000 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:46.087271+0000 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:46.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 Processor -- start 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- start start 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: init 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-c con 0x7f13d00973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-b con 0x7f13d0097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-a con 0x7f13d01400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d0099350 con 0x7f13d01400d0 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d009f690 con 0x7f13d00973c0 2024-09-16T06:43:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d0099bf0 con 0x7f13d0097eb0 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient(hunting): authenticate will time out at 1173.221105s 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13d0097eb0 0x7f13d013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44398/0 (socket says 172.21.15.76:44398) 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cffff700 1 -- 172.21.15.76:0/1813899876 learned_addr learned my addr 172.21.15.76:0/1813899876 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585138379 0 0) 0x7f13d0099bf0 con 0x7f13d0097eb0 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): my global_id is 125273 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b80035d0 con 0x7f13d0097eb0 2024-09-16T06:43:46.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756336521 0 0) 0x7f13d0099350 con 0x7f13d01400d0 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): my global_id is 154428 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0099bf0 con 0x7f13d01400d0 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061416647 0 0) 0x7f13d009f690 con 0x7f13d00973c0 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient(hunting): my global_id is 164158 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0099350 con 0x7f13d00973c0 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681903647 0 0) 0x7f13d0099bf0 con 0x7f13d01400d0 2024-09-16T06:43:46.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 >> v1:172.21.15.76:6790/0 conn(0x7f13d0097eb0 legacy=0x7f13d013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 >> v1:172.21.15.189:6789/0 conn(0x7f13d00973c0 legacy=0x7f13d00977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 monclient: found mon.noname-a 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0098360 con 0x7f13d01400d0 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 5 monclient: authenticate success, global_id 154428 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c8001970 con 0x7f13d01400d0 2024-09-16T06:43:46.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: dump: 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:46.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c8001c10 con 0x7f13d01400d0 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 4 set_mon_vals no callback set 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 1 -- 172.21.15.76:0/1813899876 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c8001fe0 con 0x7f13d01400d0 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13cf7fe700 10 monclient: dump: 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: shutdown 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- 172.21.15.76:0/1813899876 >> v1:172.21.15.76:6789/0 conn(0x7f13d01400d0 legacy=0x7f13d01424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- 172.21.15.76:0/1813899876 shutdown_connections 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- 172.21.15.76:0/1813899876 >> 172.21.15.76:0/1813899876 conn(0x7f13d00a0030 msgr2=0x7f13d00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- 172.21.15.76:0/1813899876 shutdown_connections 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 1 -- 172.21.15.76:0/1813899876 wait complete. 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: build_initial_monmap 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.087+0000 7f13d5efa700 10 monclient: monmap: 2024-09-16T06:43:46.097 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 Processor -- start 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- start start 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: init 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:46.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-c con 0x7f13d00973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-a con 0x7f13d0097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient(hunting): picked mon.noname-b con 0x7f13d01400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d0072660 con 0x7f13d0097eb0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d013fda0 con 0x7f13d01400d0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d006c9c0 con 0x7f13d00973c0 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient(hunting): authenticate will time out at 1173.225091s 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d4ef8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13d00973c0 0x7f13d01ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44412/0 (socket says 172.21.15.76:44412) 2024-09-16T06:43:46.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d4ef8700 1 -- 172.21.15.76:0/4177443181 learned_addr learned my addr 172.21.15.76:0/4177443181 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:46.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589787928 0 0) 0x7f13d006c9c0 con 0x7f13d00973c0 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): my global_id is 125276 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0035d0 con 0x7f13d00973c0 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062760529 0 0) 0x7f13d0072660 con 0x7f13d0097eb0 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): my global_id is 154431 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d006c9c0 con 0x7f13d0097eb0 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869244667 0 0) 0x7f13d013fda0 con 0x7f13d01400d0 2024-09-16T06:43:46.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): my global_id is 164161 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0072660 con 0x7f13d01400d0 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1775425112 0 0) 0x7f13bc0035d0 con 0x7f13d00973c0 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d013fda0 con 0x7f13d00973c0 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c0001970 con 0x7f13d00973c0 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.104 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396118573 0 0) 0x7f13d006c9c0 con 0x7f13d0097eb0 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc0035d0 con 0x7f13d0097eb0 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141595027 0 0) 0x7f13d0072660 con 0x7f13d01400d0 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d006c9c0 con 0x7f13d01400d0 2024-09-16T06:43:46.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c4001190 con 0x7f13d0097eb0 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.106 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c8001f20 con 0x7f13d01400d0 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.107 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240723669 0 0) 0x7f13d013fda0 con 0x7f13d00973c0 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 >> v1:172.21.15.189:6789/0 conn(0x7f13d01400d0 legacy=0x7f13d01d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 >> v1:172.21.15.76:6789/0 conn(0x7f13d0097eb0 legacy=0x7f13d01d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 monclient: found mon.c 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d009a540 con 0x7f13d00973c0 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:46.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 5 monclient: authenticate success, global_id 125276 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _renew_subs 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d006c120 con 0x7f13d00973c0 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _renew_subs 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d006bdb0 con 0x7f13d00973c0 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c0001cb0 con 0x7f13d00973c0 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:46.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13c0003b10 con 0x7f13d00973c0 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13cd7fa700 10 monclient: dump: 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.110 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.091+0000 7f13ceffd700 4 set_mon_vals no callback set 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13c0003cf0 con 0x7f13d00973c0 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13d5efa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13d5efa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13d5efa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b0005100 con 0x7f13d00973c0 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(134..134 src has 66..134) v4 ==== 6377+0+0 (unknown 1629906902 0 0) 0x7f13c0043de0 con 0x7f13d00973c0 2024-09-16T06:43:46.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.095+0000 7f13cffff700 10 monclient: get_auth_request con 0x7f13bc036710 auth_method 0 2024-09-16T06:43:46.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.103+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f13c0014830 con 0x7f13d00973c0 2024-09-16T06:43:46.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.103+0000 7f13cd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.103+0000 7f13cd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.335+0000 7f13d5efa700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:43:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.335+0000 7f13d5efa700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:43:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.335+0000 7f13d5efa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.335+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f13b0005bc0 con 0x7f13d00973c0 2024-09-16T06:43:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.335+0000 7f13cd7fa700 1 -- 172.21.15.76:0/4177443181 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f13c0012ca0 con 0x7f13d00973c0 2024-09-16T06:43:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13cd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:43:46.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13cd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:46.339 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.339 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=11,kv=11,logm=53} crc {auth=3220254852,config=1097430493,health=1971228270,kv=2388690603,logm=1856592589}) 2024-09-16T06:43:46.339 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=3979845382,config=1097430493,health=1016277833,kv=1001081567,logm=2923874186}) 2024-09-16T06:43:46.340 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.340 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=11,kv=11,logm=53} crc {auth=3220254852,config=1097430493,health=1971228270,kv=2388690603,logm=1856592589}) 2024-09-16T06:43:46.340 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=13,logm=51} crc {auth=1233966958,config=1097430493,health=1016277833,kv=1001081567,logm=1497243374}) 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13bc036710 legacy=0x7f13bc038bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 10 monclient: shutdown 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 >> v1:172.21.15.76:6790/0 conn(0x7f13d00973c0 legacy=0x7f13d01ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 shutdown_connections 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 >> 172.21.15.76:0/4177443181 conn(0x7f13d00a0030 msgr2=0x7f13d00a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 shutdown_connections 2024-09-16T06:43:46.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f13d5efa700 1 -- 172.21.15.76:0/4177443181 wait complete. 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532071137}) 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380231903}) 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532071137}) 2024-09-16T06:43:46.342 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.339+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502058410}) 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1120224855}) 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624501859}) 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1120224855}) 2024-09-16T06:43:46.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366158600}) 2024-09-16T06:43:46.345 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.345 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755659374}) 2024-09-16T06:43:46.345 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644819425}) 2024-09-16T06:43:46.346 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.346 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755659374}) 2024-09-16T06:43:46.346 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.343+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287948178}) 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497714420}) 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577759198}) 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497714420}) 2024-09-16T06:43:46.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746976075}) 2024-09-16T06:43:46.348 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.348 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842518510}) 2024-09-16T06:43:46.348 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015901980}) 2024-09-16T06:43:46.348 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.349 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842518510}) 2024-09-16T06:43:46.349 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678352681}) 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009884268}) 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111234415}) 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009884268}) 2024-09-16T06:43:46.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.347+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1192573705}) 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713391049}) 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792146388}) 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713391049}) 2024-09-16T06:43:46.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149284349}) 2024-09-16T06:43:46.352 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937481230}) 2024-09-16T06:43:46.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030507719}) 2024-09-16T06:43:46.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937481230}) 2024-09-16T06:43:46.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4172159308}) 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285177631}) 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285412358}) 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.351+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285177631}) 2024-09-16T06:43:46.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1436271061}) 2024-09-16T06:43:46.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544527987}) 2024-09-16T06:43:46.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967940991}) 2024-09-16T06:43:46.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544527987}) 2024-09-16T06:43:46.356 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135487659}) 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661507402}) 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938713485}) 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661507402}) 2024-09-16T06:43:46.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624438256}) 2024-09-16T06:43:46.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828921368}) 2024-09-16T06:43:46.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3409298634}) 2024-09-16T06:43:46.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828921368}) 2024-09-16T06:43:46.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.355+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940643140}) 2024-09-16T06:43:46.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:43:46.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164282190}) 2024-09-16T06:43:46.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696003110}) 2024-09-16T06:43:46.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164282190}) 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1762686212}) 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090351106}) 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495680492}) 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090351106}) 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1508691358}) 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392779147}) 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214333070}) 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392779147}) 2024-09-16T06:43:46.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.359+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137103561}) 2024-09-16T06:43:46.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023404100}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522443084}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023404100}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646951937}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224645502}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833666887}) 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224645502}) 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131343057}) 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096140405}) 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831596830}) 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096140405}) 2024-09-16T06:43:46.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511219494}) 2024-09-16T06:43:46.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143254494}) 2024-09-16T06:43:46.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82704820}) 2024-09-16T06:43:46.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.363+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143254494}) 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842493867}) 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616974273}) 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467361662}) 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616974273}) 2024-09-16T06:43:46.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946581615}) 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712491356}) 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666964877}) 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712491356}) 2024-09-16T06:43:46.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716226179}) 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_snap=2,osdmap=2} crc {logm=2592936548,mdsmap=1531151339,mgr=4265060939,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4155708150,mon_config_key=3917553283,monmap=1887442722,osd_metadata=2101314684,osd_snap=2033388227,osdmap=3898017584}) 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=2} crc {logm=3145077805,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4155708150,mon_config_key=4279594037,monmap=3140514009,osd_metadata=1334916596,osd_pg_creating=1421878362,osd_snap=2033388227,osdmap=3898017584}) 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_snap=2,osdmap=2} crc {logm=2592936548,mdsmap=1531151339,mgr=4265060939,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4155708150,mon_config_key=3917553283,monmap=1887442722,osd_metadata=2101314684,osd_snap=2033388227,osdmap=3898017584}) 2024-09-16T06:43:46.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.367+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=3523437971,mdsmap=1531151339,mgr=3204891192,mgr_metadata=1347778423,mgrstat=4155708150,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1421878362,osd_snap=2033388227}) 2024-09-16T06:43:46.371 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.371 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4292821841}) 2024-09-16T06:43:46.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1784699623}) 2024-09-16T06:43:46.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4292821841}) 2024-09-16T06:43:46.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.371+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2233902347}) 2024-09-16T06:43:46.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=12} crc {osdmap=2650880696}) 2024-09-16T06:43:46.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=12} crc {osdmap=1221377903}) 2024-09-16T06:43:46.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:43:46.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=12} crc {osdmap=2650880696}) 2024-09-16T06:43:46.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:43:46.375+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=12} crc {osdmap=1422017939}) 2024-09-16T06:43:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.777+0000 7fd1627fc700 1 -- 172.21.15.189:0/541522665 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v135) v1 ==== 78+0+0 (unknown 205680241 0 0) 0x7fd15c0150b0 con 0x7fd16c0a5c70 2024-09-16T06:43:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.777+0000 7fd1627fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:43:46.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.777+0000 7fd1627fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd150036150 legacy=0x7fd150038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 10 monclient: shutdown 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 >> v1:172.21.15.189:6789/0 conn(0x7fd16c0a5c70 legacy=0x7fd16c1c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 shutdown_connections 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 >> 172.21.15.189:0/541522665 conn(0x7fd16c0a79e0 msgr2=0x7fd16c0a30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 shutdown_connections 2024-09-16T06:43:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.781+0000 7fd1740c5700 1 -- 172.21.15.189:0/541522665 wait complete. 2024-09-16T06:43:46.800 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create cool 2 2024-09-16T06:43:46.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: get_monmap_and_config 2024-09-16T06:43:46.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: build_initial_monmap 2024-09-16T06:43:46.912 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: monmap: 2024-09-16T06:43:46.912 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:46.913125+0000 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:46.913125+0000 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:46.913 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 Processor -- start 2024-09-16T06:43:46.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- start start 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: init 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-b con 0x7f6c8009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-a con 0x7f6c8009c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-c con 0x7f6c80138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8009cca0 con 0x7f6c8009c5f0 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8009f750 con 0x7f6c80138da0 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80097620 con 0x7f6c8009bb00 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient(hunting): authenticate will time out at 1179.426223s 2024-09-16T06:43:46.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c86bb4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6c80138da0 0x7f6c8013b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38494/0 (socket says 172.21.15.189:38494) 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c86bb4700 1 -- 172.21.15.189:0/981797847 learned_addr learned my addr 172.21.15.189:0/981797847 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395396347 0 0) 0x7f6c8009f750 con 0x7f6c80138da0 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient(hunting): my global_id is 164164 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c0035d0 con 0x7f6c80138da0 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206331294 0 0) 0x7f6c6c0035d0 con 0x7f6c80138da0 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 >> v1:172.21.15.76:6790/0 conn(0x7f6c8009bb00 legacy=0x7f6c8009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 >> v1:172.21.15.76:6789/0 conn(0x7f6c8009c5f0 legacy=0x7f6c80137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: _finish_hunting 0 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 monclient: found mon.noname-c 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:46.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80097900 con 0x7f6c80138da0 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: _finish_auth 0 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: _check_auth_tickets 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 5 monclient: authenticate success, global_id 164164 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c78001990 con 0x7f6c80138da0 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: dump: 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c78001cb0 con 0x7f6c80138da0 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 1 -- 172.21.15.189:0/981797847 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c78002080 con 0x7f6c80138da0 2024-09-16T06:43:46.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 4 set_mon_vals no callback set 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c853b1700 10 monclient: dump: 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 10 monclient: shutdown 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- 172.21.15.189:0/981797847 >> v1:172.21.15.189:6789/0 conn(0x7f6c80138da0 legacy=0x7f6c8013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- 172.21.15.189:0/981797847 shutdown_connections 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.909+0000 7f6c88617700 1 -- 172.21.15.189:0/981797847 >> 172.21.15.189:0/981797847 conn(0x7f6c800a03e0 msgr2=0x7f6c800a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:46.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- 172.21.15.189:0/981797847 shutdown_connections 2024-09-16T06:43:46.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- 172.21.15.189:0/981797847 wait complete. 2024-09-16T06:43:46.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient: build_initial_monmap 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient: monmap: 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 Processor -- start 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- start start 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient: init 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-b con 0x7f6c8009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-c con 0x7f6c8009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient(hunting): picked mon.noname-a con 0x7f6c80138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8009a6e0 con 0x7f6c80138da0 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8012cc20 con 0x7f6c8009bb00 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8012cf00 con 0x7f6c8009c5f0 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:46.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c88617700 10 monclient(hunting): authenticate will time out at 1179.431215s 2024-09-16T06:43:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c863b3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6c8009bb00 0x7f6c801c7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38502/0 (socket says 172.21.15.189:38502) 2024-09-16T06:43:46.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.913+0000 7f6c863b3700 1 -- 172.21.15.189:0/2531335841 learned_addr learned my addr 172.21.15.189:0/2531335841 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229553220 0 0) 0x7f6c8012cc20 con 0x7f6c8009bb00 2024-09-16T06:43:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): my global_id is 164167 2024-09-16T06:43:46.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c640035d0 con 0x7f6c8009bb00 2024-09-16T06:43:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862708125 0 0) 0x7f6c8009a6e0 con 0x7f6c80138da0 2024-09-16T06:43:46.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): my global_id is 154437 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8012cc20 con 0x7f6c80138da0 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617043052 0 0) 0x7f6c640035d0 con 0x7f6c8009bb00 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c8009a6e0 con 0x7f6c8009bb00 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c74001930 con 0x7f6c8009bb00 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.931 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2215380708 0 0) 0x7f6c8009a6e0 con 0x7f6c8009bb00 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 >> v1:172.21.15.76:6790/0 conn(0x7f6c8009c5f0 legacy=0x7f6c801c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 >> v1:172.21.15.76:6789/0 conn(0x7f6c80138da0 legacy=0x7f6c801cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:46.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 monclient: found mon.b 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8012d1e0 con 0x7f6c8009bb00 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 5 monclient: authenticate success, global_id 164167 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _renew_subs 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8012b640 con 0x7f6c8009bb00 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _renew_subs 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8012b390 con 0x7f6c8009bb00 2024-09-16T06:43:46.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c740036c0 con 0x7f6c8009bb00 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c740039d0 con 0x7f6c8009bb00 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c84bb0700 4 set_mon_vals no callback set 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 10 monclient: dump: 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:46.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6c74003b30 con 0x7f6c8009bb00 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(135..135 src has 66..135) v4 ==== 6377+0+0 (unknown 1863960687 0 0) 0x7f6c74043c50 con 0x7f6c8009bb00 2024-09-16T06:43:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c68005100 con 0x7f6c8009bb00 2024-09-16T06:43:46.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.925+0000 7f6c85bb2700 10 monclient: get_auth_request con 0x7f6c64036060 auth_method 0 2024-09-16T06:43:46.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.933+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6c74004500 con 0x7f6c8009bb00 2024-09-16T06:43:46.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.933+0000 7f6c72ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:46.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:46.933+0000 7f6c72ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:47.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.105+0000 7f6c88617700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:47.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.105+0000 7f6c88617700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:47.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.105+0000 7f6c88617700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.105+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f6c68005930 con 0x7f6c8009bb00 2024-09-16T06:43:47.361 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:43:47.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:43:47.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.479+0000 7fb04b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:43:47.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.479+0000 7fb04b59e700 10 monclient: build_initial_monmap 2024-09-16T06:43:47.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.479+0000 7fb04b59e700 10 monclient: monmap: 2024-09-16T06:43:47.481 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:47.483236+0000 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:47.483236+0000 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:47.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.479+0000 7fb04b59e700 1 Processor -- start 2024-09-16T06:43:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- start start 2024-09-16T06:43:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient: init 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb04c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb04c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb04c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c098990 con 0x7fb04c097b60 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c097f40 con 0x7fb04c099090 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c09d610 con 0x7fb04c139500 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient(hunting): authenticate will time out at 1174.616564s 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb04c097b60 0x7fb04c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60736/0 (socket says 172.21.15.76:60736) 2024-09-16T06:43:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04a59c700 1 -- 172.21.15.76:0/113409646 learned_addr learned my addr 172.21.15.76:0/113409646 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047240085 0 0) 0x7fb04c098990 con 0x7fb04c097b60 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): my global_id is 154440 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0380035d0 con 0x7fb04c097b60 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562301783 0 0) 0x7fb04c097f40 con 0x7fb04c099090 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): my global_id is 164170 2024-09-16T06:43:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c098990 con 0x7fb04c099090 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344795973 0 0) 0x7fb04c09d610 con 0x7fb04c139500 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient(hunting): my global_id is 125279 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c097f40 con 0x7fb04c139500 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3317738365 0 0) 0x7fb0380035d0 con 0x7fb04c097b60 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 >> v1:172.21.15.76:6790/0 conn(0x7fb04c139500 legacy=0x7fb04c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 >> v1:172.21.15.189:6789/0 conn(0x7fb04c099090 legacy=0x7fb04c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: _finish_hunting 0 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 monclient: found mon.noname-a 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0986b0 con 0x7fb04c097b60 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: _finish_auth 0 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: _check_auth_tickets 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 5 monclient: authenticate success, global_id 154440 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb044001990 con 0x7fb04c097b60 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: dump: 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb044001c70 con 0x7fb04c097b60 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 4 set_mon_vals no callback set 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 1 -- 172.21.15.76:0/113409646 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb044002000 con 0x7fb04c097b60 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04959a700 10 monclient: dump: 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 10 monclient: shutdown 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- 172.21.15.76:0/113409646 >> v1:172.21.15.76:6789/0 conn(0x7fb04c097b60 legacy=0x7fb04c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- 172.21.15.76:0/113409646 shutdown_connections 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- 172.21.15.76:0/113409646 >> 172.21.15.76:0/113409646 conn(0x7fb04c0a0030 msgr2=0x7fb04c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- 172.21.15.76:0/113409646 shutdown_connections 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.483+0000 7fb04b59e700 1 -- 172.21.15.76:0/113409646 wait complete. 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient: build_initial_monmap 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient: monmap: 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:47.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 1 Processor -- start 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 1 -- start start 2024-09-16T06:43:47.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient: init 2024-09-16T06:43:47.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:47.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:47.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb04c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:47.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb04c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:47.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb04c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c13c350 con 0x7fb04c139500 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c08fc40 con 0x7fb04c099090 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c09a540 con 0x7fb04c097b60 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04b59e700 10 monclient(hunting): authenticate will time out at 1174.622710s 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04a59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb04c097b60 0x7fb04c138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44426/0 (socket says 172.21.15.76:44426) 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb04a59c700 1 -- 172.21.15.76:0/607706689 learned_addr learned my addr 172.21.15.76:0/607706689 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476894187 0 0) 0x7fb04c09a540 con 0x7fb04c097b60 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): my global_id is 125282 2024-09-16T06:43:47.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb02c0035d0 con 0x7fb04c097b60 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264074198 0 0) 0x7fb04c097f40 con 0x7fb04c139500 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): my global_id is 154443 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c09a540 con 0x7fb04c139500 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412508002 0 0) 0x7fb04c08fc40 con 0x7fb04c099090 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): my global_id is 164173 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c097f40 con 0x7fb04c099090 2024-09-16T06:43:47.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398530129 0 0) 0x7fb04c09a540 con 0x7fb04c139500 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb04c08fc40 con 0x7fb04c139500 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb03c001710 con 0x7fb04c139500 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.502 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500005045 0 0) 0x7fb04c097f40 con 0x7fb04c099090 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb04c09a540 con 0x7fb04c099090 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb0340018b0 con 0x7fb04c099090 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.503 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849321436 0 0) 0x7fb02c0035d0 con 0x7fb04c097b60 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb04c097f40 con 0x7fb04c097b60 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb044001950 con 0x7fb04c097b60 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.487+0000 7fb042ffd700 10 monclient(hunting): dump: 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.504 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3914817612 0 0) 0x7fb04c09a540 con 0x7fb04c099090 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 >> v1:172.21.15.76:6790/0 conn(0x7fb04c097b60 legacy=0x7fb04c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 >> v1:172.21.15.76:6789/0 conn(0x7fb04c139500 legacy=0x7fb04c1d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 monclient: found mon.b 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0a2d10 con 0x7fb04c099090 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 5 monclient: authenticate success, global_id 164173 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _renew_subs 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c08f540 con 0x7fb04c099090 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _renew_subs 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c08d450 con 0x7fb04c099090 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb034001d30 con 0x7fb04c099090 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb034003840 con 0x7fb04c099090 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 10 monclient: dump: 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb048d99700 4 set_mon_vals no callback set 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb030005100 con 0x7fb04c099090 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb03400e2b0 con 0x7fb04c099090 2024-09-16T06:43:47.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(135..135 src has 66..135) v4 ==== 6377+0+0 (unknown 1863960687 0 0) 0x7fb034042930 con 0x7fb04c099090 2024-09-16T06:43:47.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.491+0000 7fb04a59c700 10 monclient: get_auth_request con 0x7fb02c036480 auth_method 0 2024-09-16T06:43:47.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.503+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb0340131b0 con 0x7fb04c099090 2024-09-16T06:43:47.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.503+0000 7fb042ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:47.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.503+0000 7fb042ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:47.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.735+0000 7fb04b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:43:47.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.735+0000 7fb04b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:47.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.735+0000 7fb04b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:47.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.735+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb030005930 con 0x7fb04c099090 2024-09-16T06:43:47.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb042ffd700 1 -- 172.21.15.76:0/607706689 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 848317869) 0x7fb034011850 con 0x7fb04c099090 2024-09-16T06:43:47.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb042ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:43:47.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb042ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:47.739 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:43:47.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb02c036480 legacy=0x7fb02c038940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 10 monclient: shutdown 2024-09-16T06:43:47.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:47.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 >> v1:172.21.15.189:6789/0 conn(0x7fb04c099090 legacy=0x7fb04c1cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 shutdown_connections 2024-09-16T06:43:47.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 >> 172.21.15.76:0/607706689 conn(0x7fb04c0a0030 msgr2=0x7fb04c139c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:47.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 shutdown_connections 2024-09-16T06:43:47.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.739+0000 7fb04b59e700 1 -- 172.21.15.76:0/607706689 wait complete. 2024-09-16T06:43:47.752 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:43:47.752 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:43:47.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:43:47.753 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.863+0000 7f2aad588700 10 monclient: get_monmap_and_config 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.863+0000 7f2aad588700 10 monclient: build_initial_monmap 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.863+0000 7f2aad588700 10 monclient: monmap: 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.866 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:47.867810+0000 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:47.867810+0000 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:47.867 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 Processor -- start 2024-09-16T06:43:47.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- start start 2024-09-16T06:43:47.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient: init 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-a con 0x7f2aa8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-c con 0x7f2aa806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-b con 0x7f2aa806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa806c280 con 0x7f2aa8132330 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa806bfa0 con 0x7f2aa806d480 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa809ace0 con 0x7f2aa806cf50 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient(hunting): authenticate will time out at 1175.001166s 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2aa8132330 0x7f2aa8132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60752/0 (socket says 172.21.15.76:60752) 2024-09-16T06:43:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa7fff700 1 -- 172.21.15.76:0/2282608714 learned_addr learned my addr 172.21.15.76:0/2282608714 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023669037 0 0) 0x7f2aa806c280 con 0x7f2aa8132330 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): my global_id is 154446 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a940035d0 con 0x7f2aa8132330 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228870432 0 0) 0x7f2aa809ace0 con 0x7f2aa806cf50 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): my global_id is 125285 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa806c280 con 0x7f2aa806cf50 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031818562 0 0) 0x7f2aa806bfa0 con 0x7f2aa806d480 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient(hunting): my global_id is 164176 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa809ace0 con 0x7f2aa806d480 2024-09-16T06:43:47.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496445703 0 0) 0x7f2a940035d0 con 0x7f2aa8132330 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 >> v1:172.21.15.76:6790/0 conn(0x7f2aa806cf50 legacy=0x7f2aa813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 >> v1:172.21.15.189:6789/0 conn(0x7f2aa806d480 legacy=0x7f2aa813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 monclient: found mon.noname-a 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa809ae70 con 0x7f2aa8132330 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 5 monclient: authenticate success, global_id 154446 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:47.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa0001990 con 0x7f2aa8132330 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: dump: 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.876 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aa0001c70 con 0x7f2aa8132330 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:47.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 4 set_mon_vals no callback set 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 1 -- 172.21.15.76:0/2282608714 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa0001e60 con 0x7f2aa8132330 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aa6ffd700 10 monclient: dump: 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 10 monclient: shutdown 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- 172.21.15.76:0/2282608714 >> v1:172.21.15.76:6789/0 conn(0x7f2aa8132330 legacy=0x7f2aa8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- 172.21.15.76:0/2282608714 shutdown_connections 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- 172.21.15.76:0/2282608714 >> 172.21.15.76:0/2282608714 conn(0x7f2aa80a41c0 msgr2=0x7f2aa80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- 172.21.15.76:0/2282608714 shutdown_connections 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.867+0000 7f2aad588700 1 -- 172.21.15.76:0/2282608714 wait complete. 2024-09-16T06:43:47.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient: build_initial_monmap 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient: monmap: 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 1 Processor -- start 2024-09-16T06:43:47.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 1 -- start start 2024-09-16T06:43:47.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient: init 2024-09-16T06:43:47.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:47.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:43:47.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-b con 0x7f2aa806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:47.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-a con 0x7f2aa806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:47.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient(hunting): picked mon.noname-c con 0x7f2aa8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa81312d0 con 0x7f2aa806d480 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa812ff90 con 0x7f2aa806cf50 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aacd87700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2aa8132330 0x7f2aa81d1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44450/0 (socket says 172.21.15.76:44450) 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2aa806d480 0x7f2aa81c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60756/0 (socket says 172.21.15.76:60756) 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aacd87700 1 -- 172.21.15.76:0/972983235 learned_addr learned my addr 172.21.15.76:0/972983235 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:47.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa81ca600 con 0x7f2aa8132330 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aad588700 10 monclient(hunting): authenticate will time out at 1175.006606s 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820566983 0 0) 0x7f2aa809ace0 con 0x7f2aa806d480 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): my global_id is 154449 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a880035d0 con 0x7f2aa806d480 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989617315 0 0) 0x7f2aa81ca600 con 0x7f2aa8132330 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): my global_id is 125288 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa809ace0 con 0x7f2aa8132330 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216762949 0 0) 0x7f2aa812ff90 con 0x7f2aa806cf50 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): my global_id is 164179 2024-09-16T06:43:47.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa81ca600 con 0x7f2aa806cf50 2024-09-16T06:43:47.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1698323185 0 0) 0x7f2aa809ace0 con 0x7f2aa8132330 2024-09-16T06:43:47.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2aa812ff90 con 0x7f2aa8132330 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a980018b0 con 0x7f2aa8132330 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.898 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007818010 0 0) 0x7f2a880035d0 con 0x7f2aa806d480 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2aa809ace0 con 0x7f2aa806d480 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1924010916 0 0) 0x7f2aa81ca600 con 0x7f2aa806cf50 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a880035d0 con 0x7f2aa806cf50 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a900016b0 con 0x7f2aa806d480 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.899 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa00013a0 con 0x7f2aa806cf50 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.871+0000 7f2aa4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient(hunting): dump: 2024-09-16T06:43:47.900 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2333181208 0 0) 0x7f2aa812ff90 con 0x7f2aa8132330 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.189:6789/0 conn(0x7f2aa806cf50 legacy=0x7f2aa81c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6789/0 conn(0x7f2aa806d480 legacy=0x7f2aa81c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 monclient: found mon.c 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa809e8b0 con 0x7f2aa8132330 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 5 monclient: authenticate success, global_id 125288 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 10 monclient: _renew_subs 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:47.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa812fcb0 con 0x7f2aa8132330 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 10 monclient: _renew_subs 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa812dc30 con 0x7f2aa8132330 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a980036c0 con 0x7f2aa8132330 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2a980039d0 con 0x7f2aa8132330 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa67fc700 4 set_mon_vals no callback set 2024-09-16T06:43:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: dump: 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.904 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2a98003bb0 con 0x7f2aa8132330 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2a8c006590 con 0x7f2a8c002830 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(135..135 src has 66..135) v4 ==== 6377+0+0 (unknown 1863960687 0 0) 0x7f2a98043c70 con 0x7f2aa8132330 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710449521 0 0) 0x7f2a8c006590 con 0x7f2a8c002830 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient(hunting): my global_id is 154452 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa812ff90 con 0x7f2a8c002830 2024-09-16T06:43:47.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa77fe700 10 monclient: get_auth_request con 0x7f2a88036530 auth_method 0 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3891142691 0 0) 0x7f2aa812ff90 con 0x7f2a8c002830 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a8c006590 con 0x7f2a8c002830 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa0004fa0 con 0x7f2a8c002830 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: dump: 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.906 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3439313866 0 0) 0x7f2a8c006590 con 0x7f2a8c002830 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a8c00a190 con 0x7f2a8c002830 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2aa00071e0 con 0x7f2a8c002830 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6789/0 conn(0x7f2a8c002830 legacy=0x7f2a8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.875+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6789/0 conn(0x7f2a8c002830 legacy=0x7f2a8c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aad588700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2a8c009e30 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289914235 0 0) 0x7f2a8c009e30 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient(hunting): my global_id is 154455 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c006590 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1779670470 0 0) 0x7f2a8c006590 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a8c009e30 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2aa00056e0 con 0x7f2a8c009150 2024-09-16T06:43:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient: dump: 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:47.909 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2425064310 0 0) 0x7f2a8c009e30 con 0x7f2a8c009150 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a8c00c340 con 0x7f2a8c009150 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 406316792) 0x7f2aa0006ac0 con 0x7f2a8c009150 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6789/0 conn(0x7f2a8c009150 legacy=0x7f2a8c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.887+0000 7f2aa4ff9700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6789/0 conn(0x7f2a8c009150 legacy=0x7f2a8c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:47.910 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:43:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:47.923 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:47.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2a88036530 legacy=0x7f2a880389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 10 monclient: shutdown 2024-09-16T06:43:47.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:47.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 >> v1:172.21.15.76:6790/0 conn(0x7f2aa8132330 legacy=0x7f2aa81d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aacd87700 1 -- 172.21.15.76:0/972983235 reap_dead start 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aacd87700 1 -- 172.21.15.76:0/972983235 reap_dead start 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 shutdown_connections 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 >> 172.21.15.76:0/972983235 conn(0x7f2aa80a41c0 msgr2=0x7f2aa809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 shutdown_connections 2024-09-16T06:43:47.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:47.891+0000 7f2aad588700 1 -- 172.21.15.76:0/972983235 wait complete. 2024-09-16T06:43:47.925 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:47.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:43:47.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:43:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.933+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v136) v1 ==== 111+0+0 (unknown 939921657 0 0) 0x7f6c74012bf0 con 0x7f6c8009bb00 2024-09-16T06:43:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.933+0000 7f6c72ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:47.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:47.933+0000 7f6c72ffd700 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-16T06:43:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.039+0000 7fce47fff700 10 monclient: get_monmap_and_config 2024-09-16T06:43:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.039+0000 7fce47fff700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.039+0000 7fce47fff700 10 monclient: monmap: 2024-09-16T06:43:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:48.042908+0000 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:48.042908+0000 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:48.042 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 Processor -- start 2024-09-16T06:43:48.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- start start 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient: init 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-a con 0x7fce480a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-b con 0x7fce480a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-c con 0x7fce4813b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4809b3e0 con 0x7fce480a20c0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce480a3450 con 0x7fce4813b6b0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4809b1a0 con 0x7fce480a24a0 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient(hunting): authenticate will time out at 1175.176855s 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce3e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fce480a24a0 0x7fce480a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44466/0 (socket says 172.21.15.76:44466) 2024-09-16T06:43:48.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce3e7fc700 1 -- 172.21.15.76:0/1839604585 learned_addr learned my addr 172.21.15.76:0/1839604585 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344787158 0 0) 0x7fce4809b1a0 con 0x7fce480a24a0 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): my global_id is 125291 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340035d0 con 0x7fce480a24a0 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740750977 0 0) 0x7fce4809b3e0 con 0x7fce480a20c0 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): my global_id is 154458 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4809b1a0 con 0x7fce480a20c0 2024-09-16T06:43:48.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916232302 0 0) 0x7fce480a3450 con 0x7fce4813b6b0 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient(hunting): my global_id is 164182 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4809b3e0 con 0x7fce4813b6b0 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925811712 0 0) 0x7fce340035d0 con 0x7fce480a24a0 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 >> v1:172.21.15.189:6789/0 conn(0x7fce4813b6b0 legacy=0x7fce4813daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 >> v1:172.21.15.76:6789/0 conn(0x7fce480a20c0 legacy=0x7fce4806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 monclient: found mon.noname-b 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:48.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4806bd30 con 0x7fce480a24a0 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce30001610 con 0x7fce480a24a0 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 5 monclient: authenticate success, global_id 125291 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: dump: 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce30001800 con 0x7fce480a24a0 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 4 set_mon_vals no callback set 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 1 -- 172.21.15.76:0/1839604585 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce30001b90 con 0x7fce480a24a0 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce467fc700 10 monclient: dump: 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 10 monclient: shutdown 2024-09-16T06:43:48.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- 172.21.15.76:0/1839604585 >> v1:172.21.15.76:6790/0 conn(0x7fce480a24a0 legacy=0x7fce480a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- 172.21.15.76:0/1839604585 shutdown_connections 2024-09-16T06:43:48.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- 172.21.15.76:0/1839604585 >> 172.21.15.76:0/1839604585 conn(0x7fce480a41c0 msgr2=0x7fce480a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- 172.21.15.76:0/1839604585 shutdown_connections 2024-09-16T06:43:48.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.043+0000 7fce47fff700 1 -- 172.21.15.76:0/1839604585 wait complete. 2024-09-16T06:43:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.041+0000 7f6c88617700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.041+0000 7f6c88617700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.041+0000 7f6c88617700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.041+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f6c680053b0 con 0x7f6c8009bb00 2024-09-16T06:43:48.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c72ffd700 1 -- 172.21.15.189:0/2531335841 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v136) v1 ==== 118+0+0 (unknown 2570299145 0 0) 0x7f6c74014550 con 0x7f6c8009bb00 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c72ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c72ffd700 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6c64036060 legacy=0x7f6c64038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 10 monclient: shutdown 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 >> v1:172.21.15.189:6789/0 conn(0x7f6c8009bb00 legacy=0x7f6c801c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 shutdown_connections 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 >> 172.21.15.189:0/2531335841 conn(0x7f6c800a03e0 msgr2=0x7f6c800a2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 shutdown_connections 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.045+0000 7f6c88617700 1 -- 172.21.15.189:0/2531335841 wait complete. 2024-09-16T06:43:48.055 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cool' already exists 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient: monmap: 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:48.061 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 1 Processor -- start 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 1 -- start start 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient: init 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-c con 0x7fce480a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-a con 0x7fce480a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 10 monclient(hunting): picked mon.noname-b con 0x7fce4813b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce481391e0 con 0x7fce480a24a0 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce47fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce480a1420 con 0x7fce4813b6b0 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.047+0000 7fce3e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fce480a24a0 0x7fce48136d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60808/0 (socket says 172.21.15.76:60808) 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3e7fc700 1 -- 172.21.15.76:0/1774477678 learned_addr learned my addr 172.21.15.76:0/1774477678 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4809e8b0 con 0x7fce480a20c0 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient(hunting): authenticate will time out at 1175.184105s 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485956894 0 0) 0x7fce481391e0 con 0x7fce480a24a0 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): my global_id is 154461 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce280035d0 con 0x7fce480a24a0 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265452954 0 0) 0x7fce4809e8b0 con 0x7fce480a20c0 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): my global_id is 125294 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce481391e0 con 0x7fce480a20c0 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647116810 0 0) 0x7fce480a1420 con 0x7fce4813b6b0 2024-09-16T06:43:48.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): my global_id is 164185 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4809e8b0 con 0x7fce4813b6b0 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 265743 0 0) 0x7fce280035d0 con 0x7fce480a24a0 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce480a1420 con 0x7fce480a24a0 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce300015d0 con 0x7fce480a24a0 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.065 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1997528660 0 0) 0x7fce481391e0 con 0x7fce480a20c0 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce280035d0 con 0x7fce480a20c0 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce380019d0 con 0x7fce480a20c0 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): dump: 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3561684589 0 0) 0x7fce480a1420 con 0x7fce480a24a0 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6790/0 conn(0x7fce480a20c0 legacy=0x7fce48136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.189:6789/0 conn(0x7fce4813b6b0 legacy=0x7fce481d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 monclient: found mon.a 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce481374a0 con 0x7fce480a24a0 2024-09-16T06:43:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 5 monclient: authenticate success, global_id 154461 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient: _renew_subs 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce48138070 con 0x7fce480a24a0 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient: _renew_subs 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce481397a0 con 0x7fce480a24a0 2024-09-16T06:43:48.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce30004b40 con 0x7fce480a24a0 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce30001210 con 0x7fce480a24a0 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce457fa700 4 set_mon_vals no callback set 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient: dump: 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fce30004ed0 con 0x7fce480a24a0 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce2c006590 con 0x7fce2c002830 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7fce30043f80 con 0x7fce480a24a0 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce477fe700 10 monclient: get_auth_request con 0x7fce28036470 auth_method 0 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540046816 0 0) 0x7fce2c006590 con 0x7fce2c002830 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 10 monclient(hunting): my global_id is 125297 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.051+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce480a1420 con 0x7fce2c002830 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1948936174 0 0) 0x7fce480a1420 con 0x7fce2c002830 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2c006590 con 0x7fce2c002830 2024-09-16T06:43:48.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce38003840 con 0x7fce2c002830 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 10 monclient: dump: 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 222536465 0 0) 0x7fce2c006590 con 0x7fce2c002830 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce2c00a190 con 0x7fce2c002830 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fce38005a60 con 0x7fce2c002830 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6790/0 conn(0x7fce2c002830 legacy=0x7fce2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.055+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6790/0 conn(0x7fce2c002830 legacy=0x7fce2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.075 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool create cold 2 2024-09-16T06:43:48.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce47fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:48.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce2c009e50 con 0x7fce2c009170 2024-09-16T06:43:48.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143442117 0 0) 0x7fce2c009e50 con 0x7fce2c009170 2024-09-16T06:43:48.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 10 monclient(hunting): my global_id is 125300 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.059+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce2c006590 con 0x7fce2c009170 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4221570481 0 0) 0x7fce2c006590 con 0x7fce2c009170 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2c009e50 con 0x7fce2c009170 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce38003f80 con 0x7fce2c009170 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 10 monclient: dump: 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.078 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1164544820 0 0) 0x7fce2c009e50 con 0x7fce2c009170 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce2c00c340 con 0x7fce2c009170 2024-09-16T06:43:48.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 3082943083) 0x7fce38005bd0 con 0x7fce2c009170 2024-09-16T06:43:48.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:48.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:48.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6790/0 conn(0x7fce2c009170 legacy=0x7fce2c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce3ffff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6790/0 conn(0x7fce2c009170 legacy=0x7fce2c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.081 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:43:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:43:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:48.106 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.189:6812/3098810674 conn(0x7fce28036470 legacy=0x7fce28038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 10 monclient: shutdown 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 >> v1:172.21.15.76:6789/0 conn(0x7fce480a24a0 legacy=0x7fce48136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce477fe700 1 -- 172.21.15.76:0/1774477678 reap_dead start 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 shutdown_connections 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 >> 172.21.15.76:0/1774477678 conn(0x7fce480a41c0 msgr2=0x7fce4813a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 shutdown_connections 2024-09-16T06:43:48.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.063+0000 7fce47fff700 1 -- 172.21.15.76:0/1774477678 wait complete. 2024-09-16T06:43:48.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:48.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:43:48.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:43:48.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: get_monmap_and_config 2024-09-16T06:43:48.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: monmap: 2024-09-16T06:43:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:43:48.173873+0000 2024-09-16T06:43:48.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:43:48.173873+0000 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:48.175 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 1 Processor -- start 2024-09-16T06:43:48.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 1 -- start start 2024-09-16T06:43:48.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: init 2024-09-16T06:43:48.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-a con 0x7fbc10099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-c con 0x7fbc10099e40 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-b con 0x7fbc1009a250 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1009c460 con 0x7fbc10099a60 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1009c1e0 con 0x7fbc10099e40 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1009bd60 con 0x7fbc1009a250 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc16fd0700 10 monclient(hunting): authenticate will time out at 1180.687248s 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc14d6c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 0x7fbc10098070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:50312/0 (socket says 172.21.15.189:50312) 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.169+0000 7fbc14d6c700 1 -- 172.21.15.189:0/670391962 learned_addr learned my addr 172.21.15.189:0/670391962 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007242453 0 0) 0x7fbc1009c460 con 0x7fbc10099a60 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): my global_id is 154464 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc000035d0 con 0x7fbc10099a60 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631153080 0 0) 0x7fbc1009c1e0 con 0x7fbc10099e40 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): my global_id is 164188 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc14d6c700 1 -- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbc10098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1009c460 con 0x7fbc10099e40 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc14d6c700 1 -- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbc10098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc14d6c700 1 --1- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 0x7fbc10098070 :-1 s=OPENED pgs=606 cs=1 l=1).handle_message read tag failed 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc14d6c700 1 --1- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 0x7fbc10098070 :-1 s=OPENED pgs=606 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:43:48.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468645767 0 0) 0x7fbc1009bd60 con 0x7fbc1009a250 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient(hunting): my global_id is 125303 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1009c1e0 con 0x7fbc1009a250 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022979653 0 0) 0x7fbc1009c460 con 0x7fbc10099e40 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6790/0 conn(0x7fbc1009a250 legacy=0x7fbc1013a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbc10098070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 monclient: found mon.noname-c 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1009d490 con 0x7fbc10099e40 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 5 monclient: authenticate success, global_id 164188 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc040015f0 con 0x7fbc10099e40 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: dump: 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc04001890 con 0x7fbc10099e40 2024-09-16T06:43:48.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 1 -- 172.21.15.189:0/670391962 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc04001c20 con 0x7fbc10099e40 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 4 set_mon_vals no callback set 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0f7fe700 10 monclient: dump: 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.190 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: shutdown 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- 172.21.15.189:0/670391962 >> v1:172.21.15.189:6789/0 conn(0x7fbc10099e40 legacy=0x7fbc10136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- 172.21.15.189:0/670391962 shutdown_connections 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- 172.21.15.189:0/670391962 >> 172.21.15.189:0/670391962 conn(0x7fbc100a03e0 msgr2=0x7fbc100a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- 172.21.15.189:0/670391962 shutdown_connections 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- 172.21.15.189:0/670391962 wait complete. 2024-09-16T06:43:48.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: monmap: 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 Processor -- start 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- start start 2024-09-16T06:43:48.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: init 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-a con 0x7fbc10099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-c con 0x7fbc10099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient(hunting): picked mon.noname-b con 0x7fbc1009a250 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1009eab0 con 0x7fbc10099a60 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1009a6e0 con 0x7fbc1009a250 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc101b6a40 con 0x7fbc10099e40 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc16fd0700 10 monclient(hunting): authenticate will time out at 1180.691188s 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc1556d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 0x7fbc101d2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:38518/0 (socket says 172.21.15.189:38518) 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc1556d700 1 -- 172.21.15.189:0/3248132267 learned_addr learned my addr 172.21.15.189:0/3248132267 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:43:48.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558353578 0 0) 0x7fbc1009a6e0 con 0x7fbc1009a250 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): my global_id is 164191 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf40035d0 con 0x7fbc1009a250 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589920661 0 0) 0x7fbc1009eab0 con 0x7fbc10099a60 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): my global_id is 154467 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1009a6e0 con 0x7fbc10099a60 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981965725 0 0) 0x7fbc101b6a40 con 0x7fbc10099e40 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 10 monclient(hunting): my global_id is 125306 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1009eab0 con 0x7fbc10099e40 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.173+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897267439 0 0) 0x7fbbf40035d0 con 0x7fbc1009a250 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc101b6a40 con 0x7fbc1009a250 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc08001bf0 con 0x7fbc1009a250 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient(hunting): dump: 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.195 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3785807277 0 0) 0x7fbc101b6a40 con 0x7fbc1009a250 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 legacy=0x7fbc101cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbc101c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 monclient: found mon.b 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc101b6d20 con 0x7fbc1009a250 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 5 monclient: authenticate success, global_id 164191 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _renew_subs 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc101b4f80 con 0x7fbc1009a250 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _renew_subs 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc101c7000 con 0x7fbc1009a250 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08003a20 con 0x7fbc1009a250 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc08003db0 con 0x7fbc1009a250 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0effd700 4 set_mon_vals no callback set 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 10 monclient: dump: 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.198 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf0005100 con 0x7fbc1009a250 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbc08003f90 con 0x7fbc1009a250 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7fbc08044080 con 0x7fbc1009a250 2024-09-16T06:43:48.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.177+0000 7fbc14d6c700 10 monclient: get_auth_request con 0x7fbbf4036360 auth_method 0 2024-09-16T06:43:48.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.181+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbc08014b20 con 0x7fbc1009a250 2024-09-16T06:43:48.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.181+0000 7fbc0d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.181+0000 7fbc0d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:48.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.227+0000 7f519aa9f700 10 monclient: get_monmap_and_config 2024-09-16T06:43:48.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.227+0000 7f519aa9f700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.227+0000 7f519aa9f700 10 monclient: monmap: 2024-09-16T06:43:48.229 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:48.229 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:43:48.231036+0000 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:43:48.231036+0000 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:43:48.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 Processor -- start 2024-09-16T06:43:48.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- start start 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: init 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-b con 0x7f5194132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-c con 0x7f519406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-a con 0x7f519406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519406c280 con 0x7f519406cf50 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519406bfa0 con 0x7f519406d480 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519409ace0 con 0x7f5194132330 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient(hunting): authenticate will time out at 1175.365128s 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5199a9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5194132330 0x7f5194132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44500/0 (socket says 172.21.15.76:44500) 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5199a9d700 1 -- 172.21.15.76:0/3601706516 learned_addr learned my addr 172.21.15.76:0/3601706516 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964668910 0 0) 0x7f519409ace0 con 0x7f5194132330 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): my global_id is 125309 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51800035d0 con 0x7f5194132330 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281862808 0 0) 0x7f519406c280 con 0x7f519406cf50 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): my global_id is 154470 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519409ace0 con 0x7f519406cf50 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650703671 0 0) 0x7f519406bfa0 con 0x7f519406d480 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient(hunting): my global_id is 164194 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519406c280 con 0x7f519406d480 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 112241462 0 0) 0x7f51800035d0 con 0x7f5194132330 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 >> v1:172.21.15.189:6789/0 conn(0x7f519406d480 legacy=0x7f519413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 >> v1:172.21.15.76:6789/0 conn(0x7f519406cf50 legacy=0x7f519413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 monclient: found mon.noname-b 2024-09-16T06:43:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519409ae70 con 0x7f5194132330 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 5 monclient: authenticate success, global_id 125309 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5190001610 con 0x7f5194132330 2024-09-16T06:43:48.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: dump: 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:43:48.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51900018f0 con 0x7f5194132330 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 1 -- 172.21.15.76:0/3601706516 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5190001ae0 con 0x7f5194132330 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: get_monmap_and_config success 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 4 set_mon_vals no callback set 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f5198a9b700 10 monclient: dump: 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.248 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: shutdown 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- 172.21.15.76:0/3601706516 >> v1:172.21.15.76:6790/0 conn(0x7f5194132330 legacy=0x7f5194132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- 172.21.15.76:0/3601706516 shutdown_connections 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- 172.21.15.76:0/3601706516 >> 172.21.15.76:0/3601706516 conn(0x7f51940a41c0 msgr2=0x7f51940a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- 172.21.15.76:0/3601706516 shutdown_connections 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 1 -- 172.21.15.76:0/3601706516 wait complete. 2024-09-16T06:43:48.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: build_initial_monmap 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.231+0000 7f519aa9f700 10 monclient: monmap: 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 Processor -- start 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- start start 2024-09-16T06:43:48.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: init 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-b con 0x7f519406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-c con 0x7f519406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient(hunting): picked mon.noname-a con 0x7f5194132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519412cdb0 con 0x7f5194132330 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51941302f0 con 0x7f519406cf50 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519412dff0 con 0x7f519406d480 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient(hunting): authenticate will time out at 1175.368830s 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 0x7f51941d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:60814/0 (socket says 172.21.15.76:60814) 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 -- 172.21.15.76:0/1370715732 learned_addr learned my addr 172.21.15.76:0/1370715732 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:43:48.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267308462 0 0) 0x7f519412cdb0 con 0x7f5194132330 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): my global_id is 154473 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51780035d0 con 0x7f5194132330 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974201465 0 0) 0x7f519412dff0 con 0x7f519406d480 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): my global_id is 125312 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519412cdb0 con 0x7f519406d480 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514949496 0 0) 0x7f51941302f0 con 0x7f519406cf50 2024-09-16T06:43:48.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): my global_id is 164197 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519412dff0 con 0x7f519406cf50 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989916743 0 0) 0x7f51780035d0 con 0x7f5194132330 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51941302f0 con 0x7f5194132330 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f518c001c30 con 0x7f5194132330 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): dump: 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310568606 0 0) 0x7f519412cdb0 con 0x7f519406d480 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51780035d0 con 0x7f519406d480 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51840018b0 con 0x7f519406d480 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): dump: 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226907735 0 0) 0x7f519412dff0 con 0x7f519406cf50 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f519412cdb0 con 0x7f519406cf50 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3105814772 0 0) 0x7f51941302f0 con 0x7f5194132330 2024-09-16T06:43:48.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6790/0 conn(0x7f519406d480 legacy=0x7f519412ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519406cf50 legacy=0x7f519412b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 monclient: found mon.a 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519412e2d0 con 0x7f5194132330 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 5 monclient: authenticate success, global_id 154473 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _renew_subs 2024-09-16T06:43:48.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519412c6f0 con 0x7f5194132330 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 legacy=0x7f51941d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 legacy=0x7f51941d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _renew_subs 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 --1- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 0x7f51941d2450 :-1 s=OPENED pgs=609 cs=1 l=1).handle_message read tag failed 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519a29e700 1 --1- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 0x7f51941d2450 :-1 s=OPENED pgs=609 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f519409e8b0 con 0x7f5194132330 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6789/0 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5194132330 legacy=0x7f51941d2450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): picked mon.a con 0x7f5178004f30 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): picked mon.c con 0x7f519406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): picked mon.b con 0x7f519406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f51941302f0 con 0x7f5178004f30 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f51780035d0 con 0x7f519406cf50 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f519412dff0 con 0x7f519406d480 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059719919 0 0) 0x7f51941302f0 con 0x7f5178004f30 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f51780124b0 con 0x7f5178004f30 2024-09-16T06:43:48.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810730629 0 0) 0x7f519412dff0 con 0x7f519406d480 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f51941302f0 con 0x7f519406d480 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204736649 0 0) 0x7f51780035d0 con 0x7f519406cf50 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f519412dff0 con 0x7f519406cf50 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1760967191 0 0) 0x7f51780124b0 con 0x7f5178004f30 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6790/0 conn(0x7f519406d480 legacy=0x7f517800b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519406cf50 legacy=0x7f517800e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 monclient: found mon.a 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f5178013660 con 0x7f5178004f30 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5190004b80 con 0x7f5178004f30 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: dump: 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5190004ce0 con 0x7f5178004f30 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f519409e8b0 con 0x7f519412b310 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.235+0000 7f518bfff700 4 set_mon_vals no callback set 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f519000f760 con 0x7f5178004f30 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854769919 0 0) 0x7f519409e8b0 con 0x7f519412b310 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient(hunting): my global_id is 164200 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51780124b0 con 0x7f519412b310 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.0 v1:172.21.15.76:6789/0 6 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7f5190043d10 con 0x7f5178004f30 2024-09-16T06:43:48.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f519a29e700 10 monclient: get_auth_request con 0x7f5194132330 auth_method 0 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3057730913 0 0) 0x7f51780124b0 con 0x7f519412b310 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f519409e8b0 con 0x7f519412b310 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5184003ee0 con 0x7f519412b310 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient: dump: 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.263 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4231221115 0 0) 0x7f519409e8b0 con 0x7f519412b310 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f519409d520 con 0x7f519412b310 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5184006340 con 0x7f519412b310 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519412b310 legacy=0x7f519412b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.239+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519412b310 legacy=0x7f519412b6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f519aa9f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f519409a890 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169998704 0 0) 0x7f519409a890 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient(hunting): my global_id is 164203 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519409e8b0 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2900488344 0 0) 0x7f519409e8b0 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f519409a890 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5184004730 con 0x7f519412be00 2024-09-16T06:43:48.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 10 monclient: dump: 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:43:48.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2237290495 0 0) 0x7f519409a890 con 0x7f519412be00 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.251+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f51940a3450 con 0x7f519412be00 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3294943446) 0x7f5184005ab0 con 0x7f519412be00 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f518a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f518a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519412be00 legacy=0x7f519402b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f518a7fc700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6789/0 conn(0x7f519412be00 legacy=0x7f519402b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.268 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 54, 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:43:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:43:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:43:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:43:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:43:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:43:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:43:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:43:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:43:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:43:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:43:48.281 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5194132330 legacy=0x7f5178043840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 10 monclient: shutdown 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 >> v1:172.21.15.76:6789/0 conn(0x7f5178004f30 legacy=0x7f5178007a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519a29e700 1 -- 172.21.15.76:0/1370715732 reap_dead start 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519a29e700 1 -- 172.21.15.76:0/1370715732 reap_dead start 2024-09-16T06:43:48.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 shutdown_connections 2024-09-16T06:43:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 >> 172.21.15.76:0/1370715732 conn(0x7f51940a41c0 msgr2=0x7f5194129fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:43:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 shutdown_connections 2024-09-16T06:43:48.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:43:48.255+0000 7f519aa9f700 1 -- 172.21.15.76:0/1370715732 wait complete. 2024-09-16T06:43:48.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:43:48.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:43:48.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T06:43:48.283 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:43:48.283 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:43:48.284 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:43:48.284 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc1556d700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 legacy=0x7fbc101d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc1556d700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 legacy=0x7fbc101d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc1556d700 1 --1- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 0x7fbc101d2300 :-1 s=OPENED pgs=523 cs=1 l=1).handle_message read tag failed 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc1556d700 1 --1- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 0x7fbc101d2300 :-1 s=OPENED pgs=523 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbc1009a250 legacy=0x7fbc101d2300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:43:48.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.b con 0x7fbbf403ff00 addr v1:172.21.15.189:6789/0 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.c con 0x7fbc10099e40 addr v1:172.21.15.76:6790/0 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.a con 0x7fbc10099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc101b6a40 con 0x7fbc10099a60 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf40035d0 con 0x7fbbf403ff00 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf404d4e0 con 0x7fbc10099e40 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbf403ff00 legacy=0x7fbbf4042330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:43:48.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306856940 0 0) 0x7fbc101b6a40 con 0x7fbc10099a60 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbbf404b480 con 0x7fbc10099a60 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447760647 0 0) 0x7fbbf404d4e0 con 0x7fbc10099e40 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:43:48.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.285+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbc101b6a40 con 0x7fbc10099e40 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3088316067 0 0) 0x7fbc101b6a40 con 0x7fbc10099e40 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbf403ff00 legacy=0x7fbbf4042330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbbf4049920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 monclient: found mon.c 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fbbf404e760 con 0x7fbc10099e40 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:43:48.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbc08015450 con 0x7fbc10099e40 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: dump: 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:43:48.294 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08012f90 con 0x7fbc10099e40 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0effd700 4 set_mon_vals no callback set 2024-09-16T06:43:48.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.289+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbc080416f0 con 0x7fbc10099e40 2024-09-16T06:43:48.324 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:43:48.324 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T06:43:48.325 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T06:43:48.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:43:48.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.349+0000 7fbc16fd0700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:43:48.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.349+0000 7fbc16fd0700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:43:48.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.349+0000 7fbc16fd0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:43:48.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:48.349+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7fbbf0005930 con 0x7fbc10099e40 2024-09-16T06:43:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:51.173+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:51.173+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:51.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:51.173+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:51.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:51.173+0000 7fbbfbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:43:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:52.173+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:52.173+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:52.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:52.173+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:53.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:53.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:53.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:53.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:53.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:53.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:54.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:54.173+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:54.173+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:54.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:54.173+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:55.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:55.173+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:55.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:55.173+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:55.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:55.173+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:56.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:56.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:56.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:56.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:56.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:56.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:57.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:57.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:57.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:57.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:57.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:57.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:58.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:58.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:58.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:58.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:58.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:58.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:43:59.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:59.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:43:59.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:59.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:43:59.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:43:59.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:00.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:00.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:00.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:00.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:00.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:00.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:01.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:01.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:01.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:01.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:01.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:01.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:01.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:01.177+0000 7fbbfbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:02.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:02.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:02.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:02.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:02.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:02.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:03.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:03.181+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:03.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:03.181+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:03.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:03.181+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:04.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:04.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:04.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:04.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:04.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:04.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:05.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:05.181+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:05.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:05.181+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:05.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:05.181+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:06.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:06.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:06.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:06.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:06.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:06.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:07.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:07.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:07.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:07.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:07.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:07.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:08.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:08.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:08.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:08.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:08.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:08.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:08.326 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T06:44:08.326 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T06:44:08.326 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:44:08.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: get_monmap_and_config 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: build_initial_monmap 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: monmap: 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:08.438 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:08.440142+0000 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:08.440142+0000 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:08.439 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 1 Processor -- start 2024-09-16T06:44:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 1 -- start start 2024-09-16T06:44:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: init 2024-09-16T06:44:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-a con 0x7f2b50132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-b con 0x7f2b5006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-c con 0x7f2b5006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5006c280 con 0x7f2b50132330 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5006bfa0 con 0x7f2b5006cf50 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5009ace0 con 0x7f2b5006d480 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b565c8700 10 monclient(hunting): authenticate will time out at 1195.574022s 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b55dc7700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b5013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b555c6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b50132330 0x7f2b50132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42210/0 (socket says 172.21.15.76:42210) 2024-09-16T06:44:08.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.439+0000 7f2b555c6700 1 -- 172.21.15.76:0/2477957026 learned_addr learned my addr 172.21.15.76:0/2477957026 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:08.643+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b5013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:09.047+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b5013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:09.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:09.177+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:09.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:09.177+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:09.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:09.177+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:09.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:09.851+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b5013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:10.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.181+0000 7fbbfbfff700 10 monclient: tick 2024-09-16T06:44:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.181+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:10.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.181+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:10.764 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc1556d700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 legacy=0x7fbbf4045d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:44:10.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc1556d700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 legacy=0x7fbbf4045d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:44:10.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc1556d700 1 --1- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 0x7fbbf4045d80 :-1 s=OPENED pgs=795 cs=1 l=1).handle_message read tag failed 2024-09-16T06:44:10.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc1556d700 1 --1- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 0x7fbbf4045d80 :-1 s=OPENED pgs=795 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:44:10.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6790/0 2024-09-16T06:44:10.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:10.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc10099e40 legacy=0x7fbbf4045d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:10.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:10.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.b con 0x7fbbf403ff00 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:10.766 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.c con 0x7fbc1009a250 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient(hunting): picked mon.a con 0x7fbc10099a60 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbc101b6a40 con 0x7fbc10099a60 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf40035d0 con 0x7fbbf403ff00 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf404d4e0 con 0x7fbc1009a250 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.757+0000 7fbc0d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:10.767 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.761+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbf403ff00 legacy=0x7fbbf40422f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:10.763+0000 7f2b54dc5700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 legacy=0x7f2b5013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T06:44:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:10.763+0000 7f2b54dc5700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 legacy=0x7f2b5013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:44:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:10.763+0000 7f2b54dc5700 1 --1- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 0x7f2b5013a330 :-1 s=OPENED pgs=809 cs=1 l=1).handle_message read tag failed 2024-09-16T06:44:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:10.763+0000 7f2b54dc5700 1 --1- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 0x7f2b5013a330 :-1 s=OPENED pgs=809 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:44:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:10.763+0000 7f2b4ffff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:44:10.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:10.965+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbf403ff00 legacy=0x7fbbf40422f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:11.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): tick 2024-09-16T06:44:11.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 monclient(hunting): continuing hunt 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbc1009a250 legacy=0x7fbbf404d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbf403ff00 legacy=0x7fbbf40422f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6789/0 conn(0x7fbc10099a60 legacy=0x7fbbf4049920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): picked mon.b con 0x7fbbe80036c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): picked mon.c con 0x7fbbe8007350 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): picked mon.a con 0x7fbbe800b190 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf404e0a0 con 0x7fbbe800b190 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbf40035d0 con 0x7fbbe80036c0 2024-09-16T06:44:11.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbe80036c0 legacy=0x7fbbe8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:11.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbbe8011230 con 0x7fbbe8007350 2024-09-16T06:44:11.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.181+0000 7fbbfbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:11.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.385+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbe80036c0 legacy=0x7fbbe8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:11.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): tick 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 monclient(hunting): continuing hunt 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 legacy=0x7f2b5013a330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b5013da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6789/0 conn(0x7f2b50132330 legacy=0x7f2b50132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): picked mon.noname-b con 0x7f2b5006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): picked mon.noname-a con 0x7f2b30006e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): picked mon.noname-c con 0x7f2b5006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:11.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5006bfa0 con 0x7f2b30006e00 2024-09-16T06:44:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3000f2b0 con 0x7f2b5006cf50 2024-09-16T06:44:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 1 -- 172.21.15.76:0/2477957026 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b300104c0 con 0x7f2b5006d480 2024-09-16T06:44:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b4effd700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:11.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.439+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b3000cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:11.643+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b3000cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:11.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:11.789+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbe80036c0 legacy=0x7fbbe8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:12.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:12.047+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b3000cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:12.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:12.593+0000 7fbc0ffff700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbe80036c0 legacy=0x7fbbe8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:12.851+0000 7f2b55dc7700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b3000cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.369+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290645930 0 0) 0x7fbbf404e0a0 con 0x7fbbe800b190 2024-09-16T06:44:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.369+0000 7fbc0d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.369+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.369+0000 7fbc0d7fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:44:13.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.369+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbbf404e230 con 0x7fbbe800b190 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314674198 0 0) 0x7f2b5006bfa0 con 0x7f2b30006e00 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient(hunting): my global_id is 154476 2024-09-16T06:44:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b3c0035d0 con 0x7f2b30006e00 2024-09-16T06:44:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2003741788 0 0) 0x7fbbf404e230 con 0x7fbbe800b190 2024-09-16T06:44:13.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6790/0 conn(0x7fbbe8007350 legacy=0x7fbbe80097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6789/0 conn(0x7fbbe80036c0 legacy=0x7fbbe8005ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 monclient: found mon.a 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fbbe8012440 con 0x7fbbe800b190 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7fbbf40035d0 con 0x7fbbe800b190 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbbfc001cf0 con 0x7fbbe800b190 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.373+0000 7fbc0d7fa700 10 monclient: dump: 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.379 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368375845 0 0) 0x7f2b3c0035d0 con 0x7f2b30006e00 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6790/0 conn(0x7f2b5006d480 legacy=0x7f2b300056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b3000cad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 monclient: found mon.noname-a 2024-09-16T06:44:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b300116d0 con 0x7f2b30006e00 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b565c8700 5 monclient: authenticate success, global_id 154476 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b565c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b380019f0 con 0x7f2b30006e00 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.375+0000 7f2b4ffff700 10 monclient: dump: 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.379+0000 7f2b565c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.377+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc001b50 con 0x7fbbe800b190 2024-09-16T06:44:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.377+0000 7fbc0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.377+0000 7fbc0e7fc700 4 set_mon_vals no callback set 2024-09-16T06:44:13.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:13.377+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbbfc005980 con 0x7fbbe800b190 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.379+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b38001c90 con 0x7f2b30006e00 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.379+0000 7f2b4ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 4 set_mon_vals no callback set 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4ffff700 1 -- 172.21.15.76:0/2477957026 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b38002060 con 0x7f2b30006e00 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4ffff700 10 monclient: dump: 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: shutdown 2024-09-16T06:44:13.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- 172.21.15.76:0/2477957026 >> v1:172.21.15.76:6789/0 conn(0x7f2b30006e00 legacy=0x7f2b30009290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- 172.21.15.76:0/2477957026 shutdown_connections 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- 172.21.15.76:0/2477957026 >> 172.21.15.76:0/2477957026 conn(0x7f2b500a41c0 msgr2=0x7f2b500a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- 172.21.15.76:0/2477957026 shutdown_connections 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- 172.21.15.76:0/2477957026 wait complete. 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: build_initial_monmap 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: monmap: 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:13.391 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 Processor -- start 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- start start 2024-09-16T06:44:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: init 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-c con 0x7f2b30006e00 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-b con 0x7f2b5006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient(hunting): picked mon.noname-a con 0x7f2b5006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50124530 con 0x7f2b5006d480 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b50122e10 con 0x7f2b5006cf50 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500a5770 con 0x7f2b30006e00 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:13.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b565c8700 10 monclient(hunting): authenticate will time out at 1200.519558s 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b54dc5700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b501cee10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b55dc7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b5006d480 0x7f2b501d2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42256/0 (socket says 172.21.15.76:42256) 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b55dc7700 1 -- 172.21.15.76:0/3612274731 learned_addr learned my addr 172.21.15.76:0/3612274731 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716320048 0 0) 0x7f2b50124530 con 0x7f2b5006d480 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 10 monclient(hunting): my global_id is 154479 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.383+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b30006190 con 0x7f2b5006d480 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901541527 0 0) 0x7f2b30006190 con 0x7f2b5006d480 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b50124530 con 0x7f2b5006d480 2024-09-16T06:44:13.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b480042c0 con 0x7f2b5006d480 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient(hunting): dump: 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 503336323 0 0) 0x7f2b50124530 con 0x7f2b5006d480 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 >> v1:172.21.15.76:6790/0 conn(0x7f2b30006e00 legacy=0x7f2b501c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 >> v1:172.21.15.189:6789/0 conn(0x7f2b5006cf50 legacy=0x7f2b501cee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 monclient: found mon.a 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50124250 con 0x7f2b5006d480 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 5 monclient: authenticate success, global_id 154479 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _renew_subs 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b501221d0 con 0x7f2b5006d480 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _renew_subs 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b50123f70 con 0x7f2b5006d480 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b480047d0 con 0x7f2b5006d480 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b48004ae0 con 0x7f2b5006d480 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4f7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 10 monclient: dump: 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2b48004c40 con 0x7f2b5006d480 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7f2b48043e40 con 0x7f2b5006d480 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b34005100 con 0x7f2b5006d480 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.387+0000 7f2b555c6700 10 monclient: get_auth_request con 0x7f2b30036ae0 auth_method 0 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.391+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b48014740 con 0x7f2b5006d480 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.391+0000 7f2b4dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.391+0000 7f2b4dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b565c8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:44:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b565c8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b565c8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2b34005930 con 0x7f2b5006d480 2024-09-16T06:44:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b4dffb700 1 -- 172.21.15.76:0/3612274731 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1740837484) 0x7f2b48012bb0 con 0x7f2b5006d480 2024-09-16T06:44:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b4dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:13.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.627+0000 7f2b4dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:13.629 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:44:13.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2b30036ae0 legacy=0x7f2b30038fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 10 monclient: shutdown 2024-09-16T06:44:13.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 >> v1:172.21.15.76:6789/0 conn(0x7f2b5006d480 legacy=0x7f2b501d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 shutdown_connections 2024-09-16T06:44:13.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 >> 172.21.15.76:0/3612274731 conn(0x7f2b500a41c0 msgr2=0x7f2b500a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 shutdown_connections 2024-09-16T06:44:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.631+0000 7f2b565c8700 1 -- 172.21.15.76:0/3612274731 wait complete. 2024-09-16T06:44:13.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:44:13.643 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:44:13.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.747+0000 7f44ba360700 10 monclient: get_monmap_and_config 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.747+0000 7f44ba360700 10 monclient: build_initial_monmap 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.747+0000 7f44ba360700 10 monclient: monmap: 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.749 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:13.751355+0000 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:13.751355+0000 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:13.750 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 Processor -- start 2024-09-16T06:44:13.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- start start 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient: init 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-c con 0x7f44b4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-b con 0x7f44b4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-a con 0x7f44b4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4098990 con 0x7f44b4139500 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4097f40 con 0x7f44b4097b60 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b409d610 con 0x7f44b4099090 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient(hunting): authenticate will time out at 1200.885053s 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44b935e700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f44b4097b60 legacy=0x7f44b409ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44b9b5f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f44b4139500 0x7f44b413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42268/0 (socket says 172.21.15.76:42268) 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44b9b5f700 1 -- 172.21.15.76:0/3354111314 learned_addr learned my addr 172.21.15.76:0/3354111314 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051596400 0 0) 0x7f44b4098990 con 0x7f44b4139500 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): my global_id is 154482 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a00035d0 con 0x7f44b4139500 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982143459 0 0) 0x7f44b409d610 con 0x7f44b4099090 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient(hunting): my global_id is 125315 2024-09-16T06:44:13.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4098990 con 0x7f44b4099090 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922792138 0 0) 0x7f44a00035d0 con 0x7f44b4139500 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 >> v1:172.21.15.76:6790/0 conn(0x7f44b4099090 legacy=0x7f44b409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 >> v1:172.21.15.189:6789/0 conn(0x7f44b4097b60 legacy=0x7f44b409ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 monclient: found mon.noname-a 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40986b0 con 0x7f44b4139500 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44ac001990 con 0x7f44b4139500 2024-09-16T06:44:13.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: dump: 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 5 monclient: authenticate success, global_id 154482 2024-09-16T06:44:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44ac001c70 con 0x7f44b4139500 2024-09-16T06:44:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 4 set_mon_vals no callback set 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 1 -- 172.21.15.76:0/3354111314 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44ac002000 con 0x7f44b4139500 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44abfff700 10 monclient: dump: 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 10 monclient: shutdown 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- 172.21.15.76:0/3354111314 >> v1:172.21.15.76:6789/0 conn(0x7f44b4139500 legacy=0x7f44b413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- 172.21.15.76:0/3354111314 shutdown_connections 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- 172.21.15.76:0/3354111314 >> 172.21.15.76:0/3354111314 conn(0x7f44b40a0030 msgr2=0x7f44b40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- 172.21.15.76:0/3354111314 shutdown_connections 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.751+0000 7f44ba360700 1 -- 172.21.15.76:0/3354111314 wait complete. 2024-09-16T06:44:13.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: build_initial_monmap 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: monmap: 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:13.761 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 Processor -- start 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- start start 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: init 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-c con 0x7f44b4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-a con 0x7f44b4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient(hunting): picked mon.noname-b con 0x7f44b4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4139330 con 0x7f44b4099090 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4091570 con 0x7f44b4139500 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b413c0c0 con 0x7f44b4097b60 2024-09-16T06:44:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient(hunting): authenticate will time out at 1200.888982s 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44b8b5d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f44b4099090 0x7f44b4138c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42276/0 (socket says 172.21.15.76:42276) 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44b8b5d700 1 -- 172.21.15.76:0/3873500032 learned_addr learned my addr 172.21.15.76:0/3873500032 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44b9b5f700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.189:6789/0 conn(0x7f44b4139500 legacy=0x7f44b41d1f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146974088 0 0) 0x7f44b4139330 con 0x7f44b4099090 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): my global_id is 154485 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44980035d0 con 0x7f44b4099090 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145104864 0 0) 0x7f44b413c0c0 con 0x7f44b4097b60 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): my global_id is 125318 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4139330 con 0x7f44b4097b60 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693685542 0 0) 0x7f44980035d0 con 0x7f44b4099090 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b413c0c0 con 0x7f44b4099090 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44a4001910 con 0x7f44b4099090 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083268513 0 0) 0x7f44b4139330 con 0x7f44b4097b60 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44980035d0 con 0x7f44b4097b60 2024-09-16T06:44:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44b00041f0 con 0x7f44b4097b60 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.771 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3304961918 0 0) 0x7f44b413c0c0 con 0x7f44b4099090 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6790/0 conn(0x7f44b4097b60 legacy=0x7f44b4138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.189:6789/0 conn(0x7f44b4139500 legacy=0x7f44b41d1f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 monclient: found mon.a 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40a2c80 con 0x7f44b4099090 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 5 monclient: authenticate success, global_id 154485 2024-09-16T06:44:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _renew_subs 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b409a540 con 0x7f44b4099090 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _renew_subs 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b408fb60 con 0x7f44b4099090 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44a4001ca0 con 0x7f44b4099090 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44a4003b10 con 0x7f44b4099090 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ab7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient: dump: 2024-09-16T06:44:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4494006590 con 0x7f4494002830 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998168008 0 0) 0x7f4494006590 con 0x7f4494002830 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 10 monclient(hunting): my global_id is 154488 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.755+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b413c0c0 con 0x7f4494002830 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 967039366 0 0) 0x7f44b413c0c0 con 0x7f4494002830 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4494006590 con 0x7f4494002830 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44b0004830 con 0x7f4494002830 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 10 monclient: dump: 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f44a4003d30 con 0x7f44b4099090 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7f44a4043db0 con 0x7f44b4099090 2024-09-16T06:44:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 693615085 0 0) 0x7f4494006590 con 0x7f4494002830 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f449400a190 con 0x7f4494002830 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44b9b5f700 10 monclient: get_auth_request con 0x7f4498036300 auth_method 0 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f44b0006b20 con 0x7f4494002830 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6789/0 conn(0x7f4494002830 legacy=0x7f4494004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.759+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6789/0 conn(0x7f4494002830 legacy=0x7f4494004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44ba360700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4494009e50 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076178287 0 0) 0x7f4494009e50 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient(hunting): my global_id is 154491 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494006590 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2922514275 0 0) 0x7f4494006590 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4494009e50 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44b00061d0 con 0x7f4494009110 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 10 monclient: dump: 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2085411105 0 0) 0x7f4494009e50 con 0x7f4494009110 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.767+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f449400c340 con 0x7f4494009110 2024-09-16T06:44:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 314987135) 0x7f44b0006640 con 0x7f4494009110 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44a9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44a9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6789/0 conn(0x7f4494009110 legacy=0x7f44940095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44a9ffb700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6789/0 conn(0x7f4494009110 legacy=0x7f44940095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:13.781 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 56, 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.783 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.784 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:13.785 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.786 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.787 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:13.795 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4498036300 legacy=0x7f44980387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 10 monclient: shutdown 2024-09-16T06:44:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 >> v1:172.21.15.76:6789/0 conn(0x7f44b4099090 legacy=0x7f44b4138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44b9b5f700 1 -- 172.21.15.76:0/3873500032 reap_dead start 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44b9b5f700 1 -- 172.21.15.76:0/3873500032 reap_dead start 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 shutdown_connections 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 >> 172.21.15.76:0/3873500032 conn(0x7f44b40a0030 msgr2=0x7f44b4071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 shutdown_connections 2024-09-16T06:44:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.771+0000 7f44ba360700 1 -- 172.21.15.76:0/3873500032 wait complete. 2024-09-16T06:44:13.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:44:13.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:44:13.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:44:13.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: get_monmap_and_config 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: build_initial_monmap 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: monmap: 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:13.916832+0000 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:13.916832+0000 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:13.915 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:13.916 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 1 Processor -- start 2024-09-16T06:44:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 1 -- start start 2024-09-16T06:44:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: init 2024-09-16T06:44:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:13.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-b con 0x7f21d0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-a con 0x7f21d006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-c con 0x7f21d006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d006c280 con 0x7f21d006d480 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d006bfa0 con 0x7f21d006cf50 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f21d006d480 0x7f21d013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:40318/0 (socket says 172.21.15.76:40318) 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cffff700 1 -- 172.21.15.76:0/3472245028 learned_addr learned my addr 172.21.15.76:0/3472245028 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d009ace0 con 0x7f21d0132330 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d5fbc700 10 monclient(hunting): authenticate will time out at 1201.051308s 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21d57bb700 1 -- 172.21.15.76:0/3472245028 >> v1:172.21.15.189:6789/0 conn(0x7f21d006cf50 legacy=0x7f21d013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510161308 0 0) 0x7f21d006c280 con 0x7f21d006d480 2024-09-16T06:44:13.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): my global_id is 154494 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21bc0035d0 con 0x7f21d006d480 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353076947 0 0) 0x7f21d009ace0 con 0x7f21d0132330 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 10 monclient(hunting): my global_id is 125321 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.915+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d006c280 con 0x7f21d0132330 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061880298 0 0) 0x7f21bc0035d0 con 0x7f21d006d480 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 >> v1:172.21.15.76:6790/0 conn(0x7f21d0132330 legacy=0x7f21d0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 >> v1:172.21.15.189:6789/0 conn(0x7f21d006cf50 legacy=0x7f21d013da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 monclient: found mon.noname-a 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d009ae70 con 0x7f21d006d480 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 5 monclient: authenticate success, global_id 154494 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21c0001610 con 0x7f21d006d480 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: dump: 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21c00018f0 con 0x7f21d006d480 2024-09-16T06:44:13.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 4 set_mon_vals no callback set 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 1 -- 172.21.15.76:0/3472245028 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21c0001ae0 con 0x7f21d006d480 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cf7fe700 10 monclient: dump: 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.924 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.924 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: shutdown 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 >> v1:172.21.15.76:6789/0 conn(0x7f21d006d480 legacy=0x7f21d013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 shutdown_connections 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 >> 172.21.15.76:0/3472245028 conn(0x7f21d00a41c0 msgr2=0x7f21d00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 shutdown_connections 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3472245028 wait complete. 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: build_initial_monmap 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: monmap: 2024-09-16T06:44:13.930 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 Processor -- start 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- start start 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: init 2024-09-16T06:44:13.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-a con 0x7f21d006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-b con 0x7f21d006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient(hunting): picked mon.noname-c con 0x7f21d0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d012f7a0 con 0x7f21d006cf50 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d0131820 con 0x7f21d006d480 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d01312d0 con 0x7f21d0132330 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d5fbc700 10 monclient(hunting): authenticate will time out at 1201.055419s 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21cffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f21d006d480 legacy=0x7f21d01c6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d57bb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f21d0132330 0x7f21d01d1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37242/0 (socket says 172.21.15.76:37242) 2024-09-16T06:44:13.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d57bb700 1 -- 172.21.15.76:0/3839619894 learned_addr learned my addr 172.21.15.76:0/3839619894 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.919+0000 7f21d4fba700 1 --1- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6789/0 conn(0x7f21d006cf50 0x7f21d01c64a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:40322/0 (socket says 172.21.15.76:40322) 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921132004 0 0) 0x7f21d006c280 con 0x7f21d006cf50 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): my global_id is 154497 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b00035d0 con 0x7f21d006cf50 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114392734 0 0) 0x7f21d01312d0 con 0x7f21d0132330 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): my global_id is 125324 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d006c280 con 0x7f21d0132330 2024-09-16T06:44:13.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761032827 0 0) 0x7f21b00035d0 con 0x7f21d006cf50 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d01312d0 con 0x7f21d006cf50 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21b8001710 con 0x7f21d006cf50 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167665295 0 0) 0x7f21d006c280 con 0x7f21d0132330 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b00035d0 con 0x7f21d0132330 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21c8004470 con 0x7f21d0132330 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.935 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3938927890 0 0) 0x7f21b00035d0 con 0x7f21d0132330 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.189:6789/0 conn(0x7f21d006d480 legacy=0x7f21d01c6bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6789/0 conn(0x7f21d006cf50 legacy=0x7f21d01c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 monclient: found mon.c 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d01c72c0 con 0x7f21d0132330 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 5 monclient: authenticate success, global_id 125324 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 10 monclient: _renew_subs 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d0132f90 con 0x7f21d0132330 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 10 monclient: _renew_subs 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d009e910 con 0x7f21d0132330 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21c8004650 con 0x7f21d0132330 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21c80049e0 con 0x7f21d0132330 2024-09-16T06:44:13.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21ce7fc700 4 set_mon_vals no callback set 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: dump: 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.938 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f21c8004b40 con 0x7f21d0132330 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f21b4006590 con 0x7f21b4002830 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(136..136 src has 66..136) v4 ==== 6720+0+0 (unknown 455270309 0 0) 0x7f21c8043fe0 con 0x7f21d0132330 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756484626 0 0) 0x7f21b4006590 con 0x7f21b4002830 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient(hunting): my global_id is 125327 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b00035d0 con 0x7f21b4002830 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cffff700 10 monclient: get_auth_request con 0x7f21b00362b0 auth_method 0 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1661288104 0 0) 0x7f21b00035d0 con 0x7f21b4002830 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b4006590 con 0x7f21b4002830 2024-09-16T06:44:13.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21b80036a0 con 0x7f21b4002830 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.923+0000 7f21cd7fa700 10 monclient: dump: 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3924808424 0 0) 0x7f21b4006590 con 0x7f21b4002830 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b400a190 con 0x7f21b4002830 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f21b8005870 con 0x7f21b4002830 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6790/0 conn(0x7f21b4002830 legacy=0x7f21b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.927+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6790/0 conn(0x7f21b4002830 legacy=0x7f21b4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21d5fbc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f21b4009e30 con 0x7f21b4009150 2024-09-16T06:44:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612308722 0 0) 0x7f21b4009e30 con 0x7f21b4009150 2024-09-16T06:44:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient(hunting): my global_id is 125330 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b4006590 con 0x7f21b4009150 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1842760062 0 0) 0x7f21b4006590 con 0x7f21b4009150 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b4009e30 con 0x7f21b4009150 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f21b8003d70 con 0x7f21b4009150 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient: dump: 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1242435436 0 0) 0x7f21b4009e30 con 0x7f21b4009150 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21b400c340 con 0x7f21b4009150 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3870 (unknown 0 0 358277170) 0x7f21b80059e0 con 0x7f21b4009150 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6790/0 conn(0x7f21b4009150 legacy=0x7f21b40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.935+0000 7f21cd7fa700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6790/0 conn(0x7f21b4009150 legacy=0x7f21b40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 56, 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:13.945 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.946 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:13.948 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:13.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.952 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:13.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:13.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.955 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:13.956 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.189:6812/3098810674 conn(0x7f21b00362b0 legacy=0x7f21b0038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 10 monclient: shutdown 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d57bb700 1 -- 172.21.15.76:0/3839619894 reap_dead start 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 >> v1:172.21.15.76:6790/0 conn(0x7f21d0132330 legacy=0x7f21d01d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 shutdown_connections 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 >> 172.21.15.76:0/3839619894 conn(0x7f21d00a41c0 msgr2=0x7f21d009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 shutdown_connections 2024-09-16T06:44:13.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:13.939+0000 7f21d5fbc700 1 -- 172.21.15.76:0/3839619894 wait complete. 2024-09-16T06:44:13.959 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:44:13.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:44:14.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.381+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v137) v1 ==== 111+0+0 (unknown 3095812247 0 0) 0x7fbbfc016380 con 0x7fbbe800b190 2024-09-16T06:44:14.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.381+0000 7fbc0d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:44:14.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.381+0000 7fbc0d7fa700 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-16T06:44:14.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.489+0000 7fbc16fd0700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:44:14.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.489+0000 7fbc16fd0700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:44:14.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.489+0000 7fbc16fd0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.489+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7fbbf0002310 con 0x7fbbe800b190 2024-09-16T06:44:14.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc0d7fa700 1 -- 172.21.15.189:0/3248132267 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v137) v1 ==== 118+0+0 (unknown 3041623357 0 0) 0x7fbbfc014e60 con 0x7fbbe800b190 2024-09-16T06:44:14.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc0d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T06:44:14.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc0d7fa700 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-16T06:44:14.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbbf4036360 legacy=0x7fbbf4038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:14.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 10 monclient: shutdown 2024-09-16T06:44:14.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:14.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 >> v1:172.21.15.76:6789/0 conn(0x7fbbe800b190 legacy=0x7fbbe800d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:14.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 shutdown_connections 2024-09-16T06:44:14.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 >> 172.21.15.189:0/3248132267 conn(0x7fbc100a03e0 msgr2=0x7fbc1006db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:14.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 shutdown_connections 2024-09-16T06:44:14.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.493+0000 7fbc16fd0700 1 -- 172.21.15.189:0/3248132267 wait complete. 2024-09-16T06:44:14.501 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cold' already exists 2024-09-16T06:44:14.510 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool set cold size 2 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: get_monmap_and_config 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: build_initial_monmap 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: monmap: 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:14.621757+0000 2024-09-16T06:44:14.621 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:14.621757+0000 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:14.622 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 1 Processor -- start 2024-09-16T06:44:14.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 1 -- start start 2024-09-16T06:44:14.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: init 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a809f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a8099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-a con 0x7fe5a81396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a809f520 con 0x7fe5a81396a0 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a809bc10 con 0x7fe5a8099700 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a809b810 con 0x7fe5a809f140 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5b07c0700 10 monclient(hunting): authenticate will time out at 1207.134733s 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5add5b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe5a8099700 legacy=0x7fe5a809b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ae55c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe5a809f140 0x7fe5a809a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:59962/0 (socket says 172.21.15.189:59962) 2024-09-16T06:44:14.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ae55c700 1 -- 172.21.15.189:0/402431751 learned_addr learned my addr 172.21.15.189:0/402431751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552041308 0 0) 0x7fe5a809f520 con 0x7fe5a81396a0 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): my global_id is 154500 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5940035d0 con 0x7fe5a81396a0 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626670636 0 0) 0x7fe5a809b810 con 0x7fe5a809f140 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 10 monclient(hunting): my global_id is 125333 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.617+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a809f520 con 0x7fe5a809f140 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852939979 0 0) 0x7fe5a809f520 con 0x7fe5a809f140 2024-09-16T06:44:14.627 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 >> v1:172.21.15.189:6789/0 conn(0x7fe5a8099700 legacy=0x7fe5a809b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 >> v1:172.21.15.76:6789/0 conn(0x7fe5a81396a0 legacy=0x7fe5a813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 monclient: found mon.noname-b 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a809ceb0 con 0x7fe5a809f140 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: _finish_auth 0 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe5980014a0 con 0x7fe5a809f140 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 5 monclient: authenticate success, global_id 125333 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:14.628 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: dump: 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5980016c0 con 0x7fe5a809f140 2024-09-16T06:44:14.629 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 1 -- 172.21.15.189:0/402431751 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe598001a50 con 0x7fe5a809f140 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 4 set_mon_vals no callback set 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5ad55a700 10 monclient: dump: 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:14.630 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 10 monclient: shutdown 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- 172.21.15.189:0/402431751 >> v1:172.21.15.76:6790/0 conn(0x7fe5a809f140 legacy=0x7fe5a809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- 172.21.15.189:0/402431751 shutdown_connections 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- 172.21.15.189:0/402431751 >> 172.21.15.189:0/402431751 conn(0x7fe5a80a03e0 msgr2=0x7fe5a80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- 172.21.15.189:0/402431751 shutdown_connections 2024-09-16T06:44:14.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- 172.21.15.189:0/402431751 wait complete. 2024-09-16T06:44:14.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 10 monclient: build_initial_monmap 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 10 monclient: monmap: 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 Processor -- start 2024-09-16T06:44:14.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 1 -- start start 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.621+0000 7fe5b07c0700 10 monclient: init 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-a con 0x7fe5a8099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a809a6e0 con 0x7fe5a8099700 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80983a0 con 0x7fe5a809f140 2024-09-16T06:44:14.636 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5add5b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe5a809f140 legacy=0x7fe5a80705a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80a2c90 con 0x7fe5a81396a0 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient(hunting): authenticate will time out at 1207.140349s 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5ae55c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe5a8099700 0x7fe5a806fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60114/0 (socket says 172.21.15.189:60114) 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5ae55c700 1 -- 172.21.15.189:0/2834277200 learned_addr learned my addr 172.21.15.189:0/2834277200 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:14.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742464636 0 0) 0x7fe5a809a6e0 con 0x7fe5a8099700 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): my global_id is 154503 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c0035d0 con 0x7fe5a8099700 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802744426 0 0) 0x7fe5a80a2c90 con 0x7fe5a81396a0 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): my global_id is 125336 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a809a6e0 con 0x7fe5a81396a0 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210097970 0 0) 0x7fe58c0035d0 con 0x7fe5a8099700 2024-09-16T06:44:14.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a80a2c90 con 0x7fe5a8099700 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482842582 0 0) 0x7fe5a809a6e0 con 0x7fe5a81396a0 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe58c0035d0 con 0x7fe5a81396a0 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe5980019d0 con 0x7fe5a8099700 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): dump: 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe5a0001bd0 con 0x7fe5a81396a0 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient(hunting): dump: 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:14.640 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2272981965 0 0) 0x7fe5a80a2c90 con 0x7fe5a8099700 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 >> v1:172.21.15.76:6790/0 conn(0x7fe5a81396a0 legacy=0x7fe5a81d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 >> v1:172.21.15.189:6789/0 conn(0x7fe5a809f140 legacy=0x7fe5a80705a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 monclient: found mon.a 2024-09-16T06:44:14.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a8098960 con 0x7fe5a8099700 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: _finish_auth 0 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 5 monclient: authenticate success, global_id 154503 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _renew_subs 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a8132350 con 0x7fe5a8099700 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _renew_subs 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a8070cb0 con 0x7fe5a8099700 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe598004b40 con 0x7fe5a8099700 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe598004cb0 con 0x7fe5a8099700 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59ffff700 4 set_mon_vals no callback set 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe59effd700 10 monclient: dump: 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:14.643 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.625+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe590005100 con 0x7fe5a8099700 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.629+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe598004ed0 con 0x7fe5a8099700 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.629+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(137..137 src has 66..137) v4 ==== 7063+0+0 (unknown 3512353167 0 0) 0x7fe598043f20 con 0x7fe5a8099700 2024-09-16T06:44:14.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.629+0000 7fe5add5b700 10 monclient: get_auth_request con 0x7fe58c0362b0 auth_method 0 2024-09-16T06:44:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.637+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe598014610 con 0x7fe5a8099700 2024-09-16T06:44:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.637+0000 7fe59effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:14.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.637+0000 7fe59effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.805+0000 7fe5b07c0700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.805+0000 7fe5b07c0700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.805+0000 7fe5b07c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:14.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:14.805+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fe590005930 con 0x7fe5a8099700 2024-09-16T06:44:15.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.389+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v138) v1 ==== 124+0+0 (unknown 3491643649 0 0) 0x7fe598012cb0 con 0x7fe5a8099700 2024-09-16T06:44:15.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.389+0000 7fe59effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:15.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.389+0000 7fe59effd700 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-16T06:44:15.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.497+0000 7fe5b07c0700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:15.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.497+0000 7fe5b07c0700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:15.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.497+0000 7fe5b07c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:15.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:15.497+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fe5900053b0 con 0x7fe5a8099700 2024-09-16T06:44:16.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.397+0000 7fe59effd700 1 -- 172.21.15.189:0/2834277200 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v139) v1 ==== 124+0+0 (unknown 2752074814 0 0) 0x7fe598014430 con 0x7fe5a8099700 2024-09-16T06:44:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.397+0000 7fe59effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T06:44:16.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.397+0000 7fe59effd700 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-16T06:44:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe58c0362b0 legacy=0x7fe58c038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 10 monclient: shutdown 2024-09-16T06:44:16.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 >> v1:172.21.15.76:6789/0 conn(0x7fe5a8099700 legacy=0x7fe5a806fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 shutdown_connections 2024-09-16T06:44:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 >> 172.21.15.189:0/2834277200 conn(0x7fe5a80a03e0 msgr2=0x7fe5a80a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 shutdown_connections 2024-09-16T06:44:16.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.401+0000 7fe5b07c0700 1 -- 172.21.15.189:0/2834277200 wait complete. 2024-09-16T06:44:16.409 INFO:tasks.workunit.client.0.smithi189.stderr:set pool 12 size to 2 2024-09-16T06:44:16.422 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 10 monclient: get_monmap_and_config 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 10 monclient: build_initial_monmap 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 10 monclient: monmap: 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:16.535653+0000 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:16.535653+0000 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:16.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:16.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:16.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:16.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 1 Processor -- start 2024-09-16T06:44:16.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 1 -- start start 2024-09-16T06:44:16.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.529+0000 7efd34b5b700 10 monclient: init 2024-09-16T06:44:16.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-c con 0x7efd3009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-b con 0x7efd30099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-a con 0x7efd301396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3009f520 con 0x7efd301396a0 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3009bc10 con 0x7efd3009f140 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3009b810 con 0x7efd30099700 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient(hunting): authenticate will time out at 1209.048724s 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2e59c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efd3009f140 legacy=0x7efd3009a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efd301396a0 0x7efd3013ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60116/0 (socket says 172.21.15.189:60116) 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2ed9d700 1 -- 172.21.15.189:0/754949378 learned_addr learned my addr 172.21.15.189:0/754949378 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:16.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2dd9b700 1 --1- 172.21.15.189:0/754949378 >> v1:172.21.15.76:6790/0 conn(0x7efd30099700 0x7efd3009b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:59984/0 (socket says 172.21.15.189:59984) 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154425042 0 0) 0x7efd3009b810 con 0x7efd30099700 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): my global_id is 125339 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd180035d0 con 0x7efd30099700 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430388040 0 0) 0x7efd3009f520 con 0x7efd301396a0 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient(hunting): my global_id is 154506 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3009b810 con 0x7efd301396a0 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017043624 0 0) 0x7efd180035d0 con 0x7efd30099700 2024-09-16T06:44:16.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 >> v1:172.21.15.189:6789/0 conn(0x7efd3009f140 legacy=0x7efd3009a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 >> v1:172.21.15.76:6789/0 conn(0x7efd301396a0 legacy=0x7efd3013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 monclient: found mon.noname-b 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3009ceb0 con 0x7efd30099700 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: _finish_auth 0 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd24001630 con 0x7efd30099700 2024-09-16T06:44:16.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 5 monclient: authenticate success, global_id 125339 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: dump: 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:16.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd24001910 con 0x7efd30099700 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 1 -- 172.21.15.189:0/754949378 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd24001ca0 con 0x7efd30099700 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 4 set_mon_vals no callback set 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd2d59a700 10 monclient: dump: 2024-09-16T06:44:16.544 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 10 monclient: shutdown 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:16.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- 172.21.15.189:0/754949378 >> v1:172.21.15.76:6790/0 conn(0x7efd30099700 legacy=0x7efd3009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- 172.21.15.189:0/754949378 shutdown_connections 2024-09-16T06:44:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- 172.21.15.189:0/754949378 >> 172.21.15.189:0/754949378 conn(0x7efd300a03e0 msgr2=0x7efd300a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- 172.21.15.189:0/754949378 shutdown_connections 2024-09-16T06:44:16.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.533+0000 7efd34b5b700 1 -- 172.21.15.189:0/754949378 wait complete. 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient: build_initial_monmap 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient: monmap: 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:16.550 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 1 Processor -- start 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 1 -- start start 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient: init 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-a con 0x7efd30099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-c con 0x7efd3009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient(hunting): picked mon.noname-b con 0x7efd301396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd30092fc0 con 0x7efd30099700 2024-09-16T06:44:16.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd300918c0 con 0x7efd301396a0 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd300a31d0 con 0x7efd3009f140 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd2ed9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7efd301396a0 legacy=0x7efd301d2b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd34b5b700 10 monclient(hunting): authenticate will time out at 1209.055433s 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd2e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efd30099700 0x7efd30090aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60128/0 (socket says 172.21.15.189:60128) 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.537+0000 7efd2e59c700 1 -- 172.21.15.189:0/1802295619 learned_addr learned my addr 172.21.15.189:0/1802295619 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231741002 0 0) 0x7efd300a31d0 con 0x7efd3009f140 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): my global_id is 125342 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd100035d0 con 0x7efd3009f140 2024-09-16T06:44:16.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620869019 0 0) 0x7efd30092fc0 con 0x7efd30099700 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): my global_id is 154509 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd300a31d0 con 0x7efd30099700 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954484055 0 0) 0x7efd100035d0 con 0x7efd3009f140 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd30092fc0 con 0x7efd3009f140 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd240015f0 con 0x7efd3009f140 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): dump: 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:16.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1954336031 0 0) 0x7efd300a31d0 con 0x7efd30099700 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd100035d0 con 0x7efd30099700 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd20004200 con 0x7efd30099700 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:16.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient(hunting): dump: 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3706757518 0 0) 0x7efd30092fc0 con 0x7efd3009f140 2024-09-16T06:44:16.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 >> v1:172.21.15.189:6789/0 conn(0x7efd301396a0 legacy=0x7efd301d2b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 >> v1:172.21.15.76:6789/0 conn(0x7efd30099700 legacy=0x7efd30090aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 monclient: found mon.c 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3009a6e0 con 0x7efd3009f140 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: _finish_auth 0 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 5 monclient: authenticate success, global_id 125342 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _renew_subs 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd30093190 con 0x7efd3009f140 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _renew_subs 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd300946b0 con 0x7efd3009f140 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd24001250 con 0x7efd3009f140 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd24004d80 con 0x7efd3009f140 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd2cd99700 4 set_mon_vals no callback set 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 10 monclient: dump: 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:16.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efd24004ee0 con 0x7efd3009f140 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd14005100 con 0x7efd3009f140 2024-09-16T06:44:16.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(139..139 src has 66..139) v4 ==== 7063+0+0 (unknown 1106376450 0 0) 0x7efd24044040 con 0x7efd3009f140 2024-09-16T06:44:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.541+0000 7efd2e59c700 10 monclient: get_auth_request con 0x7efd100362c0 auth_method 0 2024-09-16T06:44:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.553+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efd24014790 con 0x7efd3009f140 2024-09-16T06:44:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.553+0000 7efd1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:16.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.553+0000 7efd1effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:16.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.729+0000 7efd34b5b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:16.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.729+0000 7efd34b5b700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:16.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.729+0000 7efd34b5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:16.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:16.729+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7efd140053b0 con 0x7efd3009f140 2024-09-16T06:44:17.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.401+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v140) v1 ==== 82+0+0 (unknown 537863739 0 0) 0x7efd24012e30 con 0x7efd3009f140 2024-09-16T06:44:17.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.401+0000 7efd1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:17.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.401+0000 7efd1effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:17.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd34b5b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:17.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd34b5b700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:17.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd34b5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7efd14005900 con 0x7efd3009f140 2024-09-16T06:44:17.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd1effd700 1 -- 172.21.15.189:0/1802295619 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v140) v1 ==== 115+0+0 (unknown 1754517887 0 0) 0x7efd240145b0 con 0x7efd3009f140 2024-09-16T06:44:17.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd1effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:17.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.521+0000 7efd1effd700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 >> v1:172.21.15.189:6812/3098810674 conn(0x7efd100362c0 legacy=0x7efd10038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 10 monclient: shutdown 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 >> v1:172.21.15.76:6790/0 conn(0x7efd3009f140 legacy=0x7efd300911b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 shutdown_connections 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 >> 172.21.15.189:0/1802295619 conn(0x7efd300a03e0 msgr2=0x7efd3006fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 shutdown_connections 2024-09-16T06:44:17.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.525+0000 7efd34b5b700 1 -- 172.21.15.189:0/1802295619 wait complete. 2024-09-16T06:44:17.530 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T06:44:17.539 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create cool flat 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 10 monclient: get_monmap_and_config 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 10 monclient: build_initial_monmap 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 10 monclient: monmap: 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.646 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:17.646911+0000 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:17.646911+0000 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 1 Processor -- start 2024-09-16T06:44:17.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 1 -- start start 2024-09-16T06:44:17.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.641+0000 7f50ff59e700 10 monclient: init 2024-09-16T06:44:17.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:17.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:17.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-b con 0x7f510009d830 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-c con 0x7f510009dc10 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5100137dd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510009a790 con 0x7f5100137dd0 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100071510 con 0x7f510009dc10 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100132040 con 0x7f510009d830 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient(hunting): authenticate will time out at 1210.160685s 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50f5d9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f510009dc10 legacy=0x7f51001366b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fe59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f510009d830 0x7f5100131930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:59992/0 (socket says 172.21.15.189:59992) 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fe59c700 1 -- 172.21.15.189:0/2691420827 learned_addr learned my addr 172.21.15.189:0/2691420827 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659411412 0 0) 0x7f5100132040 con 0x7f510009d830 2024-09-16T06:44:17.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): my global_id is 125345 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec0046e0 con 0x7f510009d830 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588491017 0 0) 0x7f510009a790 con 0x7f5100137dd0 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient(hunting): my global_id is 154512 2024-09-16T06:44:17.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100132040 con 0x7f5100137dd0 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852472008 0 0) 0x7f50ec0046e0 con 0x7f510009d830 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 >> v1:172.21.15.189:6789/0 conn(0x7f510009dc10 legacy=0x7f51001366b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 >> v1:172.21.15.76:6789/0 conn(0x7f5100137dd0 legacy=0x7f510013a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 monclient: found mon.noname-b 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510009e060 con 0x7f510009d830 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: _finish_auth 0 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50f0001610 con 0x7f510009d830 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 5 monclient: authenticate success, global_id 125345 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: dump: 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.658 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50f0001870 con 0x7f510009d830 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 1 -- 172.21.15.189:0/2691420827 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50f0001c40 con 0x7f510009d830 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:17.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 4 set_mon_vals no callback set 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50fdd9b700 10 monclient: dump: 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: shutdown 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- 172.21.15.189:0/2691420827 >> v1:172.21.15.76:6790/0 conn(0x7f510009d830 legacy=0x7f5100131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- 172.21.15.189:0/2691420827 shutdown_connections 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- 172.21.15.189:0/2691420827 >> 172.21.15.189:0/2691420827 conn(0x7f51000a4320 msgr2=0x7f51000a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- 172.21.15.189:0/2691420827 shutdown_connections 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 1 -- 172.21.15.189:0/2691420827 wait complete. 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: build_initial_monmap 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.645+0000 7f50ff59e700 10 monclient: monmap: 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:17.661 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 Processor -- start 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- start start 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: init 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-c con 0x7f510009d830 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-a con 0x7f510009dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5100137dd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510011cda0 con 0x7f510009dc10 2024-09-16T06:44:17.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510009f730 con 0x7f5100137dd0 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100072210 con 0x7f510009d830 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient(hunting): authenticate will time out at 1210.164723s 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50fed9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5100137dd0 legacy=0x7f51001d1fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50fe59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f510009d830 0x7f510011a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:59996/0 (socket says 172.21.15.189:59996) 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50fe59c700 1 -- 172.21.15.189:0/1318825232 learned_addr learned my addr 172.21.15.189:0/1318825232 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561122892 0 0) 0x7f5100072210 con 0x7f510009d830 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): my global_id is 125348 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e00035d0 con 0x7f510009d830 2024-09-16T06:44:17.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792864460 0 0) 0x7f510011cda0 con 0x7f510009dc10 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): my global_id is 154515 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100072210 con 0x7f510009dc10 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345341521 0 0) 0x7f50e00035d0 con 0x7f510009d830 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f510011cda0 con 0x7f510009d830 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50f0001b80 con 0x7f510009d830 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): dump: 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:17.664 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1040317165 0 0) 0x7f5100072210 con 0x7f510009dc10 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50e00035d0 con 0x7f510009dc10 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50e8004170 con 0x7f510009dc10 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:17.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient(hunting): dump: 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:17.666 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1280392218 0 0) 0x7f510011cda0 con 0x7f510009d830 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 >> v1:172.21.15.189:6789/0 conn(0x7f5100137dd0 legacy=0x7f51001d1fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 >> v1:172.21.15.76:6789/0 conn(0x7f510009dc10 legacy=0x7f510011a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 monclient: found mon.c 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510011afd0 con 0x7f510009d830 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:17.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 5 monclient: authenticate success, global_id 125348 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _renew_subs 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51000a20b0 con 0x7f510009d830 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _renew_subs 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f510011d640 con 0x7f510009d830 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50f0004ca0 con 0x7f510009d830 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50fcd99700 4 set_mon_vals no callback set 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50f0005030 con 0x7f510009d830 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:17.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.649+0000 7f50f77fe700 10 monclient: dump: 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:17.669 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50ff59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50ff59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50ff59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e4005100 con 0x7f510009d830 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f50f0005210 con 0x7f510009d830 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(140..140 src has 66..140) v4 ==== 7395+0+0 (unknown 4227307600 0 0) 0x7f50f0044380 con 0x7f510009d830 2024-09-16T06:44:17.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.653+0000 7f50f5d9b700 10 monclient: get_auth_request con 0x7f50e00360e0 auth_method 0 2024-09-16T06:44:17.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.661+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f50f0014af0 con 0x7f510009d830 2024-09-16T06:44:17.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.661+0000 7f50f77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:17.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.661+0000 7f50f77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:17.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.841+0000 7f50ff59e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:17.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.841+0000 7f50ff59e700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:17.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.841+0000 7f50ff59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:17.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:17.841+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f50e4005930 con 0x7f510009d830 2024-09-16T06:44:18.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.409+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v141) v1 ==== 107+0+0 (unknown 1480991178 0 0) 0x7f50f0012f60 con 0x7f510009d830 2024-09-16T06:44:18.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.409+0000 7f50f77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:18.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.409+0000 7f50f77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:18.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50ff59e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:18.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50ff59e700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:18.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50ff59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:18.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f50e40053b0 con 0x7f510009d830 2024-09-16T06:44:18.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50f77fe700 1 -- 172.21.15.189:0/1318825232 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v141) v1 ==== 149+0+0 (unknown 1978132493 0 0) 0x7f50f00146e0 con 0x7f510009d830 2024-09-16T06:44:18.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50f77fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T06:44:18.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.529+0000 7f50f77fe700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 >> v1:172.21.15.189:6812/3098810674 conn(0x7f50e00360e0 legacy=0x7f50e00385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 10 monclient: shutdown 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 >> v1:172.21.15.76:6790/0 conn(0x7f510009d830 legacy=0x7f510011a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 shutdown_connections 2024-09-16T06:44:18.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 >> 172.21.15.189:0/1318825232 conn(0x7f51000a4320 msgr2=0x7f51000a2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:18.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 shutdown_connections 2024-09-16T06:44:18.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.533+0000 7f50ff59e700 1 -- 172.21.15.189:0/1318825232 wait complete. 2024-09-16T06:44:18.539 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'cool' already created 2024-09-16T06:44:18.553 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create cold positional 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: get_monmap_and_config 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: build_initial_monmap 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: monmap: 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:18.661813+0000 2024-09-16T06:44:18.661 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:18.661813+0000 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:18.662 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 1 Processor -- start 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 1 -- start start 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: init 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-a con 0x7f59a009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-b con 0x7f59a009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-c con 0x7f59a0138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a009cca0 con 0x7f59a009bb00 2024-09-16T06:44:18.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a009f750 con 0x7f59a0138da0 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a0097620 con 0x7f59a009c5f0 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a6fc8700 10 monclient(hunting): authenticate will time out at 1211.174935s 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a5565700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f59a0138da0 legacy=0x7f59a013b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a4d64700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f59a009bb00 0x7f59a009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60440/0 (socket says 172.21.15.189:60440) 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f59a4d64700 1 -- 172.21.15.189:0/2721999556 learned_addr learned my addr 172.21.15.189:0/2721999556 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:18.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599ffff700 1 --1- 172.21.15.189:0/2721999556 >> v1:172.21.15.76:6790/0 conn(0x7f59a009c5f0 0x7f59a0137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52238/0 (socket says 172.21.15.189:52238) 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477042083 0 0) 0x7f59a009cca0 con 0x7f59a009bb00 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): my global_id is 154518 2024-09-16T06:44:18.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59880035d0 con 0x7f59a009bb00 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034384956 0 0) 0x7f59a0097620 con 0x7f59a009c5f0 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.657+0000 7f599f7fe700 10 monclient(hunting): my global_id is 125351 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a009cca0 con 0x7f59a009c5f0 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591053993 0 0) 0x7f59880035d0 con 0x7f59a009bb00 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 >> v1:172.21.15.76:6790/0 conn(0x7f59a009c5f0 legacy=0x7f59a0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 >> v1:172.21.15.189:6789/0 conn(0x7f59a0138da0 legacy=0x7f59a013b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:18.667 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a0097900 con 0x7f59a009bb00 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5994001610 con 0x7f59a009bb00 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: dump: 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.668 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 5 monclient: authenticate success, global_id 154518 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5994001830 con 0x7f59a009bb00 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 1 -- 172.21.15.189:0/2721999556 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5994001bc0 con 0x7f59a009bb00 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 4 set_mon_vals no callback set 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:18.669 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f599f7fe700 10 monclient: dump: 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 10 monclient: shutdown 2024-09-16T06:44:18.670 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:18.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2721999556 >> v1:172.21.15.76:6789/0 conn(0x7f59a009bb00 legacy=0x7f59a009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:18.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2721999556 shutdown_connections 2024-09-16T06:44:18.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2721999556 >> 172.21.15.189:0/2721999556 conn(0x7f59a00a03e0 msgr2=0x7f59a00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:18.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2721999556 shutdown_connections 2024-09-16T06:44:18.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2721999556 wait complete. 2024-09-16T06:44:18.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 10 monclient: build_initial_monmap 2024-09-16T06:44:18.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.661+0000 7f59a6fc8700 10 monclient: monmap: 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 Processor -- start 2024-09-16T06:44:18.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- start start 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: init 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-c con 0x7f59a009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-b con 0x7f59a009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient(hunting): picked mon.noname-a con 0x7f59a0138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a0090450 con 0x7f59a0138da0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a0091890 con 0x7f59a009c5f0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f59a009c5f0 legacy=0x7f59a01cc160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a0091b70 con 0x7f59a009bb00 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient(hunting): authenticate will time out at 1211.181975s 2024-09-16T06:44:18.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a4d64700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f59a009bb00 0x7f59a01c7190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52240/0 (socket says 172.21.15.189:52240) 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a4d64700 1 -- 172.21.15.189:0/2083196943 learned_addr learned my addr 172.21.15.189:0/2083196943 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687074376 0 0) 0x7f59a0090450 con 0x7f59a0138da0 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): my global_id is 154521 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f598c0035d0 con 0x7f59a0138da0 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203924678 0 0) 0x7f59a0091b70 con 0x7f59a009bb00 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:18.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): my global_id is 125354 2024-09-16T06:44:18.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a0090450 con 0x7f59a009bb00 2024-09-16T06:44:18.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849074559 0 0) 0x7f598c0035d0 con 0x7f59a0138da0 2024-09-16T06:44:18.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59a0091b70 con 0x7f59a0138da0 2024-09-16T06:44:18.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5998004390 con 0x7f59a0138da0 2024-09-16T06:44:18.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:18.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2703692331 0 0) 0x7f59a0090450 con 0x7f59a009bb00 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f598c0035d0 con 0x7f59a009bb00 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5994001610 con 0x7f59a009bb00 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:18.680 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1940023439 0 0) 0x7f59a0091b70 con 0x7f59a0138da0 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 >> v1:172.21.15.76:6790/0 conn(0x7f59a009bb00 legacy=0x7f59a01c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 >> v1:172.21.15.189:6789/0 conn(0x7f59a009c5f0 legacy=0x7f59a01cc160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 monclient: found mon.a 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00a2420 con 0x7f59a0138da0 2024-09-16T06:44:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 5 monclient: authenticate success, global_id 154521 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _renew_subs 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a008f810 con 0x7f59a0138da0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _renew_subs 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a0090160 con 0x7f59a0138da0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59980049f0 con 0x7f59a0138da0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5998004d00 con 0x7f59a0138da0 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599d7fa700 10 monclient: dump: 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:18.683 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.665+0000 7f599effd700 4 set_mon_vals no callback set 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f59a6fc8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f59a6fc8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f59a6fc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a0068180 con 0x7f59a0138da0 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f599800f770 con 0x7f59a0138da0 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(141..141 src has 66..141) v4 ==== 7727+0+0 (unknown 271057129 0 0) 0x7f5998044290 con 0x7f59a0138da0 2024-09-16T06:44:18.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.669+0000 7f59a4d64700 10 monclient: get_auth_request con 0x7f598c0384b0 auth_method 0 2024-09-16T06:44:18.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.681+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f59980146d0 con 0x7f59a0138da0 2024-09-16T06:44:18.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.681+0000 7f599d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:18.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.681+0000 7f599d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:18.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.857+0000 7f59a6fc8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:18.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.857+0000 7f59a6fc8700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:18.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.857+0000 7f59a6fc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:18.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:18.857+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f59a0066480 con 0x7f59a0138da0 2024-09-16T06:44:19.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.413+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v142) v1 ==== 113+0+0 (unknown 2952636617 0 0) 0x7f5998012d70 con 0x7f59a0138da0 2024-09-16T06:44:19.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.413+0000 7f599d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:19.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.413+0000 7f599d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:19.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.533+0000 7f59a6fc8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:19.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.533+0000 7f59a6fc8700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:19.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.533+0000 7f59a6fc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:19.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.533+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7f59a008ebd0 con 0x7f59a0138da0 2024-09-16T06:44:19.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f599d7fa700 1 -- 172.21.15.189:0/2083196943 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v142) v1 ==== 155+0+0 (unknown 174935603 0 0) 0x7f59980144f0 con 0x7f59a0138da0 2024-09-16T06:44:19.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f599d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T06:44:19.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f599d7fa700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-16T06:44:19.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 >> v1:172.21.15.189:6812/3098810674 conn(0x7f598c0384b0 legacy=0x7f598c03a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:19.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 10 monclient: shutdown 2024-09-16T06:44:19.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:19.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 >> v1:172.21.15.76:6789/0 conn(0x7f59a0138da0 legacy=0x7f59a01cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:19.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 shutdown_connections 2024-09-16T06:44:19.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 >> 172.21.15.189:0/2083196943 conn(0x7f59a00a03e0 msgr2=0x7f59a00a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:19.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 shutdown_connections 2024-09-16T06:44:19.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.537+0000 7f59a6fc8700 1 -- 172.21.15.189:0/2083196943 wait complete. 2024-09-16T06:44:19.544 INFO:tasks.workunit.client.0.smithi189.stderr:weight-set for pool 'cold' already created 2024-09-16T06:44:19.553 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T06:44:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.649+0000 7f81e3a67700 10 monclient: get_monmap_and_config 2024-09-16T06:44:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.649+0000 7f81e3a67700 10 monclient: build_initial_monmap 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.649+0000 7f81e3a67700 10 monclient: monmap: 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:19.656232+0000 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:19.656232+0000 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:19.656 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 Processor -- start 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 -- start start 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient: init 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-c con 0x7f81dc09f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-b con 0x7f81dc099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-a con 0x7f81dc1396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc09f520 con 0x7f81dc1396a0 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc09bc10 con 0x7f81dc09f140 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc09b810 con 0x7f81dc099700 2024-09-16T06:44:19.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient(hunting): authenticate will time out at 1212.169157s 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e1803700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f81dc09f140 legacy=0x7f81dc09a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e1002700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f81dc099700 0x7f81dc09b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52242/0 (socket says 172.21.15.189:52242) 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e2004700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f81dc1396a0 0x7f81dc13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60466/0 (socket says 172.21.15.189:60466) 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e1002700 1 -- 172.21.15.189:0/1658608815 learned_addr learned my addr 172.21.15.189:0/1658608815 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777518754 0 0) 0x7f81dc09f520 con 0x7f81dc1396a0 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _negotiate 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): my global_id is 154524 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c80035d0 con 0x7f81dc1396a0 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566734216 0 0) 0x7f81dc09b810 con 0x7f81dc099700 2024-09-16T06:44:19.661 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _negotiate 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient(hunting): my global_id is 125357 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc09f520 con 0x7f81dc099700 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098782724 0 0) 0x7f81dc09f520 con 0x7f81dc099700 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 >> v1:172.21.15.189:6789/0 conn(0x7f81dc09f140 legacy=0x7f81dc09a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 >> v1:172.21.15.76:6789/0 conn(0x7f81dc1396a0 legacy=0x7f81dc13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: _finish_hunting 0 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 monclient: found mon.noname-b 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc09ceb0 con 0x7f81dc099700 2024-09-16T06:44:19.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: _finish_auth 0 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: _check_auth_tickets 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81d80014a0 con 0x7f81dc099700 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 5 monclient: authenticate success, global_id 125357 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: dump: 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.663 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d80016c0 con 0x7f81dc099700 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:19.664 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 1 -- 172.21.15.189:0/1658608815 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81d8001a50 con 0x7f81dc099700 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 4 set_mon_vals no callback set 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e0801700 10 monclient: dump: 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:19.665 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 10 monclient: shutdown 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.653+0000 7f81e3a67700 1 -- 172.21.15.189:0/1658608815 >> v1:172.21.15.76:6790/0 conn(0x7f81dc099700 legacy=0x7f81dc09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- 172.21.15.189:0/1658608815 shutdown_connections 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- 172.21.15.189:0/1658608815 >> 172.21.15.189:0/1658608815 conn(0x7f81dc0a03e0 msgr2=0x7f81dc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- 172.21.15.189:0/1658608815 shutdown_connections 2024-09-16T06:44:19.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- 172.21.15.189:0/1658608815 wait complete. 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient: build_initial_monmap 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient: monmap: 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:19.671 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 Processor -- start 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- start start 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient: init 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-b con 0x7f81dc099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-a con 0x7f81dc09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient(hunting): picked mon.noname-c con 0x7f81dc1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc070dc0 con 0x7f81dc09f140 2024-09-16T06:44:19.672 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e1803700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f81dc099700 legacy=0x7f81dc0706b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc13c340 con 0x7f81dc099700 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc0a2e40 con 0x7f81dc1396a0 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e3a67700 10 monclient(hunting): authenticate will time out at 1212.174306s 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e2004700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f81dc1396a0 0x7f81dc1cf700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52254/0 (socket says 172.21.15.189:52254) 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81e2004700 1 -- 172.21.15.189:0/3312250040 learned_addr learned my addr 172.21.15.189:0/3312250040 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734849460 0 0) 0x7f81dc0a2e40 con 0x7f81dc1396a0 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): my global_id is 125360 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c00035d0 con 0x7f81dc1396a0 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615380409 0 0) 0x7f81dc070dc0 con 0x7f81dc09f140 2024-09-16T06:44:19.673 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): my global_id is 154527 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc0a2e40 con 0x7f81dc09f140 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1856664199 0 0) 0x7f81c00035d0 con 0x7f81dc1396a0 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81dc070dc0 con 0x7f81dc1396a0 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81d4001db0 con 0x7f81dc1396a0 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.657+0000 7f81d27fc700 10 monclient(hunting): dump: 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.674 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 667485014 0 0) 0x7f81dc0a2e40 con 0x7f81dc09f140 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c00035d0 con 0x7f81dc09f140 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81d8001830 con 0x7f81dc09f140 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient(hunting): dump: 2024-09-16T06:44:19.675 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 99729738 0 0) 0x7f81dc070dc0 con 0x7f81dc1396a0 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 >> v1:172.21.15.189:6789/0 conn(0x7f81dc099700 legacy=0x7f81dc0706b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:19.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 >> v1:172.21.15.76:6789/0 conn(0x7f81dc09f140 legacy=0x7f81dc1cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 monclient: found mon.c 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc09a6e0 con 0x7f81dc1396a0 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:19.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 5 monclient: authenticate success, global_id 125360 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _renew_subs 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc06ea90 con 0x7f81dc1396a0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _renew_subs 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc06ecd0 con 0x7f81dc1396a0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d4004ec0 con 0x7f81dc1396a0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81d40051d0 con 0x7f81dc1396a0 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d37fe700 4 set_mon_vals no callback set 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:19.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 10 monclient: dump: 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:19.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c4005100 con 0x7f81dc1396a0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f81d400fc40 con 0x7f81dc1396a0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(142..142 src has 66..142) v4 ==== 8199+0+0 (unknown 2040261617 0 0) 0x7f81d40449d0 con 0x7f81dc1396a0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.661+0000 7f81e1803700 10 monclient: get_auth_request con 0x7f81c00362a0 auth_method 0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.673+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f81d4014b50 con 0x7f81dc1396a0 2024-09-16T06:44:19.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.673+0000 7f81d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:19.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.673+0000 7f81d27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.849+0000 7f81e3a67700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.849+0000 7f81e3a67700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.849+0000 7f81e3a67700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:19.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:19.849+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f81c4005bc0 con 0x7f81dc1396a0 2024-09-16T06:44:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.428+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v143) v1 ==== 123+0+0 (unknown 980694632 0 0) 0x7f81d40131f0 con 0x7f81dc1396a0 2024-09-16T06:44:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.428+0000 7f81d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:20.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.428+0000 7f81d27fc700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T06:44:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.544+0000 7f81e3a67700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.544+0000 7f81e3a67700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.544+0000 7f81e3a67700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:20.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.544+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f81c40053b0 con 0x7f81dc1396a0 2024-09-16T06:44:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.768+0000 7f81d27fc700 1 -- 172.21.15.189:0/3312250040 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v144) v1 ==== 149+0+0 (unknown 1730350939 0 0) 0x7f81d4014970 con 0x7f81dc1396a0 2024-09-16T06:44:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.768+0000 7f81d27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:20.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.768+0000 7f81d27fc700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:20.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 >> v1:172.21.15.189:6812/3098810674 conn(0x7f81c00362a0 legacy=0x7f81c0038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:20.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 10 monclient: shutdown 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 >> v1:172.21.15.76:6790/0 conn(0x7f81dc1396a0 legacy=0x7f81dc1cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 shutdown_connections 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 >> 172.21.15.189:0/3312250040 conn(0x7f81dc0a03e0 msgr2=0x7f81dc0a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 shutdown_connections 2024-09-16T06:44:20.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.772+0000 7f81e3a67700 1 -- 172.21.15.189:0/3312250040 wait complete. 2024-09-16T06:44:20.778 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:20.794 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-16T06:44:20.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 10 monclient: get_monmap_and_config 2024-09-16T06:44:20.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 10 monclient: build_initial_monmap 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 10 monclient: monmap: 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:20.911259+0000 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:20.911259+0000 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:20.911 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 1 Processor -- start 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 1 -- start start 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.904+0000 7f2795f2b700 10 monclient: init 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-a con 0x7f279009a9f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-c con 0x7f279009a230 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-b con 0x7f279009b4e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279009bca0 con 0x7f279009a9f0 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279009bef0 con 0x7f279009a230 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279009f770 con 0x7f279009b4e0 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:20.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient(hunting): authenticate will time out at 1213.424582s 2024-09-16T06:44:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f279009a230 legacy=0x7f27901388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f279009b4e0 0x7f279013bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52258/0 (socket says 172.21.15.189:52258) 2024-09-16T06:44:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278ffff700 1 -- 172.21.15.189:0/2047428709 learned_addr learned my addr 172.21.15.189:0/2047428709 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:20.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278f7fe700 1 --1- 172.21.15.189:0/2047428709 >> v1:172.21.15.76:6789/0 conn(0x7f279009a9f0 0x7f279009add0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60490/0 (socket says 172.21.15.189:60490) 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803268494 0 0) 0x7f279009f770 con 0x7f279009b4e0 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): my global_id is 125363 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27780035d0 con 0x7f279009b4e0 2024-09-16T06:44:20.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481255647 0 0) 0x7f279009bca0 con 0x7f279009a9f0 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient(hunting): my global_id is 154530 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279009f770 con 0x7f279009a9f0 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435900822 0 0) 0x7f27780035d0 con 0x7f279009b4e0 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 >> v1:172.21.15.189:6789/0 conn(0x7f279009a230 legacy=0x7f27901388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 >> v1:172.21.15.76:6789/0 conn(0x7f279009a9f0 legacy=0x7f279009add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:20.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279009c1d0 con 0x7f279009b4e0 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2788001630 con 0x7f279009b4e0 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 5 monclient: authenticate success, global_id 125363 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: dump: 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2788001850 con 0x7f279009b4e0 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:20.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 1 -- 172.21.15.189:0/2047428709 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2788001be0 con 0x7f279009b4e0 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 4 set_mon_vals no callback set 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f278e7fc700 10 monclient: dump: 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:20.920 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 10 monclient: shutdown 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/2047428709 >> v1:172.21.15.76:6790/0 conn(0x7f279009b4e0 legacy=0x7f279013bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/2047428709 shutdown_connections 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/2047428709 >> 172.21.15.189:0/2047428709 conn(0x7f27900a03e0 msgr2=0x7f27900a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/2047428709 shutdown_connections 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/2047428709 wait complete. 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: build_initial_monmap 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: monmap: 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:20.921 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 Processor -- start 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- start start 2024-09-16T06:44:20.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: init 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-a con 0x7f279009a230 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-c con 0x7f279009a9f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient(hunting): picked mon.noname-b con 0x7f279009b4e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:20.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2790130050 con 0x7f279009a230 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2790131b80 con 0x7f279009b4e0 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279006bba0 con 0x7f279009a9f0 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient(hunting): authenticate will time out at 1213.429549s 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f278ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f279009b4e0 legacy=0x7f27901d1bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f278f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f279009a230 0x7f27901c9870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60492/0 (socket says 172.21.15.189:60492) 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f278f7fe700 1 -- 172.21.15.189:0/3789496367 learned_addr learned my addr 172.21.15.189:0/3789496367 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393840697 0 0) 0x7f2790130050 con 0x7f279009a230 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): my global_id is 154533 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f277c0035d0 con 0x7f279009a230 2024-09-16T06:44:20.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156494695 0 0) 0x7f279006bba0 con 0x7f279009a9f0 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): my global_id is 125366 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2790130050 con 0x7f279009a9f0 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126830931 0 0) 0x7f277c0035d0 con 0x7f279009a230 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f279006bba0 con 0x7f279009a230 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2784001940 con 0x7f279009a230 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): dump: 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:20.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2813245767 0 0) 0x7f2790130050 con 0x7f279009a9f0 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f277c0035d0 con 0x7f279009a9f0 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f27800044f0 con 0x7f279009a9f0 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:20.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient(hunting): dump: 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 324202376 0 0) 0x7f279006bba0 con 0x7f279009a230 2024-09-16T06:44:20.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 >> v1:172.21.15.76:6790/0 conn(0x7f279009a9f0 legacy=0x7f2790133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 >> v1:172.21.15.189:6789/0 conn(0x7f279009b4e0 legacy=0x7f27901d1bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 monclient: found mon.a 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279009a6e0 con 0x7f279009a230 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 5 monclient: authenticate success, global_id 154533 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _renew_subs 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790132140 con 0x7f279009a230 2024-09-16T06:44:20.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _renew_subs 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2790132420 con 0x7f279009a230 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2784001c90 con 0x7f279009a230 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2784004ce0 con 0x7f279009a230 2024-09-16T06:44:20.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f2777fff700 10 monclient: dump: 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:20.932 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.912+0000 7f278dffb700 4 set_mon_vals no callback set 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f278400f750 con 0x7f279009a230 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2795f2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2795f2b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2795f2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770005100 con 0x7f279009a230 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(144..144 src has 66..144) v4 ==== 8167+0+0 (unknown 3830946197 0 0) 0x7f2784044400 con 0x7f279009a230 2024-09-16T06:44:20.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.916+0000 7f278effd700 10 monclient: get_auth_request con 0x7f277c0362c0 auth_method 0 2024-09-16T06:44:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.924+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2784010d40 con 0x7f279009a230 2024-09-16T06:44:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.924+0000 7f2777fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:20.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:20.924+0000 7f2777fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:21.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.104+0000 7f2795f2b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:21.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.104+0000 7f2795f2b700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:21.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.104+0000 7f2795f2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:21.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.104+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f27700053b0 con 0x7f279009a230 2024-09-16T06:44:21.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.784+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v145) v1 ==== 119+0+0 (unknown 4140392404 0 0) 0x7f2784012d60 con 0x7f279009a230 2024-09-16T06:44:21.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.784+0000 7f2777fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:21.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.784+0000 7f2777fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:21.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.908+0000 7f2795f2b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:21.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.908+0000 7f2795f2b700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:21.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.908+0000 7f2795f2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:21.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:21.908+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f2770005650 con 0x7f279009a230 2024-09-16T06:44:22.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.788+0000 7f2777fff700 1 -- 172.21.15.189:0/3789496367 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v146) v1 ==== 119+0+0 (unknown 4230661651 0 0) 0x7f27840144e0 con 0x7f279009a230 2024-09-16T06:44:22.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.788+0000 7f2777fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T06:44:22.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.788+0000 7f2777fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 >> v1:172.21.15.189:6812/3098810674 conn(0x7f277c0362c0 legacy=0x7f277c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 10 monclient: shutdown 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 >> v1:172.21.15.76:6789/0 conn(0x7f279009a230 legacy=0x7f27901c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 shutdown_connections 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 >> 172.21.15.189:0/3789496367 conn(0x7f27900a03e0 msgr2=0x7f279006d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 shutdown_connections 2024-09-16T06:44:22.802 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.796+0000 7f2795f2b700 1 -- 172.21.15.189:0/3789496367 wait complete. 2024-09-16T06:44:22.823 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: get_monmap_and_config 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: build_initial_monmap 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: monmap: 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:22.926619+0000 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:22.926619+0000 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.926 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:22.927 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:22.927 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:22.927 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 1 Processor -- start 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 1 -- start start 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: init 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-c con 0x7f9d98098070 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-b con 0x7f9d98099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-a con 0x7f9d98138a80 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9809c460 con 0x7f9d98138a80 2024-09-16T06:44:22.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9809c1e0 con 0x7f9d98098070 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9809bd60 con 0x7f9d98099a60 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9da0621700 10 monclient(hunting): authenticate will time out at 1215.439609s 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.920+0000 7f9d9e3bd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9d98098070 legacy=0x7f9d9809f140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9ebbe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9d98138a80 0x7f9d9813ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60494/0 (socket says 172.21.15.189:60494) 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9dbbc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9d98099a60 0x7f9d98099e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52268/0 (socket says 172.21.15.189:52268) 2024-09-16T06:44:22.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9ebbe700 1 -- 172.21.15.189:0/2702905553 learned_addr learned my addr 172.21.15.189:0/2702905553 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:22.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946942824 0 0) 0x7f9d9809c460 con 0x7f9d98138a80 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): my global_id is 154536 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d840035d0 con 0x7f9d98138a80 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384485173 0 0) 0x7f9d9809bd60 con 0x7f9d98099a60 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient(hunting): my global_id is 125369 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d9809c460 con 0x7f9d98099a60 2024-09-16T06:44:22.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336968206 0 0) 0x7f9d840035d0 con 0x7f9d98138a80 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 >> v1:172.21.15.76:6790/0 conn(0x7f9d98099a60 legacy=0x7f9d98099e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 >> v1:172.21.15.189:6789/0 conn(0x7f9d98098070 legacy=0x7f9d9809f140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 monclient: found mon.noname-a 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9809d490 con 0x7f9d98138a80 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: _finish_auth 0 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d90001990 con 0x7f9d98138a80 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 5 monclient: authenticate success, global_id 154536 2024-09-16T06:44:22.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: dump: 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d90001c00 con 0x7f9d98138a80 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 1 -- 172.21.15.189:0/2702905553 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d90001fd0 con 0x7f9d98138a80 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 4 set_mon_vals no callback set 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9d9d3bb700 10 monclient: dump: 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.935 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 10 monclient: shutdown 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 1 -- 172.21.15.189:0/2702905553 >> v1:172.21.15.76:6789/0 conn(0x7f9d98138a80 legacy=0x7f9d9813ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 1 -- 172.21.15.189:0/2702905553 shutdown_connections 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 1 -- 172.21.15.189:0/2702905553 >> 172.21.15.189:0/2702905553 conn(0x7f9d980a03e0 msgr2=0x7f9d980a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 1 -- 172.21.15.189:0/2702905553 shutdown_connections 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 1 -- 172.21.15.189:0/2702905553 wait complete. 2024-09-16T06:44:22.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 10 monclient: build_initial_monmap 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.924+0000 7f9da0621700 10 monclient: monmap: 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:22.937 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 1 Processor -- start 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 1 -- start start 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient: init 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-a con 0x7f9d98098070 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-c con 0x7f9d98099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient(hunting): picked mon.noname-b con 0x7f9d98138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d98133680 con 0x7f9d98098070 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d9ebbe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9d98138a80 legacy=0x7f9d981cfc60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:22.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d98129860 con 0x7f9d98138a80 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d98071180 con 0x7f9d98099a60 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9da0621700 10 monclient(hunting): authenticate will time out at 1215.446204s 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d9e3bd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9d98098070 0x7f9d98070360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60498/0 (socket says 172.21.15.189:60498) 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d9e3bd700 1 -- 172.21.15.189:0/2560803129 learned_addr learned my addr 172.21.15.189:0/2560803129 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24972893 0 0) 0x7f9d98133680 con 0x7f9d98098070 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): my global_id is 154539 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d7c0035d0 con 0x7f9d98098070 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821507625 0 0) 0x7f9d98071180 con 0x7f9d98099a60 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:22.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): my global_id is 125372 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98133680 con 0x7f9d98099a60 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792203579 0 0) 0x7f9d7c0035d0 con 0x7f9d98098070 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d98071180 con 0x7f9d98098070 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d88004200 con 0x7f9d98098070 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): dump: 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2446804072 0 0) 0x7f9d98133680 con 0x7f9d98099a60 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d7c0035d0 con 0x7f9d98099a60 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d940019a0 con 0x7f9d98099a60 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient(hunting): dump: 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:22.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4081243540 0 0) 0x7f9d98071180 con 0x7f9d98098070 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 >> v1:172.21.15.76:6790/0 conn(0x7f9d98099a60 legacy=0x7f9d98070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 >> v1:172.21.15.189:6789/0 conn(0x7f9d98138a80 legacy=0x7f9d981cfc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:22.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 1 monclient: found mon.a 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.928+0000 7f9d8effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980993d0 con 0x7f9d98098070 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: _finish_auth 0 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 5 monclient: authenticate success, global_id 154539 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _renew_subs 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d98098ea0 con 0x7f9d98098070 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _renew_subs 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d98129580 con 0x7f9d98098070 2024-09-16T06:44:22.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d880044e0 con 0x7f9d98098070 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d88004870 con 0x7f9d98098070 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d9cbba700 4 set_mon_vals no callback set 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 10 monclient: dump: 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:22.948 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d80005100 con 0x7f9d98098070 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d88004a50 con 0x7f9d98098070 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(146..146 src has 66..146) v4 ==== 8378+0+0 (unknown 2553207946 0 0) 0x7f9d880445b0 con 0x7f9d98098070 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.932+0000 7f9d9dbbc700 10 monclient: get_auth_request con 0x7f9d7c0362a0 auth_method 0 2024-09-16T06:44:22.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.944+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d880148a0 con 0x7f9d98098070 2024-09-16T06:44:22.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.944+0000 7f9d8effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:22.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:22.944+0000 7f9d8effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.124+0000 7f9da0621700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.124+0000 7f9da0621700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.124+0000 7f9da0621700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:23.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.124+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f9d800053b0 con 0x7f9d98098070 2024-09-16T06:44:23.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.800+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v147) v1 ==== 128+0+0 (unknown 3531060156 0 0) 0x7f9d8801ce90 con 0x7f9d98098070 2024-09-16T06:44:23.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.800+0000 7f9d8effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:23.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.800+0000 7f9d8effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:23.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.920+0000 7f9da0621700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:23.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.920+0000 7f9da0621700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:23.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.920+0000 7f9da0621700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:23.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:23.920+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f9d80005af0 con 0x7f9d98098070 2024-09-16T06:44:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.812+0000 7f9d8effd700 1 -- 172.21.15.189:0/2560803129 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v148) v1 ==== 128+0+0 (unknown 3412569123 0 0) 0x7f9d88012d10 con 0x7f9d98098070 2024-09-16T06:44:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.812+0000 7f9d8effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T06:44:24.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.812+0000 7f9d8effd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9d7c0362a0 legacy=0x7f9d7c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 10 monclient: shutdown 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 >> v1:172.21.15.76:6789/0 conn(0x7f9d98098070 legacy=0x7f9d98070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 shutdown_connections 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 >> 172.21.15.189:0/2560803129 conn(0x7f9d980a03e0 msgr2=0x7f9d98137fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 shutdown_connections 2024-09-16T06:44:24.826 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.820+0000 7f9da0621700 1 -- 172.21.15.189:0/2560803129 wait complete. 2024-09-16T06:44:24.842 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: get_monmap_and_config 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: build_initial_monmap 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: monmap: 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:24.960615+0000 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:24.960615+0000 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.961 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 1 Processor -- start 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 1 -- start start 2024-09-16T06:44:24.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: init 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-c con 0x7f967406a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-a con 0x7f967406b7a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-b con 0x7f967406ae30 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:24.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409d6b0 con 0x7f967406b7a0 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967406c0a0 con 0x7f967406a340 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967406c310 con 0x7f967406ae30 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient(hunting): authenticate will time out at 1217.473672s 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9673fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f967406a340 legacy=0x7f967406a720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f96737fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f967406b7a0 0x7f967413aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60514/0 (socket says 172.21.15.189:60514) 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f96737fe700 1 -- 172.21.15.189:0/2331557288 learned_addr learned my addr 172.21.15.189:0/2331557288 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906600474 0 0) 0x7f967409d6b0 con 0x7f967406b7a0 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): my global_id is 154542 2024-09-16T06:44:24.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96600035d0 con 0x7f967406b7a0 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095460222 0 0) 0x7f967406c310 con 0x7f967406ae30 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient(hunting): my global_id is 125375 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967409d6b0 con 0x7f967406ae30 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145480930 0 0) 0x7f96600035d0 con 0x7f967406b7a0 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 >> v1:172.21.15.76:6790/0 conn(0x7f967406ae30 legacy=0x7f967413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 >> v1:172.21.15.189:6789/0 conn(0x7f967406a340 legacy=0x7f967406a720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 monclient: found mon.noname-a 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:24.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967409c9d0 con 0x7f967406b7a0 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 5 monclient: authenticate success, global_id 154542 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f965c001610 con 0x7f967406b7a0 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: dump: 2024-09-16T06:44:24.973 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:24.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f965c0018f0 con 0x7f967406b7a0 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f967a51c700 4 set_mon_vals no callback set 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 1 -- 172.21.15.189:0/2331557288 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f965c001c80 con 0x7f967406b7a0 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.956+0000 7f9672ffd700 10 monclient: dump: 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.975 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: shutdown 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- 172.21.15.189:0/2331557288 >> v1:172.21.15.76:6789/0 conn(0x7f967406b7a0 legacy=0x7f967413aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- 172.21.15.189:0/2331557288 shutdown_connections 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- 172.21.15.189:0/2331557288 >> 172.21.15.189:0/2331557288 conn(0x7f96740a2560 msgr2=0x7f967409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- 172.21.15.189:0/2331557288 shutdown_connections 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- 172.21.15.189:0/2331557288 wait complete. 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: build_initial_monmap 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: monmap: 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:24.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 Processor -- start 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- start start 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: init 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:24.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-c con 0x7f967406a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-a con 0x7f967406ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient(hunting): picked mon.noname-b con 0x7f967406b7a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409eab0 con 0x7f967406ae30 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9674094140 con 0x7f967406b7a0 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967409c620 con 0x7f967406a340 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f967a51c700 10 monclient(hunting): authenticate will time out at 1217.478700s 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9678ab9700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f967406b7a0 legacy=0x7f96741d2590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f96737fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f967406ae30 0x7f96741cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60522/0 (socket says 172.21.15.189:60522) 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f96737fe700 1 -- 172.21.15.189:0/3229298581 learned_addr learned my addr 172.21.15.189:0/3229298581 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630099399 0 0) 0x7f967409eab0 con 0x7f967406ae30 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:24.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): my global_id is 154545 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96540035d0 con 0x7f967406ae30 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531848688 0 0) 0x7f967409c620 con 0x7f967406a340 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 10 monclient(hunting): my global_id is 125378 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.960+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967409eab0 con 0x7f967406a340 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722097306 0 0) 0x7f96540035d0 con 0x7f967406ae30 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f967409c620 con 0x7f967406ae30 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f965c0015d0 con 0x7f967406ae30 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:24.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:24.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524205881 0 0) 0x7f967409eab0 con 0x7f967406a340 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96540035d0 con 0x7f967406a340 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f96640044f0 con 0x7f967406a340 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:24.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1845895749 0 0) 0x7f967409c620 con 0x7f967406ae30 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 >> v1:172.21.15.76:6790/0 conn(0x7f967406a340 legacy=0x7f9674097900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 >> v1:172.21.15.189:6789/0 conn(0x7f967406b7a0 legacy=0x7f96741d2590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 monclient: found mon.a 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:24.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674093070 con 0x7f967406ae30 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: _finish_auth 0 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 5 monclient: authenticate success, global_id 154545 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _renew_subs 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674093b10 con 0x7f967406ae30 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _renew_subs 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96740981f0 con 0x7f967406ae30 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f965c004ca0 con 0x7f967406ae30 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:24.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f965c005030 con 0x7f967406ae30 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f96727fc700 4 set_mon_vals no callback set 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 10 monclient: dump: 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:24.984 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f965c005210 con 0x7f967406ae30 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9658005100 con 0x7f967406ae30 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(148..148 src has 66..148) v4 ==== 8353+0+0 (unknown 2038009919 0 0) 0x7f965c0447e0 con 0x7f967406ae30 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.964+0000 7f9673fff700 10 monclient: get_auth_request con 0x7f96540362f0 auth_method 0 2024-09-16T06:44:24.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.972+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f965c014bf0 con 0x7f967406ae30 2024-09-16T06:44:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.972+0000 7f9670ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:24.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:24.972+0000 7f9670ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.156+0000 7f967a51c700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.156+0000 7f967a51c700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.156+0000 7f967a51c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.156+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f96580053b0 con 0x7f967406ae30 2024-09-16T06:44:25.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.836+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v149) v1 ==== 134+0+0 (unknown 962590776 0 0) 0x7f965c013060 con 0x7f967406ae30 2024-09-16T06:44:25.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.840+0000 7f9670ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:25.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.840+0000 7f9670ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:25.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.960+0000 7f967a51c700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:25.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.960+0000 7f967a51c700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:25.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.960+0000 7f967a51c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:25.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:25.960+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f9658005af0 con 0x7f967406ae30 2024-09-16T06:44:26.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.848+0000 7f9670ff9700 1 -- 172.21.15.189:0/3229298581 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v150) v1 ==== 134+0+0 (unknown 796944355 0 0) 0x7f965c0147e0 con 0x7f967406ae30 2024-09-16T06:44:26.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.848+0000 7f9670ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T06:44:26.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.848+0000 7f9670ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:26.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 >> v1:172.21.15.189:6812/3098810674 conn(0x7f96540362f0 legacy=0x7f96540387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:26.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 10 monclient: shutdown 2024-09-16T06:44:26.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:26.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 >> v1:172.21.15.76:6789/0 conn(0x7f967406ae30 legacy=0x7f96741cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:26.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 shutdown_connections 2024-09-16T06:44:26.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 >> 172.21.15.189:0/3229298581 conn(0x7f96740a2560 msgr2=0x7f967409b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:26.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 shutdown_connections 2024-09-16T06:44:26.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.852+0000 7f967a51c700 1 -- 172.21.15.189:0/3229298581 wait complete. 2024-09-16T06:44:26.872 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-16T06:44:26.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 10 monclient: get_monmap_and_config 2024-09-16T06:44:26.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 10 monclient: build_initial_monmap 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 10 monclient: monmap: 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:26.990553+0000 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:26.990553+0000 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:26.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:26.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 1 Processor -- start 2024-09-16T06:44:26.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 1 -- start start 2024-09-16T06:44:26.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.984+0000 7fc65b288700 10 monclient: init 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-a con 0x7fc6540a5c70 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-c con 0x7fc6540a7100 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-b con 0x7fc65413bab0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540a5a90 con 0x7fc6540a5c70 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540a6a90 con 0x7fc6540a7100 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc654099e40 con 0x7fc65413bab0 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient(hunting): authenticate will time out at 1219.504231s 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc658823700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc6540a7100 legacy=0x7fc65413a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc659825700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc65413bab0 0x7fc65413dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52308/0 (socket says 172.21.15.189:52308) 2024-09-16T06:44:26.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc659024700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc6540a5c70 0x7fc6540a6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60524/0 (socket says 172.21.15.189:60524) 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc659825700 1 -- 172.21.15.189:0/3986872781 learned_addr learned my addr 172.21.15.189:0/3986872781 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605655812 0 0) 0x7fc654099e40 con 0x7fc65413bab0 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): my global_id is 125381 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6400035d0 con 0x7fc65413bab0 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764360327 0 0) 0x7fc6540a5a90 con 0x7fc6540a5c70 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:26.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient(hunting): my global_id is 154548 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654099e40 con 0x7fc6540a5c70 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377658986 0 0) 0x7fc6400035d0 con 0x7fc65413bab0 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 >> v1:172.21.15.189:6789/0 conn(0x7fc6540a7100 legacy=0x7fc65413a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 >> v1:172.21.15.76:6789/0 conn(0x7fc6540a5c70 legacy=0x7fc6540a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 monclient: found mon.noname-b 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65409a120 con 0x7fc65413bab0 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:26.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 5 monclient: authenticate success, global_id 125381 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc64c001990 con 0x7fc65413bab0 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: dump: 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:27.000 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc64c001bb0 con 0x7fc65413bab0 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 1 -- 172.21.15.189:0/3986872781 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc64c001f40 con 0x7fc65413bab0 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 4 set_mon_vals no callback set 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc64bfff700 10 monclient: dump: 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:27.001 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 10 monclient: shutdown 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- 172.21.15.189:0/3986872781 >> v1:172.21.15.76:6790/0 conn(0x7fc65413bab0 legacy=0x7fc65413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- 172.21.15.189:0/3986872781 shutdown_connections 2024-09-16T06:44:27.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- 172.21.15.189:0/3986872781 >> 172.21.15.189:0/3986872781 conn(0x7fc6540a79e0 msgr2=0x7fc6540a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- 172.21.15.189:0/3986872781 shutdown_connections 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.988+0000 7fc65b288700 1 -- 172.21.15.189:0/3986872781 wait complete. 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient: build_initial_monmap 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient: monmap: 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:27.003 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 1 Processor -- start 2024-09-16T06:44:27.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 1 -- start start 2024-09-16T06:44:27.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient: init 2024-09-16T06:44:27.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-b con 0x7fc6540a5c70 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-c con 0x7fc6540a7100 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc659024700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc6540a5c70 legacy=0x7fc6541c9ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient(hunting): picked mon.noname-a con 0x7fc65413bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540a6f50 con 0x7fc65413bab0 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65409ff30 con 0x7fc6540a5c70 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65406bd50 con 0x7fc6540a7100 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc65b288700 10 monclient(hunting): authenticate will time out at 1219.510378s 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc658823700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc6540a7100 0x7fc6541cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:52322/0 (socket says 172.21.15.189:52322) 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc658823700 1 -- 172.21.15.189:0/4088880421 learned_addr learned my addr 172.21.15.189:0/4088880421 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:27.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc659825700 1 --1- 172.21.15.189:0/4088880421 >> v1:172.21.15.76:6789/0 conn(0x7fc65413bab0 0x7fc6541d25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:60534/0 (socket says 172.21.15.189:60534) 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612218248 0 0) 0x7fc65406bd50 con 0x7fc6540a7100 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): my global_id is 125384 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6380035d0 con 0x7fc6540a7100 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082163971 0 0) 0x7fc6540a6f50 con 0x7fc65413bab0 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): my global_id is 154551 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65406bd50 con 0x7fc65413bab0 2024-09-16T06:44:27.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1042762776 0 0) 0x7fc6380035d0 con 0x7fc6540a7100 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6540a6f50 con 0x7fc6540a7100 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc6440041f0 con 0x7fc6540a7100 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:27.010 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 869490687 0 0) 0x7fc65406bd50 con 0x7fc65413bab0 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6380035d0 con 0x7fc65413bab0 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc64c001890 con 0x7fc65413bab0 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.992+0000 7fc649ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:27.011 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1414486530 0 0) 0x7fc6540a6f50 con 0x7fc6540a7100 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 >> v1:172.21.15.189:6789/0 conn(0x7fc6540a5c70 legacy=0x7fc6541c9ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 >> v1:172.21.15.76:6789/0 conn(0x7fc65413bab0 legacy=0x7fc6541d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 monclient: found mon.c 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540a04b0 con 0x7fc6540a7100 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:27.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 5 monclient: authenticate success, global_id 125384 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _renew_subs 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65413acc0 con 0x7fc6540a7100 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _renew_subs 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65411f100 con 0x7fc6540a7100 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc644004410 con 0x7fc6540a7100 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc644004720 con 0x7fc6540a7100 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc64b7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:27.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 10 monclient: dump: 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc6440048e0 con 0x7fc6540a7100 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc654134da0 con 0x7fc6540a7100 2024-09-16T06:44:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(150..150 src has 66..150) v4 ==== 8353+0+0 (unknown 2806568010 0 0) 0x7fc644044400 con 0x7fc6540a7100 2024-09-16T06:44:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:26.996+0000 7fc659024700 10 monclient: get_auth_request con 0x7fc638036340 auth_method 0 2024-09-16T06:44:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.008+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc6440050e0 con 0x7fc6540a7100 2024-09-16T06:44:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.008+0000 7fc649ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:27.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.008+0000 7fc649ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.200+0000 7fc65b288700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.200+0000 7fc65b288700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.200+0000 7fc65b288700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.205 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.200+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fc654067950 con 0x7fc6540a7100 2024-09-16T06:44:27.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.856+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v151) v1 ==== 137+0+0 (unknown 110050353 0 0) 0x7fc644012b00 con 0x7fc6540a7100 2024-09-16T06:44:27.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.856+0000 7fc649ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:27.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.856+0000 7fc649ffb700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-16T06:44:27.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.968+0000 7fc65b288700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:27.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.968+0000 7fc65b288700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:27.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.968+0000 7fc65b288700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:27.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:27.968+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fc65411f2f0 con 0x7fc6540a7100 2024-09-16T06:44:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.868+0000 7fc649ffb700 1 -- 172.21.15.189:0/4088880421 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v152) v1 ==== 241+0+0 (unknown 3700707988 0 0) 0x7fc644014690 con 0x7fc6540a7100 2024-09-16T06:44:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.868+0000 7fc649ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T06:44:28.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.868+0000 7fc649ffb700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc638036340 legacy=0x7fc638038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 10 monclient: shutdown 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 >> v1:172.21.15.76:6790/0 conn(0x7fc6540a7100 legacy=0x7fc6541cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 shutdown_connections 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 >> 172.21.15.189:0/4088880421 conn(0x7fc6540a79e0 msgr2=0x7fc6540a38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 shutdown_connections 2024-09-16T06:44:28.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:28.872+0000 7fc65b288700 1 -- 172.21.15.189:0/4088880421 wait complete. 2024-09-16T06:44:28.880 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T06:44:28.896 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:28.896 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:28.896 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T06:44:28.896 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '10\.' 2024-09-16T06:44:29.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.000+0000 7f4c792d4700 10 monclient: get_monmap_and_config 2024-09-16T06:44:29.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.000+0000 7f4c792d4700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.000+0000 7f4c792d4700 10 monclient: monmap: 2024-09-16T06:44:29.006 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:29.007066+0000 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:29.007066+0000 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:29.007 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.000+0000 7f4c792d4700 1 Processor -- start 2024-09-16T06:44:29.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 1 -- start start 2024-09-16T06:44:29.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient: init 2024-09-16T06:44:29.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-a con 0x7f4c7409e6d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-c con 0x7f4c7409f9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-b con 0x7f4c7409f1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7409ff50 con 0x7f4c7409e6d0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7409b8f0 con 0x7f4c7409f9d0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7409e030 con 0x7f4c7409f1c0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient(hunting): authenticate will time out at 1221.521388s 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c727fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4c7409f9d0 legacy=0x7f4c74136aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c72ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4c7409e6d0 0x7f4c7409eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32882/0 (socket says 172.21.15.189:32882) 2024-09-16T06:44:29.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c72ffd700 1 -- 172.21.15.189:0/3667675215 learned_addr learned my addr 172.21.15.189:0/3667675215 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936052740 0 0) 0x7f4c7409ff50 con 0x7f4c7409e6d0 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): my global_id is 154554 2024-09-16T06:44:29.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c5c0046e0 con 0x7f4c7409e6d0 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024657368 0 0) 0x7f4c7409e030 con 0x7f4c7409f1c0 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient(hunting): my global_id is 125387 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c7409ff50 con 0x7f4c7409f1c0 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76864205 0 0) 0x7f4c5c0046e0 con 0x7f4c7409e6d0 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 >> v1:172.21.15.76:6790/0 conn(0x7f4c7409f1c0 legacy=0x7f4c7413a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 >> v1:172.21.15.189:6789/0 conn(0x7f4c7409f9d0 legacy=0x7f4c74136aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 monclient: found mon.noname-a 2024-09-16T06:44:29.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740a1150 con 0x7f4c7409e6d0 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 5 monclient: authenticate success, global_id 154554 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c64001540 con 0x7f4c7409e6d0 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: dump: 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.015 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c64001760 con 0x7f4c7409e6d0 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:29.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c792d4700 4 set_mon_vals no callback set 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 1 -- 172.21.15.189:0/3667675215 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c64001b30 con 0x7f4c7409e6d0 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.004+0000 7f4c71ffb700 10 monclient: dump: 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.017 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 10 monclient: shutdown 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- 172.21.15.189:0/3667675215 >> v1:172.21.15.76:6789/0 conn(0x7f4c7409e6d0 legacy=0x7f4c7409eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- 172.21.15.189:0/3667675215 shutdown_connections 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- 172.21.15.189:0/3667675215 >> 172.21.15.189:0/3667675215 conn(0x7f4c740a4410 msgr2=0x7f4c740a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- 172.21.15.189:0/3667675215 shutdown_connections 2024-09-16T06:44:29.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- 172.21.15.189:0/3667675215 wait complete. 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 10 monclient: monmap: 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:29.023 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 Processor -- start 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 1 -- start start 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.008+0000 7f4c792d4700 10 monclient: init 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-c con 0x7f4c7409e6d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-a con 0x7f4c7409f1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient(hunting): picked mon.noname-b con 0x7f4c7409f9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c74123aa0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c74121ec0 con 0x7f4c7409f9d0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c74070880 con 0x7f4c7409e6d0 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c737fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4c7409f9d0 legacy=0x7f4c741d0200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient(hunting): authenticate will time out at 1221.528452s 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c727fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4c7409f1c0 0x7f4c741ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32888/0 (socket says 172.21.15.189:32888) 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c72ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4c7409e6d0 0x7f4c741c7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46342/0 (socket says 172.21.15.189:46342) 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c727fc700 1 -- 172.21.15.189:0/579813707 learned_addr learned my addr 172.21.15.189:0/579813707 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740071343 0 0) 0x7f4c74123aa0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): my global_id is 154557 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c600035d0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149704323 0 0) 0x7f4c74070880 con 0x7f4c7409e6d0 2024-09-16T06:44:29.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): my global_id is 125390 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c74123aa0 con 0x7f4c7409e6d0 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436930129 0 0) 0x7f4c600035d0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c74070880 con 0x7f4c7409f1c0 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c680041f0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): dump: 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.026 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3864783586 0 0) 0x7f4c74123aa0 con 0x7f4c7409e6d0 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c600035d0 con 0x7f4c7409e6d0 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c64001b30 con 0x7f4c7409e6d0 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:29.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient(hunting): dump: 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639699587 0 0) 0x7f4c74070880 con 0x7f4c7409f1c0 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 >> v1:172.21.15.76:6790/0 conn(0x7f4c7409e6d0 legacy=0x7f4c741c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 >> v1:172.21.15.189:6789/0 conn(0x7f4c7409f9d0 legacy=0x7f4c741d0200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 monclient: found mon.a 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740a25d0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 5 monclient: authenticate success, global_id 154557 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _renew_subs 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7406b210 con 0x7f4c7409f1c0 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _renew_subs 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c741219b0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c68004460 con 0x7f4c7409f1c0 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4c680047f0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c717fa700 4 set_mon_vals no callback set 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 10 monclient: dump: 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4c68004a10 con 0x7f4c7409f1c0 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c54005100 con 0x7f4c7409f1c0 2024-09-16T06:44:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.012+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(152..152 src has 66..152) v4 ==== 9018+0+0 (unknown 1303383233 0 0) 0x7f4c680447a0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.016+0000 7f4c72ffd700 10 monclient: get_auth_request con 0x7f4c60036250 auth_method 0 2024-09-16T06:44:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.024+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4c680147f0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.024+0000 7f4c5b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.024+0000 7f4c5b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:29.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c792d4700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c792d4700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c792d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f4c540053b0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7f4c68012c60 con 0x7f4c7409f1c0 2024-09-16T06:44:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c5b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.208+0000 7f4c5b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:29.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c792d4700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c792d4700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c792d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f4c54005650 con 0x7f4c7409f1c0 2024-09-16T06:44:29.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c5b7fe700 1 -- 172.21.15.189:0/579813707 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7f4c680143e0 con 0x7f4c7409f1c0 2024-09-16T06:44:29.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c5b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.312+0000 7f4c5b7fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:29.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4c60036250 legacy=0x7f4c60038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 10 monclient: shutdown 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 >> v1:172.21.15.76:6789/0 conn(0x7f4c7409f1c0 legacy=0x7f4c741ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 shutdown_connections 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 >> 172.21.15.189:0/579813707 conn(0x7f4c740a4410 msgr2=0x7f4c740a2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 shutdown_connections 2024-09-16T06:44:29.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.316+0000 7f4c792d4700 1 -- 172.21.15.189:0/579813707 wait complete. 2024-09-16T06:44:29.330 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:29.330 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T06:44:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '11\.' 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: get_monmap_and_config 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: monmap: 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:29.432586+0000 2024-09-16T06:44:29.432 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:29.432586+0000 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:29.433 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 Processor -- start 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- start start 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: init 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-a con 0x7f3e0809eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-b con 0x7f3e0809f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-c con 0x7f3e08139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0809ab80 con 0x7f3e0809eba0 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0809a690 con 0x7f3e08139a50 2024-09-16T06:44:29.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e08072230 con 0x7f3e0809f690 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient(hunting): authenticate will time out at 1221.945468s 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0ce7c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3e08139a50 legacy=0x7f3e0813be40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e07fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3e0809eba0 0x7f3e0809ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32890/0 (socket says 172.21.15.189:32890) 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e07fff700 1 -- 172.21.15.189:0/3329506471 learned_addr learned my addr 172.21.15.189:0/3329506471 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e077fe700 1 --1- 172.21.15.189:0/3329506471 >> v1:172.21.15.76:6790/0 conn(0x7f3e0809f690 0x7f3e08138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46354/0 (socket says 172.21.15.189:46354) 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834776538 0 0) 0x7f3e08072230 con 0x7f3e0809f690 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): my global_id is 125393 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df40035d0 con 0x7f3e0809f690 2024-09-16T06:44:29.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803441323 0 0) 0x7f3e0809ab80 con 0x7f3e0809eba0 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient(hunting): my global_id is 154560 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e08072230 con 0x7f3e0809eba0 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635070630 0 0) 0x7f3df40035d0 con 0x7f3e0809f690 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 >> v1:172.21.15.189:6789/0 conn(0x7f3e08139a50 legacy=0x7f3e0813be40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 >> v1:172.21.15.76:6789/0 conn(0x7f3e0809eba0 legacy=0x7f3e0809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 monclient: found mon.noname-b 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:29.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0809be90 con 0x7f3e0809f690 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 5 monclient: authenticate success, global_id 125393 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e00001610 con 0x7f3e0809f690 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: dump: 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.440 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e000018f0 con 0x7f3e0809f690 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 1 -- 172.21.15.189:0/3329506471 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e00001c80 con 0x7f3e0809f690 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 4 set_mon_vals no callback set 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e06ffd700 10 monclient: dump: 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 10 monclient: shutdown 2024-09-16T06:44:29.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- 172.21.15.189:0/3329506471 >> v1:172.21.15.76:6790/0 conn(0x7f3e0809f690 legacy=0x7f3e08138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- 172.21.15.189:0/3329506471 shutdown_connections 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- 172.21.15.189:0/3329506471 >> 172.21.15.189:0/3329506471 conn(0x7f3e0809fe70 msgr2=0x7f3e080a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- 172.21.15.189:0/3329506471 shutdown_connections 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.428+0000 7f3e0d67d700 1 -- 172.21.15.189:0/3329506471 wait complete. 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: monmap: 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 Processor -- start 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- start start 2024-09-16T06:44:29.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: init 2024-09-16T06:44:29.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-a con 0x7f3e0809eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-b con 0x7f3e0809f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient(hunting): picked mon.noname-c con 0x7f3e08139a50 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0809a380 con 0x7f3e0809eba0 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0806b970 con 0x7f3e0809f690 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e077fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3e0809f690 legacy=0x7f3e081cee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e08071540 con 0x7f3e08139a50 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient(hunting): authenticate will time out at 1221.950002s 2024-09-16T06:44:29.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e07fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3e0809eba0 0x7f3e081c9e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32900/0 (socket says 172.21.15.189:32900) 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e07fff700 1 -- 172.21.15.189:0/936996993 learned_addr learned my addr 172.21.15.189:0/936996993 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582881371 0 0) 0x7f3e0809a380 con 0x7f3e0809eba0 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): my global_id is 154563 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de80035d0 con 0x7f3e0809eba0 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748751546 0 0) 0x7f3e08071540 con 0x7f3e08139a50 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): my global_id is 125396 2024-09-16T06:44:29.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e0809a380 con 0x7f3e08139a50 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1367950707 0 0) 0x7f3de80035d0 con 0x7f3e0809eba0 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e08071540 con 0x7f3e0809eba0 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3df0001870 con 0x7f3e0809eba0 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235999090 0 0) 0x7f3e0809a380 con 0x7f3e08139a50 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de80035d0 con 0x7f3e08139a50 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3df80043e0 con 0x7f3e08139a50 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.451 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165979843 0 0) 0x7f3e08071540 con 0x7f3e0809eba0 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 >> v1:172.21.15.76:6790/0 conn(0x7f3e08139a50 legacy=0x7f3e081d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 >> v1:172.21.15.189:6789/0 conn(0x7f3e0809f690 legacy=0x7f3e081cee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 monclient: found mon.a 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080916e0 con 0x7f3e0809eba0 2024-09-16T06:44:29.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e04ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 5 monclient: authenticate success, global_id 154563 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _renew_subs 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e08098d90 con 0x7f3e0809eba0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _renew_subs 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.432+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080981a0 con 0x7f3e0809eba0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3df0004890 con 0x7f3e0809eba0 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3df0004ba0 con 0x7f3e0809eba0 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e067fc700 4 set_mon_vals no callback set 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 10 monclient: dump: 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3df000f690 con 0x7f3e0809eba0 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e0d67d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e0d67d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e0d67d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dec005100 con 0x7f3e0809eba0 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(152..152 src has 66..152) v4 ==== 9018+0+0 (unknown 1303383233 0 0) 0x7f3df0044830 con 0x7f3e0809eba0 2024-09-16T06:44:29.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.436+0000 7f3e077fe700 10 monclient: get_auth_request con 0x7f3de80362a0 auth_method 0 2024-09-16T06:44:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.444+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3df0010c30 con 0x7f3e0809eba0 2024-09-16T06:44:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.444+0000 7f3e04ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.444+0000 7f3e04ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:29.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e0d67d700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e0d67d700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e0d67d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.630 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f3dec0053b0 con 0x7f3e0809eba0 2024-09-16T06:44:29.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7f3df0012cf0 con 0x7f3e0809eba0 2024-09-16T06:44:29.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e04ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.624+0000 7f3e04ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:29.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e0d67d700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e0d67d700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e0d67d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.734 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f3dec005650 con 0x7f3e0809eba0 2024-09-16T06:44:29.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e04ff9700 1 -- 172.21.15.189:0/936996993 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7f3df0014470 con 0x7f3e0809eba0 2024-09-16T06:44:29.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e04ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:29.735 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.728+0000 7f3e04ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:29.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3de80362a0 legacy=0x7f3de8038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 10 monclient: shutdown 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 >> v1:172.21.15.76:6789/0 conn(0x7f3e0809eba0 legacy=0x7f3e081c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 shutdown_connections 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 >> 172.21.15.189:0/936996993 conn(0x7f3e0809fe70 msgr2=0x7f3e08138460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 shutdown_connections 2024-09-16T06:44:29.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.732+0000 7f3e0d67d700 1 -- 172.21.15.189:0/936996993 wait complete. 2024-09-16T06:44:29.748 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:29.748 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:29.749 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:29.749 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:29.749 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep fish 2024-09-16T06:44:29.750 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '12\.' 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.844+0000 7fc5957ea700 10 monclient: get_monmap_and_config 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.844+0000 7fc5957ea700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.844+0000 7fc5957ea700 10 monclient: monmap: 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:29.851744+0000 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:29.851744+0000 2024-09-16T06:44:29.851 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.852 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.852 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:29.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:29.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 Processor -- start 2024-09-16T06:44:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- start start 2024-09-16T06:44:29.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient: init 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-a con 0x7fc59009bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-c con 0x7fc59009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-b con 0x7fc590138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59009cca0 con 0x7fc59009bb00 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59009f750 con 0x7fc59009c5f0 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc590097620 con 0x7fc590138da0 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient(hunting): authenticate will time out at 1222.365017s 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58e7fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc59009c5f0 legacy=0x7fc590137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc59009bb00 0x7fc59009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32904/0 (socket says 172.21.15.189:32904) 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58effd700 1 -- 172.21.15.189:0/3627764612 learned_addr learned my addr 172.21.15.189:0/3627764612 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515070834 0 0) 0x7fc590097620 con 0x7fc590138da0 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): my global_id is 125399 2024-09-16T06:44:29.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5780057e0 con 0x7fc590138da0 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129271538 0 0) 0x7fc59009cca0 con 0x7fc59009bb00 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient(hunting): my global_id is 154566 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc590097620 con 0x7fc59009bb00 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351348520 0 0) 0x7fc590097620 con 0x7fc59009bb00 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 >> v1:172.21.15.76:6790/0 conn(0x7fc590138da0 legacy=0x7fc59013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 >> v1:172.21.15.189:6789/0 conn(0x7fc59009c5f0 legacy=0x7fc590137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 monclient: found mon.noname-a 2024-09-16T06:44:29.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590097900 con 0x7fc59009bb00 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc584001570 con 0x7fc59009bb00 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: dump: 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.859 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 5 monclient: authenticate success, global_id 154566 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc584001790 con 0x7fc59009bb00 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 1 -- 172.21.15.189:0/3627764612 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc584001ba0 con 0x7fc59009bb00 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 4 set_mon_vals no callback set 2024-09-16T06:44:29.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc58dffb700 10 monclient: dump: 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 10 monclient: shutdown 2024-09-16T06:44:29.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- 172.21.15.189:0/3627764612 >> v1:172.21.15.76:6789/0 conn(0x7fc59009bb00 legacy=0x7fc59009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- 172.21.15.189:0/3627764612 shutdown_connections 2024-09-16T06:44:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- 172.21.15.189:0/3627764612 >> 172.21.15.189:0/3627764612 conn(0x7fc5900a03e0 msgr2=0x7fc5900a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- 172.21.15.189:0/3627764612 shutdown_connections 2024-09-16T06:44:29.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.848+0000 7fc5957ea700 1 -- 172.21.15.189:0/3627764612 wait complete. 2024-09-16T06:44:29.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: build_initial_monmap 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: monmap: 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:29.866 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 Processor -- start 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- start start 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: init 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-b con 0x7fc59009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-a con 0x7fc59009c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient(hunting): picked mon.noname-c con 0x7fc590138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59012b7b0 con 0x7fc59009c5f0 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59012a420 con 0x7fc59009bb00 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59013c1b0 con 0x7fc590138da0 2024-09-16T06:44:29.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient(hunting): authenticate will time out at 1222.369759s 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc58effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc59009bb00 legacy=0x7fc590129600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc58e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc59009c5f0 0x7fc590129d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32916/0 (socket says 172.21.15.189:32916) 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc58e7fc700 1 -- 172.21.15.189:0/3409128789 learned_addr learned my addr 172.21.15.189:0/3409128789 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527264309 0 0) 0x7fc59012b7b0 con 0x7fc59009c5f0 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): my global_id is 154569 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c0035d0 con 0x7fc59009c5f0 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071040857 0 0) 0x7fc59013c1b0 con 0x7fc590138da0 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:29.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): my global_id is 125402 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59012b7b0 con 0x7fc590138da0 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620172555 0 0) 0x7fc57c0035d0 con 0x7fc59009c5f0 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc59013c1b0 con 0x7fc59009c5f0 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc580004200 con 0x7fc59009c5f0 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): dump: 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.869 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1590040233 0 0) 0x7fc59012b7b0 con 0x7fc590138da0 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc57c0035d0 con 0x7fc590138da0 2024-09-16T06:44:29.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc588001b00 con 0x7fc590138da0 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient(hunting): dump: 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.871 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2379404828 0 0) 0x7fc59013c1b0 con 0x7fc59009c5f0 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 >> v1:172.21.15.76:6790/0 conn(0x7fc590138da0 legacy=0x7fc5901d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 >> v1:172.21.15.189:6789/0 conn(0x7fc59009bb00 legacy=0x7fc590129600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 monclient: found mon.a 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590099380 con 0x7fc59009c5f0 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:29.872 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 5 monclient: authenticate success, global_id 154569 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _renew_subs 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59012d0e0 con 0x7fc59009c5f0 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _renew_subs 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59012db50 con 0x7fc59009c5f0 2024-09-16T06:44:29.873 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc580004470 con 0x7fc59009c5f0 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc580004800 con 0x7fc59009c5f0 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc58d7fa700 4 set_mon_vals no callback set 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.852+0000 7fc5777fe700 10 monclient: dump: 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:29.874 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc580004a20 con 0x7fc59009c5f0 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5957ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5957ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5957ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc570005100 con 0x7fc59009c5f0 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(152..152 src has 66..152) v4 ==== 9018+0+0 (unknown 1303383233 0 0) 0x7fc5800447d0 con 0x7fc59009c5f0 2024-09-16T06:44:29.875 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.856+0000 7fc58effd700 10 monclient: get_auth_request con 0x7fc57c0362c0 auth_method 0 2024-09-16T06:44:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.864+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc580014820 con 0x7fc59009c5f0 2024-09-16T06:44:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.864+0000 7fc5777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:29.876 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:29.864+0000 7fc5777fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:30.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.028+0000 7fc5957ea700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.028+0000 7fc5957ea700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.028+0000 7fc5957ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:30.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.028+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc5700053b0 con 0x7fc59009c5f0 2024-09-16T06:44:30.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.032+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7fc580012c90 con 0x7fc59009c5f0 2024-09-16T06:44:30.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.032+0000 7fc5777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.032+0000 7fc5777fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:30.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5957ea700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5957ea700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5957ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:30.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc570005f60 con 0x7fc59009c5f0 2024-09-16T06:44:30.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5777fe700 1 -- 172.21.15.189:0/3409128789 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v152) v1 ==== 83+0+2175 (unknown 3934654694 0 2697006557) 0x7fc580014410 con 0x7fc59009c5f0 2024-09-16T06:44:30.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5777fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:30.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.128+0000 7fc5777fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc57c0362c0 legacy=0x7fc57c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 10 monclient: shutdown 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 >> v1:172.21.15.76:6789/0 conn(0x7fc59009c5f0 legacy=0x7fc590129d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 shutdown_connections 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 >> 172.21.15.189:0/3409128789 conn(0x7fc5900a03e0 msgr2=0x7fc5900a23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 shutdown_connections 2024-09-16T06:44:30.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.132+0000 7fc5957ea700 1 -- 172.21.15.189:0/3409128789 wait complete. 2024-09-16T06:44:30.153 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:30.153 INFO:tasks.workunit.client.0.smithi189.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:30.153 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T06:44:30.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.252+0000 7fc8a68da700 10 monclient: get_monmap_and_config 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.252+0000 7fc8a68da700 10 monclient: build_initial_monmap 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.252+0000 7fc8a68da700 10 monclient: monmap: 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:30.259538+0000 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:30.259538+0000 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:30.260 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:30.260 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:30.260 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 1 Processor -- start 2024-09-16T06:44:30.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 1 -- start start 2024-09-16T06:44:30.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient: init 2024-09-16T06:44:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:30.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-a con 0x7fc8a00a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-c con 0x7fc8a00a3d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-b con 0x7fc8a00a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0099a40 con 0x7fc8a00a27a0 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0099ef0 con 0x7fc8a00a3d10 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc8a00a3d10 legacy=0x7fc8a013a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a009a1d0 con 0x7fc8a00a2350 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient(hunting): authenticate will time out at 1222.773682s 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc8a00a27a0 0x7fc8a00a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32922/0 (socket says 172.21.15.189:32922) 2024-09-16T06:44:30.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89ffff700 1 -- 172.21.15.189:0/3154015444 learned_addr learned my addr 172.21.15.189:0/3154015444 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180696634 0 0) 0x7fc8a0099a40 con 0x7fc8a00a27a0 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): my global_id is 154572 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8880035d0 con 0x7fc8a00a27a0 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557064974 0 0) 0x7fc8a009a1d0 con 0x7fc8a00a2350 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient(hunting): my global_id is 125405 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0099a40 con 0x7fc8a00a2350 2024-09-16T06:44:30.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817479843 0 0) 0x7fc8a0099a40 con 0x7fc8a00a2350 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 >> v1:172.21.15.189:6789/0 conn(0x7fc8a00a3d10 legacy=0x7fc8a013a5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 >> v1:172.21.15.76:6789/0 conn(0x7fc8a00a27a0 legacy=0x7fc8a00a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 monclient: found mon.noname-b 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00a3290 con 0x7fc8a00a2350 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: _finish_auth 0 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc898001970 con 0x7fc8a00a2350 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 5 monclient: authenticate success, global_id 125405 2024-09-16T06:44:30.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: dump: 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.274 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc898001c90 con 0x7fc8a00a2350 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 1 -- 172.21.15.189:0/3154015444 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc898002060 con 0x7fc8a00a2350 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc8a68da700 4 set_mon_vals no callback set 2024-09-16T06:44:30.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.256+0000 7fc89effd700 10 monclient: dump: 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.276 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: shutdown 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:30.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- 172.21.15.189:0/3154015444 >> v1:172.21.15.76:6790/0 conn(0x7fc8a00a2350 legacy=0x7fc8a013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- 172.21.15.189:0/3154015444 shutdown_connections 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- 172.21.15.189:0/3154015444 >> 172.21.15.189:0/3154015444 conn(0x7fc8a00a7d70 msgr2=0x7fc8a00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- 172.21.15.189:0/3154015444 shutdown_connections 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- 172.21.15.189:0/3154015444 wait complete. 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: build_initial_monmap 2024-09-16T06:44:30.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: monmap: 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:30.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:30.280 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 Processor -- start 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- start start 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: init 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-c con 0x7fc8a00a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:30.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-a con 0x7fc8a00a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 10 monclient(hunting): picked mon.noname-b con 0x7fc8a013eff0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a009b290 con 0x7fc8a00a27a0 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a01c9d70 con 0x7fc8a013eff0 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a68da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a006bea0 con 0x7fc8a00a2350 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.260+0000 7fc8a4e77700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fc8a013eff0 legacy=0x7fc8a01d2150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient(hunting): authenticate will time out at 1222.780040s 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc8a00a2350 0x7fc8a013e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46406/0 (socket says 172.21.15.189:46406) 2024-09-16T06:44:30.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89ffff700 1 -- 172.21.15.189:0/4283707412 learned_addr learned my addr 172.21.15.189:0/4283707412 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206246024 0 0) 0x7fc8a006bea0 con 0x7fc8a00a2350 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): my global_id is 125408 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc88c0035d0 con 0x7fc8a00a2350 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855755843 0 0) 0x7fc894003ac0 con 0x7fc8a00a27a0 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:30.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): my global_id is 154575 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a006bea0 con 0x7fc8a00a27a0 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139657600 0 0) 0x7fc88c0035d0 con 0x7fc8a00a2350 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc894003ac0 con 0x7fc8a00a2350 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8900019f0 con 0x7fc8a00a2350 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): dump: 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.284 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:30.285 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1715923233 0 0) 0x7fc8a006bea0 con 0x7fc8a00a27a0 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc88c0035d0 con 0x7fc8a00a27a0 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc894004130 con 0x7fc8a00a27a0 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient(hunting): dump: 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:30.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3822259537 0 0) 0x7fc894003ac0 con 0x7fc8a00a2350 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 >> v1:172.21.15.189:6789/0 conn(0x7fc8a013eff0 legacy=0x7fc8a01d2150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 >> v1:172.21.15.76:6789/0 conn(0x7fc8a00a27a0 legacy=0x7fc8a01cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 monclient: found mon.c 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a013b050 con 0x7fc8a00a2350 2024-09-16T06:44:30.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: _finish_auth 0 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 5 monclient: authenticate success, global_id 125408 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _renew_subs 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a013cb10 con 0x7fc8a00a2350 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _renew_subs 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a013bcb0 con 0x7fc8a00a2350 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc890001d00 con 0x7fc8a00a2350 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc890004ce0 con 0x7fc8a00a2350 2024-09-16T06:44:30.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 10 monclient: dump: 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:30.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89e7fc700 4 set_mon_vals no callback set 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc89000f750 con 0x7fc8a00a2350 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(152..152 src has 66..152) v4 ==== 9018+0+0 (unknown 1303383233 0 0) 0x7fc890044910 con 0x7fc8a00a2350 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc880005100 con 0x7fc8a00a2350 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.264+0000 7fc89f7fe700 10 monclient: get_auth_request con 0x7fc88c036230 auth_method 0 2024-09-16T06:44:30.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.276+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc890014900 con 0x7fc8a00a2350 2024-09-16T06:44:30.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.276+0000 7fc89cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:30.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.276+0000 7fc89cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.436+0000 7fc8a68da700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.436+0000 7fc8a68da700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.436+0000 7fc8a68da700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.436+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fc880005bc0 con 0x7fc8a00a2350 2024-09-16T06:44:30.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.880+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v153) v1 ==== 123+0+0 (unknown 2169363404 0 0) 0x7fc890012d70 con 0x7fc8a00a2350 2024-09-16T06:44:30.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.880+0000 7fc89cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:30.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.880+0000 7fc89cff9700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T06:44:30.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.984+0000 7fc8a68da700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:30.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.984+0000 7fc8a68da700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:30.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.984+0000 7fc8a68da700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:30.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:30.984+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fc8800010a0 con 0x7fc8a00a2350 2024-09-16T06:44:31.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.896+0000 7fc89cff9700 1 -- 172.21.15.189:0/4283707412 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v154) v1 ==== 149+0+0 (unknown 3264209139 0 0) 0x7fc8900144f0 con 0x7fc8a00a2350 2024-09-16T06:44:31.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.896+0000 7fc89cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:31.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.896+0000 7fc89cff9700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc88c036230 legacy=0x7fc88c0386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 10 monclient: shutdown 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 >> v1:172.21.15.76:6790/0 conn(0x7fc8a00a2350 legacy=0x7fc8a013e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 shutdown_connections 2024-09-16T06:44:31.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 >> 172.21.15.189:0/4283707412 conn(0x7fc8a00a7d70 msgr2=0x7fc8a0137c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:31.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 shutdown_connections 2024-09-16T06:44:31.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:31.900+0000 7fc8a68da700 1 -- 172.21.15.189:0/4283707412 wait complete. 2024-09-16T06:44:31.908 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:31.923 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-16T06:44:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: get_monmap_and_config 2024-09-16T06:44:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: build_initial_monmap 2024-09-16T06:44:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: monmap: 2024-09-16T06:44:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:32.037 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:32.037927+0000 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:32.037927+0000 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:32.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 1 Processor -- start 2024-09-16T06:44:32.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 1 -- start start 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: init 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-c con 0x7f82b40a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-b con 0x7f82b409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-a con 0x7f82b40a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b409a040 con 0x7f82b40a3ca0 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40a33d0 con 0x7f82b40a2350 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4099d60 con 0x7f82b409aa20 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82ba07a700 10 monclient(hunting): authenticate will time out at 1224.551527s 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82b37fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f82b40a2350 legacy=0x7f82b40a2730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:32.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82b3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f82b40a3ca0 0x7f82b413dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32940/0 (socket says 172.21.15.189:32940) 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.032+0000 7f82b3fff700 1 -- 172.21.15.189:0/2615857803 learned_addr learned my addr 172.21.15.189:0/2615857803 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140612309 0 0) 0x7f82b409a040 con 0x7f82b40a3ca0 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): my global_id is 154578 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0035d0 con 0x7f82b40a3ca0 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206102749 0 0) 0x7f82b4099d60 con 0x7f82b409aa20 2024-09-16T06:44:32.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient(hunting): my global_id is 125411 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b409a040 con 0x7f82b409aa20 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729929915 0 0) 0x7f829c0035d0 con 0x7f82b40a3ca0 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 >> v1:172.21.15.76:6790/0 conn(0x7f82b409aa20 legacy=0x7f82b413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 >> v1:172.21.15.189:6789/0 conn(0x7f82b40a2350 legacy=0x7f82b40a2730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 monclient: found mon.noname-a 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40a3920 con 0x7f82b40a3ca0 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:32.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 5 monclient: authenticate success, global_id 154578 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82ac001990 con 0x7f82b40a3ca0 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: dump: 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ac001c70 con 0x7f82b40a3ca0 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 4 set_mon_vals no callback set 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 1 -- 172.21.15.189:0/2615857803 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82ac002000 con 0x7f82b40a3ca0 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:32.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b27fc700 10 monclient: dump: 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: shutdown 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:32.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- 172.21.15.189:0/2615857803 >> v1:172.21.15.76:6789/0 conn(0x7f82b40a3ca0 legacy=0x7f82b413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- 172.21.15.189:0/2615857803 shutdown_connections 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- 172.21.15.189:0/2615857803 >> 172.21.15.189:0/2615857803 conn(0x7f82b40a7d70 msgr2=0x7f82b40a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- 172.21.15.189:0/2615857803 shutdown_connections 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- 172.21.15.189:0/2615857803 wait complete. 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: build_initial_monmap 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: monmap: 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:32.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 Processor -- start 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- start start 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: init 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-c con 0x7f82b409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-a con 0x7f82b40a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:32.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient(hunting): picked mon.noname-b con 0x7f82b40a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b409b6b0 con 0x7f82b40a2350 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b41219c0 con 0x7f82b40a3ca0 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b406bed0 con 0x7f82b409aa20 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82ba07a700 10 monclient(hunting): authenticate will time out at 1224.555942s 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.036+0000 7f82b3fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f82b40a3ca0 legacy=0x7f82b41d2460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82b2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f82b40a2350 0x7f82b41ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:32942/0 (socket says 172.21.15.189:32942) 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82b2ffd700 1 -- 172.21.15.189:0/3785948178 learned_addr learned my addr 172.21.15.189:0/3785948178 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347923106 0 0) 0x7f82b409a040 con 0x7f82b40a2350 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:32.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): my global_id is 154581 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a00035d0 con 0x7f82b40a2350 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051588007 0 0) 0x7f82b406bed0 con 0x7f82b409aa20 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): my global_id is 125414 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b409a040 con 0x7f82b409aa20 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098402823 0 0) 0x7f82a00035d0 con 0x7f82b40a2350 2024-09-16T06:44:32.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b406bed0 con 0x7f82b40a2350 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82a4001770 con 0x7f82b40a2350 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): dump: 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.059 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413699128 0 0) 0x7f82b409a040 con 0x7f82b409aa20 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a00035d0 con 0x7f82b409aa20 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82a80040f0 con 0x7f82b409aa20 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:32.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient(hunting): dump: 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 748955092 0 0) 0x7f82b406bed0 con 0x7f82b40a2350 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 >> v1:172.21.15.76:6790/0 conn(0x7f82b409aa20 legacy=0x7f82b41345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 >> v1:172.21.15.189:6789/0 conn(0x7f82b40a3ca0 legacy=0x7f82b41d2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 monclient: found mon.a 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b409b110 con 0x7f82b40a2350 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 5 monclient: authenticate success, global_id 154581 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _renew_subs 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4121f80 con 0x7f82b40a2350 2024-09-16T06:44:32.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _renew_subs 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b411fc20 con 0x7f82b40a2350 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a40036c0 con 0x7f82b40a2350 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82a4003a50 con 0x7f82b40a2350 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82b1ffb700 4 set_mon_vals no callback set 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 10 monclient: dump: 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.064 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f82a4003c30 con 0x7f82b40a2350 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(154..154 src has 66..154) v4 ==== 8353+0+0 (unknown 1748476054 0 0) 0x7f82a4044420 con 0x7f82b40a2350 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8294005100 con 0x7f82b40a2350 2024-09-16T06:44:32.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.040+0000 7f82b37fe700 10 monclient: get_auth_request con 0x7f82a0036310 auth_method 0 2024-09-16T06:44:32.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.052+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f82a40146d0 con 0x7f82b40a2350 2024-09-16T06:44:32.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.052+0000 7f829bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:32.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.052+0000 7f829bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:32.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.228+0000 7f82ba07a700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:32.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.228+0000 7f82ba07a700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:32.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.228+0000 7f82ba07a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:32.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.228+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f82940053b0 con 0x7f82b40a2350 2024-09-16T06:44:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.904+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v155) v1 ==== 145+0+0 (unknown 172702741 0 0) 0x7f82a401cd20 con 0x7f82b40a2350 2024-09-16T06:44:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.904+0000 7f829bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:32.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:32.904+0000 7f829bfff700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-16T06:44:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.016+0000 7f82ba07a700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.016+0000 7f82ba07a700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.016+0000 7f82ba07a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:33.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.016+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f82940055c0 con 0x7f82b40a2350 2024-09-16T06:44:33.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.916+0000 7f829bfff700 1 -- 172.21.15.189:0/3785948178 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v156) v1 ==== 257+0+0 (unknown 1607106326 0 0) 0x7f82a4012b40 con 0x7f82b40a2350 2024-09-16T06:44:33.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.916+0000 7f829bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T06:44:33.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.916+0000 7f829bfff700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T06:44:33.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 >> v1:172.21.15.189:6812/3098810674 conn(0x7f82a0036310 legacy=0x7f82a00387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:33.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 10 monclient: shutdown 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 >> v1:172.21.15.76:6789/0 conn(0x7f82b40a2350 legacy=0x7f82b41ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 shutdown_connections 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 >> 172.21.15.189:0/3785948178 conn(0x7f82b40a7d70 msgr2=0x7f82b40a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.920+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 shutdown_connections 2024-09-16T06:44:33.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:33.924+0000 7f82ba07a700 1 -- 172.21.15.189:0/3785948178 wait complete. 2024-09-16T06:44:33.929 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T06:44:33.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:33.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:33.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:33.944 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '10\.' 2024-09-16T06:44:33.959 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:44:33.959 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:44:33.960 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:44:33.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:44:33.962 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:44:33.962 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T06:44:33.962 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T06:44:33.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:44:34.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: get_monmap_and_config 2024-09-16T06:44:34.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: build_initial_monmap 2024-09-16T06:44:34.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: monmap: 2024-09-16T06:44:34.057 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:34.057 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:34.057 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:34.053353+0000 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:34.053353+0000 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 1 Processor -- start 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 1 -- start start 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: init 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:34.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient(hunting): picked mon.noname-a con 0x7faea009f770 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient(hunting): picked mon.noname-c con 0x7faea012ee70 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient(hunting): picked mon.noname-b con 0x7faea01388d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea00a0f30 con 0x7faea009f770 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea009b0d0 con 0x7faea012ee70 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea009fb50 con 0x7faea01388d0 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient(hunting): authenticate will time out at 1226.566073s 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9e59c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faea012ee70 legacy=0x7faea01371b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:34.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9f59e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faea01388d0 0x7faea013acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46442/0 (socket says 172.21.15.189:46442) 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9f59e700 1 -- 172.21.15.189:0/1593521089 learned_addr learned my addr 172.21.15.189:0/1593521089 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982516563 0 0) 0x7faea009fb50 con 0x7faea01388d0 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient(hunting): my global_id is 125417 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae880035d0 con 0x7faea01388d0 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846878644 0 0) 0x7fae880035d0 con 0x7faea01388d0 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 >> v1:172.21.15.189:6789/0 conn(0x7faea012ee70 legacy=0x7faea01371b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 >> v1:172.21.15.76:6789/0 conn(0x7faea009f770 legacy=0x7faea0099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:34.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 monclient: found mon.noname-b 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00978e0 con 0x7faea01388d0 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 5 monclient: authenticate success, global_id 125417 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae98001990 con 0x7faea01388d0 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:34.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: dump: 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae98001c80 con 0x7faea01388d0 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 1 -- 172.21.15.189:0/1593521089 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae98002010 con 0x7faea01388d0 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7faea5241700 4 set_mon_vals no callback set 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.048+0000 7fae9dd9b700 10 monclient: dump: 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:34.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: shutdown 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- 172.21.15.189:0/1593521089 >> v1:172.21.15.76:6790/0 conn(0x7faea01388d0 legacy=0x7faea013acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- 172.21.15.189:0/1593521089 shutdown_connections 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- 172.21.15.189:0/1593521089 >> 172.21.15.189:0/1593521089 conn(0x7faea00a0410 msgr2=0x7faea00a0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- 172.21.15.189:0/1593521089 shutdown_connections 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- 172.21.15.189:0/1593521089 wait complete. 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: build_initial_monmap 2024-09-16T06:44:34.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: monmap: 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:34.065 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 Processor -- start 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- start start 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: init 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:34.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient(hunting): picked mon.noname-b con 0x7faea009f770 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient(hunting): picked mon.noname-c con 0x7faea012ee70 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient(hunting): picked mon.noname-a con 0x7faea01388d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea0099fe0 con 0x7faea01388d0 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea012e6f0 con 0x7faea009f770 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea01ca8a0 con 0x7faea012ee70 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7faea5241700 10 monclient(hunting): authenticate will time out at 1226.570156s 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae9ed9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7faea009f770 legacy=0x7faea01c9a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae9e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faea012ee70 0x7faea01ca190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46446/0 (socket says 172.21.15.189:46446) 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae9e59c700 1 -- 172.21.15.189:0/1168972824 learned_addr learned my addr 172.21.15.189:0/1168972824 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445500653 0 0) 0x7faea01ca8a0 con 0x7faea012ee70 2024-09-16T06:44:34.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): my global_id is 125420 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae8c0035d0 con 0x7faea012ee70 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3627478882 0 0) 0x7fae8c0035d0 con 0x7faea012ee70 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faea01ca8a0 con 0x7faea012ee70 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae94004200 con 0x7faea012ee70 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): dump: 2024-09-16T06:44:34.072 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:34.073 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:34.073 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.073 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 764606516 0 0) 0x7faea01ca8a0 con 0x7faea012ee70 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 >> v1:172.21.15.189:6789/0 conn(0x7faea009f770 legacy=0x7faea01c9a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.052+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 >> v1:172.21.15.76:6789/0 conn(0x7faea01388d0 legacy=0x7faea01d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:34.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 monclient: found mon.c 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0130a90 con 0x7faea012ee70 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 5 monclient: authenticate success, global_id 125420 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _renew_subs 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea0131a60 con 0x7faea012ee70 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _renew_subs 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faea009a6a0 con 0x7faea012ee70 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae940047f0 con 0x7faea012ee70 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae9d59a700 4 set_mon_vals no callback set 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae94004b00 con 0x7faea012ee70 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 10 monclient: dump: 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:34.076 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fae9400f5f0 con 0x7faea012ee70 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae80005100 con 0x7faea012ee70 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(156..156 src has 66..156) v4 ==== 9042+0+0 (unknown 113264351 0 0) 0x7fae940447e0 con 0x7faea012ee70 2024-09-16T06:44:34.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.056+0000 7fae9ed9d700 10 monclient: get_auth_request con 0x7fae8c0360d0 auth_method 0 2024-09-16T06:44:34.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.068+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fae940145f0 con 0x7faea012ee70 2024-09-16T06:44:34.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.068+0000 7fae877fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:34.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.068+0000 7fae877fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:34.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.244+0000 7faea5241700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:34.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.244+0000 7faea5241700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:34.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.244+0000 7faea5241700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:34.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:34.244+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fae800053b0 con 0x7faea012ee70 2024-09-16T06:44:37.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:37.056+0000 7fae867fc700 10 monclient: tick 2024-09-16T06:44:37.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:37.056+0000 7fae867fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:37.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:37.056+0000 7fae867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:37.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:37.056+0000 7fae867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:38.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:38.056+0000 7fae867fc700 10 monclient: tick 2024-09-16T06:44:38.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:38.056+0000 7fae867fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:38.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:38.056+0000 7fae867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.052+0000 7fae867fc700 10 monclient: tick 2024-09-16T06:44:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.052+0000 7fae867fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:39.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.052+0000 7fae867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.144+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7fae94012c90 con 0x7faea012ee70 2024-09-16T06:44:39.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.144+0000 7fae877fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.144+0000 7fae877fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.256+0000 7faea5241700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.256+0000 7faea5241700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.256+0000 7faea5241700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:39.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.256+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fae80005650 con 0x7faea012ee70 2024-09-16T06:44:39.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.256+0000 7fae877fe700 1 -- 172.21.15.189:0/1168972824 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7fae94014410 con 0x7faea012ee70 2024-09-16T06:44:39.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7fae877fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7fae877fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 >> v1:172.21.15.189:6812/3098810674 conn(0x7fae8c0360d0 legacy=0x7fae8c038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 10 monclient: shutdown 2024-09-16T06:44:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:39.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 >> v1:172.21.15.76:6790/0 conn(0x7faea012ee70 legacy=0x7faea01ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 shutdown_connections 2024-09-16T06:44:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 >> 172.21.15.189:0/1168972824 conn(0x7faea00a0410 msgr2=0x7faea00714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 shutdown_connections 2024-09-16T06:44:39.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.260+0000 7faea5241700 1 -- 172.21.15.189:0/1168972824 wait complete. 2024-09-16T06:44:39.277 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:39.277 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:39.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:39.278 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '11\.' 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.372+0000 7f9c1c071700 10 monclient: get_monmap_and_config 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.372+0000 7f9c1c071700 10 monclient: build_initial_monmap 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.372+0000 7f9c1c071700 10 monclient: monmap: 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:39.379893+0000 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:39.379893+0000 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:39.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 1 Processor -- start 2024-09-16T06:44:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 1 -- start start 2024-09-16T06:44:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient: init 2024-09-16T06:44:39.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-b con 0x7f9c1409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-a con 0x7f9c1409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-c con 0x7f9c14138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409cca0 con 0x7f9c1409c5f0 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1409f750 con 0x7f9c14138da0 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14097620 con 0x7f9c1409bb00 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient(hunting): authenticate will time out at 1231.892686s 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1a60e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c14138da0 0x7f9c1413b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52948/0 (socket says 172.21.15.189:52948) 2024-09-16T06:44:39.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1a60e700 1 -- 172.21.15.189:0/1198454374 learned_addr learned my addr 172.21.15.189:0/1198454374 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108527763 0 0) 0x7f9c14097620 con 0x7f9c1409bb00 2024-09-16T06:44:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient(hunting): my global_id is 125423 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c000035d0 con 0x7f9c1409bb00 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1246383554 0 0) 0x7f9c000035d0 con 0x7f9c1409bb00 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 >> v1:172.21.15.189:6789/0 conn(0x7f9c14138da0 legacy=0x7f9c1413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 >> v1:172.21.15.76:6789/0 conn(0x7f9c1409c5f0 legacy=0x7f9c14137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: _finish_hunting 0 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 monclient: found mon.noname-b 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14097900 con 0x7f9c1409bb00 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: _finish_auth 0 2024-09-16T06:44:39.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: _check_auth_tickets 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c10001610 con 0x7f9c1409bb00 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 5 monclient: authenticate success, global_id 125423 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: dump: 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.387 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c10001900 con 0x7f9c1409bb00 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 1 -- 172.21.15.189:0/1198454374 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c10001d10 con 0x7f9c1409bb00 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c18e0b700 10 monclient: dump: 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.388 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.376+0000 7f9c1c071700 4 set_mon_vals no callback set 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: shutdown 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- 172.21.15.189:0/1198454374 >> v1:172.21.15.76:6790/0 conn(0x7f9c1409bb00 legacy=0x7f9c1409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- 172.21.15.189:0/1198454374 shutdown_connections 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- 172.21.15.189:0/1198454374 >> 172.21.15.189:0/1198454374 conn(0x7f9c140a03e0 msgr2=0x7f9c140a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- 172.21.15.189:0/1198454374 shutdown_connections 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- 172.21.15.189:0/1198454374 wait complete. 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: build_initial_monmap 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: monmap: 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:39.390 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:39.391 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 Processor -- start 2024-09-16T06:44:39.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- start start 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: init 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-b con 0x7f9c1409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-a con 0x7f9c1409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient(hunting): picked mon.noname-c con 0x7f9c14138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1412aa60 con 0x7f9c1409c5f0 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1412d550 con 0x7f9c1409bb00 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141c8710 con 0x7f9c14138da0 2024-09-16T06:44:39.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c1c071700 10 monclient(hunting): authenticate will time out at 1231.899211s 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c19e0d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9c1409bb00 0x7f9c141c78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52950/0 (socket says 172.21.15.189:52950) 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.380+0000 7f9c19e0d700 1 -- 172.21.15.189:0/4012970312 learned_addr learned my addr 172.21.15.189:0/4012970312 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143065464 0 0) 0x7f9c141c8710 con 0x7f9c14138da0 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): my global_id is 125426 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80035d0 con 0x7f9c14138da0 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808175922 0 0) 0x7f9c1412d550 con 0x7f9c1409bb00 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): my global_id is 174100 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c141c8710 con 0x7f9c1409bb00 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981404653 0 0) 0x7f9c141c8710 con 0x7f9c1409bb00 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c1412d550 con 0x7f9c1409bb00 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c100015d0 con 0x7f9c1409bb00 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): dump: 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.398 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520214895 0 0) 0x7f9bf80035d0 con 0x7f9c14138da0 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c141c8710 con 0x7f9c14138da0 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c0c001a60 con 0x7f9c14138da0 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient(hunting): dump: 2024-09-16T06:44:39.399 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1431964050 0 0) 0x7f9c1412d550 con 0x7f9c1409bb00 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 >> v1:172.21.15.76:6790/0 conn(0x7f9c14138da0 legacy=0x7f9c141cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 >> v1:172.21.15.76:6789/0 conn(0x7f9c1409c5f0 legacy=0x7f9c141c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 monclient: found mon.b 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1409a6e0 con 0x7f9c1409bb00 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 5 monclient: authenticate success, global_id 174100 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _renew_subs 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140993f0 con 0x7f9c1409bb00 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _renew_subs 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1412d270 con 0x7f9c1409bb00 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c10003cd0 con 0x7f9c1409bb00 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c10004060 con 0x7f9c1409bb00 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 10 monclient: dump: 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c07fff700 4 set_mon_vals no callback set 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9c10004240 con 0x7f9c1409bb00 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc005100 con 0x7f9c1409bb00 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(156..156 src has 66..156) v4 ==== 9042+0+0 (unknown 113264351 0 0) 0x7f9c10044b70 con 0x7f9c1409bb00 2024-09-16T06:44:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.384+0000 7f9c1960c700 10 monclient: get_auth_request con 0x7f9bf803a530 auth_method 0 2024-09-16T06:44:39.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.392+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9c10011670 con 0x7f9c1409bb00 2024-09-16T06:44:39.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.392+0000 7f9c067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.392+0000 7f9c067fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.568+0000 7f9c1c071700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.568+0000 7f9c1c071700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.572+0000 7f9c1c071700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.572+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9bfc0053b0 con 0x7f9c1409bb00 2024-09-16T06:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.572+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7f9c100148e0 con 0x7f9c1409bb00 2024-09-16T06:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.572+0000 7f9c067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.572+0000 7f9c067fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:39.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.672+0000 7f9c1c071700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.672+0000 7f9c1c071700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.672+0000 7f9c1c071700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.672+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9bfc005f60 con 0x7f9c1409bb00 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c067fc700 1 -- 172.21.15.189:0/4012970312 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7f9c10014ac0 con 0x7f9c1409bb00 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c067fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c067fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9bf803a530 legacy=0x7f9bf803c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 10 monclient: shutdown 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 >> v1:172.21.15.189:6789/0 conn(0x7f9c1409bb00 legacy=0x7f9c141c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 shutdown_connections 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 >> 172.21.15.189:0/4012970312 conn(0x7f9c140a03e0 msgr2=0x7f9c140a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 shutdown_connections 2024-09-16T06:44:39.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.676+0000 7f9c1c071700 1 -- 172.21.15.189:0/4012970312 wait complete. 2024-09-16T06:44:39.693 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:39.693 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:39.693 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:39.694 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:39.694 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:39.694 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '12\.' 2024-09-16T06:44:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.792+0000 7f86007d2700 10 monclient: get_monmap_and_config 2024-09-16T06:44:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.792+0000 7f86007d2700 10 monclient: build_initial_monmap 2024-09-16T06:44:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.792+0000 7f86007d2700 10 monclient: monmap: 2024-09-16T06:44:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:39.799 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:39.799975+0000 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:39.799975+0000 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:39.800 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 Processor -- start 2024-09-16T06:44:39.803 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- start start 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: init 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-c con 0x7f85f809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-a con 0x7f85f8099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-b con 0x7f85f81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f809f520 con 0x7f85f8099700 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f809bc10 con 0x7f85f809f140 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f809b810 con 0x7f85f81396a0 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient(hunting): authenticate will time out at 1232.312933s 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fe56e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85f809f140 0x7f85f809a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52956/0 (socket says 172.21.15.189:52956) 2024-09-16T06:44:39.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fe56e700 1 -- 172.21.15.189:0/1874989304 learned_addr learned my addr 172.21.15.189:0/1874989304 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385999741 0 0) 0x7f85f809bc10 con 0x7f85f809f140 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient(hunting): my global_id is 174103 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e40035d0 con 0x7f85f809f140 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065951230 0 0) 0x7f85e40035d0 con 0x7f85f809f140 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 >> v1:172.21.15.76:6790/0 conn(0x7f85f81396a0 legacy=0x7f85f813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 >> v1:172.21.15.76:6789/0 conn(0x7f85f8099700 legacy=0x7f85f809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: _finish_hunting 0 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 monclient: found mon.noname-c 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:39.805 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f809ceb0 con 0x7f85f809f140 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: _finish_auth 0 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: _check_auth_tickets 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 5 monclient: authenticate success, global_id 174103 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85f4001610 con 0x7f85f809f140 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: dump: 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.806 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85f4001900 con 0x7f85f809f140 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 4 set_mon_vals no callback set 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 1 -- 172.21.15.189:0/1874989304 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85f4001d10 con 0x7f85f809f140 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.807 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f85fd56c700 10 monclient: dump: 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: shutdown 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:39.808 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- 172.21.15.189:0/1874989304 >> v1:172.21.15.189:6789/0 conn(0x7f85f809f140 legacy=0x7f85f809a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- 172.21.15.189:0/1874989304 shutdown_connections 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- 172.21.15.189:0/1874989304 >> 172.21.15.189:0/1874989304 conn(0x7f85f80a03e0 msgr2=0x7f85f80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- 172.21.15.189:0/1874989304 shutdown_connections 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 1 -- 172.21.15.189:0/1874989304 wait complete. 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: build_initial_monmap 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.796+0000 7f86007d2700 10 monclient: monmap: 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:39.809 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:39.810 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:39.810 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:39.810 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 Processor -- start 2024-09-16T06:44:39.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- start start 2024-09-16T06:44:39.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: init 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-a con 0x7f85f8099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-b con 0x7f85f809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient(hunting): picked mon.noname-c con 0x7f85f81396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8090690 con 0x7f85f8099700 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8092710 con 0x7f85f809f140 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8091260 con 0x7f85f81396a0 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient(hunting): authenticate will time out at 1232.317772s 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85fdd6d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f85f809f140 0x7f85f81cc850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52960/0 (socket says 172.21.15.189:52960) 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85fdd6d700 1 -- 172.21.15.189:0/1347608641 learned_addr learned my addr 172.21.15.189:0/1347608641 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:39.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332035453 0 0) 0x7f85f8092710 con 0x7f85f809f140 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): my global_id is 174106 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85dc0035d0 con 0x7f85f809f140 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271943845 0 0) 0x7f85f8091260 con 0x7f85f81396a0 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): my global_id is 125432 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8092710 con 0x7f85f81396a0 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43418280 0 0) 0x7f85dc0035d0 con 0x7f85f809f140 2024-09-16T06:44:39.815 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f8091260 con 0x7f85f809f140 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85e80018e0 con 0x7f85f809f140 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): dump: 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.816 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109632568 0 0) 0x7f85f8091260 con 0x7f85f809f140 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 >> v1:172.21.15.76:6790/0 conn(0x7f85f81396a0 legacy=0x7f85f81cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 >> v1:172.21.15.76:6789/0 conn(0x7f85f8099700 legacy=0x7f85f8095f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 monclient: found mon.b 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80929f0 con 0x7f85f809f140 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:39.817 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f85eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 5 monclient: authenticate success, global_id 174106 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _renew_subs 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f80714a0 con 0x7f85f809f140 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _renew_subs 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.800+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f8090fe0 con 0x7f85f809f140 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e8001c90 con 0x7f85f809f140 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85e8003af0 con 0x7f85f809f140 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85fcd6b700 4 set_mon_vals no callback set 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:39.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 10 monclient: dump: 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f85e8003c50 con 0x7f85f809f140 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f86007d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.819 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(156..156 src has 66..156) v4 ==== 9042+0+0 (unknown 113264351 0 0) 0x7f85e8044780 con 0x7f85f809f140 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f86007d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f86007d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0005100 con 0x7f85f809f140 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.804+0000 7f85fe56e700 10 monclient: get_auth_request con 0x7f85dc036250 auth_method 0 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.808+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f85e8014810 con 0x7f85f809f140 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.808+0000 7f85eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:39.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.808+0000 7f85eeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:39.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f86007d2700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f86007d2700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f86007d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f85e00053b0 con 0x7f85f809f140 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7f85e801ce60 con 0x7f85f809f140 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f85eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:39.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:39.984+0000 7f85eeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f86007d2700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f86007d2700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f86007d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f85e0005af0 con 0x7f85f809f140 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f85eeffd700 1 -- 172.21.15.189:0/1347608641 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v156) v1 ==== 83+0+2378 (unknown 2042302870 0 657427554) 0x7f85e8012c80 con 0x7f85f809f140 2024-09-16T06:44:40.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f85eeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:40.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.092+0000 7f85eeffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 >> v1:172.21.15.189:6812/3098810674 conn(0x7f85dc036250 legacy=0x7f85dc038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 10 monclient: shutdown 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 >> v1:172.21.15.189:6789/0 conn(0x7f85f809f140 legacy=0x7f85f81cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 shutdown_connections 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 >> 172.21.15.189:0/1347608641 conn(0x7f85f80a03e0 msgr2=0x7f85f80a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 shutdown_connections 2024-09-16T06:44:40.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.096+0000 7f86007d2700 1 -- 172.21.15.189:0/1347608641 wait complete. 2024-09-16T06:44:40.114 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:40.114 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T06:44:40.115 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-16T06:44:40.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: get_monmap_and_config 2024-09-16T06:44:40.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: build_initial_monmap 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: monmap: 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:40.212944+0000 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:40.212944+0000 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:40.213 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 Processor -- start 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 -- start start 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: init 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-b con 0x7f13c01352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-a con 0x7f13c0135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-c con 0x7f13c00a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c009b050 con 0x7f13c0135db0 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c006ab20 con 0x7f13c00a2030 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c006a840 con 0x7f13c01352c0 2024-09-16T06:44:40.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient(hunting): authenticate will time out at 1232.725953s 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13c00a2030 0x7f13c013df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52964/0 (socket says 172.21.15.189:52964) 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bf7fe700 1 -- 172.21.15.189:0/3730678992 learned_addr learned my addr 172.21.15.189:0/3730678992 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276270392 0 0) 0x7f13c006ab20 con 0x7f13c00a2030 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient(hunting): my global_id is 174109 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a80035d0 con 0x7f13c00a2030 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3081978014 0 0) 0x7f13a80035d0 con 0x7f13c00a2030 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 >> v1:172.21.15.76:6790/0 conn(0x7f13c01352c0 legacy=0x7f13c01356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 >> v1:172.21.15.76:6789/0 conn(0x7f13c0135db0 legacy=0x7f13c013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:40.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 monclient: found mon.noname-c 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c006c1b0 con 0x7f13c00a2030 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13b8001990 con 0x7f13c00a2030 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 5 monclient: authenticate success, global_id 174109 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: dump: 2024-09-16T06:44:40.218 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b8001c80 con 0x7f13c00a2030 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:40.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 4 set_mon_vals no callback set 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 1 -- 172.21.15.189:0/3730678992 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13b8002090 con 0x7f13c00a2030 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bdffb700 10 monclient: dump: 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:40.220 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 10 monclient: shutdown 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.208+0000 7f13bffff700 1 -- 172.21.15.189:0/3730678992 >> v1:172.21.15.189:6789/0 conn(0x7f13c00a2030 legacy=0x7f13c013df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3730678992 shutdown_connections 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3730678992 >> 172.21.15.189:0/3730678992 conn(0x7f13c00a4550 msgr2=0x7f13c00a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3730678992 shutdown_connections 2024-09-16T06:44:40.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3730678992 wait complete. 2024-09-16T06:44:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: build_initial_monmap 2024-09-16T06:44:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: monmap: 2024-09-16T06:44:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:40.224 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 Processor -- start 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- start start 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: init 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:40.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-c con 0x7f13c00a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-b con 0x7f13c01352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient(hunting): picked mon.noname-a con 0x7f13c0135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c0131e70 con 0x7f13c0135db0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c0130960 con 0x7f13c01352c0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c00a6130 con 0x7f13c00a2030 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient(hunting): authenticate will time out at 1232.729669s 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13be7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f13c01352c0 0x7f13c01ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52978/0 (socket says 172.21.15.189:52978) 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13be7fc700 1 -- 172.21.15.189:0/3627396079 learned_addr learned my addr 172.21.15.189:0/3627396079 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128821292 0 0) 0x7f13c0130960 con 0x7f13c01352c0 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:40.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): my global_id is 174112 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac0035d0 con 0x7f13c01352c0 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166166480 0 0) 0x7f13c00a6130 con 0x7f13c00a2030 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): my global_id is 125438 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0130960 con 0x7f13c00a2030 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2006401464 0 0) 0x7f13ac0035d0 con 0x7f13c01352c0 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13c00a6130 con 0x7f13c01352c0 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13b00018e0 con 0x7f13c01352c0 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): dump: 2024-09-16T06:44:40.227 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2808591468 0 0) 0x7f13c00a6130 con 0x7f13c01352c0 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 >> v1:172.21.15.76:6790/0 conn(0x7f13c00a2030 legacy=0x7f13c0132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 >> v1:172.21.15.76:6789/0 conn(0x7f13c0135db0 legacy=0x7f13c01d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:40.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 monclient: found mon.b 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c009e820 con 0x7f13c01352c0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 5 monclient: authenticate success, global_id 174112 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _renew_subs 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c012ee30 con 0x7f13c01352c0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _renew_subs 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c0130f20 con 0x7f13c01352c0 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b0001c90 con 0x7f13c01352c0 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13b0003af0 con 0x7f13c01352c0 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13bd7fa700 4 set_mon_vals no callback set 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.212+0000 7f13a7fff700 10 monclient: dump: 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:40.230 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:40.232 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:40.232 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13bffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13bffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13bffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a0005100 con 0x7f13c01352c0 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13b0003cd0 con 0x7f13c01352c0 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(156..156 src has 66..156) v4 ==== 9042+0+0 (unknown 113264351 0 0) 0x7f13b0044870 con 0x7f13c01352c0 2024-09-16T06:44:40.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.216+0000 7f13beffd700 10 monclient: get_auth_request con 0x7f13ac036190 auth_method 0 2024-09-16T06:44:40.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.220+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f13b0014650 con 0x7f13c01352c0 2024-09-16T06:44:40.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.220+0000 7f13a7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:40.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.220+0000 7f13a7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:40.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.396+0000 7f13bffff700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:40.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.396+0000 7f13bffff700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:40.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.396+0000 7f13bffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:40.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:40.396+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f13a00053b0 con 0x7f13c01352c0 2024-09-16T06:44:41.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.212+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v157) v1 ==== 118+0+0 (unknown 1949066829 0 0) 0x7f13b0012cf0 con 0x7f13c01352c0 2024-09-16T06:44:41.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.212+0000 7f13a7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:41.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.212+0000 7f13a7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:41.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.328+0000 7f13bffff700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:41.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.328+0000 7f13bffff700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:41.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.328+0000 7f13bffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:41.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:41.328+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f13a0005680 con 0x7f13c01352c0 2024-09-16T06:44:42.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.216+0000 7f13a7fff700 1 -- 172.21.15.189:0/3627396079 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v158) v1 ==== 118+0+0 (unknown 3004103463 0 0) 0x7f13b0014470 con 0x7f13c01352c0 2024-09-16T06:44:42.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.216+0000 7f13a7fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T06:44:42.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.216+0000 7f13a7fff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13ac036190 legacy=0x7f13ac038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 10 monclient: shutdown 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 >> v1:172.21.15.189:6789/0 conn(0x7f13c01352c0 legacy=0x7f13c01ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 shutdown_connections 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 >> 172.21.15.189:0/3627396079 conn(0x7f13c00a4550 msgr2=0x7f13c00a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:42.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 shutdown_connections 2024-09-16T06:44:42.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.220+0000 7f13bffff700 1 -- 172.21.15.189:0/3627396079 wait complete. 2024-09-16T06:44:42.242 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-16T06:44:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 10 monclient: get_monmap_and_config 2024-09-16T06:44:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 10 monclient: build_initial_monmap 2024-09-16T06:44:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 10 monclient: monmap: 2024-09-16T06:44:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:42.359045+0000 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:42.359045+0000 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:42.359 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 1 Processor -- start 2024-09-16T06:44:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 1 -- start start 2024-09-16T06:44:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.352+0000 7f68ae936700 10 monclient: init 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-a con 0x7f68a809f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-c con 0x7f68a809c210 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-b con 0x7f68a8138690 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a809f520 con 0x7f68a809f140 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8099ee0 con 0x7f68a809c210 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8097620 con 0x7f68a8138690 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient(hunting): authenticate will time out at 1234.872756s 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68a809c210 0x7f68a809c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52988/0 (socket says 172.21.15.189:52988) 2024-09-16T06:44:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a77fe700 1 -- 172.21.15.189:0/742295200 learned_addr learned my addr 172.21.15.189:0/742295200 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315186177 0 0) 0x7f68a8099ee0 con 0x7f68a809c210 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): my global_id is 174115 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68900035d0 con 0x7f68a809c210 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380977087 0 0) 0x7f68a8097620 con 0x7f68a8138690 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:42.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient(hunting): my global_id is 125441 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a8099ee0 con 0x7f68a8138690 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715857497 0 0) 0x7f68900035d0 con 0x7f68a809c210 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 >> v1:172.21.15.76:6790/0 conn(0x7f68a8138690 legacy=0x7f68a813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 >> v1:172.21.15.76:6789/0 conn(0x7f68a809f140 legacy=0x7f68a809bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 monclient: found mon.noname-c 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:42.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8097900 con 0x7f68a809c210 2024-09-16T06:44:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:42.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 5 monclient: authenticate success, global_id 174115 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c001610 con 0x7f68a809c210 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: dump: 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:42.368 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c001830 con 0x7f68a809c210 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 1 -- 172.21.15.189:0/742295200 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c001bc0 con 0x7f68a809c210 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 4 set_mon_vals no callback set 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68a6ffd700 10 monclient: dump: 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 10 monclient: shutdown 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- 172.21.15.189:0/742295200 >> v1:172.21.15.189:6789/0 conn(0x7f68a809c210 legacy=0x7f68a809c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- 172.21.15.189:0/742295200 shutdown_connections 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- 172.21.15.189:0/742295200 >> 172.21.15.189:0/742295200 conn(0x7f68a80a03e0 msgr2=0x7f68a80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- 172.21.15.189:0/742295200 shutdown_connections 2024-09-16T06:44:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.360+0000 7f68ae936700 1 -- 172.21.15.189:0/742295200 wait complete. 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: build_initial_monmap 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: monmap: 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 Processor -- start 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- start start 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: init 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-a con 0x7f68a809c210 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-b con 0x7f68a809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient(hunting): picked mon.noname-c con 0x7f68a8138690 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8090f90 con 0x7f68a809c210 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80a1fa0 con 0x7f68a809f140 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8071990 con 0x7f68a8138690 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient(hunting): authenticate will time out at 1234.878131s 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68a809f140 0x7f68a81cc850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52996/0 (socket says 172.21.15.189:52996) 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a77fe700 1 -- 172.21.15.189:0/9065200 learned_addr learned my addr 172.21.15.189:0/9065200 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222083583 0 0) 0x7f68a80a1fa0 con 0x7f68a809f140 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): my global_id is 174118 2024-09-16T06:44:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940035d0 con 0x7f68a809f140 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223257875 0 0) 0x7f68a8071990 con 0x7f68a8138690 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): my global_id is 125444 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a80a1fa0 con 0x7f68a8138690 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091123340 0 0) 0x7f68940035d0 con 0x7f68a809f140 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a8071990 con 0x7f68a809f140 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c0015d0 con 0x7f68a809f140 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:42.378 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710812836 0 0) 0x7f68a80a1fa0 con 0x7f68a8138690 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68940035d0 con 0x7f68a8138690 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68a0001c50 con 0x7f68a8138690 2024-09-16T06:44:42.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 829720980 0 0) 0x7f68a8071990 con 0x7f68a809f140 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 >> v1:172.21.15.76:6790/0 conn(0x7f68a8138690 legacy=0x7f68a81cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 >> v1:172.21.15.76:6789/0 conn(0x7f68a809c210 legacy=0x7f68a8096710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 monclient: found mon.b 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a809a6e0 con 0x7f68a809f140 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 5 monclient: authenticate success, global_id 174118 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _renew_subs 2024-09-16T06:44:42.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a8092b00 con 0x7f68a809f140 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _renew_subs 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a809b650 con 0x7f68a809f140 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c004bc0 con 0x7f68a809f140 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c004f50 con 0x7f68a809f140 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a67fc700 4 set_mon_vals no callback set 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 10 monclient: dump: 2024-09-16T06:44:42.382 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6888005100 con 0x7f68a809f140 2024-09-16T06:44:42.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f689c005130 con 0x7f68a809f140 2024-09-16T06:44:42.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(158..158 src has 66..158) v4 ==== 9042+0+0 (unknown 1060767036 0 0) 0x7f689c044a00 con 0x7f68a809f140 2024-09-16T06:44:42.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.364+0000 7f68a7fff700 10 monclient: get_auth_request con 0x7f68940362f0 auth_method 0 2024-09-16T06:44:42.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.372+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f689c014ac0 con 0x7f68a809f140 2024-09-16T06:44:42.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.372+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:42.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.372+0000 7f68a4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:42.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.552+0000 7f68ae936700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:42.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.552+0000 7f68ae936700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:42.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.552+0000 7f68ae936700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:42.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:42.552+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f68880053b0 con 0x7f68a809f140 2024-09-16T06:44:43.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.228+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v159) v1 ==== 127+0+0 (unknown 543606421 0 0) 0x7f689c0146b0 con 0x7f68a809f140 2024-09-16T06:44:43.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.228+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:43.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.228+0000 7f68a4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:43.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.352+0000 7f68ae936700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:43.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.352+0000 7f68ae936700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:43.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.352+0000 7f68ae936700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:43.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:43.352+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f6888005af0 con 0x7f68a809f140 2024-09-16T06:44:44.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.240+0000 7f68a4ff9700 1 -- 172.21.15.189:0/9065200 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v160) v1 ==== 127+0+0 (unknown 1757210535 0 0) 0x7f689c014890 con 0x7f68a809f140 2024-09-16T06:44:44.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.244+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T06:44:44.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.244+0000 7f68a4ff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 >> v1:172.21.15.189:6812/3098810674 conn(0x7f68940362f0 legacy=0x7f68940387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 10 monclient: shutdown 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 >> v1:172.21.15.189:6789/0 conn(0x7f68a809f140 legacy=0x7f68a81cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 shutdown_connections 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 >> 172.21.15.189:0/9065200 conn(0x7f68a80a03e0 msgr2=0x7f68a809ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 shutdown_connections 2024-09-16T06:44:44.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.248+0000 7f68ae936700 1 -- 172.21.15.189:0/9065200 wait complete. 2024-09-16T06:44:44.261 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-16T06:44:44.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: get_monmap_and_config 2024-09-16T06:44:44.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: build_initial_monmap 2024-09-16T06:44:44.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: monmap: 2024-09-16T06:44:44.364 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:44.365091+0000 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:44.365091+0000 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:44.365 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 1 Processor -- start 2024-09-16T06:44:44.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 1 -- start start 2024-09-16T06:44:44.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: init 2024-09-16T06:44:44.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-b con 0x7fd6a409f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-c con 0x7fd6a4099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-a con 0x7fd6a41396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a409f520 con 0x7fd6a41396a0 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a409bc10 con 0x7fd6a4099700 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a409b810 con 0x7fd6a409f140 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 10 monclient(hunting): authenticate will time out at 1236.878768s 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd69a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd6a4099700 0x7fd6a409b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53016/0 (socket says 172.21.15.189:53016) 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd69a59c700 1 -- 172.21.15.189:0/1277511575 learned_addr learned my addr 172.21.15.189:0/1277511575 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896114887 0 0) 0x7fd6a409bc10 con 0x7fd6a4099700 2024-09-16T06:44:44.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _negotiate 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): my global_id is 174121 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6900035d0 con 0x7fd6a4099700 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598614459 0 0) 0x7fd6a409b810 con 0x7fd6a409f140 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _negotiate 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient(hunting): my global_id is 125447 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a409bc10 con 0x7fd6a409f140 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799376342 0 0) 0x7fd6900035d0 con 0x7fd6a4099700 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 >> v1:172.21.15.76:6790/0 conn(0x7fd6a409f140 legacy=0x7fd6a409a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 >> v1:172.21.15.76:6789/0 conn(0x7fd6a41396a0 legacy=0x7fd6a413ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: _finish_hunting 0 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 monclient: found mon.noname-c 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a409ceb0 con 0x7fd6a4099700 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: _finish_auth 0 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: _check_auth_tickets 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 5 monclient: authenticate success, global_id 174121 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd68c001610 con 0x7fd6a4099700 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a259c700 10 monclient: dump: 2024-09-16T06:44:44.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.364+0000 7fd6a90d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd68c0018f0 con 0x7fd6a4099700 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:44.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 1 -- 172.21.15.189:0/1277511575 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd68c001c80 con 0x7fd6a4099700 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 4 set_mon_vals no callback set 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a259c700 10 monclient: dump: 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:44.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: shutdown 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1277511575 >> v1:172.21.15.189:6789/0 conn(0x7fd6a4099700 legacy=0x7fd6a409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1277511575 shutdown_connections 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1277511575 >> 172.21.15.189:0/1277511575 conn(0x7fd6a40a03e0 msgr2=0x7fd6a40a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1277511575 shutdown_connections 2024-09-16T06:44:44.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1277511575 wait complete. 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: build_initial_monmap 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: monmap: 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:44.378 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 Processor -- start 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- start start 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: init 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-a con 0x7fd6a4099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-b con 0x7fd6a409f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient(hunting): picked mon.noname-c con 0x7fd6a41396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a4094990 con 0x7fd6a4099700 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a4093270 con 0x7fd6a409f140 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a4071610 con 0x7fd6a41396a0 2024-09-16T06:44:44.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient(hunting): authenticate will time out at 1236.882790s 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd6a409f140 0x7fd6a40910f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53018/0 (socket says 172.21.15.189:53018) 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69a59c700 1 -- 172.21.15.189:0/1507807167 learned_addr learned my addr 172.21.15.189:0/1507807167 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061491448 0 0) 0x7fd6a4093270 con 0x7fd6a409f140 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): my global_id is 174124 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6840035d0 con 0x7fd6a409f140 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783842724 0 0) 0x7fd6a4071610 con 0x7fd6a41396a0 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:44.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): my global_id is 125450 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a4093270 con 0x7fd6a41396a0 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963646213 0 0) 0x7fd6840035d0 con 0x7fd6a409f140 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6a4071610 con 0x7fd6a409f140 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd68c0015d0 con 0x7fd6a409f140 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): dump: 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.381 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1527811352 0 0) 0x7fd6a4071610 con 0x7fd6a409f140 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 >> v1:172.21.15.76:6790/0 conn(0x7fd6a41396a0 legacy=0x7fd6a4091800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 >> v1:172.21.15.76:6789/0 conn(0x7fd6a4099700 legacy=0x7fd6a40909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 monclient: found mon.b 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40946b0 con 0x7fd6a409f140 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd69bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 5 monclient: authenticate success, global_id 174124 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.368+0000 7fd6a90d6700 10 monclient: _renew_subs 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a4092630 con 0x7fd6a409f140 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 monclient: _renew_subs 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a40943d0 con 0x7fd6a409f140 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd68c004ca0 con 0x7fd6a409f140 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:44.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd68c005030 con 0x7fd6a409f140 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 10 monclient: dump: 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:44.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a1d9b700 4 set_mon_vals no callback set 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd68c005210 con 0x7fd6a409f140 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd688005100 con 0x7fd6a409f140 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(160..160 src has 66..160) v4 ==== 9005+0+0 (unknown 2237298182 0 0) 0x7fd68c044a50 con 0x7fd6a409f140 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.372+0000 7fd6a2d9d700 10 monclient: get_auth_request con 0x7fd6840361d0 auth_method 0 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.380+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd68c014b70 con 0x7fd6a409f140 2024-09-16T06:44:44.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.380+0000 7fd69bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:44.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.380+0000 7fd69bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:44.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.556+0000 7fd6a90d6700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:44.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.556+0000 7fd6a90d6700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:44.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.556+0000 7fd6a90d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:44.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:44.556+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7fd6880053b0 con 0x7fd6a409f140 2024-09-16T06:44:45.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.256+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v161) v1 ==== 132+0+0 (unknown 185118270 0 0) 0x7fd68c012fe0 con 0x7fd6a409f140 2024-09-16T06:44:45.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.256+0000 7fd69bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:45.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.256+0000 7fd69bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:45.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.376+0000 7fd6a90d6700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:45.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.376+0000 7fd6a90d6700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:45.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.376+0000 7fd6a90d6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.376+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7fd688005af0 con 0x7fd6a409f140 2024-09-16T06:44:45.780 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.780+0000 7fd69bfff700 1 -- 172.21.15.189:0/1507807167 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v162) v1 ==== 132+0+0 (unknown 2760931569 0 0) 0x7fd68c014760 con 0x7fd6a409f140 2024-09-16T06:44:45.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.780+0000 7fd69bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T06:44:45.781 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.780+0000 7fd69bfff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:45.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd6840361d0 legacy=0x7fd684038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 10 monclient: shutdown 2024-09-16T06:44:45.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:45.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 >> v1:172.21.15.189:6789/0 conn(0x7fd6a409f140 legacy=0x7fd6a40910f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 shutdown_connections 2024-09-16T06:44:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 >> 172.21.15.189:0/1507807167 conn(0x7fd6a40a03e0 msgr2=0x7fd6a406fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 shutdown_connections 2024-09-16T06:44:45.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.784+0000 7fd6a90d6700 1 -- 172.21.15.189:0/1507807167 wait complete. 2024-09-16T06:44:45.800 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:45.801 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:45.801 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:45.801 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '7\.' 2024-09-16T06:44:45.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: get_monmap_and_config 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: build_initial_monmap 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: monmap: 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:45.909513+0000 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:45.909513+0000 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:45.909 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:45.910 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 1 Processor -- start 2024-09-16T06:44:45.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 1 -- start start 2024-09-16T06:44:45.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: init 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-a con 0x7fc784073b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-b con 0x7fc784073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-c con 0x7fc78413b860 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc784066710 con 0x7fc784073b60 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78990e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc78413b860 0x7fc78413dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53034/0 (socket says 172.21.15.189:53034) 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78990e700 1 -- 172.21.15.189:0/2737566869 learned_addr learned my addr 172.21.15.189:0/2737566869 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.908+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7840723d0 con 0x7fc78413b860 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc784071e00 con 0x7fc784073f40 2024-09-16T06:44:45.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient(hunting): authenticate will time out at 1238.424145s 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103554299 0 0) 0x7fc7840723d0 con 0x7fc78413b860 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): my global_id is 174127 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7700035d0 con 0x7fc78413b860 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438346543 0 0) 0x7fc784071e00 con 0x7fc784073f40 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:45.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient(hunting): my global_id is 125453 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7840723d0 con 0x7fc784073f40 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032216113 0 0) 0x7fc7700035d0 con 0x7fc78413b860 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 >> v1:172.21.15.76:6790/0 conn(0x7fc784073f40 legacy=0x7fc784074320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 >> v1:172.21.15.76:6789/0 conn(0x7fc784073b60 legacy=0x7fc78406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 monclient: found mon.noname-c 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78409bac0 con 0x7fc78413b860 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:45.916 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 5 monclient: authenticate success, global_id 174127 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc77c001990 con 0x7fc78413b860 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: dump: 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:45.917 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc77c001c70 con 0x7fc78413b860 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 1 -- 172.21.15.189:0/2737566869 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc77c002000 con 0x7fc78413b860 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 4 set_mon_vals no callback set 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc77bfff700 10 monclient: dump: 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.918 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient: shutdown 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 >> v1:172.21.15.189:6789/0 conn(0x7fc78413b860 legacy=0x7fc78413dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 shutdown_connections 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 >> 172.21.15.189:0/2737566869 conn(0x7fc7840a1100 msgr2=0x7fc7840a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 shutdown_connections 2024-09-16T06:44:45.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- 172.21.15.189:0/2737566869 wait complete. 2024-09-16T06:44:45.926 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient: build_initial_monmap 2024-09-16T06:44:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient: monmap: 2024-09-16T06:44:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:45.927 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:45.928 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 Processor -- start 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 1 -- start start 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.912+0000 7fc78b371700 10 monclient: init 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:45.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-b con 0x7fc784073b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-a con 0x7fc784073f40 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient(hunting): picked mon.noname-c con 0x7fc78413b860 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78412ac00 con 0x7fc784073f40 2024-09-16T06:44:45.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78409aed0 con 0x7fc784073b60 2024-09-16T06:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78409ab10 con 0x7fc78413b860 2024-09-16T06:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient(hunting): authenticate will time out at 1238.428410s 2024-09-16T06:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78910d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc784073b60 0x7fc7841ca4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53040/0 (socket says 172.21.15.189:53040) 2024-09-16T06:44:45.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78910d700 1 -- 172.21.15.189:0/2129247574 learned_addr learned my addr 172.21.15.189:0/2129247574 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442159268 0 0) 0x7fc78409aed0 con 0x7fc784073b60 2024-09-16T06:44:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:45.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): my global_id is 174130 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7680035d0 con 0x7fc784073b60 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1783333633 0 0) 0x7fc7680035d0 con 0x7fc784073b60 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc78409aed0 con 0x7fc784073b60 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7800018e0 con 0x7fc784073b60 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:45.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.934 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:45.935 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:45.936 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:45.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405414077 0 0) 0x7fc78409ab10 con 0x7fc78413b860 2024-09-16T06:44:45.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:45.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient(hunting): my global_id is 125456 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7680035d0 con 0x7fc78413b860 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2932226492 0 0) 0x7fc78409aed0 con 0x7fc784073b60 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 >> v1:172.21.15.76:6790/0 conn(0x7fc78413b860 legacy=0x7fc7841d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 >> v1:172.21.15.76:6789/0 conn(0x7fc784073f40 legacy=0x7fc7841cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:45.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:45.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 monclient: found mon.b 2024-09-16T06:44:45.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:45.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78409b610 con 0x7fc784073b60 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 5 monclient: authenticate success, global_id 174130 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _renew_subs 2024-09-16T06:44:45.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78412aee0 con 0x7fc784073b60 2024-09-16T06:44:45.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _renew_subs 2024-09-16T06:44:45.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78b371700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc784139f00 con 0x7fc784073b60 2024-09-16T06:44:45.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc780001d10 con 0x7fc784073b60 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc780003b70 con 0x7fc784073b60 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 10 monclient: dump: 2024-09-16T06:44:45.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:45.942 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:45.943 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:45.943 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:45.943 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:45.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc77b7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:45.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc780003d50 con 0x7fc784073b60 2024-09-16T06:44:45.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(162..162 src has 66..162) v4 ==== 9005+0+0 (unknown 1663024974 0 0) 0x7fc7800448c0 con 0x7fc784073b60 2024-09-16T06:44:45.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc78890c700 10 monclient: get_auth_request con 0x7fc768036200 auth_method 0 2024-09-16T06:44:45.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc76f7fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:45.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc76f7fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:45.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc76f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.916+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c005100 con 0x7fc784073b60 2024-09-16T06:44:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.924+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc780014940 con 0x7fc784073b60 2024-09-16T06:44:45.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.924+0000 7fc779ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:45.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:45.924+0000 7fc779ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc76f7fe700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc76f7fe700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc76f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc75c0053b0 con 0x7fc784073b60 2024-09-16T06:44:46.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) v1 ==== 83+0+2320 (unknown 1241242665 0 1435408216) 0x7fc780012db0 con 0x7fc784073b60 2024-09-16T06:44:46.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc779ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.104+0000 7fc779ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:46.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc76f7fe700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc76f7fe700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc76f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc75c005bc0 con 0x7fc784073b60 2024-09-16T06:44:46.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc779ffb700 1 -- 172.21.15.189:0/2129247574 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) v1 ==== 83+0+2320 (unknown 1241242665 0 1435408216) 0x7fc780014530 con 0x7fc784073b60 2024-09-16T06:44:46.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc779ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.216+0000 7fc779ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:46.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc768036200 legacy=0x7fc7680386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 10 monclient: shutdown 2024-09-16T06:44:46.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 >> v1:172.21.15.189:6789/0 conn(0x7fc784073b60 legacy=0x7fc7841ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 shutdown_connections 2024-09-16T06:44:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 >> 172.21.15.189:0/2129247574 conn(0x7fc7840a1100 msgr2=0x7fc78413aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 shutdown_connections 2024-09-16T06:44:46.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.220+0000 7fc76f7fe700 1 -- 172.21.15.189:0/2129247574 wait complete. 2024-09-16T06:44:46.231 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:46.231 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:46.231 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:46.232 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '8\.' 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: get_monmap_and_config 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: monmap: 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:46.340803+0000 2024-09-16T06:44:46.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:46.340803+0000 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:46.341 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 Processor -- start 2024-09-16T06:44:46.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- start start 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: init 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient(hunting): picked mon.noname-c con 0x7f6784133230 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient(hunting): picked mon.noname-b con 0x7f6784132d00 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient(hunting): picked mon.noname-a con 0x7f6784139740 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678409adc0 con 0x7f6784139740 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678409b450 con 0x7f6784133230 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678409c760 con 0x7f6784132d00 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient(hunting): authenticate will time out at 1238.853786s 2024-09-16T06:44:46.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6784133230 0x7f6784133610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53046/0 (socket says 172.21.15.189:53046) 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678259c700 1 -- 172.21.15.189:0/351210538 learned_addr learned my addr 172.21.15.189:0/351210538 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870726052 0 0) 0x7f678409b450 con 0x7f6784133230 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): my global_id is 174133 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f676c0035d0 con 0x7f6784133230 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899887820 0 0) 0x7f678409c760 con 0x7f6784132d00 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient(hunting): my global_id is 125459 2024-09-16T06:44:46.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678409b450 con 0x7f6784132d00 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664970453 0 0) 0x7f676c0035d0 con 0x7f6784133230 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 >> v1:172.21.15.76:6790/0 conn(0x7f6784132d00 legacy=0x7f6784138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 >> v1:172.21.15.76:6789/0 conn(0x7f6784139740 legacy=0x7f678413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 monclient: found mon.noname-c 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678409b1d0 con 0x7f6784133230 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: _finish_auth 0 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 5 monclient: authenticate success, global_id 174133 2024-09-16T06:44:46.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f677c001990 con 0x7f6784133230 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: dump: 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f677c001c70 con 0x7f6784133230 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 1 -- 172.21.15.189:0/351210538 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f677c002000 con 0x7f6784133230 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 4 set_mon_vals no callback set 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678159a700 10 monclient: dump: 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.352 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 10 monclient: shutdown 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- 172.21.15.189:0/351210538 >> v1:172.21.15.189:6789/0 conn(0x7f6784133230 legacy=0x7f6784133610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- 172.21.15.189:0/351210538 shutdown_connections 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- 172.21.15.189:0/351210538 >> 172.21.15.189:0/351210538 conn(0x7f67840a3950 msgr2=0x7f678409fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- 172.21.15.189:0/351210538 shutdown_connections 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.340+0000 7f678359e700 1 -- 172.21.15.189:0/351210538 wait complete. 2024-09-16T06:44:46.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: monmap: 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 Processor -- start 2024-09-16T06:44:46.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- start start 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: init 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient(hunting): picked mon.noname-a con 0x7f6784132d00 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient(hunting): picked mon.noname-c con 0x7f6784133230 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient(hunting): picked mon.noname-b con 0x7f6784139740 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678406d4d0 con 0x7f6784132d00 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678409dea0 con 0x7f6784139740 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678409a490 con 0x7f6784133230 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient(hunting): authenticate will time out at 1238.857010s 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6782d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6784139740 0x7f67841d2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53052/0 (socket says 172.21.15.189:53052) 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6782d9d700 1 -- 172.21.15.189:0/2179195992 learned_addr learned my addr 172.21.15.189:0/2179195992 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592907611 0 0) 0x7f678409dea0 con 0x7f6784139740 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): my global_id is 174136 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67640035d0 con 0x7f6784139740 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623489888 0 0) 0x7f678409a490 con 0x7f6784133230 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): my global_id is 125462 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678409dea0 con 0x7f6784133230 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406164502 0 0) 0x7f67640035d0 con 0x7f6784139740 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f678409a490 con 0x7f6784139740 2024-09-16T06:44:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6774001830 con 0x7f6784139740 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1175010780 0 0) 0x7f678409dea0 con 0x7f6784133230 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67640035d0 con 0x7f6784133230 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f67780018d0 con 0x7f6784133230 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902618124 0 0) 0x7f678409a490 con 0x7f6784139740 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 >> v1:172.21.15.76:6790/0 conn(0x7f6784133230 legacy=0x7f67841cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 >> v1:172.21.15.76:6789/0 conn(0x7f6784132d00 legacy=0x7f678412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 monclient: found mon.b 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784138980 con 0x7f6784139740 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:46.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 5 monclient: authenticate success, global_id 174136 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _renew_subs 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6784138c60 con 0x7f6784139740 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _renew_subs 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6784137410 con 0x7f6784139740 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67740036a0 con 0x7f6784139740 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6774003a30 con 0x7f6784139740 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6772ffd700 10 monclient: dump: 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f6780d99700 4 set_mon_vals no callback set 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.344+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6768005100 con 0x7f6784139740 2024-09-16T06:44:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.348+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6774003c10 con 0x7f6784139740 2024-09-16T06:44:46.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.348+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(162..162 src has 66..162) v4 ==== 9005+0+0 (unknown 1663024974 0 0) 0x7f67740447c0 con 0x7f6784139740 2024-09-16T06:44:46.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.348+0000 7f678259c700 10 monclient: get_auth_request con 0x7f67640362a0 auth_method 0 2024-09-16T06:44:46.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.352+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f67740147e0 con 0x7f6784139740 2024-09-16T06:44:46.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.352+0000 7f6772ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.352+0000 7f6772ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:46.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.528+0000 7f678359e700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.528+0000 7f678359e700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.528+0000 7f678359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.528+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f67680053b0 con 0x7f6784139740 2024-09-16T06:44:46.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.532+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) v1 ==== 83+0+2320 (unknown 1241242665 0 1435408216) 0x7f677401ce30 con 0x7f6784139740 2024-09-16T06:44:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.532+0000 7f6772ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.532+0000 7f6772ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.644+0000 7f678359e700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.644+0000 7f678359e700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.644+0000 7f678359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.644+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f6768005650 con 0x7f6784139740 2024-09-16T06:44:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f6772ffd700 1 -- 172.21.15.189:0/2179195992 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) v1 ==== 83+0+2320 (unknown 1241242665 0 1435408216) 0x7f6774012c50 con 0x7f6784139740 2024-09-16T06:44:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f6772ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f6772ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:46.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 >> v1:172.21.15.189:6812/3098810674 conn(0x7f67640362a0 legacy=0x7f6764038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 10 monclient: shutdown 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 >> v1:172.21.15.189:6789/0 conn(0x7f6784139740 legacy=0x7f67841d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 shutdown_connections 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 >> 172.21.15.189:0/2179195992 conn(0x7f67840a3950 msgr2=0x7f67840a10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 shutdown_connections 2024-09-16T06:44:46.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.648+0000 7f678359e700 1 -- 172.21.15.189:0/2179195992 wait complete. 2024-09-16T06:44:46.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:46.660 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T06:44:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep 'osd\.0' 2024-09-16T06:44:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep globster 2024-09-16T06:44:46.661 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep '6\.' 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: get_monmap_and_config 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: build_initial_monmap 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: monmap: 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:46.765799+0000 2024-09-16T06:44:46.765 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:46.765799+0000 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:46.766 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 1 Processor -- start 2024-09-16T06:44:46.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 1 -- start start 2024-09-16T06:44:46.769 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: init 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc09ce10 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc09d1f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc071ee0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc097fb0 con 0x7ff3bc09ce10 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc09d670 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc099690 con 0x7ff3bc071ee0 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient(hunting): authenticate will time out at 1239.278438s 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c0d53700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3bc09d1f0 0x7ff3bc0978a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53062/0 (socket says 172.21.15.189:53062) 2024-09-16T06:44:46.770 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c0d53700 1 -- 172.21.15.189:0/452580220 learned_addr learned my addr 172.21.15.189:0/452580220 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835942141 0 0) 0x7ff3bc09d670 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): my global_id is 174139 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a40035d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068149444 0 0) 0x7ff3bc099690 con 0x7ff3bc071ee0 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient(hunting): my global_id is 125465 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc09d670 con 0x7ff3bc071ee0 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816799534 0 0) 0x7ff3a40035d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 >> v1:172.21.15.76:6790/0 conn(0x7ff3bc071ee0 legacy=0x7ff3bc1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 >> v1:172.21.15.76:6789/0 conn(0x7ff3bc09ce10 legacy=0x7ff3bc097190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 monclient: found mon.noname-c 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc098660 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 5 monclient: authenticate success, global_id 174139 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3b4001990 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: dump: 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:46.773 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b4001c00 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3c2556700 4 set_mon_vals no callback set 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 1 -- 172.21.15.189:0/452580220 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3b4001f90 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:46.774 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.764+0000 7ff3bbfff700 10 monclient: dump: 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.775 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.776 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.776 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: shutdown 2024-09-16T06:44:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- 172.21.15.189:0/452580220 >> v1:172.21.15.189:6789/0 conn(0x7ff3bc09d1f0 legacy=0x7ff3bc0978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- 172.21.15.189:0/452580220 shutdown_connections 2024-09-16T06:44:46.777 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- 172.21.15.189:0/452580220 >> 172.21.15.189:0/452580220 conn(0x7ff3bc09fe70 msgr2=0x7ff3bc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- 172.21.15.189:0/452580220 shutdown_connections 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- 172.21.15.189:0/452580220 wait complete. 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: build_initial_monmap 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: monmap: 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:46.778 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:46.779 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 Processor -- start 2024-09-16T06:44:46.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- start start 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: init 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc071ee0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc09ce10 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc09d1f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc09a380 con 0x7ff3bc09ce10 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc09c0d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc1aedc0 con 0x7ff3bc071ee0 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c2556700 10 monclient(hunting): authenticate will time out at 1239.283798s 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c1d55700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3bc09d1f0 0x7ff3bc1d66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53072/0 (socket says 172.21.15.189:53072) 2024-09-16T06:44:46.785 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.768+0000 7ff3c1d55700 1 -- 172.21.15.189:0/2835082901 learned_addr learned my addr 172.21.15.189:0/2835082901 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135767947 0 0) 0x7ff3bc09d670 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): my global_id is 174142 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a80035d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268749820 0 0) 0x7ff3bc1aedc0 con 0x7ff3bc071ee0 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): my global_id is 125468 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc09d670 con 0x7ff3bc071ee0 2024-09-16T06:44:46.786 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 771996274 0 0) 0x7ff3a80035d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3bc1aedc0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3b0001770 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.787 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780427224 0 0) 0x7ff3bc09d670 con 0x7ff3bc071ee0 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a80035d0 con 0x7ff3bc071ee0 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3ac0017f0 con 0x7ff3bc071ee0 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.788 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 704120742 0 0) 0x7ff3bc1aedc0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 >> v1:172.21.15.76:6790/0 conn(0x7ff3bc071ee0 legacy=0x7ff3bc1c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 >> v1:172.21.15.76:6789/0 conn(0x7ff3bc09ce10 legacy=0x7ff3bc1d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 monclient: found mon.b 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc1ca430 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.789 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 5 monclient: authenticate success, global_id 174142 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _renew_subs 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc1ad2f0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _renew_subs 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc1cb160 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b00036c0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3b00039d0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:46.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3bb7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 10 monclient: dump: 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff3b0003bb0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.791 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c005100 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(162..162 src has 66..162) v4 ==== 9005+0+0 (unknown 1663024974 0 0) 0x7ff3b0044670 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.772+0000 7ff3c1554700 10 monclient: get_auth_request con 0x7ff3a80362f0 auth_method 0 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.776+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff3b00146f0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.776+0000 7ff3b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:46.792 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.776+0000 7ff3b9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:46.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3c2556700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3c2556700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3c2556700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:46.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7ff39c0053b0 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2320 (unknown 1832164469 0 1435408216) 0x7ff3b0012b60 con 0x7ff3bc09d1f0 2024-09-16T06:44:46.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:46.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:46.956+0000 7ff3b9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:47.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3c2556700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:47.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3c2556700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:47.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3c2556700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7ff39c005f60 con 0x7ff3bc09d1f0 2024-09-16T06:44:47.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3b9ffb700 1 -- 172.21.15.189:0/2835082901 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2320 (unknown 1832164469 0 1435408216) 0x7ff3b0012b60 con 0x7ff3bc09d1f0 2024-09-16T06:44:47.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3b9ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T06:44:47.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.056+0000 7ff3b9ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff3a80362f0 legacy=0x7ff3a80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 10 monclient: shutdown 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 >> v1:172.21.15.189:6789/0 conn(0x7ff3bc09d1f0 legacy=0x7ff3bc1d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 shutdown_connections 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 >> 172.21.15.189:0/2835082901 conn(0x7ff3bc09fe70 msgr2=0x7ff3bc096140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 shutdown_connections 2024-09-16T06:44:47.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.060+0000 7ff3c2556700 1 -- 172.21.15.189:0/2835082901 wait complete. 2024-09-16T06:44:47.070 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:47.070 INFO:tasks.workunit.client.0.smithi189.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T06:44:47.071 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 10 monclient: get_monmap_and_config 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 10 monclient: build_initial_monmap 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 10 monclient: monmap: 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:47.174506+0000 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:47.174506+0000 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:47.174 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:47.175 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:47.175 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 1 Processor -- start 2024-09-16T06:44:47.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 1 -- start start 2024-09-16T06:44:47.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.172+0000 7fb861323700 10 monclient: init 2024-09-16T06:44:47.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:47.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:47.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient(hunting): picked mon.noname-a con 0x7fb85c0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:47.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient(hunting): picked mon.noname-c con 0x7fb85c09aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:47.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient(hunting): picked mon.noname-b con 0x7fb85c0a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c09a040 con 0x7fb85c0a2350 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c0a33d0 con 0x7fb85c09aa20 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c099d60 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient(hunting): authenticate will time out at 1239.688254s 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb85a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb85c09aa20 0x7fb85c13a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53082/0 (socket says 172.21.15.189:53082) 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb85a7fc700 1 -- 172.21.15.189:0/469117232 learned_addr learned my addr 172.21.15.189:0/469117232 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975355445 0 0) 0x7fb85c0a33d0 con 0x7fb85c09aa20 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:47.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): my global_id is 174145 2024-09-16T06:44:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8440035d0 con 0x7fb85c09aa20 2024-09-16T06:44:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128105631 0 0) 0x7fb85c099d60 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:47.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient(hunting): my global_id is 125471 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c0a33d0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789783887 0 0) 0x7fb8440035d0 con 0x7fb85c09aa20 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 >> v1:172.21.15.76:6790/0 conn(0x7fb85c0a3ca0 legacy=0x7fb85c13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 >> v1:172.21.15.76:6789/0 conn(0x7fb85c0a2350 legacy=0x7fb85c0a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 monclient: found mon.noname-c 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c0a3920 con 0x7fb85c09aa20 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:47.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 5 monclient: authenticate success, global_id 174145 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb850001610 con 0x7fb85c09aa20 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: dump: 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:47.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8500018f0 con 0x7fb85c09aa20 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 1 -- 172.21.15.189:0/469117232 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb850001c80 con 0x7fb85c09aa20 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 4 set_mon_vals no callback set 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb859ffb700 10 monclient: dump: 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:47.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:47.183 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: shutdown 2024-09-16T06:44:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- 172.21.15.189:0/469117232 >> v1:172.21.15.189:6789/0 conn(0x7fb85c09aa20 legacy=0x7fb85c13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- 172.21.15.189:0/469117232 shutdown_connections 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- 172.21.15.189:0/469117232 >> 172.21.15.189:0/469117232 conn(0x7fb85c0a7d70 msgr2=0x7fb85c0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- 172.21.15.189:0/469117232 shutdown_connections 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 1 -- 172.21.15.189:0/469117232 wait complete. 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: build_initial_monmap 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.176+0000 7fb861323700 10 monclient: monmap: 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:47.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 Processor -- start 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- start start 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: init 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient(hunting): picked mon.noname-c con 0x7fb85c09aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient(hunting): picked mon.noname-a con 0x7fb85c0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient(hunting): picked mon.noname-b con 0x7fb85c0a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c06bed0 con 0x7fb85c0a2350 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c09b660 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c120ab0 con 0x7fb85c09aa20 2024-09-16T06:44:47.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient(hunting): authenticate will time out at 1239.694177s 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb85b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb85c0a3ca0 0x7fb85c1d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53084/0 (socket says 172.21.15.189:53084) 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb85b7fe700 1 -- 172.21.15.189:0/1929562001 learned_addr learned my addr 172.21.15.189:0/1929562001 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842406036 0 0) 0x7fb85c09b660 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): my global_id is 174148 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8480035d0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979745826 0 0) 0x7fb85c120ab0 con 0x7fb85c09aa20 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:47.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): my global_id is 125474 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c09b660 con 0x7fb85c09aa20 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345342299 0 0) 0x7fb8480035d0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb85c120ab0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb854001c90 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): dump: 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:47.191 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163343621 0 0) 0x7fb85c120ab0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 >> v1:172.21.15.76:6790/0 conn(0x7fb85c09aa20 legacy=0x7fb85c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 >> v1:172.21.15.76:6789/0 conn(0x7fb85c0a2350 legacy=0x7fb85c1cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 monclient: found mon.b 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c136170 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:47.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 5 monclient: authenticate success, global_id 174148 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _renew_subs 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c122b30 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _renew_subs 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c121400 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb854002020 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb854003e90 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8597fa700 4 set_mon_vals no callback set 2024-09-16T06:44:47.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.180+0000 7fb8437fe700 10 monclient: dump: 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb861323700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb861323700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:47.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb861323700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c005100 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb854003ff0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(163..163 src has 66..163) v4 ==== 8980+0+0 (unknown 818012645 0 0) 0x7fb854044b00 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.184+0000 7fb85affd700 10 monclient: get_auth_request con 0x7fb848036270 auth_method 0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.192+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb854013030 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.192+0000 7fb8437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:47.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.192+0000 7fb8437fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.364+0000 7fb861323700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.364+0000 7fb861323700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.364+0000 7fb861323700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.364+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fb83c005bc0 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.788+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v164) v1 ==== 123+0+0 (unknown 2491997209 0 0) 0x7fb854013030 con 0x7fb85c0a3ca0 2024-09-16T06:44:47.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.788+0000 7fb8437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:47.790 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.788+0000 7fb8437fe700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T06:44:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.900+0000 7fb861323700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.900+0000 7fb861323700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.900+0000 7fb861323700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:47.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:47.900+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fb83c0053b0 con 0x7fb85c0a3ca0 2024-09-16T06:44:48.793 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.792+0000 7fb8437fe700 1 -- 172.21.15.189:0/1929562001 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v165) v1 ==== 149+0+0 (unknown 551761756 0 0) 0x7fb8540154f0 con 0x7fb85c0a3ca0 2024-09-16T06:44:48.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.792+0000 7fb8437fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T06:44:48.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.792+0000 7fb8437fe700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb848036270 legacy=0x7fb848038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 10 monclient: shutdown 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 >> v1:172.21.15.189:6789/0 conn(0x7fb85c0a3ca0 legacy=0x7fb85c1d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 shutdown_connections 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 >> 172.21.15.189:0/1929562001 conn(0x7fb85c0a7d70 msgr2=0x7fb85c1bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:48.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 shutdown_connections 2024-09-16T06:44:48.799 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.796+0000 7fb861323700 1 -- 172.21.15.189:0/1929562001 wait complete. 2024-09-16T06:44:48.799 INFO:tasks.workunit.client.0.smithi189.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T06:44:48.813 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: get_monmap_and_config 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: build_initial_monmap 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: monmap: 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:48.924810+0000 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:48.924810+0000 2024-09-16T06:44:48.924 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:48.925 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.925 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:48.925 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:48.925 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:48.925 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 Processor -- start 2024-09-16T06:44:48.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- start start 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: init 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-b con 0x7fc5d00a5c70 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-c con 0x7fc5d00a7100 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-a con 0x7fc5d013bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00a5a90 con 0x7fc5d013bab0 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00a6a90 con 0x7fc5d00a7100 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d0099e40 con 0x7fc5d00a5c70 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient(hunting): authenticate will time out at 1241.437812s 2024-09-16T06:44:48.929 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d5569700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5d00a7100 0x7fc5d013a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46860/0 (socket says 172.21.15.189:46860) 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d5569700 1 -- 172.21.15.189:0/1320256144 learned_addr learned my addr 172.21.15.189:0/1320256144 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120337549 0 0) 0x7fc5d00a6a90 con 0x7fc5d00a7100 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _negotiate 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): my global_id is 174151 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc0035d0 con 0x7fc5d00a7100 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507838511 0 0) 0x7fc5d0099e40 con 0x7fc5d00a5c70 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _negotiate 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient(hunting): my global_id is 125477 2024-09-16T06:44:48.930 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d00a6a90 con 0x7fc5d00a5c70 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898394291 0 0) 0x7fc5bc0035d0 con 0x7fc5d00a7100 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 >> v1:172.21.15.76:6790/0 conn(0x7fc5d00a5c70 legacy=0x7fc5d00a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 >> v1:172.21.15.76:6789/0 conn(0x7fc5d013bab0 legacy=0x7fc5d013dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: _finish_hunting 0 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 monclient: found mon.noname-c 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d009a120 con 0x7fc5d00a7100 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: _finish_auth 0 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: _check_auth_tickets 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 5 monclient: authenticate success, global_id 174151 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:48.931 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc001610 con 0x7fc5d00a7100 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: dump: 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:48.932 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5cc001830 con 0x7fc5d00a7100 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 4 set_mon_vals no callback set 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 1 -- 172.21.15.189:0/1320256144 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc001bc0 con 0x7fc5d00a7100 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d4d68700 10 monclient: dump: 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.933 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 10 monclient: shutdown 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- 172.21.15.189:0/1320256144 >> v1:172.21.15.189:6789/0 conn(0x7fc5d00a7100 legacy=0x7fc5d013a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- 172.21.15.189:0/1320256144 shutdown_connections 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- 172.21.15.189:0/1320256144 >> 172.21.15.189:0/1320256144 conn(0x7fc5d00a79e0 msgr2=0x7fc5d00a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- 172.21.15.189:0/1320256144 shutdown_connections 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.924+0000 7fc5d7fce700 1 -- 172.21.15.189:0/1320256144 wait complete. 2024-09-16T06:44:48.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: build_initial_monmap 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: monmap: 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 Processor -- start 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- start start 2024-09-16T06:44:48.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: init 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-a con 0x7fc5d00a5c70 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-b con 0x7fc5d00a7100 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient(hunting): picked mon.noname-c con 0x7fc5d013bab0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00a6f50 con 0x7fc5d00a5c70 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d011f1b0 con 0x7fc5d00a7100 2024-09-16T06:44:48.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d006bd50 con 0x7fc5d013bab0 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient(hunting): authenticate will time out at 1241.441817s 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d5569700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5d00a7100 0x7fc5d009ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46870/0 (socket says 172.21.15.189:46870) 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d5569700 1 -- 172.21.15.189:0/2768247255 learned_addr learned my addr 172.21.15.189:0/2768247255 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636581139 0 0) 0x7fc5d011f1b0 con 0x7fc5d00a7100 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): my global_id is 174154 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b40035d0 con 0x7fc5d00a7100 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447824200 0 0) 0x7fc5d006bd50 con 0x7fc5d013bab0 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:48.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): my global_id is 125480 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d011f1b0 con 0x7fc5d013bab0 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023413649 0 0) 0x7fc5b40035d0 con 0x7fc5d00a7100 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5d006bd50 con 0x7fc5d00a7100 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc0015d0 con 0x7fc5d00a7100 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): dump: 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.941 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166158692 0 0) 0x7fc5d011f1b0 con 0x7fc5d013bab0 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b40035d0 con 0x7fc5d013bab0 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5c8001af0 con 0x7fc5d013bab0 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient(hunting): dump: 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:48.942 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1441587530 0 0) 0x7fc5d006bd50 con 0x7fc5d00a7100 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 >> v1:172.21.15.76:6790/0 conn(0x7fc5d013bab0 legacy=0x7fc5d01d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 >> v1:172.21.15.76:6789/0 conn(0x7fc5d00a5c70 legacy=0x7fc5d009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 monclient: found mon.b 2024-09-16T06:44:48.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00a4550 con 0x7fc5d00a7100 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 5 monclient: authenticate success, global_id 174154 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _renew_subs 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d011e500 con 0x7fc5d00a7100 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _renew_subs 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d009ff80 con 0x7fc5d00a7100 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5cc004c80 con 0x7fc5d00a7100 2024-09-16T06:44:48.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc005010 con 0x7fc5d00a7100 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c67fc700 10 monclient: dump: 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:48.945 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.928+0000 7fc5c77fe700 4 set_mon_vals no callback set 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc5cc006260 con 0x7fc5d00a7100 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5d7fce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5d7fce700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5d7fce700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(165..165 src has 66..165) v4 ==== 8291+0+0 (unknown 2125616204 0 0) 0x7fc5cc044820 con 0x7fc5d00a7100 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d013af50 con 0x7fc5d00a7100 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.932+0000 7fc5d5d6a700 10 monclient: get_auth_request con 0x7fc5b40362f0 auth_method 0 2024-09-16T06:44:48.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.940+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc5cc0149e0 con 0x7fc5d00a7100 2024-09-16T06:44:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.940+0000 7fc5c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:48.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:48.940+0000 7fc5c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:49.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.104+0000 7fc5d7fce700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.104+0000 7fc5d7fce700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.104+0000 7fc5d7fce700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:49.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.104+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5d0067950 con 0x7fc5d00a7100 2024-09-16T06:44:49.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.796+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v166) v1 ==== 148+0+0 (unknown 737826526 0 0) 0x7fc5cc013080 con 0x7fc5d00a7100 2024-09-16T06:44:49.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.796+0000 7fc5c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.798 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.796+0000 7fc5c67fc700 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-16T06:44:49.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.904+0000 7fc5d7fce700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.904+0000 7fc5d7fce700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.904+0000 7fc5d7fce700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:49.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.904+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5d009f410 con 0x7fc5d00a7100 2024-09-16T06:44:49.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5c67fc700 1 -- 172.21.15.189:0/2768247255 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v166) v1 ==== 155+0+0 (unknown 2594891778 0 0) 0x7fc5cc014800 con 0x7fc5d00a7100 2024-09-16T06:44:49.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5c67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:49.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5c67fc700 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-16T06:44:49.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc5b40362f0 legacy=0x7fc5b40387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:49.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 10 monclient: shutdown 2024-09-16T06:44:49.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:49.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 >> v1:172.21.15.189:6789/0 conn(0x7fc5d00a7100 legacy=0x7fc5d009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:49.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 shutdown_connections 2024-09-16T06:44:49.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 >> 172.21.15.189:0/2768247255 conn(0x7fc5d00a79e0 msgr2=0x7fc5d013a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:49.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 shutdown_connections 2024-09-16T06:44:49.911 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:49.908+0000 7fc5d7fce700 1 -- 172.21.15.189:0/2768247255 wait complete. 2024-09-16T06:44:49.912 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cool' does not exist 2024-09-16T06:44:49.921 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-16T06:44:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: get_monmap_and_config 2024-09-16T06:44:50.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: build_initial_monmap 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: monmap: 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:50.030202+0000 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:50.030202+0000 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:50.030 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 1 Processor -- start 2024-09-16T06:44:50.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 1 -- start start 2024-09-16T06:44:50.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: init 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-a con 0x7ff2901352c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-c con 0x7ff290135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-b con 0x7ff2900a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29009b050 con 0x7ff2901352c0 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29006ab20 con 0x7ff290135db0 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29006a840 con 0x7ff2900a2030 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.028+0000 7ff2964d7700 10 monclient(hunting): authenticate will time out at 1242.543973s 2024-09-16T06:44:50.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff294cd4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff290135db0 0x7ff29013a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46882/0 (socket says 172.21.15.189:46882) 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff294cd4700 1 -- 172.21.15.189:0/1882148875 learned_addr learned my addr 172.21.15.189:0/1882148875 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941859430 0 0) 0x7ff29006ab20 con 0x7ff290135db0 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): my global_id is 174157 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27c0035d0 con 0x7ff290135db0 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392195147 0 0) 0x7ff29006a840 con 0x7ff2900a2030 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient(hunting): my global_id is 125483 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29006ab20 con 0x7ff2900a2030 2024-09-16T06:44:50.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684121618 0 0) 0x7ff27c0035d0 con 0x7ff290135db0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 >> v1:172.21.15.76:6790/0 conn(0x7ff2900a2030 legacy=0x7ff29013df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 >> v1:172.21.15.76:6789/0 conn(0x7ff2901352c0 legacy=0x7ff2901356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 monclient: found mon.noname-c 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29006c1b0 con 0x7ff290135db0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff280001610 con 0x7ff290135db0 2024-09-16T06:44:50.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 5 monclient: authenticate success, global_id 174157 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: dump: 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:50.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2800018f0 con 0x7ff290135db0 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 1 -- 172.21.15.189:0/1882148875 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff280001c80 con 0x7ff290135db0 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 4 set_mon_vals no callback set 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff287fff700 10 monclient: dump: 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.038 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 10 monclient: shutdown 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- 172.21.15.189:0/1882148875 >> v1:172.21.15.189:6789/0 conn(0x7ff290135db0 legacy=0x7ff29013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- 172.21.15.189:0/1882148875 shutdown_connections 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- 172.21.15.189:0/1882148875 >> 172.21.15.189:0/1882148875 conn(0x7ff2900a4550 msgr2=0x7ff2900a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- 172.21.15.189:0/1882148875 shutdown_connections 2024-09-16T06:44:50.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- 172.21.15.189:0/1882148875 wait complete. 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 10 monclient: build_initial_monmap 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 10 monclient: monmap: 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:50.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 Processor -- start 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 1 -- start start 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.032+0000 7ff2964d7700 10 monclient: init 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-a con 0x7ff2900a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-c con 0x7ff2901352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient(hunting): picked mon.noname-b con 0x7ff290135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2900a6130 con 0x7ff2900a2030 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29013a5a0 con 0x7ff290135db0 2024-09-16T06:44:50.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2900a6680 con 0x7ff2901352c0 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient(hunting): authenticate will time out at 1242.548882s 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff295cd6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff290135db0 0x7ff2901d1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46888/0 (socket says 172.21.15.189:46888) 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff295cd6700 1 -- 172.21.15.189:0/3965496827 learned_addr learned my addr 172.21.15.189:0/3965496827 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836844652 0 0) 0x7ff29013a5a0 con 0x7ff290135db0 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): my global_id is 174160 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2740035d0 con 0x7ff290135db0 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628409043 0 0) 0x7ff2900a6680 con 0x7ff2901352c0 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:50.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): my global_id is 125486 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29013a5a0 con 0x7ff2901352c0 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220741597 0 0) 0x7ff2740035d0 con 0x7ff290135db0 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2900a6680 con 0x7ff290135db0 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff288001c90 con 0x7ff290135db0 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:50.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2812201907 0 0) 0x7ff2900a6680 con 0x7ff290135db0 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 >> v1:172.21.15.76:6790/0 conn(0x7ff2901352c0 legacy=0x7ff29013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 >> v1:172.21.15.76:6789/0 conn(0x7ff2900a2030 legacy=0x7ff2901c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 monclient: found mon.b 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29009e820 con 0x7ff290135db0 2024-09-16T06:44:50.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 5 monclient: authenticate success, global_id 174160 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _renew_subs 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29013a2c0 con 0x7ff290135db0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _renew_subs 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29013e3e0 con 0x7ff290135db0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff288002020 con 0x7ff290135db0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff288003e90 con 0x7ff290135db0 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2877fe700 4 set_mon_vals no callback set 2024-09-16T06:44:50.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 10 monclient: dump: 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff288004070 con 0x7ff290135db0 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:50.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff270005100 con 0x7ff290135db0 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.036+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(166..166 src has 66..166) v4 ==== 7620+0+0 (unknown 356038422 0 0) 0x7ff288044630 con 0x7ff290135db0 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.040+0000 7ff2954d5700 10 monclient: get_auth_request con 0x7ff274036200 auth_method 0 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.048+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff288014c00 con 0x7ff290135db0 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.048+0000 7ff285ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:50.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.048+0000 7ff285ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:50.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.216+0000 7ff2964d7700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.216+0000 7ff2964d7700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.216+0000 7ff2964d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.216+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff2700053b0 con 0x7ff290135db0 2024-09-16T06:44:50.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.812+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v167) v1 ==== 148+0+0 (unknown 1614740769 0 0) 0x7ff288019b20 con 0x7ff290135db0 2024-09-16T06:44:50.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.812+0000 7ff285ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.814 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.812+0000 7ff285ffb700 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-16T06:44:50.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.920+0000 7ff2964d7700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.920+0000 7ff2964d7700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.920+0000 7ff2964d7700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:50.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.920+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff270005af0 con 0x7ff290135db0 2024-09-16T06:44:50.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff285ffb700 1 -- 172.21.15.189:0/3965496827 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v167) v1 ==== 155+0+0 (unknown 1403278884 0 0) 0x7ff288013070 con 0x7ff290135db0 2024-09-16T06:44:50.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff285ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T06:44:50.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff285ffb700 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff274036200 legacy=0x7ff2740386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 10 monclient: shutdown 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 >> v1:172.21.15.189:6789/0 conn(0x7ff290135db0 legacy=0x7ff2901d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 shutdown_connections 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 >> 172.21.15.189:0/3965496827 conn(0x7ff2900a4550 msgr2=0x7ff2900a24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:50.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 shutdown_connections 2024-09-16T06:44:50.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:50.924+0000 7ff2964d7700 1 -- 172.21.15.189:0/3965496827 wait complete. 2024-09-16T06:44:50.928 INFO:tasks.workunit.client.0.smithi189.stderr:pool 'cold' does not exist 2024-09-16T06:44:50.937 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: get_monmap_and_config 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: build_initial_monmap 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: monmap: 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:51.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:51.041836+0000 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:51.041836+0000 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:51.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 1 Processor -- start 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 1 -- start start 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: init 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c09f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:51.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c09f520 con 0x7f5f5c099700 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c09bc10 con 0x7f5f5c09f140 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c09b810 con 0x7f5f5c1396a0 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient(hunting): authenticate will time out at 1243.554640s 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f6114a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5f5c09f140 0x7f5f5c09a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46898/0 (socket says 172.21.15.189:46898) 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f6114a700 1 -- 172.21.15.189:0/1035206803 learned_addr learned my addr 172.21.15.189:0/1035206803 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652882035 0 0) 0x7f5f5c09bc10 con 0x7f5f5c09f140 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:51.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient(hunting): my global_id is 174163 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f440035d0 con 0x7f5f5c09f140 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109564211 0 0) 0x7f5f440035d0 con 0x7f5f5c09f140 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 >> v1:172.21.15.76:6790/0 conn(0x7f5f5c1396a0 legacy=0x7f5f5c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 >> v1:172.21.15.76:6789/0 conn(0x7f5f5c099700 legacy=0x7f5f5c09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 monclient: found mon.noname-c 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c09ceb0 con 0x7f5f5c09f140 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:51.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f50001610 con 0x7f5f5c09f140 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: dump: 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:51.048 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 5 monclient: authenticate success, global_id 174163 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f50001900 con 0x7f5f5c09f140 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 1 -- 172.21.15.189:0/1035206803 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f50001d10 con 0x7f5f5c09f140 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f5bfff700 10 monclient: dump: 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:51.049 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.040+0000 7f5f633ae700 4 set_mon_vals no callback set 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 10 monclient: shutdown 2024-09-16T06:44:51.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:51.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- 172.21.15.189:0/1035206803 >> v1:172.21.15.189:6789/0 conn(0x7f5f5c09f140 legacy=0x7f5f5c09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:51.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- 172.21.15.189:0/1035206803 shutdown_connections 2024-09-16T06:44:51.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- 172.21.15.189:0/1035206803 >> 172.21.15.189:0/1035206803 conn(0x7f5f5c0a03e0 msgr2=0x7f5f5c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:51.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- 172.21.15.189:0/1035206803 shutdown_connections 2024-09-16T06:44:51.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- 172.21.15.189:0/1035206803 wait complete. 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 10 monclient: build_initial_monmap 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 10 monclient: monmap: 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:51.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 Processor -- start 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 1 -- start start 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.044+0000 7f5f633ae700 10 monclient: init 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:51.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c09f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c1396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c090690 con 0x7f5f5c1396a0 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c092710 con 0x7f5f5c09f140 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c091260 con 0x7f5f5c099700 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f60949700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5f5c09f140 0x7f5f5c1cc850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46908/0 (socket says 172.21.15.189:46908) 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient(hunting): authenticate will time out at 1243.560427s 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f60949700 1 -- 172.21.15.189:0/876376590 learned_addr learned my addr 172.21.15.189:0/876376590 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828965920 0 0) 0x7f5f5c092710 con 0x7f5f5c09f140 2024-09-16T06:44:51.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): my global_id is 174166 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f480035d0 con 0x7f5f5c09f140 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457997766 0 0) 0x7f5f5c091260 con 0x7f5f5c099700 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): my global_id is 125492 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5c092710 con 0x7f5f5c099700 2024-09-16T06:44:51.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252399253 0 0) 0x7f5f480035d0 con 0x7f5f5c09f140 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f5c091260 con 0x7f5f5c09f140 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f4c001c60 con 0x7f5f5c09f140 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:51.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1488453546 0 0) 0x7f5f5c091260 con 0x7f5f5c09f140 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 >> v1:172.21.15.76:6790/0 conn(0x7f5f5c099700 legacy=0x7f5f5c095f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 >> v1:172.21.15.76:6789/0 conn(0x7f5f5c1396a0 legacy=0x7f5f5c1cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 monclient: found mon.b 2024-09-16T06:44:51.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c0929f0 con 0x7f5f5c09f140 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 5 monclient: authenticate success, global_id 174166 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _renew_subs 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c0714a0 con 0x7f5f5c09f140 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _renew_subs 2024-09-16T06:44:51.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c090fe0 con 0x7f5f5c09f140 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f4c003a20 con 0x7f5f5c09f140 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f4c003d30 con 0x7f5f5c09f140 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 10 monclient: dump: 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:51.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f5b7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5f4c003f10 con 0x7f5f5c09f140 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c005100 con 0x7f5f5c09f140 2024-09-16T06:44:51.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(167..167 src has 141..167) v4 ==== 6813+0+0 (unknown 3200957990 0 0) 0x7f5f4c044250 con 0x7f5f5c09f140 2024-09-16T06:44:51.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.048+0000 7f5f6114a700 10 monclient: get_auth_request con 0x7f5f48036250 auth_method 0 2024-09-16T06:44:51.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.056+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5f4c014af0 con 0x7f5f5c09f140 2024-09-16T06:44:51.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.056+0000 7f5f59ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:51.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.056+0000 7f5f59ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:51.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.232+0000 7f5f633ae700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:51.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.232+0000 7f5f633ae700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:51.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.232+0000 7f5f633ae700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.232+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f5f3c0053b0 con 0x7f5f5c09f140 2024-09-16T06:44:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.820+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v168) v1 ==== 78+0+0 (unknown 3731923353 0 0) 0x7f5f4c012f60 con 0x7f5f5c09f140 2024-09-16T06:44:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.820+0000 7f5f59ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:51.820 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.820+0000 7f5f59ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.940+0000 7f5f633ae700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.940+0000 7f5f633ae700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.940+0000 7f5f633ae700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:51.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:51.940+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f5f3c0023e0 con 0x7f5f5c09f140 2024-09-16T06:44:52.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.836+0000 7f5f59ffb700 1 -- 172.21.15.189:0/876376590 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v169) v1 ==== 78+0+0 (unknown 2679237564 0 0) 0x7f5f4c0148c0 con 0x7f5f5c09f140 2024-09-16T06:44:52.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.836+0000 7f5f59ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:44:52.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.836+0000 7f5f59ffb700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:44:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5f48036250 legacy=0x7f5f48038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 10 monclient: shutdown 2024-09-16T06:44:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:52.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 >> v1:172.21.15.189:6789/0 conn(0x7f5f5c09f140 legacy=0x7f5f5c1cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 shutdown_connections 2024-09-16T06:44:52.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 >> 172.21.15.189:0/876376590 conn(0x7f5f5c0a03e0 msgr2=0x7f5f5c0a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:52.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 shutdown_connections 2024-09-16T06:44:52.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.848+0000 7f5f633ae700 1 -- 172.21.15.189:0/876376590 wait complete. 2024-09-16T06:44:52.864 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T06:44:52.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: get_monmap_and_config 2024-09-16T06:44:52.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: monmap: 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:52.965188+0000 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:52.965188+0000 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:52.965 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 1 Processor -- start 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 1 -- start start 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: init 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-c con 0x7f3e1409f570 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3e140a0060 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-a con 0x7f3e14138a80 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1409b010 con 0x7f3e14138a80 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1409c950 con 0x7f3e1409f570 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1409c0b0 con 0x7f3e140a0060 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1359e700 10 monclient(hunting): authenticate will time out at 1245.478837s 2024-09-16T06:44:52.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e1409f570 0x7f3e1409f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46916/0 (socket says 172.21.15.189:46916) 2024-09-16T06:44:52.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1259c700 1 -- 172.21.15.189:0/1242504338 learned_addr learned my addr 172.21.15.189:0/1242504338 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983382796 0 0) 0x7f3e1409c950 con 0x7f3e1409f570 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): my global_id is 174169 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dfc0035d0 con 0x7f3e1409f570 2024-09-16T06:44:52.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566277425 0 0) 0x7f3e1409c0b0 con 0x7f3e140a0060 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 10 monclient(hunting): my global_id is 125495 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.964+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1409c950 con 0x7f3e140a0060 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279577310 0 0) 0x7f3dfc0035d0 con 0x7f3e1409f570 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 >> v1:172.21.15.76:6790/0 conn(0x7f3e140a0060 legacy=0x7f3e14137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 >> v1:172.21.15.76:6789/0 conn(0x7f3e14138a80 legacy=0x7f3e1413ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 monclient: found mon.noname-c 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:52.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e140667c0 con 0x7f3e1409f570 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: _finish_auth 0 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e04001610 con 0x7f3e1409f570 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 5 monclient: authenticate success, global_id 174169 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: dump: 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e040018f0 con 0x7f3e1409f570 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 4 set_mon_vals no callback set 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 1 -- 172.21.15.189:0/1242504338 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e04001c80 con 0x7f3e1409f570 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:52.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1159a700 10 monclient: dump: 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: shutdown 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1242504338 >> v1:172.21.15.189:6789/0 conn(0x7f3e1409f570 legacy=0x7f3e1409f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1242504338 shutdown_connections 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1242504338 >> 172.21.15.189:0/1242504338 conn(0x7f3e140a4320 msgr2=0x7f3e140a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1242504338 shutdown_connections 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1242504338 wait complete. 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: monmap: 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:52.979 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 Processor -- start 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- start start 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: init 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-a con 0x7f3e140a0060 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-c con 0x7f3e14138a80 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3e14138080 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e14072210 con 0x7f3e140a0060 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140a2440 con 0x7f3e14138080 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1412a0c0 con 0x7f3e14138a80 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:52.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e1359e700 10 monclient(hunting): authenticate will time out at 1245.483178s 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e12d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e14138080 0x7f3e141d67c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46930/0 (socket says 172.21.15.189:46930) 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e12d9d700 1 -- 172.21.15.189:0/1857981068 learned_addr learned my addr 172.21.15.189:0/1857981068 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e11d9b700 1 --1- 172.21.15.189:0/1857981068 >> v1:172.21.15.76:6790/0 conn(0x7f3e14138a80 0x7f3e141d3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:56462/0 (socket says 172.21.15.189:56462) 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058567290 0 0) 0x7f3e1412a0c0 con 0x7f3e14138a80 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): my global_id is 125498 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df40035d0 con 0x7f3e14138a80 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971271546 0 0) 0x7f3e140a2440 con 0x7f3e14138080 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:52.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 10 monclient(hunting): my global_id is 174172 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.968+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1412a0c0 con 0x7f3e14138080 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466127870 0 0) 0x7f3e1412a0c0 con 0x7f3e14138080 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e140a2440 con 0x7f3e14138080 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e0c001bb0 con 0x7f3e14138080 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:52.982 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108539310 0 0) 0x7f3df40035d0 con 0x7f3e14138a80 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e1412a0c0 con 0x7f3e14138a80 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e08001850 con 0x7f3e14138a80 2024-09-16T06:44:52.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1682657186 0 0) 0x7f3e140a2440 con 0x7f3e14138080 2024-09-16T06:44:52.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 >> v1:172.21.15.76:6790/0 conn(0x7f3e14138a80 legacy=0x7f3e141d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 >> v1:172.21.15.76:6789/0 conn(0x7f3e140a0060 legacy=0x7f3e14137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 monclient: found mon.b 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14127fd0 con 0x7f3e14138080 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 5 monclient: authenticate success, global_id 174172 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _renew_subs 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14070e70 con 0x7f3e14138080 2024-09-16T06:44:52.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _renew_subs 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14129b00 con 0x7f3e14138080 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e0c003a20 con 0x7f3e14138080 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e0c003db0 con 0x7f3e14138080 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e03fff700 4 set_mon_vals no callback set 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 10 monclient: dump: 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:52.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df8005100 con 0x7f3e14138080 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3e0c003f90 con 0x7f3e14138080 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(169..169 src has 141..169) v4 ==== 6386+0+0 (unknown 526347892 0 0) 0x7f3e0c0440b0 con 0x7f3e14138080 2024-09-16T06:44:52.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.972+0000 7f3e1259c700 10 monclient: get_auth_request con 0x7f3df40362c0 auth_method 0 2024-09-16T06:44:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.980+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e0c014b60 con 0x7f3e14138080 2024-09-16T06:44:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.980+0000 7f3e02ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:52.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:52.980+0000 7f3e02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:53.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.156+0000 7f3e1359e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.156+0000 7f3e1359e700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.159 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.156+0000 7f3e1359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:53.160 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.156+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f3df80053b0 con 0x7f3e14138080 2024-09-16T06:44:53.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.848+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v170) v1 ==== 82+0+0 (unknown 3990475870 0 0) 0x7f3e0c01d1a0 con 0x7f3e14138080 2024-09-16T06:44:53.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.848+0000 7f3e02ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.848+0000 7f3e02ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:53.963 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T06:44:53.964 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T06:44:53.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:44:53.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:44:53.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e1359e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e1359e700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e1359e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:53.965 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f3df8005900 con 0x7f3e14138080 2024-09-16T06:44:53.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e02ffd700 1 -- 172.21.15.189:0/1857981068 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v170) v1 ==== 115+0+0 (unknown 878742931 0 0) 0x7f3e0c012fd0 con 0x7f3e14138080 2024-09-16T06:44:53.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e02ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:44:53.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.964+0000 7f3e02ffd700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T06:44:53.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3df40362c0 legacy=0x7f3df4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:53.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 10 monclient: shutdown 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 >> v1:172.21.15.189:6789/0 conn(0x7f3e14138080 legacy=0x7f3e141d67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 shutdown_connections 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 >> 172.21.15.189:0/1857981068 conn(0x7f3e140a4320 msgr2=0x7f3e14099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 shutdown_connections 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:53.968+0000 7f3e1359e700 1 -- 172.21.15.189:0/1857981068 wait complete. 2024-09-16T06:44:53.969 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T06:44:53.978 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket fooo host 2024-09-16T06:44:54.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: monmap: 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:54.076928+0000 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:54.076928+0000 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.075 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 1 Processor -- start 2024-09-16T06:44:54.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 1 -- start start 2024-09-16T06:44:54.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: init 2024-09-16T06:44:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f4097470 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f4097850 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-a con 0x7fb1f4097c90 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f409d760 con 0x7fb1f4097c90 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f40a0ab0 con 0x7fb1f4097470 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f409d110 con 0x7fb1f4097850 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1fa0bb700 10 monclient(hunting): authenticate will time out at 1241.211648s 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1f98ba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb1f4097c90 0x7fb1f413a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38002/0 (socket says 172.21.15.76:38002) 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.074+0000 7fb1f98ba700 1 -- 172.21.15.76:0/2959083013 learned_addr learned my addr 172.21.15.76:0/2959083013 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605079224 0 0) 0x7fb1f40a0ab0 con 0x7fb1f4097470 2024-09-16T06:44:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): my global_id is 174175 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e00035d0 con 0x7fb1f4097470 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346627984 0 0) 0x7fb1f409d110 con 0x7fb1f4097850 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): my global_id is 125501 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f40a0ab0 con 0x7fb1f4097850 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006397530 0 0) 0x7fb1f409d760 con 0x7fb1f4097c90 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient(hunting): my global_id is 154584 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f409d110 con 0x7fb1f4097c90 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1713627117 0 0) 0x7fb1e00035d0 con 0x7fb1f4097470 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 >> v1:172.21.15.76:6790/0 conn(0x7fb1f4097850 legacy=0x7fb1f4136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 >> v1:172.21.15.76:6789/0 conn(0x7fb1f4097c90 legacy=0x7fb1f413a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 monclient: found mon.noname-c 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f409d3f0 con 0x7fb1f4097470 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1f0001610 con 0x7fb1f4097470 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: dump: 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 5 monclient: authenticate success, global_id 174175 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f00018f0 con 0x7fb1f4097470 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 4 set_mon_vals no callback set 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 1 -- 172.21.15.76:0/2959083013 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1f0001c80 con 0x7fb1f4097470 2024-09-16T06:44:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1ebfff700 10 monclient: dump: 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 10 monclient: shutdown 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/2959083013 >> v1:172.21.15.189:6789/0 conn(0x7fb1f4097470 legacy=0x7fb1f409e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/2959083013 shutdown_connections 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/2959083013 >> 172.21.15.76:0/2959083013 conn(0x7fb1f40a1120 msgr2=0x7fb1f40a1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/2959083013 shutdown_connections 2024-09-16T06:44:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/2959083013 wait complete. 2024-09-16T06:44:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.078+0000 7fb1fa0bb700 10 monclient: monmap: 2024-09-16T06:44:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 Processor -- start 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- start start 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: init 2024-09-16T06:44:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f4097470 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f4097850 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient(hunting): picked mon.noname-a con 0x7fb1f4097c90 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406f640 con 0x7fb1f4097c90 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406b730 con 0x7fb1f4097850 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f406f360 con 0x7fb1f4097470 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1f98ba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb1f4097c90 0x7fb1f41d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38008/0 (socket says 172.21.15.76:38008) 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1f98ba700 1 -- 172.21.15.76:0/226849142 learned_addr learned my addr 172.21.15.76:0/226849142 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient(hunting): authenticate will time out at 1241.217621s 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024657672 0 0) 0x7fb1f406f640 con 0x7fb1f4097c90 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): my global_id is 154587 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d80035d0 con 0x7fb1f4097c90 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178593026 0 0) 0x7fb1f406f360 con 0x7fb1f4097470 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): my global_id is 125504 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f406f640 con 0x7fb1f4097470 2024-09-16T06:44:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596607093 0 0) 0x7fb1f406b730 con 0x7fb1f4097850 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): my global_id is 174178 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f406f360 con 0x7fb1f4097850 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305360466 0 0) 0x7fb1d80035d0 con 0x7fb1f4097c90 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f406b730 con 0x7fb1f4097c90 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1ec001bb0 con 0x7fb1f4097c90 2024-09-16T06:44:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219124609 0 0) 0x7fb1f406f640 con 0x7fb1f4097470 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1d80035d0 con 0x7fb1f4097470 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1f00015d0 con 0x7fb1f4097470 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.102 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761023511 0 0) 0x7fb1f406f360 con 0x7fb1f4097850 2024-09-16T06:44:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f406f640 con 0x7fb1f4097850 2024-09-16T06:44:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1e4001760 con 0x7fb1f4097850 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.103 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3085145886 0 0) 0x7fb1f406b730 con 0x7fb1f4097c90 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 >> v1:172.21.15.76:6790/0 conn(0x7fb1f4097470 legacy=0x7fb1f406a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 >> v1:172.21.15.189:6789/0 conn(0x7fb1f4097850 legacy=0x7fb1f406b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 monclient: found mon.a 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f406f920 con 0x7fb1f4097c90 2024-09-16T06:44:54.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 5 monclient: authenticate success, global_id 154587 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _renew_subs 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f406e690 con 0x7fb1f4097c90 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _renew_subs 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f406dd80 con 0x7fb1f4097c90 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1ec0020a0 con 0x7fb1f4097c90 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb1ec004db0 con 0x7fb1f4097c90 2024-09-16T06:44:54.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1e9ffb700 10 monclient: dump: 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.106 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.082+0000 7fb1eb7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:54.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1fa0bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1fa0bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1fa0bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d4005100 con 0x7fb1f4097c90 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb1ec004f90 con 0x7fb1f4097c90 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(170..170 src has 141..170) v4 ==== 6714+0+0 (unknown 4275550958 0 0) 0x7fb1ec044080 con 0x7fb1f4097c90 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.086+0000 7fb1f90b9700 10 monclient: get_auth_request con 0x7fb1d80364b0 auth_method 0 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.090+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb1ec005710 con 0x7fb1f4097c90 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.090+0000 7fb1e9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.090+0000 7fb1e9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: monmap: 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:54.086333+0000 2024-09-16T06:44:54.111 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:54.086333+0000 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 1 Processor -- start 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 1 -- start start 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: init 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:54.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-c con 0x7f459409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-b con 0x7f459409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-a con 0x7f4594138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459409cca0 con 0x7f4594138da0 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459409f750 con 0x7f459409bb00 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594097620 con 0x7f459409c5f0 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f459a08f700 10 monclient(hunting): authenticate will time out at 1246.599502s 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f4592ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f459409c5f0 0x7f4594137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:56464/0 (socket says 172.21.15.189:56464) 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f45937fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f459409bb00 0x7f459409bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46932/0 (socket says 172.21.15.189:46932) 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.084+0000 7f4592ffd700 1 -- 172.21.15.189:0/3533149767 learned_addr learned my addr 172.21.15.189:0/3533149767 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774866836 0 0) 0x7f459409f750 con 0x7f459409bb00 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): my global_id is 174181 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c0057e0 con 0x7f459409bb00 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711115635 0 0) 0x7f4594097620 con 0x7f459409c5f0 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient(hunting): my global_id is 125507 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459409f750 con 0x7f459409c5f0 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3900220834 0 0) 0x7f457c0057e0 con 0x7f459409bb00 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 >> v1:172.21.15.76:6790/0 conn(0x7f459409c5f0 legacy=0x7f4594137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 >> v1:172.21.15.76:6789/0 conn(0x7f4594138da0 legacy=0x7f459413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 monclient: found mon.noname-c 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594097900 con 0x7f459409bb00 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 5 monclient: authenticate success, global_id 174181 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4584001990 con 0x7f459409bb00 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: dump: 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4584001c70 con 0x7f459409bb00 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 1 -- 172.21.15.189:0/3533149767 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4584002040 con 0x7f459409bb00 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 4 set_mon_vals no callback set 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f45927fc700 10 monclient: dump: 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.117 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: shutdown 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- 172.21.15.189:0/3533149767 >> v1:172.21.15.189:6789/0 conn(0x7f459409bb00 legacy=0x7f459409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- 172.21.15.189:0/3533149767 shutdown_connections 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- 172.21.15.189:0/3533149767 >> 172.21.15.189:0/3533149767 conn(0x7f45940a03e0 msgr2=0x7f45940a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- 172.21.15.189:0/3533149767 shutdown_connections 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- 172.21.15.189:0/3533149767 wait complete. 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: monmap: 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 Processor -- start 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- start start 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: init 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-a con 0x7f459409bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-b con 0x7f459409c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient(hunting): picked mon.noname-c con 0x7f4594138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594090f20 con 0x7f459409bb00 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594099460 con 0x7f459409c5f0 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459408f790 con 0x7f4594138da0 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f459a08f700 10 monclient(hunting): authenticate will time out at 1246.602860s 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f4593fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4594138da0 0x7f45941ce880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:56476/0 (socket says 172.21.15.189:56476) 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f4593fff700 1 -- 172.21.15.189:0/1076508548 learned_addr learned my addr 172.21.15.189:0/1076508548 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f4592ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f459409c5f0 0x7f45941cb150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46942/0 (socket says 172.21.15.189:46942) 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967284577 0 0) 0x7f459408f790 con 0x7f4594138da0 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): my global_id is 125510 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45800035d0 con 0x7f4594138da0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688065557 0 0) 0x7f4594099460 con 0x7f459409c5f0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): my global_id is 174184 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459408f790 con 0x7f459409c5f0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954902936 0 0) 0x7f45800035d0 con 0x7f4594138da0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4594099460 con 0x7f4594138da0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f458c001830 con 0x7f4594138da0 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): dump: 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 284346914 0 0) 0x7f459408f790 con 0x7f459409c5f0 2024-09-16T06:44:54.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45800035d0 con 0x7f459409c5f0 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4588001950 con 0x7f459409c5f0 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.088+0000 7f457bfff700 10 monclient(hunting): dump: 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3049241136 0 0) 0x7f4594099460 con 0x7f4594138da0 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 >> v1:172.21.15.189:6789/0 conn(0x7f459409c5f0 legacy=0x7f45941cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 >> v1:172.21.15.76:6789/0 conn(0x7f459409bb00 legacy=0x7f45941c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 monclient: found mon.c 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594090c40 con 0x7f4594138da0 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 5 monclient: authenticate success, global_id 125510 2024-09-16T06:44:54.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _renew_subs 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459408ebc0 con 0x7f4594138da0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _renew_subs 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45940a2d90 con 0x7f4594138da0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f458c0036c0 con 0x7f4594138da0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f458c003a50 con 0x7f4594138da0 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 10 monclient: dump: 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.125 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f45917fa700 4 set_mon_vals no callback set 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f458c003c30 con 0x7f4594138da0 2024-09-16T06:44:54.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4574005100 con 0x7f4594138da0 2024-09-16T06:44:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(170..170 src has 141..170) v4 ==== 6714+0+0 (unknown 4275550958 0 0) 0x7f458c043d70 con 0x7f4594138da0 2024-09-16T06:44:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.092+0000 7f45937fe700 10 monclient: get_auth_request con 0x7f45800362c0 auth_method 0 2024-09-16T06:44:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.100+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f458c0145a0 con 0x7f4594138da0 2024-09-16T06:44:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.100+0000 7f457bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.100+0000 7f457bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.276+0000 7f459a08f700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.276+0000 7f459a08f700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.276+0000 7f459a08f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.276+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f4574005930 con 0x7f4594138da0 2024-09-16T06:44:54.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1fa0bb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1fa0bb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1fa0bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb1d4005930 con 0x7fb1f4097c90 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1e9ffb700 1 -- 172.21.15.76:0/226849142 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3493916760) 0x7fb1ec01d030 con 0x7fb1f4097c90 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1e9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:54.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.322+0000 7fb1e9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:44:54.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb1d80364b0 legacy=0x7fb1d8038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 10 monclient: shutdown 2024-09-16T06:44:54.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 >> v1:172.21.15.76:6789/0 conn(0x7fb1f4097c90 legacy=0x7fb1f41d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 shutdown_connections 2024-09-16T06:44:54.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 >> 172.21.15.76:0/226849142 conn(0x7fb1f40a1120 msgr2=0x7fb1f4131810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.326+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 shutdown_connections 2024-09-16T06:44:54.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.330+0000 7fb1fa0bb700 1 -- 172.21.15.76:0/226849142 wait complete. 2024-09-16T06:44:54.345 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:44:54.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:44:54.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:44:54.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: monmap: 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:54.464073+0000 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:54.464073+0000 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 1 Processor -- start 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 1 -- start start 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: init 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-a con 0x7f5bd809ee40 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-b con 0x7f5bd8138570 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-c con 0x7f5bd813c080 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd809f930 con 0x7f5bd809ee40 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd806d9b0 con 0x7f5bd813c080 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd806cb30 con 0x7f5bd8138570 2024-09-16T06:44:54.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient(hunting): authenticate will time out at 1241.597281s 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdd24b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5bd8138570 0x7f5bd813a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43072/0 (socket says 172.21.15.76:43072) 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdda4c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5bd809ee40 0x7f5bd809f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38020/0 (socket says 172.21.15.76:38020) 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdd24b700 1 -- 172.21.15.76:0/593341406 learned_addr learned my addr 172.21.15.76:0/593341406 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085858828 0 0) 0x7f5bc8001110 con 0x7f5bd809ee40 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): my global_id is 154590 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc40035d0 con 0x7f5bd809ee40 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906173690 0 0) 0x7f5bd806cb30 con 0x7f5bd8138570 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): my global_id is 125513 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc8001110 con 0x7f5bd8138570 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265103458 0 0) 0x7f5bd806d9b0 con 0x7f5bd813c080 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient(hunting): my global_id is 174187 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd806cb30 con 0x7f5bd813c080 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2100919483 0 0) 0x7f5bc8001110 con 0x7f5bd8138570 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 >> v1:172.21.15.189:6789/0 conn(0x7f5bd813c080 legacy=0x7f5bd813e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 >> v1:172.21.15.76:6789/0 conn(0x7f5bd809ee40 legacy=0x7f5bd809f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 monclient: found mon.noname-b 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd809d7d0 con 0x7f5bd8138570 2024-09-16T06:44:54.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd40015f0 con 0x7f5bd8138570 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 5 monclient: authenticate success, global_id 125513 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: dump: 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd4001890 con 0x7f5bd8138570 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 1 -- 172.21.15.76:0/593341406 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4001c60 con 0x7f5bd8138570 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdea4e700 4 set_mon_vals no callback set 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.462+0000 7f5bdca4a700 10 monclient: dump: 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: shutdown 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/593341406 >> v1:172.21.15.76:6790/0 conn(0x7f5bd8138570 legacy=0x7f5bd813a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/593341406 shutdown_connections 2024-09-16T06:44:54.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/593341406 >> 172.21.15.76:0/593341406 conn(0x7f5bd80a2300 msgr2=0x7f5bd80a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/593341406 shutdown_connections 2024-09-16T06:44:54.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/593341406 wait complete. 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: monmap: 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.478 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 Processor -- start 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- start start 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: init 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-a con 0x7f5bd809ee40 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-b con 0x7f5bd8138570 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient(hunting): picked mon.noname-c con 0x7f5bd813c080 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd8071d50 con 0x7f5bd809ee40 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd8090800 con 0x7f5bd8138570 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdda4c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5bd809ee40 0x7f5bd8137a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38036/0 (socket says 172.21.15.76:38036) 2024-09-16T06:44:54.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdda4c700 1 -- 172.21.15.76:0/1066609385 learned_addr learned my addr 172.21.15.76:0/1066609385 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd808fb70 con 0x7f5bd813c080 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.466+0000 7f5bdea4e700 10 monclient(hunting): authenticate will time out at 1241.603880s 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075724724 0 0) 0x7f5bd8071d50 con 0x7f5bd809ee40 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): my global_id is 154593 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc0035d0 con 0x7f5bd809ee40 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126238848 0 0) 0x7f5bd808fb70 con 0x7f5bd813c080 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): my global_id is 125516 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd8071d50 con 0x7f5bd813c080 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421034784 0 0) 0x7f5bbc0035d0 con 0x7f5bd809ee40 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bd808fb70 con 0x7f5bd809ee40 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bc8001d80 con 0x7f5bd809ee40 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): dump: 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.481 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477920750 0 0) 0x7f5bd8090800 con 0x7f5bd8138570 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): my global_id is 174190 2024-09-16T06:44:54.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc0035d0 con 0x7f5bd8138570 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3808071277 0 0) 0x7f5bd808fb70 con 0x7f5bd809ee40 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6790/0 conn(0x7f5bd813c080 legacy=0x7f5bd81d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.189:6789/0 conn(0x7f5bd8138570 legacy=0x7f5bd81cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 monclient: found mon.a 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8090240 con 0x7f5bd809ee40 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 5 monclient: authenticate success, global_id 154593 2024-09-16T06:44:54.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 10 monclient: _renew_subs 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd8090520 con 0x7f5bd809ee40 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 10 monclient: _renew_subs 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd808f400 con 0x7f5bd809ee40 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc80015d0 con 0x7f5bd809ee40 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bc8004ce0 con 0x7f5bd809ee40 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bcf7fe700 4 set_mon_vals no callback set 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: dump: 2024-09-16T06:44:54.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5bc800f750 con 0x7f5bd809ee40 2024-09-16T06:44:54.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bc0006590 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(170..170 src has 141..170) v4 ==== 6714+0+0 (unknown 4275550958 0 0) 0x7f5bc8043f50 con 0x7f5bd809ee40 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505785451 0 0) 0x7f5bc0006590 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient(hunting): my global_id is 154596 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd808fb70 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bde24d700 10 monclient: get_auth_request con 0x7f5bbc0362f0 auth_method 0 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 134326478 0 0) 0x7f5bd808fb70 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc0006590 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4001a00 con 0x7f5bc0002830 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.470+0000 7f5bce7fc700 10 monclient: dump: 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3452306006 0 0) 0x7f5bc0006590 con 0x7f5bc0002830 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bc000a190 con 0x7f5bc0002830 2024-09-16T06:44:54.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5bd4004890 con 0x7f5bc0002830 2024-09-16T06:44:54.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6789/0 conn(0x7f5bc0002830 legacy=0x7f5bc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.474+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6789/0 conn(0x7f5bc0002830 legacy=0x7f5bc0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bdea4e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bc0009e50 con 0x7f5bc0009170 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5652187 0 0) 0x7f5bc0009e50 con 0x7f5bc0009170 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient(hunting): my global_id is 154599 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0006590 con 0x7f5bc0009170 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1220555385 0 0) 0x7f5bc0006590 con 0x7f5bc0009170 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc0009e50 con 0x7f5bc0009170 2024-09-16T06:44:54.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4004e10 con 0x7f5bc0009170 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 10 monclient: dump: 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2942790211 0 0) 0x7f5bc0009e50 con 0x7f5bc0009170 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.482+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bc000c340 con 0x7f5bc0009170 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 634833934) 0x7f5bd4008fe0 con 0x7f5bc0009170 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bce7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6789/0 conn(0x7f5bc0009170 legacy=0x7f5bc0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bce7fc700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6789/0 conn(0x7f5bc0009170 legacy=0x7f5bc0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:54.492 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.493 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.494 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:54.495 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.496 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:54.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.500 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:54.504 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:54.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5bbc0362f0 legacy=0x7f5bbc0387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 10 monclient: shutdown 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 >> v1:172.21.15.76:6789/0 conn(0x7f5bd809ee40 legacy=0x7f5bd8137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bde24d700 1 -- 172.21.15.76:0/1066609385 reap_dead start 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bde24d700 1 -- 172.21.15.76:0/1066609385 reap_dead start 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 shutdown_connections 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 >> 172.21.15.76:0/1066609385 conn(0x7f5bd80a2300 msgr2=0x7f5bd809e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 shutdown_connections 2024-09-16T06:44:54.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.486+0000 7f5bdea4e700 1 -- 172.21.15.76:0/1066609385 wait complete. 2024-09-16T06:44:54.506 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:54.507 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:44:54.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:44:54.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.622+0000 7fc08b468700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.622+0000 7fc08b468700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.622+0000 7fc08b468700 10 monclient: monmap: 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:54.627778+0000 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:54.627778+0000 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.626 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 Processor -- start 2024-09-16T06:44:54.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- start start 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient: init 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-b con 0x7fc08409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-a con 0x7fc084098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-c con 0x7fc084099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08409d560 con 0x7fc084098be0 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08409f140 con 0x7fc084099030 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08409d790 con 0x7fc08409ed60 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient(hunting): authenticate will time out at 1241.761432s 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089c65700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc084098be0 0x7fc0841374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38054/0 (socket says 172.21.15.76:38054) 2024-09-16T06:44:54.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089c65700 1 -- 172.21.15.76:0/63623745 learned_addr learned my addr 172.21.15.76:0/63623745 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044964839 0 0) 0x7fc08409d790 con 0x7fc08409ed60 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): my global_id is 125519 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0700035d0 con 0x7fc08409ed60 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699113568 0 0) 0x7fc08409d560 con 0x7fc084098be0 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): my global_id is 154602 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08409d790 con 0x7fc084098be0 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470149343 0 0) 0x7fc08409f140 con 0x7fc084099030 2024-09-16T06:44:54.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient(hunting): my global_id is 174193 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08409d560 con 0x7fc084099030 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344391886 0 0) 0x7fc0700035d0 con 0x7fc08409ed60 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 >> v1:172.21.15.189:6789/0 conn(0x7fc084099030 legacy=0x7fc08413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 >> v1:172.21.15.76:6789/0 conn(0x7fc084098be0 legacy=0x7fc0841374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 monclient: found mon.noname-b 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08409f620 con 0x7fc08409ed60 2024-09-16T06:44:54.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc080001610 con 0x7fc08409ed60 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: dump: 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 5 monclient: authenticate success, global_id 125519 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0800018f0 con 0x7fc08409ed60 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 1 -- 172.21.15.76:0/63623745 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc080001c80 con 0x7fc08409ed60 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 4 set_mon_vals no callback set 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc089464700 10 monclient: dump: 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 10 monclient: shutdown 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- 172.21.15.76:0/63623745 >> v1:172.21.15.76:6790/0 conn(0x7fc08409ed60 legacy=0x7fc0840984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- 172.21.15.76:0/63623745 shutdown_connections 2024-09-16T06:44:54.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- 172.21.15.76:0/63623745 >> 172.21.15.76:0/63623745 conn(0x7fc0840a0030 msgr2=0x7fc0840a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- 172.21.15.76:0/63623745 shutdown_connections 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.626+0000 7fc08b468700 1 -- 172.21.15.76:0/63623745 wait complete. 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: monmap: 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.637 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 Processor -- start 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- start start 2024-09-16T06:44:54.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: init 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-a con 0x7fc084098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-c con 0x7fc084099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient(hunting): picked mon.noname-b con 0x7fc08409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840a29b0 con 0x7fc084098be0 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08408f3c0 con 0x7fc08409ed60 2024-09-16T06:44:54.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc084090f30 con 0x7fc084099030 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient(hunting): authenticate will time out at 1241.766403s 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc089c65700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc084099030 0x7fc08408da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43108/0 (socket says 172.21.15.76:43108) 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08a466700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc084098be0 0x7fc08408d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38064/0 (socket says 172.21.15.76:38064) 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc089c65700 1 -- 172.21.15.76:0/1158635587 learned_addr learned my addr 172.21.15.76:0/1158635587 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346885507 0 0) 0x7fc084090f30 con 0x7fc084099030 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): my global_id is 125522 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0680035d0 con 0x7fc084099030 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975192869 0 0) 0x7fc0840a29b0 con 0x7fc084098be0 2024-09-16T06:44:54.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): my global_id is 154605 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc084090f30 con 0x7fc084098be0 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571163626 0 0) 0x7fc08408f3c0 con 0x7fc08409ed60 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): my global_id is 174196 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0840a29b0 con 0x7fc08409ed60 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4240044877 0 0) 0x7fc0680035d0 con 0x7fc084099030 2024-09-16T06:44:54.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc08408f3c0 con 0x7fc084099030 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc07c001b30 con 0x7fc084099030 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.647 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068066187 0 0) 0x7fc084090f30 con 0x7fc084098be0 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0680035d0 con 0x7fc084098be0 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc080001c00 con 0x7fc084098be0 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient(hunting): dump: 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570222122 0 0) 0x7fc08408f3c0 con 0x7fc084099030 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.189:6789/0 conn(0x7fc08409ed60 legacy=0x7fc0841cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6789/0 conn(0x7fc084098be0 legacy=0x7fc08408d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 monclient: found mon.c 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08408e110 con 0x7fc084099030 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc076ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 5 monclient: authenticate success, global_id 125522 2024-09-16T06:44:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _renew_subs 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0840900f0 con 0x7fc084099030 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _renew_subs 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.630+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc084090c50 con 0x7fc084099030 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc07c0020b0 con 0x7fc084099030 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc07c004db0 con 0x7fc084099030 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: dump: 2024-09-16T06:44:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc088c63700 4 set_mon_vals no callback set 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc08b468700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc06c006590 con 0x7fc06c002830 2024-09-16T06:44:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc07c00f820 con 0x7fc084099030 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(170..170 src has 141..170) v4 ==== 6714+0+0 (unknown 4275550958 0 0) 0x7fc07c044050 con 0x7fc084099030 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157236990 0 0) 0x7fc06c006590 con 0x7fc06c002830 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient(hunting): my global_id is 125525 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08408f3c0 con 0x7fc06c002830 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc08ac67700 10 monclient: get_auth_request con 0x7fc0680383a0 auth_method 0 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3226278617 0 0) 0x7fc08408f3c0 con 0x7fc06c002830 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc06c006590 con 0x7fc06c002830 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc080003990 con 0x7fc06c002830 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: dump: 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3260231140 0 0) 0x7fc06c006590 con 0x7fc06c002830 2024-09-16T06:44:54.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc06c00a190 con 0x7fc06c002830 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc0800058f0 con 0x7fc06c002830 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6790/0 conn(0x7fc06c002830 legacy=0x7fc06c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.634+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6790/0 conn(0x7fc06c002830 legacy=0x7fc06c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc06c009e50 con 0x7fc06c009170 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967331559 0 0) 0x7fc06c009e50 con 0x7fc06c009170 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient(hunting): my global_id is 125528 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06c006590 con 0x7fc06c009170 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1685659425 0 0) 0x7fc06c006590 con 0x7fc06c009170 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc06c009e50 con 0x7fc06c009170 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0800040f0 con 0x7fc06c009170 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient: dump: 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1053685753 0 0) 0x7fc06c009e50 con 0x7fc06c009170 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc06c00c340 con 0x7fc06c009170 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 1733178217) 0x7fc0800043f0 con 0x7fc06c009170 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6790/0 conn(0x7fc06c009170 legacy=0x7fc06c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc076ffd700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6790/0 conn(0x7fc06c009170 legacy=0x7fc06c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:54.669 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc0680383a0 legacy=0x7fc06803a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 10 monclient: shutdown 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 >> v1:172.21.15.76:6790/0 conn(0x7fc084099030 legacy=0x7fc08408da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08ac67700 1 -- 172.21.15.76:0/1158635587 reap_dead start 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 shutdown_connections 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 >> 172.21.15.76:0/1158635587 conn(0x7fc0840a0030 msgr2=0x7fc0840a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 shutdown_connections 2024-09-16T06:44:54.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.642+0000 7fc08b468700 1 -- 172.21.15.76:0/1158635587 wait complete. 2024-09-16T06:44:54.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:54.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:44:54.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: monmap: 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:54.792893+0000 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:54.792893+0000 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.792 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 1 Processor -- start 2024-09-16T06:44:54.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 1 -- start start 2024-09-16T06:44:54.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: init 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient(hunting): picked mon.noname-a con 0x7f35340973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient(hunting): picked mon.noname-c con 0x7f3534097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient(hunting): picked mon.noname-b con 0x7f35341400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534099350 con 0x7f35340973c0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353409f690 con 0x7f3534097eb0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534099bf0 con 0x7f35341400d0 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient(hunting): authenticate will time out at 1241.926256s 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f3532d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f35341400d0 0x7f35341424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43142/0 (socket says 172.21.15.76:43142) 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f3532d9d700 1 -- 172.21.15.76:0/3750421570 learned_addr learned my addr 172.21.15.76:0/3750421570 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743713510 0 0) 0x7f3534099bf0 con 0x7f35341400d0 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): my global_id is 125531 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35200035d0 con 0x7f35341400d0 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105396401 0 0) 0x7f353409f690 con 0x7f3534097eb0 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): my global_id is 174199 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3534099bf0 con 0x7f3534097eb0 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607645964 0 0) 0x7f3534099350 con 0x7f35340973c0 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient(hunting): my global_id is 154608 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353409f690 con 0x7f35340973c0 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939221805 0 0) 0x7f35200035d0 con 0x7f35341400d0 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 >> v1:172.21.15.189:6789/0 conn(0x7f3534097eb0 legacy=0x7f353413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 >> v1:172.21.15.76:6789/0 conn(0x7f35340973c0 legacy=0x7f35340977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 monclient: found mon.noname-b 2024-09-16T06:44:54.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3534098360 con 0x7f35341400d0 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f352c001610 con 0x7f35341400d0 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 5 monclient: authenticate success, global_id 125531 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: dump: 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.798 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f352c0018f0 con 0x7f35341400d0 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 1 -- 172.21.15.76:0/3750421570 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f352c001c80 con 0x7f35341400d0 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353359e700 4 set_mon_vals no callback set 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.790+0000 7f353159a700 10 monclient: dump: 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.800 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: shutdown 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- 172.21.15.76:0/3750421570 >> v1:172.21.15.76:6790/0 conn(0x7f35341400d0 legacy=0x7f35341424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- 172.21.15.76:0/3750421570 shutdown_connections 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- 172.21.15.76:0/3750421570 >> 172.21.15.76:0/3750421570 conn(0x7f35340a0030 msgr2=0x7f35340a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- 172.21.15.76:0/3750421570 shutdown_connections 2024-09-16T06:44:54.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- 172.21.15.76:0/3750421570 wait complete. 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: monmap: 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 Processor -- start 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- start start 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: init 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient(hunting): picked mon.noname-b con 0x7f35340973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient(hunting): picked mon.noname-a con 0x7f3534097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient(hunting): picked mon.noname-c con 0x7f35341400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534072660 con 0x7f3534097eb0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353413ec00 con 0x7f35340973c0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353413fb50 con 0x7f35341400d0 2024-09-16T06:44:54.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.794+0000 7f353359e700 10 monclient(hunting): authenticate will time out at 1241.931942s 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f3532d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f35341400d0 0x7f35341d62a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43152/0 (socket says 172.21.15.76:43152) 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f3532d9d700 1 -- 172.21.15.76:0/549415918 learned_addr learned my addr 172.21.15.76:0/549415918 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631666613 0 0) 0x7f353413fb50 con 0x7f35341400d0 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): my global_id is 125534 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35140035d0 con 0x7f35341400d0 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367670811 0 0) 0x7f3534072660 con 0x7f3534097eb0 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): my global_id is 154611 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353413fb50 con 0x7f3534097eb0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134452958 0 0) 0x7f353413ec00 con 0x7f35340973c0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): my global_id is 174202 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3534072660 con 0x7f35340973c0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694122830 0 0) 0x7f35140035d0 con 0x7f35341400d0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f353413ec00 con 0x7f35341400d0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525079690 0 0) 0x7f353413fb50 con 0x7f3534097eb0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35140035d0 con 0x7f3534097eb0 2024-09-16T06:44:54.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f352c001430 con 0x7f35341400d0 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): dump: 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.811 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3524001bb0 con 0x7f3534097eb0 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient(hunting): dump: 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3318906039 0 0) 0x7f35140035d0 con 0x7f3534097eb0 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.76:6790/0 conn(0x7f35341400d0 legacy=0x7f35341d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6789/0 conn(0x7f35340973c0 legacy=0x7f353413c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 monclient: found mon.a 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353409a540 con 0x7f3534097eb0 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 5 monclient: authenticate success, global_id 154611 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 10 monclient: _renew_subs 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353413dfc0 con 0x7f3534097eb0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 10 monclient: _renew_subs 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f353406c580 con 0x7f3534097eb0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3524001d50 con 0x7f3534097eb0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3524003bc0 con 0x7f3534097eb0 2024-09-16T06:44:54.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f3530d99700 4 set_mon_vals no callback set 2024-09-16T06:44:54.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 10 monclient: dump: 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.816 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f352400e630 con 0x7f3534097eb0 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.798+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3518006590 con 0x7f3518002830 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(170..170 src has 141..170) v4 ==== 6714+0+0 (unknown 4275550958 0 0) 0x7f3524042e20 con 0x7f3534097eb0 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f3532d9d700 10 monclient: get_auth_request con 0x7f3514036510 auth_method 0 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145623114 0 0) 0x7f3518006590 con 0x7f3518002830 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient(hunting): my global_id is 174205 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35140035d0 con 0x7f3518002830 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2279911164 0 0) 0x7f35140035d0 con 0x7f3518002830 2024-09-16T06:44:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3518006590 con 0x7f3518002830 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f351c001810 con 0x7f3518002830 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient: dump: 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.818 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3300587985 0 0) 0x7f3518006590 con 0x7f3518002830 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f351800a190 con 0x7f3518002830 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f351c001bf0 con 0x7f3518002830 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6789/0 conn(0x7f3518002830 legacy=0x7f3518004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.802+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6789/0 conn(0x7f3518002830 legacy=0x7f3518004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f353359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3518009e50 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799194585 0 0) 0x7f3518009e50 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient(hunting): my global_id is 174208 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3518006590 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1691146371 0 0) 0x7f3518006590 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3518009e50 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f351c003a30 con 0x7f3518009170 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.814+0000 7f352affd700 10 monclient: dump: 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2428982981 0 0) 0x7f3518009e50 con 0x7f3518009170 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f351800c340 con 0x7f3518009170 2024-09-16T06:44:54.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2814559769) 0x7f351c003c40 con 0x7f3518009170 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6789/0 conn(0x7f3518009170 legacy=0x7f3518009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f352affd700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6789/0 conn(0x7f3518009170 legacy=0x7f3518009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:44:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:54.835 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3514036510 legacy=0x7f35140389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 10 monclient: shutdown 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 >> v1:172.21.15.76:6789/0 conn(0x7f3534097eb0 legacy=0x7f353413cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f3532d9d700 1 -- 172.21.15.76:0/549415918 reap_dead start 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 shutdown_connections 2024-09-16T06:44:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 >> 172.21.15.76:0/549415918 conn(0x7f35340a0030 msgr2=0x7f35340a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 shutdown_connections 2024-09-16T06:44:54.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.818+0000 7f353359e700 1 -- 172.21.15.76:0/549415918 wait complete. 2024-09-16T06:44:54.837 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:54.837 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:44:54.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:44:54.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.860+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v171) v1 ==== 140+0+0 (unknown 1996196586 0 0) 0x7f458c012c40 con 0x7f4594138da0 2024-09-16T06:44:54.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.860+0000 7f457bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.860+0000 7f457bfff700 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: get_monmap_and_config 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: monmap: 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:54.956350+0000 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:54.956350+0000 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:54.954 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:54.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 1 Processor -- start 2024-09-16T06:44:54.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 1 -- start start 2024-09-16T06:44:54.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: init 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-c con 0x7f0d14131910 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-a con 0x7f0d141347f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-b con 0x7f0d14138300 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d14132400 con 0x7f0d141347f0 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1409fcd0 con 0x7f0d14131910 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1409b300 con 0x7f0d14138300 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d19dea700 10 monclient(hunting): authenticate will time out at 1242.091335s 2024-09-16T06:44:54.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d13fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0d141347f0 0x7f0d14136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38108/0 (socket says 172.21.15.76:38108) 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d195e9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0d14138300 0x7f0d1413a6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43156/0 (socket says 172.21.15.76:43156) 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d13fff700 1 -- 172.21.15.76:0/1094779553 learned_addr learned my addr 172.21.15.76:0/1094779553 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973822643 0 0) 0x7f0d14132400 con 0x7f0d141347f0 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): my global_id is 154614 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d000035d0 con 0x7f0d141347f0 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050778841 0 0) 0x7f0d1409b300 con 0x7f0d14138300 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 10 monclient(hunting): my global_id is 125537 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14132400 con 0x7f0d14138300 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.954+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452385983 0 0) 0x7f0d1409fcd0 con 0x7f0d14131910 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient(hunting): my global_id is 174211 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1409b300 con 0x7f0d14131910 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887216888 0 0) 0x7f0d000035d0 con 0x7f0d141347f0 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 >> v1:172.21.15.76:6790/0 conn(0x7f0d14138300 legacy=0x7f0d1413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 >> v1:172.21.15.189:6789/0 conn(0x7f0d14131910 legacy=0x7f0d14131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 monclient: found mon.noname-a 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1409b7e0 con 0x7f0d141347f0 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 5 monclient: authenticate success, global_id 154614 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0cfc001540 con 0x7f0d141347f0 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: dump: 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cfc001820 con 0x7f0d141347f0 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 1 -- 172.21.15.76:0/1094779553 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0cfc001bb0 con 0x7f0d141347f0 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 4 set_mon_vals no callback set 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d137fe700 10 monclient: dump: 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: shutdown 2024-09-16T06:44:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- 172.21.15.76:0/1094779553 >> v1:172.21.15.76:6789/0 conn(0x7f0d141347f0 legacy=0x7f0d14136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- 172.21.15.76:0/1094779553 shutdown_connections 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- 172.21.15.76:0/1094779553 >> 172.21.15.76:0/1094779553 conn(0x7f0d140a4430 msgr2=0x7f0d140a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- 172.21.15.76:0/1094779553 shutdown_connections 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- 172.21.15.76:0/1094779553 wait complete. 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: build_initial_monmap 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: monmap: 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 Processor -- start 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- start start 2024-09-16T06:44:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: init 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-c con 0x7f0d14131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-b con 0x7f0d141347f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient(hunting): picked mon.noname-a con 0x7f0d14138300 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:54.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1409d590 con 0x7f0d14138300 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1411b670 con 0x7f0d141347f0 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d14137e90 con 0x7f0d14131910 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d19dea700 10 monclient(hunting): authenticate will time out at 1242.095022s 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d18de8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0d14131910 0x7f0d1412dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43168/0 (socket says 172.21.15.76:43168) 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d18de8700 1 -- 172.21.15.76:0/2786937356 learned_addr learned my addr 172.21.15.76:0/2786937356 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652223868 0 0) 0x7f0d14137e90 con 0x7f0d14131910 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): my global_id is 125540 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf40035d0 con 0x7f0d14131910 2024-09-16T06:44:54.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755759989 0 0) 0x7f0d14132400 con 0x7f0d14138300 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): my global_id is 154617 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14137e90 con 0x7f0d14138300 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756776379 0 0) 0x7f0d1411b670 con 0x7f0d141347f0 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 10 monclient(hunting): my global_id is 174214 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14132400 con 0x7f0d141347f0 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309450589 0 0) 0x7f0d14137e90 con 0x7f0d14138300 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d1411b670 con 0x7f0d14138300 2024-09-16T06:44:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.958+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101400084 0 0) 0x7f0cf40035d0 con 0x7f0d14131910 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d14137e90 con 0x7f0d14131910 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c001a90 con 0x7f0d14138300 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): dump: 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d04001890 con 0x7f0d14131910 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): dump: 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.976 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705575245 0 0) 0x7f0d14132400 con 0x7f0d141347f0 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf40035d0 con 0x7f0d141347f0 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0cfc001500 con 0x7f0d141347f0 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient(hunting): dump: 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2089663534 0 0) 0x7f0d1411b670 con 0x7f0d14138300 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 >> v1:172.21.15.76:6790/0 conn(0x7f0d14131910 legacy=0x7f0d1412dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 >> v1:172.21.15.189:6789/0 conn(0x7f0d141347f0 legacy=0x7f0d141cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 monclient: found mon.a 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1409b0f0 con 0x7f0d14138300 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 5 monclient: authenticate success, global_id 154617 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _renew_subs 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14071860 con 0x7f0d14138300 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _renew_subs 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1412e330 con 0x7f0d14138300 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140678b0 con 0x7f0d14138300 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d0c002030 con 0x7f0d14138300 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c004db0 con 0x7f0d14138300 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 10 monclient: dump: 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d127fc700 4 set_mon_vals no callback set 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0d0c00f820 con 0x7f0d14138300 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(171..171 src has 141..171) v4 ==== 6717+0+0 (unknown 674616593 0 0) 0x7f0d0c043df0 con 0x7f0d14138300 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.962+0000 7f0d18de8700 10 monclient: get_auth_request con 0x7f0cf4036550 auth_method 0 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.966+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0d0c014580 con 0x7f0d14138300 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.966+0000 7f0d117fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:54.966+0000 7f0d117fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f459a08f700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f459a08f700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f459a08f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f45740053b0 con 0x7f4594138da0 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f457bfff700 1 -- 172.21.15.189:0/1076508548 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v171) v1 ==== 128+0+0 (unknown 1685209675 0 0) 0x7f458c0143c0 con 0x7f4594138da0 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f457bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.968+0000 7f457bfff700 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 >> v1:172.21.15.189:6812/3098810674 conn(0x7f45800362c0 legacy=0x7f4580038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 10 monclient: shutdown 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 >> v1:172.21.15.76:6790/0 conn(0x7f4594138da0 legacy=0x7f45941ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 shutdown_connections 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 >> 172.21.15.189:0/1076508548 conn(0x7f45940a03e0 msgr2=0x7f459409a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 shutdown_connections 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:54.972+0000 7f459a08f700 1 -- 172.21.15.189:0/1076508548 wait complete. 2024-09-16T06:44:54.983 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'fooo' already exists 2024-09-16T06:44:54.984 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move fooo root=default 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 10 monclient: get_monmap_and_config 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 10 monclient: build_initial_monmap 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 10 monclient: monmap: 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:55.091748+0000 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:55.091748+0000 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:55.091 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.092 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:55.092 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:55.092 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:55.092 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 1 Processor -- start 2024-09-16T06:44:55.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 1 -- start start 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.088+0000 7fcf5e9e6700 10 monclient: init 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-a con 0x7fcf581352c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-b con 0x7fcf58135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-c con 0x7fcf580a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5809b050 con 0x7fcf581352c0 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5806ab20 con 0x7fcf580a2030 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5806a840 con 0x7fcf58135db0 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient(hunting): authenticate will time out at 1247.604552s 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e1e5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcf580a2030 0x7fcf5813df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46950/0 (socket says 172.21.15.189:46950) 2024-09-16T06:44:55.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e1e5700 1 -- 172.21.15.189:0/936753829 learned_addr learned my addr 172.21.15.189:0/936753829 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829054739 0 0) 0x7fcf5806ab20 con 0x7fcf580a2030 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): my global_id is 174217 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf440035d0 con 0x7fcf580a2030 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663621821 0 0) 0x7fcf5809b050 con 0x7fcf581352c0 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): my global_id is 154620 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5806ab20 con 0x7fcf581352c0 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652097798 0 0) 0x7fcf5806a840 con 0x7fcf58135db0 2024-09-16T06:44:55.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient(hunting): my global_id is 125543 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5809b050 con 0x7fcf58135db0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655256371 0 0) 0x7fcf5806ab20 con 0x7fcf581352c0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 >> v1:172.21.15.76:6790/0 conn(0x7fcf58135db0 legacy=0x7fcf5813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 >> v1:172.21.15.189:6789/0 conn(0x7fcf580a2030 legacy=0x7fcf5813df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: _finish_hunting 0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 monclient: found mon.noname-a 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5806c1b0 con 0x7fcf581352c0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: _finish_auth 0 2024-09-16T06:44:55.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: _check_auth_tickets 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 5 monclient: authenticate success, global_id 154620 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcf54001970 con 0x7fcf581352c0 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: dump: 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.098 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf54001c10 con 0x7fcf581352c0 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 4 set_mon_vals no callback set 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 1 -- 172.21.15.189:0/936753829 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcf54001fe0 con 0x7fcf581352c0 2024-09-16T06:44:55.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5c9e2700 10 monclient: dump: 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.100 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: shutdown 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/936753829 >> v1:172.21.15.76:6789/0 conn(0x7fcf581352c0 legacy=0x7fcf581356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/936753829 shutdown_connections 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/936753829 >> 172.21.15.189:0/936753829 conn(0x7fcf580a4550 msgr2=0x7fcf580a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/936753829 shutdown_connections 2024-09-16T06:44:55.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/936753829 wait complete. 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: build_initial_monmap 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.092+0000 7fcf5e9e6700 10 monclient: monmap: 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:55.105 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 Processor -- start 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- start start 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: init 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-b con 0x7fcf580a2030 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-a con 0x7fcf581352c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient(hunting): picked mon.noname-c con 0x7fcf58135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:55.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5813b050 con 0x7fcf581352c0 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5809e820 con 0x7fcf580a2030 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf580a6730 con 0x7fcf58135db0 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient(hunting): authenticate will time out at 1247.608822s 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5d9e4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcf580a2030 0x7fcf5813e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46964/0 (socket says 172.21.15.189:46964) 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5d9e4700 1 -- 172.21.15.189:0/2038933 learned_addr learned my addr 172.21.15.189:0/2038933 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90692026 0 0) 0x7fcf5813b050 con 0x7fcf581352c0 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): my global_id is 154623 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c0035d0 con 0x7fcf581352c0 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626424900 0 0) 0x7fcf5809e820 con 0x7fcf580a2030 2024-09-16T06:44:55.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): my global_id is 174220 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5813b050 con 0x7fcf580a2030 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563117657 0 0) 0x7fcf580a6730 con 0x7fcf58135db0 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): my global_id is 125546 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5809e820 con 0x7fcf58135db0 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3977588963 0 0) 0x7fcf3c0035d0 con 0x7fcf581352c0 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf580a6730 con 0x7fcf581352c0 2024-09-16T06:44:55.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcf480018b0 con 0x7fcf581352c0 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): dump: 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439874800 0 0) 0x7fcf5813b050 con 0x7fcf580a2030 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf3c0035d0 con 0x7fcf580a2030 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcf54001870 con 0x7fcf580a2030 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient(hunting): dump: 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:55.110 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1341257717 0 0) 0x7fcf580a6730 con 0x7fcf581352c0 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 >> v1:172.21.15.76:6790/0 conn(0x7fcf58135db0 legacy=0x7fcf581d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 >> v1:172.21.15.189:6789/0 conn(0x7fcf580a2030 legacy=0x7fcf5813e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 monclient: found mon.a 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf580a6200 con 0x7fcf581352c0 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:55.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 5 monclient: authenticate success, global_id 154623 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _renew_subs 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5813a7b0 con 0x7fcf581352c0 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf480036c0 con 0x7fcf581352c0 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcf480039d0 con 0x7fcf581352c0 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 10 monclient: dump: 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.112 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _renew_subs 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5813ecb0 con 0x7fcf581352c0 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4ffff700 4 set_mon_vals no callback set 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf58066d00 con 0x7fcf581352c0 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.096+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcf4800e440 con 0x7fcf581352c0 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.100+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(171..171 src has 141..171) v4 ==== 6717+0+0 (unknown 674616593 0 0) 0x7fcf48042bb0 con 0x7fcf581352c0 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.100+0000 7fcf5d9e4700 10 monclient: get_auth_request con 0x7fcf3c036550 auth_method 0 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.108+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcf480132e0 con 0x7fcf581352c0 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.108+0000 7fcf4e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:55.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.108+0000 7fcf4e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:55.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d19dea700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:44:55.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d19dea700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:44:55.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d19dea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0d1412ff60 con 0x7f0d14138300 2024-09-16T06:44:55.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d117fa700 1 -- 172.21.15.76:0/2786937356 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0d0c012c20 con 0x7f0d14138300 2024-09-16T06:44:55.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d117fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:44:55.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.210+0000 7f0d117fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:55.214 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1589320288,config=1097430493,health=2428946659,kv=4135318919,logm=3319776241}) 2024-09-16T06:44:55.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=3324874918,config=1097430493,health=3235778646,kv=1001081567,logm=1716336336}) 2024-09-16T06:44:55.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1589320288,config=1097430493,health=2428946659,kv=4135318919,logm=3319776241}) 2024-09-16T06:44:55.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=1,health=11,kv=12,logm=54} crc {auth=2419184296,config=1228700967,health=2449408819,kv=4212347341,logm=3223381286}) 2024-09-16T06:44:55.216 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.216 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2762414115}) 2024-09-16T06:44:55.216 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286505058}) 2024-09-16T06:44:55.217 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.217 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2762414115}) 2024-09-16T06:44:55.217 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792704319}) 2024-09-16T06:44:55.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0cf4036550 legacy=0x7f0cf4038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f0d19dea700 10 monclient: shutdown 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f0d19dea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.214+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 >> v1:172.21.15.76:6789/0 conn(0x7f0d14138300 legacy=0x7f0d141d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 shutdown_connections 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 >> 172.21.15.76:0/2786937356 conn(0x7f0d140a4430 msgr2=0x7f0d140a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 shutdown_connections 2024-09-16T06:44:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f0d19dea700 1 -- 172.21.15.76:0/2786937356 wait complete. 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078346134}) 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146032175}) 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078346134}) 2024-09-16T06:44:55.219 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271759044}) 2024-09-16T06:44:55.220 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.220 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783694458}) 2024-09-16T06:44:55.220 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766963505}) 2024-09-16T06:44:55.220 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.220 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783694458}) 2024-09-16T06:44:55.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.218+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350169541}) 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2722006126}) 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641309105}) 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2722006126}) 2024-09-16T06:44:55.222 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699022022}) 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232922555}) 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373892343}) 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232922555}) 2024-09-16T06:44:55.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377710663}) 2024-09-16T06:44:55.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930221526}) 2024-09-16T06:44:55.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848432002}) 2024-09-16T06:44:55.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.222+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930221526}) 2024-09-16T06:44:55.226 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699460160}) 2024-09-16T06:44:55.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054944204}) 2024-09-16T06:44:55.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379989353}) 2024-09-16T06:44:55.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.228 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054944204}) 2024-09-16T06:44:55.228 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=833237471}) 2024-09-16T06:44:55.228 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.228 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038938398}) 2024-09-16T06:44:55.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833819726}) 2024-09-16T06:44:55.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038938398}) 2024-09-16T06:44:55.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068378704}) 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016713278}) 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676951202}) 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.226+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016713278}) 2024-09-16T06:44:55.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868364090}) 2024-09-16T06:44:55.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554212742}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449262160}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554212742}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3078354701}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4060015554}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174166814}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4060015554}) 2024-09-16T06:44:55.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=144941428}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774196619}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653090231}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774196619}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.230+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505920852}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367627485}) 2024-09-16T06:44:55.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551267965}) 2024-09-16T06:44:55.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367627485}) 2024-09-16T06:44:55.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=9851224}) 2024-09-16T06:44:55.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367446546}) 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354074413}) 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367446546}) 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996141760}) 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939663379}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4084964981}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939663379}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2271226719}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=561148305}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267520002}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=561148305}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.234+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017884051}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522863702}) 2024-09-16T06:44:55.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463881030}) 2024-09-16T06:44:55.240 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.240 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522863702}) 2024-09-16T06:44:55.240 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692572348}) 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236306908}) 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087446990}) 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236306908}) 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407363407}) 2024-09-16T06:44:55.241 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2072778795}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329527864}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2072778795}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.238+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549374409}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970563565}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341544398}) 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970563565}) 2024-09-16T06:44:55.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388062470}) 2024-09-16T06:44:55.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689980809}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243362329}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689980809}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470804455}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657779834}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734870057}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657779834}) 2024-09-16T06:44:55.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2435987787}) 2024-09-16T06:44:55.245 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517727520}) 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032075812}) 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517727520}) 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.242+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183330288}) 2024-09-16T06:44:55.246 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480357591}) 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595025966}) 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480357591}) 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769853394}) 2024-09-16T06:44:55.247 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.248 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067029561}) 2024-09-16T06:44:55.248 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707638279}) 2024-09-16T06:44:55.248 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.248 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067029561}) 2024-09-16T06:44:55.248 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828260105}) 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619147697}) 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094691816}) 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619147697}) 2024-09-16T06:44:55.249 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.246+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=673520377}) 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564522722}) 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868014635}) 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564522722}) 2024-09-16T06:44:55.250 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977713364}) 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2058587020,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1624244666,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1782621429,osd_snap=553062916,osdmap=1025894205}) 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=38} crc {logm=670660328,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1695515033,mon_config_key=1714515979,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1782621429,osd_snap=2033388227,osdmap=1348312906}) 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2058587020,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1624244666,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=1782621429,osd_snap=553062916,osdmap=1025894205}) 2024-09-16T06:44:55.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.250+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=38} crc {logm=2058587020,mdsmap=2394902515,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4025871340,mon_config_key=263294865,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=1782621429,osd_snap=2033388228,osdmap=3246504300}) 2024-09-16T06:44:55.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=785116269}) 2024-09-16T06:44:55.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=1313536039}) 2024-09-16T06:44:55.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:44:55.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=785116269}) 2024-09-16T06:44:55.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:44:55.258+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=3766776476}) 2024-09-16T06:44:55.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.284+0000 7fcf5e9e6700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.284+0000 7fcf5e9e6700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.284+0000 7fcf5e9e6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.284+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fcf5813ef60 con 0x7fcf581352c0 2024-09-16T06:44:55.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.856+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v172) v1 ==== 173+0+0 (unknown 622584503 0 0) 0x7fcf48013100 con 0x7fcf581352c0 2024-09-16T06:44:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.856+0000 7fcf4e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.856+0000 7fcf4e7fc700 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T06:44:55.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.968+0000 7fcf5e9e6700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.968+0000 7fcf5e9e6700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.968+0000 7fcf5e9e6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:55.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.968+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fcf580008d0 con 0x7fcf581352c0 2024-09-16T06:44:55.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf4e7fc700 1 -- 172.21.15.189:0/2038933 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v172) v1 ==== 183+0+0 (unknown 1074569446 0 0) 0x7fcf48013e40 con 0x7fcf581352c0 2024-09-16T06:44:55.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf4e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T06:44:55.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf4e7fc700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T06:44:55.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcf3c036550 legacy=0x7fcf3c038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf5e9e6700 10 monclient: shutdown 2024-09-16T06:44:55.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf5e9e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:55.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.972+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 >> v1:172.21.15.76:6789/0 conn(0x7fcf581352c0 legacy=0x7fcf581cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.976+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 shutdown_connections 2024-09-16T06:44:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.976+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 >> 172.21.15.189:0/2038933 conn(0x7fcf580a4550 msgr2=0x7fcf580a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.976+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 shutdown_connections 2024-09-16T06:44:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:55.976+0000 7fcf5e9e6700 1 -- 172.21.15.189:0/2038933 wait complete. 2024-09-16T06:44:55.977 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T06:44:55.991 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket barr rack 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: get_monmap_and_config 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: monmap: 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:56.101609+0000 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:56.101609+0000 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.101 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 1 Processor -- start 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 1 -- start start 2024-09-16T06:44:56.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: init 2024-09-16T06:44:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-a con 0x7f259009b1d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-c con 0x7f259009bcc0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-b con 0x7f2590138b80 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259009d470 con 0x7f259009b1d0 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2590066610 con 0x7f259009bcc0 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259009c4c0 con 0x7f2590138b80 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f25966d2700 10 monclient(hunting): authenticate will time out at 1248.615955s 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.100+0000 7f2587fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f259009bcc0 0x7f2590137460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46976/0 (socket says 172.21.15.189:46976) 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f2587fff700 1 -- 172.21.15.189:0/2234827812 learned_addr learned my addr 172.21.15.189:0/2234827812 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104696622 0 0) 0x7f2590066610 con 0x7f259009bcc0 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): my global_id is 174223 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c0035d0 con 0x7f259009bcc0 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447968553 0 0) 0x7f259009d470 con 0x7f259009b1d0 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): my global_id is 154626 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590066610 con 0x7f259009b1d0 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947251050 0 0) 0x7f259009c4c0 con 0x7f2590138b80 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient(hunting): my global_id is 125549 2024-09-16T06:44:56.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259009d470 con 0x7f2590138b80 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848198137 0 0) 0x7f257c0035d0 con 0x7f259009bcc0 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 >> v1:172.21.15.76:6790/0 conn(0x7f2590138b80 legacy=0x7f259013af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 >> v1:172.21.15.76:6789/0 conn(0x7f259009b1d0 legacy=0x7f259009b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 monclient: found mon.noname-c 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25900669a0 con 0x7f259009bcc0 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 5 monclient: authenticate success, global_id 174223 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2578001610 con 0x7f259009bcc0 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: dump: 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.110 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25780018f0 con 0x7f259009bcc0 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:56.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 4 set_mon_vals no callback set 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 1 -- 172.21.15.189:0/2234827812 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2578001c80 con 0x7f259009bcc0 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f258f7fe700 10 monclient: dump: 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.112 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 10 monclient: shutdown 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 1 -- 172.21.15.189:0/2234827812 >> v1:172.21.15.189:6789/0 conn(0x7f259009bcc0 legacy=0x7f2590137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 1 -- 172.21.15.189:0/2234827812 shutdown_connections 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 1 -- 172.21.15.189:0/2234827812 >> 172.21.15.189:0/2234827812 conn(0x7f25900a0760 msgr2=0x7f25900a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 1 -- 172.21.15.189:0/2234827812 shutdown_connections 2024-09-16T06:44:56.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 1 -- 172.21.15.189:0/2234827812 wait complete. 2024-09-16T06:44:56.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.104+0000 7f25966d2700 10 monclient: monmap: 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:56.118 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 Processor -- start 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- start start 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: init 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-b con 0x7f259009b1d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-a con 0x7f259009bcc0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient(hunting): picked mon.noname-c con 0x7f2590138b80 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900719c0 con 0x7f259009bcc0 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259011f5d0 con 0x7f259009b1d0 2024-09-16T06:44:56.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2590120a10 con 0x7f2590138b80 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient(hunting): authenticate will time out at 1248.622000s 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f259009b1d0 0x7f259011de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:46992/0 (socket says 172.21.15.189:46992) 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258ffff700 1 -- 172.21.15.189:0/3441560117 learned_addr learned my addr 172.21.15.189:0/3441560117 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419337344 0 0) 0x7f259011f5d0 con 0x7f259009b1d0 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): my global_id is 174226 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25700035d0 con 0x7f259009b1d0 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329615216 0 0) 0x7f2590120a10 con 0x7f2590138b80 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): my global_id is 125552 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259011f5d0 con 0x7f2590138b80 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109346089 0 0) 0x7f25900719c0 con 0x7f259009bcc0 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): my global_id is 154629 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590120a10 con 0x7f259009bcc0 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235325938 0 0) 0x7f25700035d0 con 0x7f259009b1d0 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25900719c0 con 0x7f259009b1d0 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f25800018f0 con 0x7f259009b1d0 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232221624 0 0) 0x7f259011f5d0 con 0x7f2590138b80 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25700035d0 con 0x7f2590138b80 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2588001c30 con 0x7f2590138b80 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271450468 0 0) 0x7f2590120a10 con 0x7f259009bcc0 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f259011f5d0 con 0x7f259009bcc0 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2578001430 con 0x7f259009bcc0 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.124 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3577340869 0 0) 0x7f25900719c0 con 0x7f259009b1d0 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 >> v1:172.21.15.76:6790/0 conn(0x7f2590138b80 legacy=0x7f25901cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 >> v1:172.21.15.76:6789/0 conn(0x7f259009bcc0 legacy=0x7f259011e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 monclient: found mon.b 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259011ec90 con 0x7f259009b1d0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 5 monclient: authenticate success, global_id 174226 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _renew_subs 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25900a0d40 con 0x7f259009b1d0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _renew_subs 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2590120730 con 0x7f259009b1d0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2580001c80 con 0x7f259009b1d0 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2580003b10 con 0x7f259009b1d0 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258effd700 4 set_mon_vals no callback set 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.108+0000 7f258d7fa700 10 monclient: dump: 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2580003c70 con 0x7f259009b1d0 2024-09-16T06:44:56.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f25966d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f25966d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f25966d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2574005100 con 0x7f259009b1d0 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(172..172 src has 146..172) v4 ==== 6905+0+0 (unknown 3687040836 0 0) 0x7f2580043d10 con 0x7f259009b1d0 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.112+0000 7f2587fff700 10 monclient: get_auth_request con 0x7f2570036460 auth_method 0 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.120+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f25800146f0 con 0x7f259009b1d0 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.120+0000 7f258d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.120+0000 7f258d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:56.237 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:44:56.238 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:44:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.292+0000 7f25966d2700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.292+0000 7f25966d2700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.292+0000 7f25966d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.292+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7f2574005930 con 0x7f259009b1d0 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.354+0000 7f5bbdeed700 10 monclient: get_monmap_and_config 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.354+0000 7f5bbdeed700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.354+0000 7f5bbdeed700 10 monclient: monmap: 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:56.359353+0000 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:56.359353+0000 2024-09-16T06:44:56.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:56.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:56.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 Processor -- start 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- start start 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient: init 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb806c280 con 0x7f5bb806d480 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb806bfa0 con 0x7f5bb806cf50 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb809ace0 con 0x7f5bb8132330 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient(hunting): authenticate will time out at 1243.492938s 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5bb806d480 0x7f5bb813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38140/0 (socket says 172.21.15.76:38140) 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb7fff700 1 -- 172.21.15.76:0/3591160538 learned_addr learned my addr 172.21.15.76:0/3591160538 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289843883 0 0) 0x7f5bb806c280 con 0x7f5bb806d480 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): my global_id is 154632 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba80035d0 con 0x7f5bb806d480 2024-09-16T06:44:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529004882 0 0) 0x7f5bb809ace0 con 0x7f5bb8132330 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): my global_id is 125555 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb806c280 con 0x7f5bb8132330 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624866121 0 0) 0x7f5bb806bfa0 con 0x7f5bb806cf50 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient(hunting): my global_id is 174229 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb809ace0 con 0x7f5bb806cf50 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884499366 0 0) 0x7f5bb806c280 con 0x7f5bb8132330 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 >> v1:172.21.15.189:6789/0 conn(0x7f5bb806cf50 legacy=0x7f5bb813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 >> v1:172.21.15.76:6789/0 conn(0x7f5bb806d480 legacy=0x7f5bb813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 monclient: found mon.noname-b 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb809ae70 con 0x7f5bb8132330 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 5 monclient: authenticate success, global_id 125555 2024-09-16T06:44:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bac0015f0 con 0x7f5bb8132330 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: dump: 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bac001910 con 0x7f5bb8132330 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 1 -- 172.21.15.76:0/3591160538 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bac001b00 con 0x7f5bb8132330 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 4 set_mon_vals no callback set 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bb77fe700 10 monclient: dump: 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 10 monclient: shutdown 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- 172.21.15.76:0/3591160538 >> v1:172.21.15.76:6790/0 conn(0x7f5bb8132330 legacy=0x7f5bb8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- 172.21.15.76:0/3591160538 shutdown_connections 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- 172.21.15.76:0/3591160538 >> 172.21.15.76:0/3591160538 conn(0x7f5bb80a41c0 msgr2=0x7f5bb80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- 172.21.15.76:0/3591160538 shutdown_connections 2024-09-16T06:44:56.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.358+0000 7f5bbdeed700 1 -- 172.21.15.76:0/3591160538 wait complete. 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: monmap: 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:56.371 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 Processor -- start 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- start start 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: init 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb8132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8131690 con 0x7f5bb806d480 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb809e8b0 con 0x7f5bb8132330 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb812e0b0 con 0x7f5bb806cf50 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient(hunting): authenticate will time out at 1243.497575s 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbceeb700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5bb806cf50 0x7f5bb812d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43180/0 (socket says 172.21.15.76:43180) 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbceeb700 1 -- 172.21.15.76:0/1747033029 learned_addr learned my addr 172.21.15.76:0/1747033029 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147061759 0 0) 0x7f5ba80035d0 con 0x7f5bb806d480 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): my global_id is 154635 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c0035d0 con 0x7f5bb806d480 2024-09-16T06:44:56.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258678639 0 0) 0x7f5bb812e0b0 con 0x7f5bb806cf50 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): my global_id is 125558 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba80035d0 con 0x7f5bb806cf50 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746172135 0 0) 0x7f5bb809e8b0 con 0x7f5bb8132330 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): my global_id is 174232 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb812e0b0 con 0x7f5bb8132330 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481112132 0 0) 0x7f5b9c0035d0 con 0x7f5bb806d480 2024-09-16T06:44:56.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb809e8b0 con 0x7f5bb806d480 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ba4001730 con 0x7f5bb806d480 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): dump: 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.375 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199601349 0 0) 0x7f5bb812e0b0 con 0x7f5bb8132330 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b9c0035d0 con 0x7f5bb8132330 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571612894 0 0) 0x7f5ba80035d0 con 0x7f5bb806cf50 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb812e0b0 con 0x7f5bb806cf50 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939069201 0 0) 0x7f5bb809e8b0 con 0x7f5bb806d480 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 >> v1:172.21.15.76:6790/0 conn(0x7f5bb806cf50 legacy=0x7f5bb812d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 >> v1:172.21.15.189:6789/0 conn(0x7f5bb8132330 legacy=0x7f5bb81d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 monclient: found mon.a 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:56.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb81310d0 con 0x7f5bb806d480 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 5 monclient: authenticate success, global_id 154635 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _renew_subs 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb8130d50 con 0x7f5bb806d480 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _renew_subs 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb812ee80 con 0x7f5bb806d480 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ba4001c80 con 0x7f5bb806d480 2024-09-16T06:44:56.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ba4004a30 con 0x7f5bb806d480 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb6ffd700 4 set_mon_vals no callback set 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.362+0000 7f5bb57fa700 10 monclient: dump: 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5ba400f4a0 con 0x7f5bb806d480 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(172..172 src has 146..172) v4 ==== 6905+0+0 (unknown 3687040836 0 0) 0x7f5ba4043ba0 con 0x7f5bb806d480 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bbdeed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bbdeed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bbdeed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b98005100 con 0x7f5bb806d480 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.366+0000 7f5bbceeb700 10 monclient: get_auth_request con 0x7f5b9c036430 auth_method 0 2024-09-16T06:44:56.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.374+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5ba4014270 con 0x7f5bb806d480 2024-09-16T06:44:56.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.374+0000 7f5bb57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.374+0000 7f5bb57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bbdeed700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:44:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bbdeed700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bbdeed700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5b98005930 con 0x7f5bb806d480 2024-09-16T06:44:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bb57fa700 1 -- 172.21.15.76:0/1747033029 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2756589783) 0x7f5ba4012910 con 0x7f5bb806d480 2024-09-16T06:44:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bb57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.610+0000 7f5bb57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:56.613 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5b9c036430 legacy=0x7f5b9c0388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 10 monclient: shutdown 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 >> v1:172.21.15.76:6789/0 conn(0x7f5bb806d480 legacy=0x7f5bb812d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 shutdown_connections 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 >> 172.21.15.76:0/1747033029 conn(0x7f5bb80a41c0 msgr2=0x7f5bb809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 shutdown_connections 2024-09-16T06:44:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.614+0000 7f5bbdeed700 1 -- 172.21.15.76:0/1747033029 wait complete. 2024-09-16T06:44:56.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:44:56.633 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:44:56.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:44:56.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: get_monmap_and_config 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: monmap: 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:56.745408+0000 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:56.745408+0000 2024-09-16T06:44:56.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.744 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.744 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.744 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:56.744 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:56.744 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 1 Processor -- start 2024-09-16T06:44:56.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 1 -- start start 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: init 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-b con 0x7fa9e0097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-c con 0x7fa9e0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-a con 0x7fa9e0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e0098990 con 0x7fa9e0139500 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e0097f40 con 0x7fa9e0099090 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e009d610 con 0x7fa9e0097b60 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient(hunting): authenticate will time out at 1243.878624s 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e5b54700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa9e0139500 0x7fa9e013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38154/0 (socket says 172.21.15.76:38154) 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e5b54700 1 -- 172.21.15.76:0/2369031778 learned_addr learned my addr 172.21.15.76:0/2369031778 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e5353700 1 --1- 172.21.15.76:0/2369031778 >> v1:172.21.15.76:6790/0 conn(0x7fa9e0097b60 0x7fa9e009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43184/0 (socket says 172.21.15.76:43184) 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000468819 0 0) 0x7fa9e009d610 con 0x7fa9e0097b60 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): my global_id is 125561 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc0035d0 con 0x7fa9e0097b60 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053259586 0 0) 0x7fa9e0098990 con 0x7fa9e0139500 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient(hunting): my global_id is 154638 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e009d610 con 0x7fa9e0139500 2024-09-16T06:44:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849711528 0 0) 0x7fa9cc0035d0 con 0x7fa9e0097b60 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 >> v1:172.21.15.189:6789/0 conn(0x7fa9e0099090 legacy=0x7fa9e009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 >> v1:172.21.15.76:6789/0 conn(0x7fa9e0139500 legacy=0x7fa9e013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 monclient: found mon.noname-b 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00986b0 con 0x7fa9e0097b60 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 5 monclient: authenticate success, global_id 125561 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9d0001610 con 0x7fa9e0097b60 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: dump: 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9d00018f0 con 0x7fa9e0097b60 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 1 -- 172.21.15.76:0/2369031778 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9d0001c80 con 0x7fa9e0097b60 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9e6355700 4 set_mon_vals no callback set 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.742+0000 7fa9d7fff700 10 monclient: dump: 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.751 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: shutdown 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/2369031778 >> v1:172.21.15.76:6790/0 conn(0x7fa9e0097b60 legacy=0x7fa9e009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/2369031778 shutdown_connections 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/2369031778 >> 172.21.15.76:0/2369031778 conn(0x7fa9e00a0030 msgr2=0x7fa9e00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/2369031778 shutdown_connections 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/2369031778 wait complete. 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: monmap: 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:56.754 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 Processor -- start 2024-09-16T06:44:56.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- start start 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: init 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-b con 0x7fa9e0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-c con 0x7fa9e0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient(hunting): picked mon.noname-a con 0x7fa9e0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e006e100 con 0x7fa9e0139500 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e006fd70 con 0x7fa9e0097b60 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e0070330 con 0x7fa9e0099090 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient(hunting): authenticate will time out at 1243.882565s 2024-09-16T06:44:56.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e4b52700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa9e0099090 0x7fa9e01cbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43186/0 (socket says 172.21.15.76:43186) 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e4b52700 1 -- 172.21.15.76:0/3471890880 learned_addr learned my addr 172.21.15.76:0/3471890880 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712522682 0 0) 0x7fa9e0070330 con 0x7fa9e0099090 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): my global_id is 125564 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c40035d0 con 0x7fa9e0099090 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886125041 0 0) 0x7fa9e006e100 con 0x7fa9e0139500 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): my global_id is 154641 2024-09-16T06:44:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e0070330 con 0x7fa9e0139500 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3730707505 0 0) 0x7fa9c40035d0 con 0x7fa9e0099090 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e006e100 con 0x7fa9e0099090 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9dc001890 con 0x7fa9e0099090 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538545792 0 0) 0x7fa9e006fd70 con 0x7fa9e0097b60 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): my global_id is 174238 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c40035d0 con 0x7fa9e0097b60 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3551383586 0 0) 0x7fa9e0070330 con 0x7fa9e0139500 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e006fd70 con 0x7fa9e0139500 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9d8001c30 con 0x7fa9e0139500 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient(hunting): dump: 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 69291881 0 0) 0x7fa9e006e100 con 0x7fa9e0099090 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.189:6789/0 conn(0x7fa9e0097b60 legacy=0x7fa9e00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6789/0 conn(0x7fa9e0139500 legacy=0x7fa9e01cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 monclient: found mon.c 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0070050 con 0x7fa9e0099090 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 5 monclient: authenticate success, global_id 125564 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _renew_subs 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e006eda0 con 0x7fa9e0099090 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _renew_subs 2024-09-16T06:44:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.746+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e0138e30 con 0x7fa9e0099090 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9dc0036c0 con 0x7fa9e0099090 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d77fe700 4 set_mon_vals no callback set 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9dc003a50 con 0x7fa9e0099090 2024-09-16T06:44:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: dump: 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9e6355700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa9c0006590 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa9dc003cb0 con 0x7fa9e0099090 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730196025 0 0) 0x7fa9c0006590 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient(hunting): my global_id is 154644 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e006e100 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(172..172 src has 146..172) v4 ==== 6905+0+0 (unknown 3687040836 0 0) 0x7fa9dc043e60 con 0x7fa9e0099090 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9e5b54700 10 monclient: get_auth_request con 0x7fa9c4036450 auth_method 0 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4251064769 0 0) 0x7fa9e006e100 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c0006590 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9d0001cc0 con 0x7fa9c0002830 2024-09-16T06:44:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: dump: 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 593579485 0 0) 0x7fa9c0006590 con 0x7fa9c0002830 2024-09-16T06:44:56.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9c000a190 con 0x7fa9c0002830 2024-09-16T06:44:56.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa9d00015d0 con 0x7fa9c0002830 2024-09-16T06:44:56.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:56.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6789/0 conn(0x7fa9c0002830 legacy=0x7fa9c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.750+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6789/0 conn(0x7fa9c0002830 legacy=0x7fa9c0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:56.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.758+0000 7fa9e6355700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:56.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.758+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa9c0009e50 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966236164 0 0) 0x7fa9c0009e50 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient(hunting): my global_id is 154647 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0006590 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4231712104 0 0) 0x7fa9c0006590 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c0009e50 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9d0004d40 con 0x7fa9c0009170 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:56.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient: dump: 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2787392228 0 0) 0x7fa9c0009e50 con 0x7fa9c0009170 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa9c000c340 con 0x7fa9c0009170 2024-09-16T06:44:56.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1356534758) 0x7fa9d0005060 con 0x7fa9c0009170 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6789/0 conn(0x7fa9c0009170 legacy=0x7fa9c0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9d5ffb700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6789/0 conn(0x7fa9c0009170 legacy=0x7fa9c0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:56.777 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:56.778 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.779 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:56.781 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.782 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:56.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:56.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:44:56.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:56.790 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa9c4036450 legacy=0x7fa9c4038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 10 monclient: shutdown 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 >> v1:172.21.15.76:6790/0 conn(0x7fa9e0099090 legacy=0x7fa9e01cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e5b54700 1 -- 172.21.15.76:0/3471890880 reap_dead start 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 shutdown_connections 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 >> 172.21.15.76:0/3471890880 conn(0x7fa9e00a0030 msgr2=0x7fa9e00a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 shutdown_connections 2024-09-16T06:44:56.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.762+0000 7fa9e6355700 1 -- 172.21.15.76:0/3471890880 wait complete. 2024-09-16T06:44:56.792 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:56.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:44:56.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:44:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.880+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v173) v1 ==== 140+0+0 (unknown 3712102420 0 0) 0x7f25800195b0 con 0x7f259009b1d0 2024-09-16T06:44:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.880+0000 7f258d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.880+0000 7f258d7fa700 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-16T06:44:56.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.898+0000 7fb4f0d85700 10 monclient: get_monmap_and_config 2024-09-16T06:44:56.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.898+0000 7fb4f0d85700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.898+0000 7fb4f0d85700 10 monclient: monmap: 2024-09-16T06:44:56.901 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.901 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:56.903755+0000 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:56.903755+0000 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:56.902 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 1 Processor -- start 2024-09-16T06:44:56.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 1 -- start start 2024-09-16T06:44:56.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient: init 2024-09-16T06:44:56.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-a con 0x7fb4ec132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-c con 0x7fb4ec06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-b con 0x7fb4ec06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec06c280 con 0x7fb4ec132330 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec06bfa0 con 0x7fb4ec06d480 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4eb7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb4ec132330 0x7fb4ec132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38184/0 (socket says 172.21.15.76:38184) 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec09ace0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4eb7fe700 1 -- 172.21.15.76:0/4248565139 learned_addr learned my addr 172.21.15.76:0/4248565139 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient(hunting): authenticate will time out at 1244.038164s 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280170910 0 0) 0x7fb4ec09ace0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): my global_id is 125567 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d40035d0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651559048 0 0) 0x7fb4ec06c280 con 0x7fb4ec132330 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): my global_id is 154650 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec09ace0 con 0x7fb4ec132330 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854507280 0 0) 0x7fb4ec06bfa0 con 0x7fb4ec06d480 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient(hunting): my global_id is 174241 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec06c280 con 0x7fb4ec06d480 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3247292114 0 0) 0x7fb4d40035d0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 >> v1:172.21.15.189:6789/0 conn(0x7fb4ec06d480 legacy=0x7fb4ec13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 >> v1:172.21.15.76:6789/0 conn(0x7fb4ec132330 legacy=0x7fb4ec132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 monclient: found mon.noname-b 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec09ae70 con 0x7fb4ec06cf50 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e4001990 con 0x7fb4ec06cf50 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: dump: 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.909 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 5 monclient: authenticate success, global_id 125567 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e4001bb0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 1 -- 172.21.15.76:0/4248565139 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e4001da0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 4 set_mon_vals no callback set 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4ea7fc700 10 monclient: dump: 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.902+0000 7fb4f0d85700 10 monclient: shutdown 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- 172.21.15.76:0/4248565139 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec06cf50 legacy=0x7fb4ec13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- 172.21.15.76:0/4248565139 shutdown_connections 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- 172.21.15.76:0/4248565139 >> 172.21.15.76:0/4248565139 conn(0x7fb4ec0a41c0 msgr2=0x7fb4ec0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- 172.21.15.76:0/4248565139 shutdown_connections 2024-09-16T06:44:56.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- 172.21.15.76:0/4248565139 wait complete. 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient: build_initial_monmap 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient: monmap: 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:56.916 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 Processor -- start 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- start start 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient: init 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-c con 0x7fb4ec06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-b con 0x7fb4ec06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:56.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient(hunting): picked mon.noname-a con 0x7fb4ec132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec137730 con 0x7fb4ec132330 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec13ed50 con 0x7fb4ec06d480 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec1313a0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4f0d85700 10 monclient(hunting): authenticate will time out at 1244.043121s 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4eb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb4ec06cf50 0x7fb4ec1350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43200/0 (socket says 172.21.15.76:43200) 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4eb7fe700 1 -- 172.21.15.76:0/1355053539 learned_addr learned my addr 172.21.15.76:0/1355053539 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776744684 0 0) 0x7fb4ec1313a0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): my global_id is 125570 2024-09-16T06:44:56.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d80035d0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375434654 0 0) 0x7fb4ec137730 con 0x7fb4ec132330 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): my global_id is 154653 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec1313a0 con 0x7fb4ec132330 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027552796 0 0) 0x7fb4ec13ed50 con 0x7fb4ec06d480 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): my global_id is 174244 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec137730 con 0x7fb4ec06d480 2024-09-16T06:44:56.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 568930661 0 0) 0x7fb4d80035d0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4ec13ed50 con 0x7fb4ec06cf50 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4dc0018f0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): dump: 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.920 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.906+0000 7fb4d3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1819425226 0 0) 0x7fb4ec13ed50 con 0x7fb4ec06cf50 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.189:6789/0 conn(0x7fb4ec06d480 legacy=0x7fb4ec1357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6789/0 conn(0x7fb4ec132330 legacy=0x7fb4ec1d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 monclient: found mon.c 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec09e8b0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:56.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 5 monclient: authenticate success, global_id 125570 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 10 monclient: _renew_subs 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec135f00 con 0x7fb4ec06cf50 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 10 monclient: _renew_subs 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec136d30 con 0x7fb4ec06cf50 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4dc004890 con 0x7fb4ec06cf50 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4dc004ba0 con 0x7fb4ec06cf50 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: dump: 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4e9ffb700 4 set_mon_vals no callback set 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb4ec1318b0 con 0x7fb4ec02c820 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb4dc00f610 con 0x7fb4ec06cf50 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7fb4dc043e10 con 0x7fb4ec06cf50 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794924559 0 0) 0x7fb4ec1318b0 con 0x7fb4ec02c820 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient(hunting): my global_id is 125573 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec13ed50 con 0x7fb4ec02c820 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4ebfff700 10 monclient: get_auth_request con 0x7fb4d8036320 auth_method 0 2024-09-16T06:44:56.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3776219364 0 0) 0x7fb4ec13ed50 con 0x7fb4ec02c820 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4ec1318b0 con 0x7fb4ec02c820 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e00017c0 con 0x7fb4ec02c820 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: dump: 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.925 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3891302178 0 0) 0x7fb4ec1318b0 con 0x7fb4ec02c820 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4ec0a3450 con 0x7fb4ec02c820 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb4e0001bb0 con 0x7fb4ec02c820 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec02c820 legacy=0x7fb4ec061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.910+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec02c820 legacy=0x7fb4ec061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:56.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:56.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb4ec1389c0 con 0x7fb4ec09d520 2024-09-16T06:44:56.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059591615 0 0) 0x7fb4ec1389c0 con 0x7fb4ec09d520 2024-09-16T06:44:56.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient(hunting): my global_id is 125576 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec1318b0 con 0x7fb4ec09d520 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3339760025 0 0) 0x7fb4ec1318b0 con 0x7fb4ec09d520 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4ec1389c0 con 0x7fb4ec09d520 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb4e0003a50 con 0x7fb4ec09d520 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient: dump: 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:56.928 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3378707219 0 0) 0x7fb4ec1389c0 con 0x7fb4ec09d520 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4ec02cc00 con 0x7fb4ec09d520 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 984411146) 0x7fb4e0004fe0 con 0x7fb4ec09d520 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:56.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec09d520 legacy=0x7fb4ec1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4d3fff700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec09d520 legacy=0x7fb4ec1d7380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:56.930 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.932 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:56.933 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:56.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:56.937 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:56.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:56.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:56.941 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb4d8036320 legacy=0x7fb4d80387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 10 monclient: shutdown 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 >> v1:172.21.15.76:6790/0 conn(0x7fb4ec06cf50 legacy=0x7fb4ec1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4ebfff700 1 -- 172.21.15.76:0/1355053539 reap_dead start 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.922+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 shutdown_connections 2024-09-16T06:44:56.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.926+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 >> 172.21.15.76:0/1355053539 conn(0x7fb4ec0a41c0 msgr2=0x7fb4ec0a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.926+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 shutdown_connections 2024-09-16T06:44:56.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:56.926+0000 7fb4f0d85700 1 -- 172.21.15.76:0/1355053539 wait complete. 2024-09-16T06:44:56.944 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:56.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:44:56.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:44:56.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f25966d2700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f25966d2700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f25966d2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:56.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7f25740055c0 con 0x7f259009b1d0 2024-09-16T06:44:56.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f258d7fa700 1 -- 172.21.15.189:0/3441560117 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v173) v1 ==== 128+0+0 (unknown 2631605423 0 0) 0x7f25800142e0 con 0x7f259009b1d0 2024-09-16T06:44:56.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f258d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T06:44:56.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.992+0000 7f258d7fa700 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2570036460 legacy=0x7f2570038920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 10 monclient: shutdown 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 >> v1:172.21.15.189:6789/0 conn(0x7f259009b1d0 legacy=0x7f259011de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 shutdown_connections 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 >> 172.21.15.189:0/3441560117 conn(0x7f25900a0760 msgr2=0x7f2590070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 shutdown_connections 2024-09-16T06:44:56.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:56.996+0000 7f25966d2700 1 -- 172.21.15.189:0/3441560117 wait complete. 2024-09-16T06:44:56.998 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'barr' already exists 2024-09-16T06:44:57.014 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move barr root=default 2024-09-16T06:44:57.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: get_monmap_and_config 2024-09-16T06:44:57.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: monmap: 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:57.065943+0000 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:57.065943+0000 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:57.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 1 Processor -- start 2024-09-16T06:44:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 1 -- start start 2024-09-16T06:44:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: init 2024-09-16T06:44:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-c con 0x7f0c2009b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-b con 0x7f0c200964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-a con 0x7f0c20138870 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c20098560 con 0x7f0c20138870 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c20099ec0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c20097030 con 0x7f0c200964f0 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c25ea7700 10 monclient(hunting): authenticate will time out at 1244.199734s 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c1ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0c200964f0 0x7f0c20096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43230/0 (socket says 172.21.15.76:43230) 2024-09-16T06:44:57.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.062+0000 7f0c1ffff700 1 -- 172.21.15.76:0/2666045808 learned_addr learned my addr 172.21.15.76:0/2666045808 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309055800 0 0) 0x7f0c20097030 con 0x7f0c200964f0 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): my global_id is 125579 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c080035d0 con 0x7f0c200964f0 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097760511 0 0) 0x7f0c20098560 con 0x7f0c20138870 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): my global_id is 154656 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20097030 con 0x7f0c20138870 2024-09-16T06:44:57.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094898492 0 0) 0x7f0c20099ec0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient(hunting): my global_id is 174247 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20098560 con 0x7f0c2009b1c0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263111431 0 0) 0x7f0c080035d0 con 0x7f0c200964f0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 >> v1:172.21.15.189:6789/0 conn(0x7f0c2009b1c0 legacy=0x7f0c2009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 >> v1:172.21.15.76:6789/0 conn(0x7f0c20138870 legacy=0x7f0c2013ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2009b5a0 con 0x7f0c200964f0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c14001610 con 0x7f0c200964f0 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: dump: 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 5 monclient: authenticate success, global_id 125579 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c140018f0 con 0x7f0c200964f0 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 4 set_mon_vals no callback set 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 1 -- 172.21.15.76:0/2666045808 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c14001c80 con 0x7f0c200964f0 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:57.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c1f7fe700 10 monclient: dump: 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: shutdown 2024-09-16T06:44:57.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- 172.21.15.76:0/2666045808 >> v1:172.21.15.76:6790/0 conn(0x7f0c200964f0 legacy=0x7f0c20096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- 172.21.15.76:0/2666045808 shutdown_connections 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- 172.21.15.76:0/2666045808 >> 172.21.15.76:0/2666045808 conn(0x7f0c200a0030 msgr2=0x7f0c200a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- 172.21.15.76:0/2666045808 shutdown_connections 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- 172.21.15.76:0/2666045808 wait complete. 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: monmap: 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.079 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 Processor -- start 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 1 -- start start 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: init 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-b con 0x7f0c200964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.066+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-a con 0x7f0c2009b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient(hunting): picked mon.noname-c con 0x7f0c20138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2008b740 con 0x7f0c2009b1c0 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2008c6a0 con 0x7f0c200964f0 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2008c910 con 0x7f0c20138870 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient(hunting): authenticate will time out at 1244.204170s 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c256a6700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0c20138870 0x7f0c201d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43246/0 (socket says 172.21.15.76:43246) 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c256a6700 1 -- 172.21.15.76:0/444369693 learned_addr learned my addr 172.21.15.76:0/444369693 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940030822 0 0) 0x7f0c2008b740 con 0x7f0c2009b1c0 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): my global_id is 154659 2024-09-16T06:44:57.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c0c0035d0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787394430 0 0) 0x7f0c2008c910 con 0x7f0c20138870 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): my global_id is 125582 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2008b740 con 0x7f0c20138870 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305508943 0 0) 0x7f0c2008c6a0 con 0x7f0c200964f0 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): my global_id is 174250 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2008c910 con 0x7f0c200964f0 2024-09-16T06:44:57.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41784231 0 0) 0x7f0c0c0035d0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c2008c6a0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c140015d0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644898053 0 0) 0x7f0c2008b740 con 0x7f0c20138870 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c0c0035d0 con 0x7f0c20138870 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c18001bb0 con 0x7f0c20138870 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): dump: 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2005686710 0 0) 0x7f0c2008c6a0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.76:6790/0 conn(0x7f0c20138870 legacy=0x7f0c201d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6789/0 conn(0x7f0c200964f0 legacy=0x7f0c201328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 monclient: found mon.a 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2008cbf0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 5 monclient: authenticate success, global_id 154659 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient: _renew_subs 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2008b0d0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient: _renew_subs 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2008da90 con 0x7f0c2009b1c0 2024-09-16T06:44:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c14004b40 con 0x7f0c2009b1c0 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c14004ed0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1e7fc700 4 set_mon_vals no callback set 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient: dump: 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0c200716c0 con 0x7f0c200712e0 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0c1400f940 con 0x7f0c2009b1c0 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f0c14043fe0 con 0x7f0c2009b1c0 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c256a6700 10 monclient: get_auth_request con 0x7f0c0c0362b0 auth_method 0 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871858428 0 0) 0x7f0c200716c0 con 0x7f0c200712e0 2024-09-16T06:44:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 10 monclient(hunting): my global_id is 174253 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.070+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2008c6a0 con 0x7f0c200712e0 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2357219658 0 0) 0x7f0c2008c6a0 con 0x7f0c200712e0 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c200716c0 con 0x7f0c200712e0 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c10003840 con 0x7f0c200712e0 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 10 monclient: dump: 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2275889952 0 0) 0x7f0c200716c0 con 0x7f0c200712e0 2024-09-16T06:44:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c2009a1a0 con 0x7f0c200712e0 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0c100039a0 con 0x7f0c200712e0 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6789/0 conn(0x7f0c200712e0 legacy=0x7f0c2006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.074+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6789/0 conn(0x7f0c200712e0 legacy=0x7f0c2006be90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.082+0000 7f0c25ea7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.082+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0c2008b2c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011224704 0 0) 0x7f0c2008b2c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient(hunting): my global_id is 174256 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c200716c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 890913456 0 0) 0x7f0c200716c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c2008b2c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c10004040 con 0x7f0c20132fc0 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient: dump: 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 737088196 0 0) 0x7f0c2008b2c0 con 0x7f0c20132fc0 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0c2009f470 con 0x7f0c20132fc0 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 4203897561) 0x7f0c10004320 con 0x7f0c20132fc0 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6789/0 conn(0x7f0c20132fc0 legacy=0x7f0c2002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c1d7fa700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6789/0 conn(0x7f0c20132fc0 legacy=0x7f0c2002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 60, 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:57.095 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:57.096 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:44:57.097 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:44:57.098 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:57.099 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.100 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:44:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:57.102 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:57.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:44:57.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:44:57.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:44:57.106 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:44:57.106 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0c0c0362b0 legacy=0x7f0c0c038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 10 monclient: shutdown 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 >> v1:172.21.15.76:6789/0 conn(0x7f0c2009b1c0 legacy=0x7f0c201cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c256a6700 1 -- 172.21.15.76:0/444369693 reap_dead start 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c256a6700 1 -- 172.21.15.76:0/444369693 reap_dead start 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 shutdown_connections 2024-09-16T06:44:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 >> 172.21.15.76:0/444369693 conn(0x7f0c200a0030 msgr2=0x7f0c2009b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 shutdown_connections 2024-09-16T06:44:57.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.086+0000 7f0c25ea7700 1 -- 172.21.15.76:0/444369693 wait complete. 2024-09-16T06:44:57.108 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:44:57.108 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:44:57.108 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:44:57.108 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:44:57.108 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:44:57.109 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:44:57.109 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:44:57.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.120+0000 7f863a285700 10 monclient: get_monmap_and_config 2024-09-16T06:44:57.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.120+0000 7f863a285700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.120+0000 7f863a285700 10 monclient: monmap: 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:44:57.123349+0000 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:44:57.123349+0000 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:57.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 Processor -- start 2024-09-16T06:44:57.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- start start 2024-09-16T06:44:57.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient: init 2024-09-16T06:44:57.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:44:57.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient(hunting): picked mon.noname-b con 0x7f863409b920 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient(hunting): picked mon.noname-a con 0x7f863409c410 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient(hunting): picked mon.noname-c con 0x7f863409c820 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86340728d0 con 0x7f863409c410 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863409d230 con 0x7f863409c820 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863409dd50 con 0x7f863409b920 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient(hunting): authenticate will time out at 1249.637131s 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f86337fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f863409c410 legacy=0x7f86341375a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8638822700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f863409c820 0x7f863413acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47054/0 (socket says 172.21.15.189:47054) 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8638822700 1 -- 172.21.15.189:0/335220108 learned_addr learned my addr 172.21.15.189:0/335220108 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900699332 0 0) 0x7f863409dd50 con 0x7f863409b920 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): my global_id is 125585 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c0035d0 con 0x7f863409b920 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170759851 0 0) 0x7f863409d230 con 0x7f863409c820 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient(hunting): my global_id is 174259 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f863409dd50 con 0x7f863409c820 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3701846916 0 0) 0x7f861c0035d0 con 0x7f863409b920 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 >> v1:172.21.15.189:6789/0 conn(0x7f863409c820 legacy=0x7f863413acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 >> v1:172.21.15.76:6789/0 conn(0x7f863409c410 legacy=0x7f86341375a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 monclient: found mon.noname-b 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863409d790 con 0x7f863409b920 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 5 monclient: authenticate success, global_id 125585 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8624001610 con 0x7f863409b920 2024-09-16T06:44:57.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: dump: 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86240018f0 con 0x7f863409b920 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 1 -- 172.21.15.189:0/335220108 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8624001cc0 con 0x7f863409b920 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f8632ffd700 10 monclient: dump: 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.131 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.132 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:57.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 4 set_mon_vals no callback set 2024-09-16T06:44:57.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 10 monclient: shutdown 2024-09-16T06:44:57.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- 172.21.15.189:0/335220108 >> v1:172.21.15.76:6790/0 conn(0x7f863409b920 legacy=0x7f863409bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- 172.21.15.189:0/335220108 shutdown_connections 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- 172.21.15.189:0/335220108 >> 172.21.15.189:0/335220108 conn(0x7f86340a0f20 msgr2=0x7f86340a1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- 172.21.15.189:0/335220108 shutdown_connections 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.124+0000 7f863a285700 1 -- 172.21.15.189:0/335220108 wait complete. 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: monmap: 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.137 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 Processor -- start 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- start start 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: init 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient(hunting): picked mon.noname-a con 0x7f863409b920 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient(hunting): picked mon.noname-b con 0x7f863409c410 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient(hunting): picked mon.noname-c con 0x7f863409c820 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863409b610 con 0x7f863409b920 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8634099e50 con 0x7f863409c410 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863409fdd0 con 0x7f863409c820 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient(hunting): authenticate will time out at 1249.641430s 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f86337fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f863409c410 0x7f86341cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:47064/0 (socket says 172.21.15.189:47064) 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f86337fe700 1 -- 172.21.15.189:0/4104154660 learned_addr learned my addr 172.21.15.189:0/4104154660 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8633fff700 1 -- 172.21.15.189:0/4104154660 >> v1:172.21.15.76:6789/0 conn(0x7f863409b920 legacy=0x7f8634097350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496740541 0 0) 0x7f8634099e50 con 0x7f863409c410 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): my global_id is 174262 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86200035d0 con 0x7f863409c410 2024-09-16T06:44:57.139 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2968876287 0 0) 0x7f86200035d0 con 0x7f863409c410 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8634099e50 con 0x7f863409c410 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8628004230 con 0x7f863409c410 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): dump: 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:57.140 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269843234 0 0) 0x7f863409fdd0 con 0x7f863409c820 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient(hunting): my global_id is 125588 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86200035d0 con 0x7f863409c820 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1602697227 0 0) 0x7f8634099e50 con 0x7f863409c410 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 >> v1:172.21.15.76:6790/0 conn(0x7f863409c820 legacy=0x7f86341d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 >> v1:172.21.15.76:6789/0 conn(0x7f863409b920 legacy=0x7f8634097350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 monclient: found mon.b 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634092df0 con 0x7f863409c410 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 5 monclient: authenticate success, global_id 174262 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _renew_subs 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8634092b10 con 0x7f863409c410 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _renew_subs 2024-09-16T06:44:57.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8634097a60 con 0x7f863409c410 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8628004550 con 0x7f863409c410 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f86280048e0 con 0x7f863409c410 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f86327fc700 4 set_mon_vals no callback set 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 10 monclient: dump: 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.143 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8628004a40 con 0x7f863409c410 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f8628044000 con 0x7f863409c410 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f8633fff700 10 monclient: get_auth_request con 0x7f86200361d0 auth_method 0 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.128+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8614005100 con 0x7f863409c410 2024-09-16T06:44:57.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.136+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8628005210 con 0x7f863409c410 2024-09-16T06:44:57.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.136+0000 7f8630ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.136+0000 7f8630ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:57.146 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:44:57.146 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:44:57.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.302+0000 7f24849c0700 10 monclient: get_monmap_and_config 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.302+0000 7f24849c0700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.302+0000 7f24849c0700 10 monclient: monmap: 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:44:57.307308+0000 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:44:57.307308+0000 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:44:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:44:57.306 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 Processor -- start 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- start start 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient: init 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-a con 0x7f2480132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-b con 0x7f248006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-c con 0x7f248006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248006c280 con 0x7f2480132330 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248006bfa0 con 0x7f248006cf50 2024-09-16T06:44:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248009ace0 con 0x7f248006d480 2024-09-16T06:44:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient(hunting): authenticate will time out at 1244.441407s 2024-09-16T06:44:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2480132330 legacy=0x7f2480132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:44:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f248006d480 0x7f248013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43260/0 (socket says 172.21.15.76:43260) 2024-09-16T06:44:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247effd700 1 -- 172.21.15.76:0/183599793 learned_addr learned my addr 172.21.15.76:0/183599793 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466251096 0 0) 0x7f248009ace0 con 0x7f248006d480 2024-09-16T06:44:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): my global_id is 125591 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24680035d0 con 0x7f248006d480 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073895454 0 0) 0x7f248006bfa0 con 0x7f248006cf50 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient(hunting): my global_id is 174265 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f248009ace0 con 0x7f248006cf50 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170911163 0 0) 0x7f24680035d0 con 0x7f248006d480 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 >> v1:172.21.15.189:6789/0 conn(0x7f248006cf50 legacy=0x7f248013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 >> v1:172.21.15.76:6789/0 conn(0x7f2480132330 legacy=0x7f2480132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248009ae70 con 0x7f248006d480 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 5 monclient: authenticate success, global_id 125591 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2474001610 con 0x7f248006d480 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: dump: 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24740018f0 con 0x7f248006d480 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 1 -- 172.21.15.76:0/183599793 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2474001ae0 con 0x7f248006d480 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient: get_monmap_and_config success 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 4 set_mon_vals no callback set 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f247e7fc700 10 monclient: dump: 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 10 monclient: shutdown 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- 172.21.15.76:0/183599793 >> v1:172.21.15.76:6790/0 conn(0x7f248006d480 legacy=0x7f248013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- 172.21.15.76:0/183599793 shutdown_connections 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- 172.21.15.76:0/183599793 >> 172.21.15.76:0/183599793 conn(0x7f24800a41c0 msgr2=0x7f24800a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- 172.21.15.76:0/183599793 shutdown_connections 2024-09-16T06:44:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.306+0000 7f24849c0700 1 -- 172.21.15.76:0/183599793 wait complete. 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient: build_initial_monmap 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient: monmap: 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:44:57.317 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.308+0000 7f863a285700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:44:57.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.308+0000 7f863a285700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:44:57.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.308+0000 7f863a285700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:44:57.308+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f8614005930 con 0x7f863409c410 2024-09-16T06:44:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 1 Processor -- start 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 1 -- start start 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient: init 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-a con 0x7f248006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-b con 0x7f248006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient(hunting): picked mon.noname-c con 0x7f2480132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248013ede0 con 0x7f248006cf50 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f247f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f248006cf50 legacy=0x7f24801c66a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24801310d0 con 0x7f248006d480 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2480131550 con 0x7f2480132330 2024-09-16T06:44:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f24849c0700 10 monclient(hunting): authenticate will time out at 1244.446141s 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f247effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f248006d480 0x7f24801cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:47120/0 (socket says 172.21.15.76:47120) 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f247effd700 1 -- 172.21.15.76:0/2075929054 learned_addr learned my addr 172.21.15.76:0/2075929054 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400336458 0 0) 0x7f24801310d0 con 0x7f248006d480 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): my global_id is 174268 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c0035d0 con 0x7f248006d480 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612439669 0 0) 0x7f246c0035d0 con 0x7f248006d480 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24801310d0 con 0x7f248006d480 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2474004890 con 0x7f248006d480 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): dump: 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:44:57.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761242340 0 0) 0x7f2480131550 con 0x7f2480132330 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _negotiate 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 10 monclient(hunting): my global_id is 125594 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.310+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c0035d0 con 0x7f2480132330 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 308314157 0 0) 0x7f24801310d0 con 0x7f248006d480 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 >> v1:172.21.15.76:6790/0 conn(0x7f2480132330 legacy=0x7f24801d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 >> v1:172.21.15.76:6789/0 conn(0x7f248006cf50 legacy=0x7f24801c66a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: _finish_hunting 0 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 monclient: found mon.b 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480130ba0 con 0x7f248006d480 2024-09-16T06:44:57.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: _finish_auth 0 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: _check_auth_tickets 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 5 monclient: authenticate success, global_id 174268 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _renew_subs 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248012fb80 con 0x7f248006d480 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _renew_subs 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f248012eff0 con 0x7f248006d480 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2474004ca0 con 0x7f248006d480 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f247dffb700 4 set_mon_vals no callback set 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2474004fb0 con 0x7f248006d480 2024-09-16T06:44:57.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 10 monclient: dump: 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:44:57.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460005100 con 0x7f248006d480 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f247400fa20 con 0x7f248006d480 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f2474044090 con 0x7f248006d480 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.314+0000 7f247f7fe700 10 monclient: get_auth_request con 0x7f246c036270 auth_method 0 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.326+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f24740148e0 con 0x7f248006d480 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.326+0000 7f2467fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:44:57.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.326+0000 7f2467fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:44:57.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f24849c0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:44:57.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f24849c0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:57.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f24849c0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:44:57.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2460005930 con 0x7f248006d480 2024-09-16T06:44:57.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f2467fff700 1 -- 172.21.15.76:0/2075929054 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 482306024) 0x7f2474014700 con 0x7f248006d480 2024-09-16T06:44:57.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f2467fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:44:57.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.566+0000 7f2467fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:44:57.569 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:44:57.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 >> v1:172.21.15.189:6812/3098810674 conn(0x7f246c036270 legacy=0x7f246c038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 10 monclient: shutdown 2024-09-16T06:44:57.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:44:57.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 >> v1:172.21.15.189:6789/0 conn(0x7f248006d480 legacy=0x7f24801cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:44:57.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 shutdown_connections 2024-09-16T06:44:57.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 >> 172.21.15.76:0/2075929054 conn(0x7f24800a41c0 msgr2=0x7f248009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:44:57.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 shutdown_connections 2024-09-16T06:44:57.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:44:57.570+0000 7f24849c0700 1 -- 172.21.15.76:0/2075929054 wait complete. 2024-09-16T06:45:00.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:00.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:00.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:00.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:00.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:00.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:00.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:00.128+0000 7f861b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:00.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: get_monmap_and_config 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: build_initial_monmap 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: monmap: 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:00.701953+0000 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:00.701953+0000 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:00.700 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 1 Processor -- start 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 1 -- start start 2024-09-16T06:45:00.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: init 2024-09-16T06:45:00.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-b con 0x7f65c4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-c con 0x7f65c4099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-a con 0x7f65c4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c4098990 con 0x7f65c4139500 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c4097f40 con 0x7f65c4099090 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c409d610 con 0x7f65c4097b60 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65ca38c700 10 monclient(hunting): authenticate will time out at 1247.835173s 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65c9b8b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f65c4139500 legacy=0x7f65c413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65c938a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f65c4097b60 0x7f65c409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43284/0 (socket says 172.21.15.76:43284) 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65c938a700 1 -- 172.21.15.76:0/3292371663 learned_addr learned my addr 172.21.15.76:0/3292371663 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509554848 0 0) 0x7f65c409d610 con 0x7f65c4097b60 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:00.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 10 monclient(hunting): my global_id is 125597 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.697+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b00035d0 con 0x7f65c4097b60 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366623265 0 0) 0x7f65b00035d0 con 0x7f65c4097b60 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 >> v1:172.21.15.189:6789/0 conn(0x7f65c4099090 legacy=0x7f65c409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 >> v1:172.21.15.76:6789/0 conn(0x7f65c4139500 legacy=0x7f65c413b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 monclient: found mon.noname-b 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c40986b0 con 0x7f65c4097b60 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: _finish_auth 0 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 5 monclient: authenticate success, global_id 125597 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65c0001610 con 0x7f65c4097b60 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: dump: 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:00.708 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:00.709 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65c0001930 con 0x7f65c4097b60 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:00.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 1 -- 172.21.15.76:0/3292371663 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65c0001d00 con 0x7f65c4097b60 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 4 set_mon_vals no callback set 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65bbfff700 10 monclient: dump: 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:00.711 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 10 monclient: shutdown 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:00.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 1 -- 172.21.15.76:0/3292371663 >> v1:172.21.15.76:6790/0 conn(0x7f65c4097b60 legacy=0x7f65c409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:00.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 1 -- 172.21.15.76:0/3292371663 shutdown_connections 2024-09-16T06:45:00.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 1 -- 172.21.15.76:0/3292371663 >> 172.21.15.76:0/3292371663 conn(0x7f65c40a0030 msgr2=0x7f65c40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:00.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 1 -- 172.21.15.76:0/3292371663 shutdown_connections 2024-09-16T06:45:00.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 1 -- 172.21.15.76:0/3292371663 wait complete. 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 10 monclient: build_initial_monmap 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.701+0000 7f65ca38c700 10 monclient: monmap: 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 Processor -- start 2024-09-16T06:45:00.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- start start 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: init 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-a con 0x7f65c4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-b con 0x7f65c4099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient(hunting): picked mon.noname-c con 0x7f65c4139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:00.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65c938a700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f65c4097b60 legacy=0x7f65c4136cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c40a2c50 con 0x7f65c4097b60 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c409a540 con 0x7f65c4099090 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c406cc40 con 0x7f65c4139500 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient(hunting): authenticate will time out at 1247.841555s 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65c9b8b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f65c4139500 0x7f65c4137ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43286/0 (socket says 172.21.15.76:43286) 2024-09-16T06:45:00.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65c9b8b700 1 -- 172.21.15.76:0/4070026576 learned_addr learned my addr 172.21.15.76:0/4070026576 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855589089 0 0) 0x7f65c406cc40 con 0x7f65c4139500 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): my global_id is 125600 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a80035d0 con 0x7f65c4139500 2024-09-16T06:45:00.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283738156 0 0) 0x7f65a80035d0 con 0x7f65c4139500 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65c406cc40 con 0x7f65c4139500 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65bc004460 con 0x7f65c4139500 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:00.725 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:00.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1772364228 0 0) 0x7f65c406cc40 con 0x7f65c4139500 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 >> v1:172.21.15.189:6789/0 conn(0x7f65c4099090 legacy=0x7f65c41373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 >> v1:172.21.15.76:6789/0 conn(0x7f65c4097b60 legacy=0x7f65c4136cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 monclient: found mon.c 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c41387f0 con 0x7f65c4139500 2024-09-16T06:45:00.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 5 monclient: authenticate success, global_id 125600 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _renew_subs 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c4071b90 con 0x7f65c4139500 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _renew_subs 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c41d10e0 con 0x7f65c4139500 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65bc004780 con 0x7f65c4139500 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65bc004b10 con 0x7f65c4139500 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65b9ffb700 10 monclient: dump: 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:00.729 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:00.730 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65bb7fe700 4 set_mon_vals no callback set 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.705+0000 7f65ca38c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65ca38c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65ca38c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a4005100 con 0x7f65c4139500 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f65bc004cf0 con 0x7f65c4139500 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f65bc044180 con 0x7f65c4139500 2024-09-16T06:45:00.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.709+0000 7f65c938a700 10 monclient: get_auth_request con 0x7f65a8035f50 auth_method 0 2024-09-16T06:45:00.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.713+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f65bc0054f0 con 0x7f65c4139500 2024-09-16T06:45:00.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.713+0000 7f65b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:00.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.713+0000 7f65b9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65ca38c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65ca38c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65ca38c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f65a4005930 con 0x7f65c4139500 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65b9ffb700 1 -- 172.21.15.76:0/4070026576 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1280551369) 0x7f65bc012ed0 con 0x7f65c4139500 2024-09-16T06:45:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.945+0000 7f65b9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:00.949 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:00.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 >> v1:172.21.15.189:6812/3098810674 conn(0x7f65a8035f50 legacy=0x7f65a8038410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 10 monclient: shutdown 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 >> v1:172.21.15.76:6790/0 conn(0x7f65c4139500 legacy=0x7f65c4137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 shutdown_connections 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 >> 172.21.15.76:0/4070026576 conn(0x7f65c40a0030 msgr2=0x7f65c40a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 shutdown_connections 2024-09-16T06:45:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:00.949+0000 7f65ca38c700 1 -- 172.21.15.76:0/4070026576 wait complete. 2024-09-16T06:45:01.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:01.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:01.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:01.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:01.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:02.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:02.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:02.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:02.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:02.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:02.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:03.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:03.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:03.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:03.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:03.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:03.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:03.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:04.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: get_monmap_and_config 2024-09-16T06:45:04.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: build_initial_monmap 2024-09-16T06:45:04.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: monmap: 2024-09-16T06:45:04.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:04.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:04.077813+0000 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:04.077813+0000 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:04.076 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:04.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 1 Processor -- start 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 1 -- start start 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: init 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-c con 0x7f6ff0098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009d560 con 0x7f6ff0099030 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009f140 con 0x7f6ff0098be0 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009d790 con 0x7f6ff009ed60 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff6088700 10 monclient(hunting): authenticate will time out at 1251.211219s 2024-09-16T06:45:04.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff5887700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff013ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:04.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff5086700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6ff009ed60 0x7f6ff00984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37570/0 (socket says 172.21.15.76:37570) 2024-09-16T06:45:04.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.073+0000 7f6ff5086700 1 -- 172.21.15.76:0/1691737923 learned_addr learned my addr 172.21.15.76:0/1691737923 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:04.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:04.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:04.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:04.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:04.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:04.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.277+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff013ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:04.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:04.681+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff013ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:05.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:05.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:05.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:05.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:05.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:05.485+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff013ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:06.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:06.128+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:06.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:06.128+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:06.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:06.128+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:07.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): tick 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6790/0 conn(0x7f6ff009ed60 legacy=0x7f6ff00984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.189:6789/0 conn(0x7f6ff0098be0 legacy=0x7f6ff01374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff013ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd4003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009f620 con 0x7f6ff0099030 2024-09-16T06:45:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009d560 con 0x7f6fd4003740 2024-09-16T06:45:07.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40103f0 con 0x7f6ff009ed60 2024-09-16T06:45:07.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6fe6ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:07.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.077+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:07.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:07.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:07.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:07.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:07.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:07.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:07.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.281+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:07.685+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:08.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:08.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:08.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:08.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:08.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:08.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:08.489+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:09.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:09.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:09.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:09.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:09.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:09.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:10.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): tick 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6790/0 conn(0x7f6ff009ed60 legacy=0x7f6fd4009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.189:6789/0 conn(0x7f6fd4003740 legacy=0x7f6fd4005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd400f470 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f6fd4010580 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd4011600 con 0x7f6ff0099030 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009f620 con 0x7f6fd400f470 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd4011090 con 0x7f6fd4010580 2024-09-16T06:45:10.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6fe6ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:10.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.077+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:10.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:10.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:10.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:10.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:10.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:10.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:10.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:10.127+0000 7f861b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:10.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.281+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:10.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:10.685+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:11.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:11.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:11.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:11.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:11.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:11.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:11.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.489+0000 7f6ff5887700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742696233 0 0) 0x7f6ff009f620 con 0x7f6fd400f470 2024-09-16T06:45:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient(hunting): my global_id is 174271 2024-09-16T06:45:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc0035d0 con 0x7f6fd400f470 2024-09-16T06:45:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748988715 0 0) 0x7f6fdc0035d0 con 0x7f6fd400f470 2024-09-16T06:45:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6790/0 conn(0x7f6fd4010580 legacy=0x7f6fd4010980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6fd400cc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 monclient: found mon.noname-c 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd4011220 con 0x7f6fd400f470 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe0001510 con 0x7f6fd400f470 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6ff6088700 5 monclient: authenticate success, global_id 174271 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6ff6088700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: dump: 2024-09-16T06:45:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6ff6088700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe0001730 con 0x7f6fd400f470 2024-09-16T06:45:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 1 -- 172.21.15.76:0/1691737923 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe0001b40 con 0x7f6fd400f470 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6ff6088700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6ff6088700 4 set_mon_vals no callback set 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.917+0000 7f6fe7fff700 10 monclient: dump: 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 10 monclient: shutdown 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- 172.21.15.76:0/1691737923 >> v1:172.21.15.189:6789/0 conn(0x7f6fd400f470 legacy=0x7f6fd400f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- 172.21.15.76:0/1691737923 shutdown_connections 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- 172.21.15.76:0/1691737923 >> 172.21.15.76:0/1691737923 conn(0x7f6ff00a0030 msgr2=0x7f6ff00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- 172.21.15.76:0/1691737923 shutdown_connections 2024-09-16T06:45:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- 172.21.15.76:0/1691737923 wait complete. 2024-09-16T06:45:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 10 monclient: build_initial_monmap 2024-09-16T06:45:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 10 monclient: monmap: 2024-09-16T06:45:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 Processor -- start 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 1 -- start start 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.921+0000 7f6ff6088700 10 monclient: init 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd400f470 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-b con 0x7f6fd4010580 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00702a0 con 0x7f6ff0099030 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff009a540 con 0x7f6fd4010580 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff5887700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff01d2270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00951b0 con 0x7f6fd400f470 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient(hunting): authenticate will time out at 1259.061018s 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff5086700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6fd400f470 0x7f6ff00966e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37588/0 (socket says 172.21.15.76:37588) 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff5086700 1 -- 172.21.15.76:0/1716906314 learned_addr learned my addr 172.21.15.76:0/1716906314 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337692111 0 0) 0x7f6ff009a540 con 0x7f6fd4010580 2024-09-16T06:45:11.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): my global_id is 174274 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4011600 con 0x7f6fd4010580 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478727632 0 0) 0x7f6fd4011600 con 0x7f6fd4010580 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff009a540 con 0x7f6fd4010580 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec001720 con 0x7f6fd4010580 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:11.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3342817858 0 0) 0x7f6ff009a540 con 0x7f6fd4010580 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 >> v1:172.21.15.76:6790/0 conn(0x7f6fd400f470 legacy=0x7f6ff00966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 >> v1:172.21.15.76:6789/0 conn(0x7f6ff0099030 legacy=0x7f6ff01d2270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:11.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 monclient: found mon.b 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0094910 con 0x7f6fd4010580 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 5 monclient: authenticate success, global_id 174274 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _renew_subs 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff0094bf0 con 0x7f6fd4010580 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _renew_subs 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:11.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff0093040 con 0x7f6fd4010580 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fec001c70 con 0x7f6fd4010580 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec003840 con 0x7f6fd4010580 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe6ffd700 4 set_mon_vals no callback set 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.925+0000 7f6fe5ffb700 10 monclient: dump: 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:11.936 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6fec0039a0 con 0x7f6fd4010580 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6ff6088700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6ff6088700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6ff6088700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0005100 con 0x7f6fd4010580 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f6fec042b40 con 0x7f6fd4010580 2024-09-16T06:45:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.929+0000 7f6ff5086700 10 monclient: get_auth_request con 0x7f6fd4037680 auth_method 0 2024-09-16T06:45:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.937+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6fec013190 con 0x7f6fd4010580 2024-09-16T06:45:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.937+0000 7f6fe5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:11.937+0000 7f6fe5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:12.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.127+0000 7f861b7fe700 10 monclient: tick 2024-09-16T06:45:12.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.127+0000 7f861b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:12.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.127+0000 7f861b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:12.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6ff6088700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:12.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6ff6088700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:12.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6ff6088700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:12.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6fd0005930 con 0x7f6fd4010580 2024-09-16T06:45:12.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6fe5ffb700 1 -- 172.21.15.76:0/1716906314 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 542890547) 0x7f6fec011830 con 0x7f6fd4010580 2024-09-16T06:45:12.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6fe5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:12.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.173+0000 7f6fe5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:12.292 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":64,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6fd4037680 legacy=0x7f6fd4039b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 10 monclient: shutdown 2024-09-16T06:45:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 >> v1:172.21.15.189:6789/0 conn(0x7f6fd4010580 legacy=0x7f6ff01ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 shutdown_connections 2024-09-16T06:45:12.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 >> 172.21.15.76:0/1716906314 conn(0x7f6ff00a0030 msgr2=0x7f6ff008aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:12.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 shutdown_connections 2024-09-16T06:45:12.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.177+0000 7f6ff6088700 1 -- 172.21.15.76:0/1716906314 wait complete. 2024-09-16T06:45:12.294 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:45:12.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:45:12.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:45:12.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: get_monmap_and_config 2024-09-16T06:45:12.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: build_initial_monmap 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: monmap: 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:12.417105+0000 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:12.417105+0000 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:12.415 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 1 Processor -- start 2024-09-16T06:45:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 1 -- start start 2024-09-16T06:45:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: init 2024-09-16T06:45:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-a con 0x7f00f409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-b con 0x7f00f409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-c con 0x7f00f4139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f409cd80 con 0x7f00f409f230 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f4099fd0 con 0x7f00f4139b70 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f409a410 con 0x7f00f409fd20 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient(hunting): authenticate will time out at 1259.550337s 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00f409f230 legacy=0x7f00f409f610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f00f409fd20 0x7f00f4138450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37600/0 (socket says 172.21.15.76:37600) 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f2ffd700 1 -- 172.21.15.76:0/1103096280 learned_addr learned my addr 172.21.15.76:0/1103096280 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750870257 0 0) 0x7f00f409a410 con 0x7f00f409fd20 2024-09-16T06:45:12.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): my global_id is 125603 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e00035d0 con 0x7f00f409fd20 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817517841 0 0) 0x7f00f4099fd0 con 0x7f00f4139b70 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient(hunting): my global_id is 174277 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f409a410 con 0x7f00f4139b70 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945512171 0 0) 0x7f00e00035d0 con 0x7f00f409fd20 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 >> v1:172.21.15.189:6789/0 conn(0x7f00f4139b70 legacy=0x7f00f413bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 >> v1:172.21.15.76:6789/0 conn(0x7f00f409f230 legacy=0x7f00f409f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 monclient: found mon.noname-b 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f409de90 con 0x7f00f409fd20 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00e4001610 con 0x7f00f409fd20 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 5 monclient: authenticate success, global_id 125603 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: dump: 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e4001880 con 0x7f00f409fd20 2024-09-16T06:45:12.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f9e59700 4 set_mon_vals no callback set 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 1 -- 172.21.15.76:0/1103096280 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00e4001c10 con 0x7f00f409fd20 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.413+0000 7f00f27fc700 10 monclient: dump: 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.423 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: shutdown 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- 172.21.15.76:0/1103096280 >> v1:172.21.15.76:6790/0 conn(0x7f00f409fd20 legacy=0x7f00f4138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- 172.21.15.76:0/1103096280 shutdown_connections 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- 172.21.15.76:0/1103096280 >> 172.21.15.76:0/1103096280 conn(0x7f00f40a3e50 msgr2=0x7f00f40a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- 172.21.15.76:0/1103096280 shutdown_connections 2024-09-16T06:45:12.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- 172.21.15.76:0/1103096280 wait complete. 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: build_initial_monmap 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: monmap: 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:12.429 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 Processor -- start 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- start start 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: init 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-c con 0x7f00f409f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-a con 0x7f00f409fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:12.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient(hunting): picked mon.noname-b con 0x7f00f4139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f2ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f00f409fd20 legacy=0x7f00f411a580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f411ac90 con 0x7f00f409fd20 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f4072450 con 0x7f00f4139b70 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f409c690 con 0x7f00f409f230 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.417+0000 7f00f9e59700 10 monclient(hunting): authenticate will time out at 1259.555947s 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f00f409f230 0x7f00f4119e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37616/0 (socket says 172.21.15.76:37616) 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f37fe700 1 -- 172.21.15.76:0/3233013796 learned_addr learned my addr 172.21.15.76:0/3233013796 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417930587 0 0) 0x7f00f409c690 con 0x7f00f409f230 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): my global_id is 125606 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d40035d0 con 0x7f00f409f230 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135090824 0 0) 0x7f00d40035d0 con 0x7f00f409f230 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f409c690 con 0x7f00f409f230 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00dc004200 con 0x7f00f409f230 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): dump: 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.432 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1518697338 0 0) 0x7f00f409c690 con 0x7f00f409f230 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.189:6789/0 conn(0x7f00f4139b70 legacy=0x7f00f41cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6789/0 conn(0x7f00f409fd20 legacy=0x7f00f411a580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 monclient: found mon.c 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f411bf00 con 0x7f00f409f230 2024-09-16T06:45:12.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: _finish_auth 0 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 5 monclient: authenticate success, global_id 125606 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 10 monclient: _renew_subs 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f4138370 con 0x7f00f409f230 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 10 monclient: _renew_subs 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f4138ce0 con 0x7f00f409f230 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00dc0044b0 con 0x7f00f409f230 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f1ffb700 4 set_mon_vals no callback set 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00dc004840 con 0x7f00f409f230 2024-09-16T06:45:12.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient: dump: 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00d8006590 con 0x7f00d8002830 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f00dc004a20 con 0x7f00f409f230 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f00dc043ee0 con 0x7f00f409f230 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842442676 0 0) 0x7f00d8006590 con 0x7f00d8002830 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 10 monclient(hunting): my global_id is 125609 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f409c690 con 0x7f00d8002830 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00f3fff700 10 monclient: get_auth_request con 0x7f00d40360a0 auth_method 0 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.421+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3599013770 0 0) 0x7f00f409c690 con 0x7f00d8002830 2024-09-16T06:45:12.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d8006590 con 0x7f00d8002830 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00e4001070 con 0x7f00d8002830 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 10 monclient: dump: 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.437 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2256745662 0 0) 0x7f00d8006590 con 0x7f00d8002830 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00d800a190 con 0x7f00d8002830 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f00e4001c50 con 0x7f00d8002830 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6790/0 conn(0x7f00d8002830 legacy=0x7f00d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.425+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6790/0 conn(0x7f00d8002830 legacy=0x7f00d8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:12.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00f9e59700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:12.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00d8009e30 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770362509 0 0) 0x7f00d8009e30 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient(hunting): my global_id is 125612 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d8006590 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2385518005 0 0) 0x7f00d8006590 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d8009e30 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00e4004ce0 con 0x7f00d8009150 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.433+0000 7f00ebfff700 10 monclient: dump: 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3438700205 0 0) 0x7f00d8009e30 con 0x7f00d8009150 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f00d800c340 con 0x7f00d8009150 2024-09-16T06:45:12.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 605641214) 0x7f00e4005fd0 con 0x7f00d8009150 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6790/0 conn(0x7f00d8009150 legacy=0x7f00d80095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00ebfff700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6790/0 conn(0x7f00d8009150 legacy=0x7f00d80095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 64, 2024-09-16T06:45:12.442 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.443 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.444 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:12.445 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.446 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.447 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:12.449 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.450 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.451 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.452 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:12.453 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:12.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00d40360a0 legacy=0x7f00d4038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 10 monclient: shutdown 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 >> v1:172.21.15.76:6790/0 conn(0x7f00f409f230 legacy=0x7f00f4119e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f3fff700 1 -- 172.21.15.76:0/3233013796 reap_dead start 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f3fff700 1 -- 172.21.15.76:0/3233013796 reap_dead start 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 shutdown_connections 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 >> 172.21.15.76:0/3233013796 conn(0x7f00f40a3e50 msgr2=0x7f00f40702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 shutdown_connections 2024-09-16T06:45:12.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.437+0000 7f00f9e59700 1 -- 172.21.15.76:0/3233013796 wait complete. 2024-09-16T06:45:12.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:45:12.456 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:45:12.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:45:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: get_monmap_and_config 2024-09-16T06:45:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: build_initial_monmap 2024-09-16T06:45:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: monmap: 2024-09-16T06:45:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:12.567 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:12.569741+0000 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:12.569741+0000 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:12.568 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 1 Processor -- start 2024-09-16T06:45:12.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 1 -- start start 2024-09-16T06:45:12.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: init 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-b con 0x7f4f38132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-a con 0x7f4f3806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-c con 0x7f4f3806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3806c280 con 0x7f4f3806d480 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3806bfa0 con 0x7f4f3806cf50 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3809ace0 con 0x7f4f38132330 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3e3a3700 10 monclient(hunting): authenticate will time out at 1259.703964s 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.565+0000 7f4f3cba0700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4f3806d480 legacy=0x7f4f3813a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3d3a1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4f38132330 0x7f4f38132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37632/0 (socket says 172.21.15.76:37632) 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3d3a1700 1 -- 172.21.15.76:0/919904959 learned_addr learned my addr 172.21.15.76:0/919904959 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:12.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697124368 0 0) 0x7f4f3809ace0 con 0x7f4f38132330 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient(hunting): my global_id is 125615 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f240035d0 con 0x7f4f38132330 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778527886 0 0) 0x7f4f240035d0 con 0x7f4f38132330 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 >> v1:172.21.15.189:6789/0 conn(0x7f4f3806cf50 legacy=0x7f4f3813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 >> v1:172.21.15.76:6789/0 conn(0x7f4f3806d480 legacy=0x7f4f3813a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 monclient: found mon.noname-b 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3809ae70 con 0x7f4f38132330 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: _finish_auth 0 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 5 monclient: authenticate success, global_id 125615 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f34001610 con 0x7f4f38132330 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: dump: 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.573 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f34001930 con 0x7f4f38132330 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 1 -- 172.21.15.76:0/919904959 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f34001b20 con 0x7f4f38132330 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 4 set_mon_vals no callback set 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f2ffff700 10 monclient: dump: 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 10 monclient: shutdown 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/919904959 >> v1:172.21.15.76:6790/0 conn(0x7f4f38132330 legacy=0x7f4f38132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/919904959 shutdown_connections 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/919904959 >> 172.21.15.76:0/919904959 conn(0x7f4f380a41c0 msgr2=0x7f4f380a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/919904959 shutdown_connections 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/919904959 wait complete. 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 10 monclient: build_initial_monmap 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 10 monclient: monmap: 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:12.581 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 Processor -- start 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 1 -- start start 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.569+0000 7f4f3e3a3700 10 monclient: init 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-b con 0x7f4f3806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-c con 0x7f4f3806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient(hunting): picked mon.noname-a con 0x7f4f38132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:12.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f381316e0 con 0x7f4f38132330 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38130c00 con 0x7f4f3806cf50 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3812f330 con 0x7f4f3806d480 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient(hunting): authenticate will time out at 1259.708672s 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3dba2700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4f38132330 legacy=0x7f4f381d2430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3cba0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4f3806d480 0x7f4f381ced00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37638/0 (socket says 172.21.15.76:37638) 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3cba0700 1 -- 172.21.15.76:0/238659837 learned_addr learned my addr 172.21.15.76:0/238659837 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896271554 0 0) 0x7f4f3812f330 con 0x7f4f3806d480 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): my global_id is 125618 2024-09-16T06:45:12.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c0035d0 con 0x7f4f3806d480 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761040374 0 0) 0x7f4f1c0035d0 con 0x7f4f3806d480 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f3812f330 con 0x7f4f3806d480 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f28004200 con 0x7f4f3806d480 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): dump: 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268172544 0 0) 0x7f4f38130c00 con 0x7f4f3806cf50 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): my global_id is 174286 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c0035d0 con 0x7f4f3806cf50 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3243535431 0 0) 0x7f4f3812f330 con 0x7f4f3806d480 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6789/0 conn(0x7f4f3806cf50 legacy=0x7f4f3813e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.76:6789/0 conn(0x7f4f38132330 legacy=0x7f4f381d2430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 monclient: found mon.c 2024-09-16T06:45:12.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3812f5b0 con 0x7f4f3806d480 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 5 monclient: authenticate success, global_id 125618 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _renew_subs 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3812fc50 con 0x7f4f3806d480 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f28004420 con 0x7f4f3806d480 2024-09-16T06:45:12.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _renew_subs 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3813ece0 con 0x7f4f3806d480 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f28004730 con 0x7f4f3806d480 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient: dump: 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.587 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2f7fe700 4 set_mon_vals no callback set 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4f28004950 con 0x7f4f3806d480 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f18006590 con 0x7f4f18002830 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(173..173 src has 146..173) v4 ==== 6933+0+0 (unknown 2903559206 0 0) 0x7f4f28043f40 con 0x7f4f3806d480 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f3dba2700 10 monclient: get_auth_request con 0x7f4f1c0361d0 auth_method 0 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998774327 0 0) 0x7f4f18006590 con 0x7f4f18002830 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 10 monclient(hunting): my global_id is 174289 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.573+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3812f330 con 0x7f4f18002830 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3959383352 0 0) 0x7f4f3812f330 con 0x7f4f18002830 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f18006590 con 0x7f4f18002830 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f340015d0 con 0x7f4f18002830 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 10 monclient: dump: 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.589 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3187173817 0 0) 0x7f4f18006590 con 0x7f4f18002830 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f1800a190 con 0x7f4f18002830 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4f34004890 con 0x7f4f18002830 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6789/0 conn(0x7f4f18002830 legacy=0x7f4f18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.577+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6789/0 conn(0x7f4f18002830 legacy=0x7f4f18004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:12.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f3e3a3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:12.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f18009e50 con 0x7f4f18009170 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805636232 0 0) 0x7f4f18009e50 con 0x7f4f18009170 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient(hunting): my global_id is 174292 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f18006590 con 0x7f4f18009170 2024-09-16T06:45:12.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2755848954 0 0) 0x7f4f18006590 con 0x7f4f18009170 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f18009e50 con 0x7f4f18009170 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f34004e10 con 0x7f4f18009170 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient: dump: 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:12.593 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3936777066 0 0) 0x7f4f18009e50 con 0x7f4f18009170 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4f1800c340 con 0x7f4f18009170 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 3233101224) 0x7f4f34005020 con 0x7f4f18009170 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6789/0 conn(0x7f4f18009170 legacy=0x7f4f18009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.589+0000 7f4f2dffb700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6789/0 conn(0x7f4f18009170 legacy=0x7f4f18009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 64, 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:12.606 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4f1c0361d0 legacy=0x7f4f1c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 10 monclient: shutdown 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 >> v1:172.21.15.76:6790/0 conn(0x7f4f3806d480 legacy=0x7f4f381ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3dba2700 1 -- 172.21.15.76:0/238659837 reap_dead start 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3dba2700 1 -- 172.21.15.76:0/238659837 reap_dead start 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 shutdown_connections 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 >> 172.21.15.76:0/238659837 conn(0x7f4f380a41c0 msgr2=0x7f4f3809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 shutdown_connections 2024-09-16T06:45:12.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:12.593+0000 7f4f3e3a3700 1 -- 172.21.15.76:0/238659837 wait complete. 2024-09-16T06:45:12.609 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:45:12.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:45:12.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.919+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v174) v1 ==== 173+0+0 (unknown 557494537 0 0) 0x7f862801ce70 con 0x7f863409c410 2024-09-16T06:45:12.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.919+0000 7f8630ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:45:12.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:12.919+0000 7f8630ff9700 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T06:45:13.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.031+0000 7f863a285700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:45:13.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.031+0000 7f863a285700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:45:13.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.031+0000 7f863a285700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:13.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.031+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f86140053b0 con 0x7f863409c410 2024-09-16T06:45:13.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f8630ff9700 1 -- 172.21.15.189:0/4104154660 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v174) v1 ==== 183+0+0 (unknown 2338795732 0 0) 0x7f8628012cf0 con 0x7f863409c410 2024-09-16T06:45:13.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f8630ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T06:45:13.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f8630ff9700 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T06:45:13.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 >> v1:172.21.15.189:6812/3098810674 conn(0x7f86200361d0 legacy=0x7f8620038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 10 monclient: shutdown 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 >> v1:172.21.15.189:6789/0 conn(0x7f863409c410 legacy=0x7f86341cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 shutdown_connections 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 >> 172.21.15.189:0/4104154660 conn(0x7f86340a0f20 msgr2=0x7f8634098b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 shutdown_connections 2024-09-16T06:45:13.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.035+0000 7f863a285700 1 -- 172.21.15.189:0/4104154660 wait complete. 2024-09-16T06:45:13.040 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T06:45:13.049 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush move fooo rack=barr 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: get_monmap_and_config 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: build_initial_monmap 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: monmap: 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:13.157645+0000 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:13.157645+0000 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.157 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:13.158 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:13.158 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:13.158 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 1 Processor -- start 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 1 -- start start 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: init 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:13.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-b con 0x7fae5409eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-c con 0x7fae5409f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-a con 0x7fae54139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5409ab80 con 0x7fae54139a50 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5409a690 con 0x7fae5409f690 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae54072230 con 0x7fae5409eba0 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 10 monclient(hunting): authenticate will time out at 1265.670825s 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59b75700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae5409f690 0x7fae54138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50330/0 (socket says 172.21.15.189:50330) 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59b75700 1 -- 172.21.15.189:0/1769493561 learned_addr learned my addr 172.21.15.189:0/1769493561 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5ab77700 1 -- 172.21.15.189:0/1769493561 >> v1:172.21.15.76:6789/0 conn(0x7fae54139a50 legacy=0x7fae5413be40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766630514 0 0) 0x7fae5409a690 con 0x7fae5409f690 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _negotiate 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:13.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): my global_id is 174295 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae400035d0 con 0x7fae5409f690 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221636051 0 0) 0x7fae54072230 con 0x7fae5409eba0 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _negotiate 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient(hunting): my global_id is 125621 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5409a690 con 0x7fae5409eba0 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248734397 0 0) 0x7fae400035d0 con 0x7fae5409f690 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 >> v1:172.21.15.76:6790/0 conn(0x7fae5409eba0 legacy=0x7fae5409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 >> v1:172.21.15.76:6789/0 conn(0x7fae54139a50 legacy=0x7fae5413be40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:13.163 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: _finish_hunting 0 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 monclient: found mon.noname-c 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5409be90 con 0x7fae5409f690 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: _finish_auth 0 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: _check_auth_tickets 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae50001610 con 0x7fae5409f690 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 5 monclient: authenticate success, global_id 174295 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae59374700 10 monclient: dump: 2024-09-16T06:45:13.164 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.155+0000 7fae5b378700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae500018f0 con 0x7fae5409f690 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:13.165 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 1 -- 172.21.15.189:0/1769493561 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae50001c80 con 0x7fae5409f690 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59374700 10 monclient: dump: 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:13.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 4 set_mon_vals no callback set 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: shutdown 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- 172.21.15.189:0/1769493561 >> v1:172.21.15.189:6789/0 conn(0x7fae5409f690 legacy=0x7fae54138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- 172.21.15.189:0/1769493561 shutdown_connections 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- 172.21.15.189:0/1769493561 >> 172.21.15.189:0/1769493561 conn(0x7fae5409fe70 msgr2=0x7fae540a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- 172.21.15.189:0/1769493561 shutdown_connections 2024-09-16T06:45:13.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- 172.21.15.189:0/1769493561 wait complete. 2024-09-16T06:45:13.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: build_initial_monmap 2024-09-16T06:45:13.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: monmap: 2024-09-16T06:45:13.172 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:13.172 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:13.172 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 Processor -- start 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- start start 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: init 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-c con 0x7fae5409eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:13.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-b con 0x7fae5409f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient(hunting): picked mon.noname-a con 0x7fae54139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae54094b60 con 0x7fae54139a50 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5409a380 con 0x7fae5409f690 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5406b750 con 0x7fae5409eba0 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5b378700 10 monclient(hunting): authenticate will time out at 1265.675754s 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae5ab77700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fae54139a50 legacy=0x7fae541d2060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59b75700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae5409f690 0x7fae54138cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50332/0 (socket says 172.21.15.189:50332) 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.159+0000 7fae59b75700 1 -- 172.21.15.189:0/1605909213 learned_addr learned my addr 172.21.15.189:0/1605909213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014394227 0 0) 0x7fae5406b750 con 0x7fae5409eba0 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:13.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): my global_id is 125624 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae380035d0 con 0x7fae5409eba0 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523132295 0 0) 0x7fae5409a380 con 0x7fae5409f690 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): my global_id is 174298 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5406b750 con 0x7fae5409f690 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887981686 0 0) 0x7fae380035d0 con 0x7fae5409eba0 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae5409a380 con 0x7fae5409eba0 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae440018b0 con 0x7fae5409eba0 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): dump: 2024-09-16T06:45:13.175 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431885961 0 0) 0x7fae5406b750 con 0x7fae5409f690 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae380035d0 con 0x7fae5409f690 2024-09-16T06:45:13.176 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae500015d0 con 0x7fae5409f690 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient(hunting): dump: 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448044256 0 0) 0x7fae5409a380 con 0x7fae5409eba0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 >> v1:172.21.15.189:6789/0 conn(0x7fae5409f690 legacy=0x7fae54138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 >> v1:172.21.15.76:6789/0 conn(0x7fae54139a50 legacy=0x7fae541d2060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 monclient: found mon.c 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae54071300 con 0x7fae5409eba0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: _finish_auth 0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 5 monclient: authenticate success, global_id 125624 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _renew_subs 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:13.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae54094880 con 0x7fae5409eba0 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _renew_subs 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae54098cd0 con 0x7fae5409eba0 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae440036c0 con 0x7fae5409eba0 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fae44003a50 con 0x7fae5409eba0 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae58b73700 4 set_mon_vals no callback set 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 10 monclient: dump: 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.179 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c005100 con 0x7fae5409eba0 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fae4400e540 con 0x7fae5409eba0 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.163+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(174..174 src has 146..174) v4 ==== 7121+0+0 (unknown 1740853683 0 0) 0x7fae44042ed0 con 0x7fae5409eba0 2024-09-16T06:45:13.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.167+0000 7fae59b75700 10 monclient: get_auth_request con 0x7fae380362a0 auth_method 0 2024-09-16T06:45:13.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.175+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fae44013460 con 0x7fae5409eba0 2024-09-16T06:45:13.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.175+0000 7fae4affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:13.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.175+0000 7fae4affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:13.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.351+0000 7fae5b378700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:13.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.351+0000 7fae5b378700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:13.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.351+0000 7fae5b378700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:13.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.351+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fae3c005930 con 0x7fae5409eba0 2024-09-16T06:45:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.935+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v175) v1 ==== 167+0+0 (unknown 2588870590 0 0) 0x7fae4401bc80 con 0x7fae5409eba0 2024-09-16T06:45:13.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.935+0000 7fae4affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:13.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:13.935+0000 7fae4affd700 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T06:45:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae5b378700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:14.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae5b378700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae5b378700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fae3c0053b0 con 0x7fae5409eba0 2024-09-16T06:45:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae4affd700 1 -- 172.21.15.189:0/1605909213 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v175) v1 ==== 177+0+0 (unknown 1343789255 0 0) 0x7fae44011b00 con 0x7fae5409eba0 2024-09-16T06:45:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae4affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T06:45:14.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.047+0000 7fae4affd700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 >> v1:172.21.15.189:6812/3098810674 conn(0x7fae380362a0 legacy=0x7fae38038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 10 monclient: shutdown 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 >> v1:172.21.15.76:6790/0 conn(0x7fae5409eba0 legacy=0x7fae541385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 shutdown_connections 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 >> 172.21.15.189:0/1605909213 conn(0x7fae5409fe70 msgr2=0x7fae5406bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 shutdown_connections 2024-09-16T06:45:14.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.051+0000 7fae5b378700 1 -- 172.21.15.189:0/1605909213 wait complete. 2024-09-16T06:45:14.054 INFO:tasks.workunit.client.0.smithi189.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T06:45:14.064 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm fooo 2024-09-16T06:45:14.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: get_monmap_and_config 2024-09-16T06:45:14.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: build_initial_monmap 2024-09-16T06:45:14.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: monmap: 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:14.162007+0000 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:14.162007+0000 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:14.162 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 1 Processor -- start 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 1 -- start start 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: init 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-c con 0x7f57a409eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-a con 0x7f57a409beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-b con 0x7f57a409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:14.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a4099ed0 con 0x7f57a409beb0 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a409d2e0 con 0x7f57a409eba0 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a409ef80 con 0x7f57a409c2e0 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a96fb700 10 monclient(hunting): authenticate will time out at 1266.675712s 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57a409eba0 0x7f57a409b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50340/0 (socket says 172.21.15.189:50340) 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f57a409beb0 legacy=0x7f57a4137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.159+0000 7f57a3fff700 1 -- 172.21.15.189:0/2660268665 learned_addr learned my addr 172.21.15.189:0/2660268665 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492391181 0 0) 0x7f57a409d2e0 con 0x7f57a409eba0 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:14.168 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): my global_id is 174301 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57900035d0 con 0x7f57a409eba0 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682767571 0 0) 0x7f57a409ef80 con 0x7f57a409c2e0 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient(hunting): my global_id is 125627 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a409d2e0 con 0x7f57a409c2e0 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414474136 0 0) 0x7f57900035d0 con 0x7f57a409eba0 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 >> v1:172.21.15.76:6790/0 conn(0x7f57a409c2e0 legacy=0x7f57a413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:14.169 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 >> v1:172.21.15.76:6789/0 conn(0x7f57a409beb0 legacy=0x7f57a4137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 monclient: found mon.noname-c 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a409cc70 con 0x7f57a409eba0 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:14.170 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 5 monclient: authenticate success, global_id 174301 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f578c001610 con 0x7f57a409eba0 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: dump: 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:14.171 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f578c001880 con 0x7f57a409eba0 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 1 -- 172.21.15.189:0/2660268665 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f578c001c10 con 0x7f57a409eba0 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 4 set_mon_vals no callback set 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:14.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a2ffd700 10 monclient: dump: 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 10 monclient: shutdown 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 1 -- 172.21.15.189:0/2660268665 >> v1:172.21.15.189:6789/0 conn(0x7f57a409eba0 legacy=0x7f57a409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:14.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 1 -- 172.21.15.189:0/2660268665 shutdown_connections 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 1 -- 172.21.15.189:0/2660268665 >> 172.21.15.189:0/2660268665 conn(0x7f57a409fe70 msgr2=0x7f57a40a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 1 -- 172.21.15.189:0/2660268665 shutdown_connections 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 1 -- 172.21.15.189:0/2660268665 wait complete. 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 10 monclient: build_initial_monmap 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.163+0000 7f57a96fb700 10 monclient: monmap: 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:14.174 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 Processor -- start 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- start start 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: init 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-c con 0x7f57a409beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-b con 0x7f57a409c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient(hunting): picked mon.noname-a con 0x7f57a409eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a412daa0 con 0x7f57a409eba0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a412dfd0 con 0x7f57a409c2e0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a409b1c0 con 0x7f57a409beb0 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:14.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient(hunting): authenticate will time out at 1266.681419s 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57a409c2e0 0x7f57a41cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50352/0 (socket says 172.21.15.189:50352) 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a37fe700 1 -- 172.21.15.189:0/3080996744 learned_addr learned my addr 172.21.15.189:0/3080996744 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a8efa700 1 -- 172.21.15.189:0/3080996744 >> v1:172.21.15.76:6789/0 conn(0x7f57a409eba0 legacy=0x7f57a41d0190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891335083 0 0) 0x7f57a412dfd0 con 0x7f57a409c2e0 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): my global_id is 174304 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57840035d0 con 0x7f57a409c2e0 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375781432 0 0) 0x7f57a409b1c0 con 0x7f57a409beb0 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:14.181 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): my global_id is 125630 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a412dfd0 con 0x7f57a409beb0 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749853822 0 0) 0x7f57840035d0 con 0x7f57a409c2e0 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57a409b1c0 con 0x7f57a409c2e0 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57940045b0 con 0x7f57a409c2e0 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.182 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:14.183 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926639699 0 0) 0x7f57a412dfd0 con 0x7f57a409beb0 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57840035d0 con 0x7f57a409beb0 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f578c0015d0 con 0x7f57a409beb0 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:14.184 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018612210 0 0) 0x7f57a409b1c0 con 0x7f57a409c2e0 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 >> v1:172.21.15.76:6790/0 conn(0x7f57a409beb0 legacy=0x7f57a41327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 >> v1:172.21.15.76:6789/0 conn(0x7f57a409eba0 legacy=0x7f57a41d0190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 monclient: found mon.b 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40a2620 con 0x7f57a409c2e0 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:14.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 5 monclient: authenticate success, global_id 174304 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _renew_subs 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a40a20f0 con 0x7f57a409c2e0 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _renew_subs 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a412b970 con 0x7f57a409c2e0 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5794004820 con 0x7f57a409c2e0 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:14.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5794004bb0 con 0x7f57a409c2e0 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a27fc700 4 set_mon_vals no callback set 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.167+0000 7f57a0ff9700 10 monclient: dump: 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:14.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a96fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a96fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a96fb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5788005100 con 0x7f57a409c2e0 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5794004e10 con 0x7f57a409c2e0 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(175..175 src has 146..175) v4 ==== 7121+0+0 (unknown 2975821317 0 0) 0x7f5794044460 con 0x7f57a409c2e0 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.171+0000 7f57a3fff700 10 monclient: get_auth_request con 0x7f57840362a0 auth_method 0 2024-09-16T06:45:14.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.175+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5794014bd0 con 0x7f57a409c2e0 2024-09-16T06:45:14.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.175+0000 7f57a0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:14.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.175+0000 7f57a0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:14.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.351+0000 7f57a96fb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:14.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.351+0000 7f57a96fb700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:14.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.351+0000 7f57a96fb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:14.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.351+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f57880053b0 con 0x7f57a409c2e0 2024-09-16T06:45:14.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.939+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v176) v1 ==== 122+0+0 (unknown 1383660573 0 0) 0x7f5794013040 con 0x7f57a409c2e0 2024-09-16T06:45:14.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.939+0000 7f57a0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:14.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:14.939+0000 7f57a0ff9700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-16T06:45:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.051+0000 7f57a96fb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.051+0000 7f57a96fb700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.051+0000 7f57a96fb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.051+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f5788005650 con 0x7f57a409c2e0 2024-09-16T06:45:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.055+0000 7f57a0ff9700 1 -- 172.21.15.189:0/3080996744 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v176) v1 ==== 122+0+0 (unknown 2061036236 0 0) 0x7f57940149a0 con 0x7f57a409c2e0 2024-09-16T06:45:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.055+0000 7f57a0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T06:45:15.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.055+0000 7f57a0ff9700 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 >> v1:172.21.15.189:6812/3098810674 conn(0x7f57840362a0 legacy=0x7f5784038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 10 monclient: shutdown 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 >> v1:172.21.15.189:6789/0 conn(0x7f57a409c2e0 legacy=0x7f57a41cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 shutdown_connections 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 >> 172.21.15.189:0/3080996744 conn(0x7f57a409fe70 msgr2=0x7f57a409db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:15.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 shutdown_connections 2024-09-16T06:45:15.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.059+0000 7f57a96fb700 1 -- 172.21.15.189:0/3080996744 wait complete. 2024-09-16T06:45:15.061 INFO:tasks.workunit.client.0.smithi189.stderr:device 'fooo' does not appear in the crush map 2024-09-16T06:45:15.072 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm barr 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.175+0000 7f3b60890700 10 monclient: get_monmap_and_config 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.175+0000 7f3b60890700 10 monclient: build_initial_monmap 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: monmap: 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:15.180527+0000 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:15.180527+0000 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:15.180 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:15.181 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:15.181 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:15.181 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 Processor -- start 2024-09-16T06:45:15.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- start start 2024-09-16T06:45:15.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: init 2024-09-16T06:45:15.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:15.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-b con 0x7f3b5c09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-c con 0x7f3b5c09beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c099ed0 con 0x7f3b5c09c2e0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c09d2e0 con 0x7f3b5c09beb0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c09ef80 con 0x7f3b5c09eba0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient(hunting): authenticate will time out at 1267.693435s 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b5b59e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3b5c09c2e0 legacy=0x7f3b5c13ac60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b5a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b5c09beb0 0x7f3b5c137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50368/0 (socket says 172.21.15.189:50368) 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b5a59c700 1 -- 172.21.15.189:0/1885802907 learned_addr learned my addr 172.21.15.189:0/1885802907 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321851980 0 0) 0x7f3b5c09d2e0 con 0x7f3b5c09beb0 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:45:15.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): my global_id is 174307 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b440057e0 con 0x7f3b5c09beb0 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273198340 0 0) 0x7f3b5c09ef80 con 0x7f3b5c09eba0 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient(hunting): my global_id is 125633 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b5c09d2e0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215828584 0 0) 0x7f3b440057e0 con 0x7f3b5c09beb0 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 >> v1:172.21.15.76:6790/0 conn(0x7f3b5c09eba0 legacy=0x7f3b5c09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 >> v1:172.21.15.76:6789/0 conn(0x7f3b5c09c2e0 legacy=0x7f3b5c13ac60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:15.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 monclient: found mon.noname-c 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c09cc70 con 0x7f3b5c09beb0 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: _finish_auth 0 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b50001610 con 0x7f3b5c09beb0 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 5 monclient: authenticate success, global_id 174307 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:15.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: dump: 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b50001880 con 0x7f3b5c09beb0 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:15.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 1 -- 172.21.15.189:0/1885802907 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b50001c10 con 0x7f3b5c09beb0 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 4 set_mon_vals no callback set 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b59d9b700 10 monclient: dump: 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:15.188 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 10 monclient: shutdown 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- 172.21.15.189:0/1885802907 >> v1:172.21.15.189:6789/0 conn(0x7f3b5c09beb0 legacy=0x7f3b5c137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- 172.21.15.189:0/1885802907 shutdown_connections 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- 172.21.15.189:0/1885802907 >> 172.21.15.189:0/1885802907 conn(0x7f3b5c09fe70 msgr2=0x7f3b5c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- 172.21.15.189:0/1885802907 shutdown_connections 2024-09-16T06:45:15.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.179+0000 7f3b60890700 1 -- 172.21.15.189:0/1885802907 wait complete. 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: build_initial_monmap 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: monmap: 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:15.193 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 Processor -- start 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- start start 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: init 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-c con 0x7f3b5c09beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient(hunting): picked mon.noname-b con 0x7f3b5c09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c1330e0 con 0x7f3b5c09c2e0 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0a1610 con 0x7f3b5c09eba0 2024-09-16T06:45:15.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0a1100 con 0x7f3b5c09beb0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient(hunting): authenticate will time out at 1267.697210s 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b5b59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b5c09eba0 0x7f3b5c1ce070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50372/0 (socket says 172.21.15.189:50372) 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b5b59e700 1 -- 172.21.15.189:0/1827139286 learned_addr learned my addr 172.21.15.189:0/1827139286 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981436631 0 0) 0x7f3b5c0a1610 con 0x7f3b5c09eba0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): my global_id is 174310 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b480035d0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1320778287 0 0) 0x7f3b480035d0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b5c0a1610 con 0x7f3b5c09eba0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b4c0040e0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): dump: 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683102798 0 0) 0x7f3b5c0a1100 con 0x7f3b5c09beb0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient(hunting): my global_id is 125636 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b480035d0 con 0x7f3b5c09beb0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 78288851 0 0) 0x7f3b5c0a1610 con 0x7f3b5c09eba0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 >> v1:172.21.15.76:6790/0 conn(0x7f3b5c09beb0 legacy=0x7f3b5c1329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 >> v1:172.21.15.76:6789/0 conn(0x7f3b5c09c2e0 legacy=0x7f3b5c1ca940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 monclient: found mon.b 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c09a380 con 0x7f3b5c09eba0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 5 monclient: authenticate success, global_id 174310 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _renew_subs 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c12d460 con 0x7f3b5c09eba0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _renew_subs 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c12bef0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b4c0043d0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b4c004760 con 0x7f3b5c09eba0 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:15.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b5959a700 4 set_mon_vals no callback set 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 10 monclient: dump: 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:15.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c005100 con 0x7f3b5c09eba0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3b4c004980 con 0x7f3b5c09eba0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(176..176 src has 146..176) v4 ==== 6905+0+0 (unknown 194181137 0 0) 0x7f3b4c043ea0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.183+0000 7f3b5ad9d700 10 monclient: get_auth_request con 0x7f3b48036220 auth_method 0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.191+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3b4c005180 con 0x7f3b5c09eba0 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.191+0000 7f3b437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:15.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.191+0000 7f3b437fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.351+0000 7f3b60890700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.351+0000 7f3b60890700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.351+0000 7f3b60890700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:15.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.351+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f3b3c0053b0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.959+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v177) v1 ==== 122+0+0 (unknown 1898091658 0 0) 0x7f3b4c012bb0 con 0x7f3b5c09eba0 2024-09-16T06:45:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.959+0000 7f3b437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:15.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:15.959+0000 7f3b437fe700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-16T06:45:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.063+0000 7f3b60890700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.063+0000 7f3b60890700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.063+0000 7f3b60890700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.063+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f3b3c005650 con 0x7f3b5c09eba0 2024-09-16T06:45:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b437fe700 1 -- 172.21.15.189:0/1827139286 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v177) v1 ==== 122+0+0 (unknown 989720853 0 0) 0x7f3b4c014330 con 0x7f3b5c09eba0 2024-09-16T06:45:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b437fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T06:45:16.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b437fe700 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3b48036220 legacy=0x7f3b480386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b60890700 10 monclient: shutdown 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b60890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.067+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 >> v1:172.21.15.189:6789/0 conn(0x7f3b5c09eba0 legacy=0x7f3b5c1ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.071+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 shutdown_connections 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.071+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 >> 172.21.15.189:0/1827139286 conn(0x7f3b5c09fe70 msgr2=0x7f3b5c09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.071+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 shutdown_connections 2024-09-16T06:45:16.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.071+0000 7f3b60890700 1 -- 172.21.15.189:0/1827139286 wait complete. 2024-09-16T06:45:16.073 INFO:tasks.workunit.client.0.smithi189.stderr:device 'barr' does not appear in the crush map 2024-09-16T06:45:16.082 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: get_monmap_and_config 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: build_initial_monmap 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: monmap: 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:16.188600+0000 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:16.188600+0000 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.188 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:16.189 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:16.189 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 Processor -- start 2024-09-16T06:45:16.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- start start 2024-09-16T06:45:16.193 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: init 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-a con 0x7fdc600707a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-b con 0x7fdc60070b80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-c con 0x7fdc6013b510 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc600a12a0 con 0x7fdc600707a0 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc600a0670 con 0x7fdc6013b510 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6009f380 con 0x7fdc60070b80 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient(hunting): authenticate will time out at 1268.701586s 2024-09-16T06:45:16.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc648c6700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc6013b510 0x7fdc6013d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50388/0 (socket says 172.21.15.189:50388) 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fdc600707a0 legacy=0x7fdc6009ff60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc648c6700 1 -- 172.21.15.189:0/1522565758 learned_addr learned my addr 172.21.15.189:0/1522565758 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657968541 0 0) 0x7fdc600a0670 con 0x7fdc6013b510 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): my global_id is 174313 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc400035d0 con 0x7fdc6013b510 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923744388 0 0) 0x7fdc6009f380 con 0x7fdc60070b80 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:16.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient(hunting): my global_id is 125639 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc600a0670 con 0x7fdc60070b80 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748509974 0 0) 0x7fdc400035d0 con 0x7fdc6013b510 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 >> v1:172.21.15.76:6790/0 conn(0x7fdc60070b80 legacy=0x7fdc60070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 >> v1:172.21.15.76:6789/0 conn(0x7fdc600707a0 legacy=0x7fdc6009ff60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 monclient: found mon.noname-c 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6009f660 con 0x7fdc6013b510 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: _finish_auth 0 2024-09-16T06:45:16.196 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 5 monclient: authenticate success, global_id 174313 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc58001990 con 0x7fdc6013b510 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: dump: 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:16.197 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc58001c00 con 0x7fdc6013b510 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 4 set_mon_vals no callback set 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 1 -- 172.21.15.189:0/1522565758 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc58001f90 con 0x7fdc6013b510 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc5effd700 10 monclient: dump: 2024-09-16T06:45:16.198 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 10 monclient: shutdown 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- 172.21.15.189:0/1522565758 >> v1:172.21.15.189:6789/0 conn(0x7fdc6013b510 legacy=0x7fdc6013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- 172.21.15.189:0/1522565758 shutdown_connections 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- 172.21.15.189:0/1522565758 >> 172.21.15.189:0/1522565758 conn(0x7fdc600a44a0 msgr2=0x7fdc600a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:16.199 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- 172.21.15.189:0/1522565758 shutdown_connections 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.187+0000 7fdc66329700 1 -- 172.21.15.189:0/1522565758 wait complete. 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient: build_initial_monmap 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient: monmap: 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:16.200 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 1 Processor -- start 2024-09-16T06:45:16.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 1 -- start start 2024-09-16T06:45:16.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient: init 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-c con 0x7fdc600707a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-b con 0x7fdc60070b80 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient(hunting): picked mon.noname-a con 0x7fdc6013b510 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc600a33a0 con 0x7fdc6013b510 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6009b0d0 con 0x7fdc60070b80 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc60137b80 con 0x7fdc600707a0 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc66329700 10 monclient(hunting): authenticate will time out at 1268.706749s 2024-09-16T06:45:16.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc60070b80 0x7fdc6013d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50400/0 (socket says 172.21.15.189:50400) 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc648c6700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fdc6013b510 legacy=0x7fdc601d1fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5f7fe700 1 -- 172.21.15.189:0/1771795227 learned_addr learned my addr 172.21.15.189:0/1771795227 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234636719 0 0) 0x7fdc6009b0d0 con 0x7fdc60070b80 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): my global_id is 174316 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc440035d0 con 0x7fdc60070b80 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048792770 0 0) 0x7fdc60137b80 con 0x7fdc600707a0 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): my global_id is 125642 2024-09-16T06:45:16.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6009b0d0 con 0x7fdc600707a0 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 85585164 0 0) 0x7fdc440035d0 con 0x7fdc60070b80 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc60137b80 con 0x7fdc60070b80 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc50001910 con 0x7fdc60070b80 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): dump: 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:16.208 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334824543 0 0) 0x7fdc6009b0d0 con 0x7fdc600707a0 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.191+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc440035d0 con 0x7fdc600707a0 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc480041a0 con 0x7fdc600707a0 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient(hunting): dump: 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.209 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1091348069 0 0) 0x7fdc60137b80 con 0x7fdc60070b80 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 >> v1:172.21.15.76:6790/0 conn(0x7fdc600707a0 legacy=0x7fdc6013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 >> v1:172.21.15.76:6789/0 conn(0x7fdc6013b510 legacy=0x7fdc601d1fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 monclient: found mon.b 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc601367c0 con 0x7fdc60070b80 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 5 monclient: authenticate success, global_id 174316 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _renew_subs 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6013e360 con 0x7fdc60070b80 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _renew_subs 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6013d750 con 0x7fdc60070b80 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc50001ca0 con 0x7fdc60070b80 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdc50003b10 con 0x7fdc60070b80 2024-09-16T06:45:16.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5e7fc700 4 set_mon_vals no callback set 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 10 monclient: dump: 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:16.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdc50003cf0 con 0x7fdc60070b80 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38005100 con 0x7fdc60070b80 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.195+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(177..177 src has 151..177) v4 ==== 6689+0+0 (unknown 1757117662 0 0) 0x7fdc50043df0 con 0x7fdc60070b80 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.199+0000 7fdc5ffff700 10 monclient: get_auth_request con 0x7fdc44036120 auth_method 0 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.203+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdc500148c0 con 0x7fdc60070b80 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.203+0000 7fdc5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:16.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.203+0000 7fdc5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.379+0000 7fdc66329700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.379+0000 7fdc66329700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.379+0000 7fdc66329700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:16.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.379+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdc380053b0 con 0x7fdc60070b80 2024-09-16T06:45:16.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.963+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v178) v1 ==== 78+0+0 (unknown 1401286797 0 0) 0x7fdc50012d30 con 0x7fdc60070b80 2024-09-16T06:45:16.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.967+0000 7fdc5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:16.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:16.967+0000 7fdc5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.079+0000 7fdc66329700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.079+0000 7fdc66329700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.079+0000 7fdc66329700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:17.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.079+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fdc38005900 con 0x7fdc60070b80 2024-09-16T06:45:17.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.975+0000 7fdc5cff9700 1 -- 172.21.15.189:0/1771795227 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v179) v1 ==== 78+0+0 (unknown 306494120 0 0) 0x7fdc500144b0 con 0x7fdc60070b80 2024-09-16T06:45:17.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.975+0000 7fdc5cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:45:17.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.975+0000 7fdc5cff9700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdc44036120 legacy=0x7fdc440385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 10 monclient: shutdown 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 >> v1:172.21.15.189:6789/0 conn(0x7fdc60070b80 legacy=0x7fdc6013d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 shutdown_connections 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.979+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 >> 172.21.15.189:0/1771795227 conn(0x7fdc600a44a0 msgr2=0x7fdc60130d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.983+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 shutdown_connections 2024-09-16T06:45:17.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:17.983+0000 7fdc66329700 1 -- 172.21.15.189:0/1771795227 wait complete. 2024-09-16T06:45:17.998 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T06:45:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: get_monmap_and_config 2024-09-16T06:45:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: build_initial_monmap 2024-09-16T06:45:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: monmap: 2024-09-16T06:45:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:18.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:18.105594+0000 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:18.105594+0000 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:18.106 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 1 Processor -- start 2024-09-16T06:45:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 1 -- start start 2024-09-16T06:45:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: init 2024-09-16T06:45:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:18.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-a con 0x7febbc09f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-c con 0x7febbc099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-b con 0x7febbc1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc09f520 con 0x7febbc09f140 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc09bc10 con 0x7febbc099700 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc09b810 con 0x7febbc1396a0 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 10 monclient(hunting): authenticate will time out at 1270.618992s 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febb2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7febbc099700 0x7febbc09b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50406/0 (socket says 172.21.15.189:50406) 2024-09-16T06:45:18.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febb2ffd700 1 -- 172.21.15.189:0/3299856417 learned_addr learned my addr 172.21.15.189:0/3299856417 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbb7fe700 1 -- 172.21.15.189:0/3299856417 >> v1:172.21.15.76:6789/0 conn(0x7febbc09f140 legacy=0x7febbc09a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428945521 0 0) 0x7febbc09bc10 con 0x7febbc099700 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): my global_id is 174319 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba80035d0 con 0x7febbc099700 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081396464 0 0) 0x7febbc09b810 con 0x7febbc1396a0 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:18.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient(hunting): my global_id is 125645 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc09bc10 con 0x7febbc1396a0 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431359833 0 0) 0x7feba80035d0 con 0x7febbc099700 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 >> v1:172.21.15.76:6790/0 conn(0x7febbc1396a0 legacy=0x7febbc13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 >> v1:172.21.15.76:6789/0 conn(0x7febbc09f140 legacy=0x7febbc09a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 monclient: found mon.noname-c 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc09ceb0 con 0x7febbc099700 2024-09-16T06:45:18.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba4001610 con 0x7febbc099700 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 5 monclient: authenticate success, global_id 174319 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febc1c69700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.103+0000 7febbaffd700 10 monclient: dump: 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.113 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba40018f0 con 0x7febbc099700 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 1 -- 172.21.15.189:0/3299856417 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba4001c80 con 0x7febbc099700 2024-09-16T06:45:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 4 set_mon_vals no callback set 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbaffd700 10 monclient: dump: 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:18.115 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: shutdown 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- 172.21.15.189:0/3299856417 >> v1:172.21.15.189:6789/0 conn(0x7febbc099700 legacy=0x7febbc09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- 172.21.15.189:0/3299856417 shutdown_connections 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- 172.21.15.189:0/3299856417 >> 172.21.15.189:0/3299856417 conn(0x7febbc0a03e0 msgr2=0x7febbc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- 172.21.15.189:0/3299856417 shutdown_connections 2024-09-16T06:45:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- 172.21.15.189:0/3299856417 wait complete. 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: build_initial_monmap 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: monmap: 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:18.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 Processor -- start 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- start start 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: init 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-a con 0x7febbc099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-b con 0x7febbc09f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient(hunting): picked mon.noname-c con 0x7febbc1396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0a14f0 con 0x7febbc099700 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc071160 con 0x7febbc09f140 2024-09-16T06:45:18.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febbb7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7febbc099700 legacy=0x7febbc095bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc0962e0 con 0x7febbc1396a0 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febc1c69700 10 monclient(hunting): authenticate will time out at 1270.623596s 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7febbc09f140 0x7febbc1ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50420/0 (socket says 172.21.15.189:50420) 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb2ffd700 1 -- 172.21.15.189:0/2452880814 learned_addr learned my addr 172.21.15.189:0/2452880814 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114455872 0 0) 0x7febbc071160 con 0x7febbc09f140 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 10 monclient(hunting): my global_id is 174322 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.107+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c0035d0 con 0x7febbc09f140 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3299250953 0 0) 0x7feb9c0035d0 con 0x7febbc09f140 2024-09-16T06:45:18.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febbc071160 con 0x7febbc09f140 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba40015d0 con 0x7febbc09f140 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:18.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 980314450 0 0) 0x7febbc071160 con 0x7febbc09f140 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 >> v1:172.21.15.76:6790/0 conn(0x7febbc1396a0 legacy=0x7febbc1d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 >> v1:172.21.15.76:6789/0 conn(0x7febbc099700 legacy=0x7febbc095bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 monclient: found mon.b 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc0946a0 con 0x7febbc09f140 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 5 monclient: authenticate success, global_id 174322 2024-09-16T06:45:18.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _renew_subs 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc09a6e0 con 0x7febbc09f140 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _renew_subs 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc0943c0 con 0x7febbc09f140 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba4004ca0 con 0x7febbc09f140 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feba4005030 con 0x7febbc09f140 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 10 monclient: dump: 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.125 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febba7fc700 4 set_mon_vals no callback set 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7feba4005260 con 0x7febbc09f140 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(179..179 src has 151..179) v4 ==== 6361+0+0 (unknown 128975357 0 0) 0x7feba4043ff0 con 0x7febbc09f140 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febbb7fe700 10 monclient: get_auth_request con 0x7feb9c0360a0 auth_method 0 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.111+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba0005100 con 0x7febbc09f140 2024-09-16T06:45:18.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.119+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feba40068a0 con 0x7febbc09f140 2024-09-16T06:45:18.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.119+0000 7febb8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:18.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.119+0000 7febb8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:18.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.303+0000 7febc1c69700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:18.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.303+0000 7febc1c69700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:18.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.303+0000 7febc1c69700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:18.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.303+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7feba00053b0 con 0x7febbc09f140 2024-09-16T06:45:18.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.979+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v180) v1 ==== 82+0+0 (unknown 3647244962 0 0) 0x7feba4014780 con 0x7febbc09f140 2024-09-16T06:45:18.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.979+0000 7febb8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:18.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:18.979+0000 7febb8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:19.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febc1c69700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:19.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febc1c69700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:19.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febc1c69700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7feba00023e0 con 0x7febbc09f140 2024-09-16T06:45:19.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febb8ff9700 1 -- 172.21.15.189:0/2452880814 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v180) v1 ==== 115+0+0 (unknown 856466574 0 0) 0x7feba4014960 con 0x7febbc09f140 2024-09-16T06:45:19.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febb8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T06:45:19.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.091+0000 7febb8ff9700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 >> v1:172.21.15.189:6812/3098810674 conn(0x7feb9c0360a0 legacy=0x7feb9c038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 10 monclient: shutdown 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 >> v1:172.21.15.189:6789/0 conn(0x7febbc09f140 legacy=0x7febbc1ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 shutdown_connections 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 >> 172.21.15.189:0/2452880814 conn(0x7febbc0a03e0 msgr2=0x7febbc06fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 shutdown_connections 2024-09-16T06:45:19.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.095+0000 7febc1c69700 1 -- 172.21.15.189:0/2452880814 wait complete. 2024-09-16T06:45:19.098 INFO:tasks.workunit.client.0.smithi189.stderr:compat weight-set already created 2024-09-16T06:45:19.107 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-16T06:45:19.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 10 monclient: get_monmap_and_config 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 10 monclient: build_initial_monmap 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 10 monclient: monmap: 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:19.214412+0000 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:19.214412+0000 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:19.214 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 1 Processor -- start 2024-09-16T06:45:19.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 1 -- start start 2024-09-16T06:45:19.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.211+0000 7ff1ecc90700 10 monclient: init 2024-09-16T06:45:19.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:19.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-a con 0x7ff1e8133380 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-b con 0x7ff1e809f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-c con 0x7ff1e80a1190 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e8132ec0 con 0x7ff1e8133380 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e809e4c0 con 0x7ff1e80a1190 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e809e7a0 con 0x7ff1e809f760 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient(hunting): authenticate will time out at 1271.728874s 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e659c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff1e8133380 legacy=0x7ff1e8133760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e6d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff1e80a1190 0x7ff1e813b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35950/0 (socket says 172.21.15.189:35950) 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e6d9d700 1 -- 172.21.15.189:0/1944048207 learned_addr learned my addr 172.21.15.189:0/1944048207 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:19.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773370749 0 0) 0x7ff1e809e4c0 con 0x7ff1e80a1190 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): my global_id is 174325 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1cc0035d0 con 0x7ff1e80a1190 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293281347 0 0) 0x7ff1e809e7a0 con 0x7ff1e809f760 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient(hunting): my global_id is 125651 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e809e4c0 con 0x7ff1e809f760 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938039417 0 0) 0x7ff1cc0035d0 con 0x7ff1e80a1190 2024-09-16T06:45:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 >> v1:172.21.15.76:6790/0 conn(0x7ff1e809f760 legacy=0x7ff1e8138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 >> v1:172.21.15.76:6789/0 conn(0x7ff1e8133380 legacy=0x7ff1e8133760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: _finish_hunting 0 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 monclient: found mon.noname-c 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e80a1cb0 con 0x7ff1e80a1190 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: _finish_auth 0 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: _check_auth_tickets 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 5 monclient: authenticate success, global_id 174325 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1e0001990 con 0x7ff1e80a1190 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: dump: 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:19.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e0001c70 con 0x7ff1e80a1190 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 4 set_mon_vals no callback set 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 1 -- 172.21.15.189:0/1944048207 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1e0002000 con 0x7ff1e80a1190 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1e559a700 10 monclient: dump: 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:19.228 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: shutdown 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- 172.21.15.189:0/1944048207 >> v1:172.21.15.189:6789/0 conn(0x7ff1e80a1190 legacy=0x7ff1e813b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- 172.21.15.189:0/1944048207 shutdown_connections 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- 172.21.15.189:0/1944048207 >> 172.21.15.189:0/1944048207 conn(0x7ff1e80a21f0 msgr2=0x7ff1e80a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- 172.21.15.189:0/1944048207 shutdown_connections 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 1 -- 172.21.15.189:0/1944048207 wait complete. 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: build_initial_monmap 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.215+0000 7ff1ecc90700 10 monclient: monmap: 2024-09-16T06:45:19.229 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 Processor -- start 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- start start 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: init 2024-09-16T06:45:19.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-a con 0x7ff1e809f760 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-c con 0x7ff1e80a1190 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient(hunting): picked mon.noname-b con 0x7ff1e8133380 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e80a4bf0 con 0x7ff1e809f760 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e8132d50 con 0x7ff1e8133380 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e809c700 con 0x7ff1e80a1190 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient(hunting): authenticate will time out at 1271.733070s 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1e6d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff1e8133380 0x7ff1e81d2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35954/0 (socket says 172.21.15.189:35954) 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1e6d9d700 1 -- 172.21.15.189:0/2174249361 learned_addr learned my addr 172.21.15.189:0/2174249361 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1e659c700 1 -- 172.21.15.189:0/2174249361 >> v1:172.21.15.76:6789/0 conn(0x7ff1e809f760 legacy=0x7ff1e812f0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:19.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630881993 0 0) 0x7ff1e8132d50 con 0x7ff1e8133380 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): my global_id is 174328 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d00035d0 con 0x7ff1e8133380 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105413457 0 0) 0x7ff1e809c700 con 0x7ff1e80a1190 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): my global_id is 125654 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8132d50 con 0x7ff1e80a1190 2024-09-16T06:45:19.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1158734754 0 0) 0x7ff1d00035d0 con 0x7ff1e8133380 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1e809c700 con 0x7ff1e8133380 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1e0001de0 con 0x7ff1e8133380 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): dump: 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:19.234 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633644849 0 0) 0x7ff1e809c700 con 0x7ff1e8133380 2024-09-16T06:45:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 >> v1:172.21.15.76:6790/0 conn(0x7ff1e80a1190 legacy=0x7ff1e812f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 >> v1:172.21.15.76:6789/0 conn(0x7ff1e809f760 legacy=0x7ff1e812f0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 monclient: found mon.b 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e812c7e0 con 0x7ff1e8133380 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 5 monclient: authenticate success, global_id 174328 2024-09-16T06:45:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _renew_subs 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e812ff50 con 0x7ff1e8133380 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _renew_subs 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e812c500 con 0x7ff1e8133380 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e00020a0 con 0x7ff1e8133380 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1e00051c0 con 0x7ff1e8133380 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1e4d99700 4 set_mon_vals no callback set 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 10 monclient: dump: 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff1e000fc30 con 0x7ff1e8133380 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.219+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4005100 con 0x7ff1e8133380 2024-09-16T06:45:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.223+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(180..180 src has 151..180) v4 ==== 6689+0+0 (unknown 3757704175 0 0) 0x7ff1e00442f0 con 0x7ff1e8133380 2024-09-16T06:45:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.223+0000 7ff1e659c700 10 monclient: get_auth_request con 0x7ff1d0036180 auth_method 0 2024-09-16T06:45:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.227+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff1e0014af0 con 0x7ff1e8133380 2024-09-16T06:45:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.227+0000 7ff1deffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.227+0000 7ff1deffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:19.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.395+0000 7ff1ecc90700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:19.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.395+0000 7ff1ecc90700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:19.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.395+0000 7ff1ecc90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:19.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.395+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7ff1c4005930 con 0x7ff1e8133380 2024-09-16T06:45:20.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.999+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v181) v1 ==== 176+0+0 (unknown 1254437603 0 0) 0x7ff1e0013190 con 0x7ff1e8133380 2024-09-16T06:45:20.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.999+0000 7ff1deffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:20.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:19.999+0000 7ff1deffd700 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-16T06:45:20.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.107+0000 7ff1ecc90700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:20.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.107+0000 7ff1ecc90700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:20.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.107+0000 7ff1ecc90700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.107+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7ff1c40053b0 con 0x7ff1e8133380 2024-09-16T06:45:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1deffd700 1 -- 172.21.15.189:0/2174249361 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v181) v1 ==== 150+0+0 (unknown 3707880678 0 0) 0x7ff1e0014910 con 0x7ff1e8133380 2024-09-16T06:45:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1deffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T06:45:20.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1deffd700 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff1d0036180 legacy=0x7ff1d0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 10 monclient: shutdown 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 >> v1:172.21.15.189:6789/0 conn(0x7ff1e8133380 legacy=0x7ff1e81d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 shutdown_connections 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 >> 172.21.15.189:0/2174249361 conn(0x7ff1e80a21f0 msgr2=0x7ff1e813c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 shutdown_connections 2024-09-16T06:45:20.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.111+0000 7ff1ecc90700 1 -- 172.21.15.189:0/2174249361 wait complete. 2024-09-16T06:45:20.116 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'r1' already exists 2024-09-16T06:45:20.126 INFO:tasks.workunit.client.0.smithi189.stderr:++ seq 1 32 2024-09-16T06:45:20.142 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:20.143 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-16T06:45:20.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.247+0000 7efd80599700 10 monclient: get_monmap_and_config 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.247+0000 7efd80599700 10 monclient: build_initial_monmap 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.247+0000 7efd80599700 10 monclient: monmap: 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:20.251353+0000 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:20.251353+0000 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:20.251 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 Processor -- start 2024-09-16T06:45:20.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- start start 2024-09-16T06:45:20.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient: init 2024-09-16T06:45:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient(hunting): picked mon.noname-c con 0x7efd780a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient(hunting): picked mon.noname-b con 0x7efd7809aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient(hunting): picked mon.noname-a con 0x7efd780a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7809a040 con 0x7efd780a3ca0 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780a33d0 con 0x7efd780a2350 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78099d60 con 0x7efd7809aa20 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient(hunting): authenticate will time out at 1272.764928s 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7e335700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efd780a2350 0x7efd780a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35956/0 (socket says 172.21.15.189:35956) 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7e335700 1 -- 172.21.15.189:0/4252642328 learned_addr learned my addr 172.21.15.189:0/4252642328 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7eb36700 1 -- 172.21.15.189:0/4252642328 >> v1:172.21.15.76:6789/0 conn(0x7efd780a3ca0 legacy=0x7efd7813dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209412503 0 0) 0x7efd780a33d0 con 0x7efd780a2350 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient(hunting): _negotiate 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient(hunting): my global_id is 174331 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640035d0 con 0x7efd780a2350 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3041584456 0 0) 0x7efd640035d0 con 0x7efd780a2350 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 >> v1:172.21.15.76:6790/0 conn(0x7efd7809aa20 legacy=0x7efd7813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 >> v1:172.21.15.76:6789/0 conn(0x7efd780a3ca0 legacy=0x7efd7813dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: _finish_hunting 0 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 monclient: found mon.noname-c 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780a3920 con 0x7efd780a2350 2024-09-16T06:45:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: _finish_auth 0 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: _check_auth_tickets 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd74001610 con 0x7efd780a2350 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: dump: 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 5 monclient: authenticate success, global_id 174331 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd74001930 con 0x7efd780a2350 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 4 set_mon_vals no callback set 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 1 -- 172.21.15.189:0/4252642328 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd74001cc0 con 0x7efd780a2350 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd7d333700 10 monclient: dump: 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 10 monclient: shutdown 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- 172.21.15.189:0/4252642328 >> v1:172.21.15.189:6789/0 conn(0x7efd780a2350 legacy=0x7efd780a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- 172.21.15.189:0/4252642328 shutdown_connections 2024-09-16T06:45:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- 172.21.15.189:0/4252642328 >> 172.21.15.189:0/4252642328 conn(0x7efd780a7d70 msgr2=0x7efd780a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- 172.21.15.189:0/4252642328 shutdown_connections 2024-09-16T06:45:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.251+0000 7efd80599700 1 -- 172.21.15.189:0/4252642328 wait complete. 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: build_initial_monmap 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: monmap: 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:20.264 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 Processor -- start 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- start start 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: init 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient(hunting): picked mon.noname-b con 0x7efd7809aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient(hunting): picked mon.noname-a con 0x7efd780a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient(hunting): picked mon.noname-c con 0x7efd780a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78121450 con 0x7efd780a2350 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7806bed0 con 0x7efd7809aa20 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78120a50 con 0x7efd780a3ca0 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd7db34700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7efd780a2350 legacy=0x7efd781ceb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd7e335700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efd7809aa20 0x7efd781353f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35960/0 (socket says 172.21.15.189:35960) 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd7e335700 1 -- 172.21.15.189:0/203144099 learned_addr learned my addr 172.21.15.189:0/203144099 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient(hunting): authenticate will time out at 1272.770138s 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510740243 0 0) 0x7efd7806bed0 con 0x7efd7809aa20 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): my global_id is 174334 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c0035d0 con 0x7efd7809aa20 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652473109 0 0) 0x7efd78120a50 con 0x7efd780a3ca0 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): my global_id is 125657 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7806bed0 con 0x7efd780a3ca0 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1005386968 0 0) 0x7efd5c0035d0 con 0x7efd7809aa20 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd78120a50 con 0x7efd7809aa20 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd740015d0 con 0x7efd7809aa20 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): dump: 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3653819620 0 0) 0x7efd78120a50 con 0x7efd7809aa20 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 >> v1:172.21.15.76:6790/0 conn(0x7efd780a3ca0 legacy=0x7efd781d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 >> v1:172.21.15.76:6789/0 conn(0x7efd780a2350 legacy=0x7efd781ceb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 monclient: found mon.b 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd781361b0 con 0x7efd7809aa20 2024-09-16T06:45:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: _finish_auth 0 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 5 monclient: authenticate success, global_id 174334 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _renew_subs 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7809b6c0 con 0x7efd7809aa20 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _renew_subs 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd78135b00 con 0x7efd7809aa20 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd740011b0 con 0x7efd7809aa20 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd74004e90 con 0x7efd7809aa20 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd7cb32700 4 set_mon_vals no callback set 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.255+0000 7efd6effd700 10 monclient: dump: 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd80599700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd80599700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd80599700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd60005100 con 0x7efd7809aa20 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efd7400f900 con 0x7efd7809aa20 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(181..181 src has 151..181) v4 ==== 6899+0+0 (unknown 4211287241 0 0) 0x7efd74044100 con 0x7efd7809aa20 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.259+0000 7efd7db34700 10 monclient: get_auth_request con 0x7efd5c036180 auth_method 0 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.267+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efd74014910 con 0x7efd7809aa20 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.267+0000 7efd6effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:20.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.267+0000 7efd6effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.439+0000 7efd80599700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.439+0000 7efd80599700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.439+0000 7efd80599700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:20.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:20.439+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efd600053b0 con 0x7efd7809aa20 2024-09-16T06:45:21.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.007+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v182) v1 ==== 166+0+0 (unknown 2275456256 0 0) 0x7efd74012fb0 con 0x7efd7809aa20 2024-09-16T06:45:21.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.007+0000 7efd6effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.007+0000 7efd6effd700 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-16T06:45:21.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.115+0000 7efd80599700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.115+0000 7efd80599700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.115+0000 7efd80599700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.115+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efd60000bc0 con 0x7efd7809aa20 2024-09-16T06:45:21.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd6effd700 1 -- 172.21.15.189:0/203144099 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v182) v1 ==== 145+0+0 (unknown 2051414444 0 0) 0x7efd74014730 con 0x7efd7809aa20 2024-09-16T06:45:21.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd6effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd6effd700 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-16T06:45:21.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 >> v1:172.21.15.189:6812/3098810674 conn(0x7efd5c036180 legacy=0x7efd5c038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:21.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 10 monclient: shutdown 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 >> v1:172.21.15.189:6789/0 conn(0x7efd7809aa20 legacy=0x7efd781353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 shutdown_connections 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 >> 172.21.15.189:0/203144099 conn(0x7efd780a7d70 msgr2=0x7efd780a6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 shutdown_connections 2024-09-16T06:45:21.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.119+0000 7efd80599700 1 -- 172.21.15.189:0/203144099 wait complete. 2024-09-16T06:45:21.125 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h1' already exists 2024-09-16T06:45:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: get_monmap_and_config 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: build_initial_monmap 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: monmap: 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:21.246 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:21.246721+0000 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:21.246721+0000 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:21.247 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 1 Processor -- start 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 1 -- start start 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: init 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-c con 0x7f703c0a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-a con 0x7f703c09aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:21.250 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-b con 0x7f703c0a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.243+0000 7f7040c5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c09a040 con 0x7f703c09aa20 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c0a33d0 con 0x7f703c0a2350 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c099d60 con 0x7f703c0a3ca0 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient(hunting): authenticate will time out at 1273.760077s 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f703c0a2350 0x7f703c0a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35968/0 (socket says 172.21.15.189:35968) 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703a59c700 1 -- 172.21.15.189:0/2187727452 learned_addr learned my addr 172.21.15.189:0/2187727452 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7039d9b700 1 -- 172.21.15.189:0/2187727452 >> v1:172.21.15.76:6789/0 conn(0x7f703c09aa20 legacy=0x7f703c13a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201760277 0 0) 0x7f703c0a33d0 con 0x7f703c0a2350 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:21.251 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient(hunting): my global_id is 174337 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70280035d0 con 0x7f703c0a2350 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755261704 0 0) 0x7f70280035d0 con 0x7f703c0a2350 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 >> v1:172.21.15.76:6790/0 conn(0x7f703c0a3ca0 legacy=0x7f703c13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 >> v1:172.21.15.76:6789/0 conn(0x7f703c09aa20 legacy=0x7f703c13a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: _finish_hunting 0 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 monclient: found mon.noname-c 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0a3920 con 0x7f703c0a2350 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: _finish_auth 0 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: _check_auth_tickets 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7024001610 con 0x7f703c0a2350 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 5 monclient: authenticate success, global_id 174337 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: dump: 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:21.253 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7024001930 con 0x7f703c0a2350 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 4 set_mon_vals no callback set 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 1 -- 172.21.15.189:0/2187727452 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7024001cc0 con 0x7f703c0a2350 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f703959a700 10 monclient: dump: 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient: shutdown 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- 172.21.15.189:0/2187727452 >> v1:172.21.15.189:6789/0 conn(0x7f703c0a2350 legacy=0x7f703c0a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- 172.21.15.189:0/2187727452 shutdown_connections 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- 172.21.15.189:0/2187727452 >> 172.21.15.189:0/2187727452 conn(0x7f703c0a7d70 msgr2=0x7f703c0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- 172.21.15.189:0/2187727452 shutdown_connections 2024-09-16T06:45:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- 172.21.15.189:0/2187727452 wait complete. 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient: build_initial_monmap 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient: monmap: 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 Processor -- start 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 1 -- start start 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.247+0000 7f7040c5d700 10 monclient: init 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-a con 0x7f703c09aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-b con 0x7f703c0a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient(hunting): picked mon.noname-c con 0x7f703c0a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c09b740 con 0x7f703c09aa20 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c120700 con 0x7f703c0a2350 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c1361b0 con 0x7f703c0a3ca0 2024-09-16T06:45:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient(hunting): authenticate will time out at 1273.764934s 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f703a59c700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f703c09aa20 legacy=0x7f703c135840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7039d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f703c0a2350 0x7f703c1cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35984/0 (socket says 172.21.15.189:35984) 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7039d9b700 1 -- 172.21.15.189:0/1524170425 learned_addr learned my addr 172.21.15.189:0/1524170425 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182837503 0 0) 0x7f703c120700 con 0x7f703c0a2350 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): my global_id is 174340 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c0035d0 con 0x7f703c0a2350 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583320665 0 0) 0x7f703c1361b0 con 0x7f703c0a3ca0 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): my global_id is 125663 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f703c120700 con 0x7f703c0a3ca0 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1677080870 0 0) 0x7f701c0035d0 con 0x7f703c0a2350 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f703c1361b0 con 0x7f703c0a2350 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f702c004230 con 0x7f703c0a2350 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1577060443 0 0) 0x7f703c1361b0 con 0x7f703c0a2350 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 >> v1:172.21.15.76:6790/0 conn(0x7f703c0a3ca0 legacy=0x7f703c1d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 >> v1:172.21.15.76:6789/0 conn(0x7f703c09aa20 legacy=0x7f703c135840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 monclient: found mon.b 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c06bed0 con 0x7f703c0a2350 2024-09-16T06:45:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 5 monclient: authenticate success, global_id 174340 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _renew_subs 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c123200 con 0x7f703c0a2350 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _renew_subs 2024-09-16T06:45:21.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c121c90 con 0x7f703c0a2350 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f702c004450 con 0x7f703c0a2350 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f702c0047e0 con 0x7f703c0a2350 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7038d99700 4 set_mon_vals no callback set 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 10 monclient: dump: 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:21.267 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020005100 con 0x7f703c0a2350 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.251+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f702c004940 con 0x7f703c0a2350 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.255+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(182..182 src has 151..182) v4 ==== 7109+0+0 (unknown 3888305008 0 0) 0x7f702c043ed0 con 0x7f703c0a2350 2024-09-16T06:45:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.255+0000 7f703a59c700 10 monclient: get_auth_request con 0x7f701c036220 auth_method 0 2024-09-16T06:45:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.259+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f702c005110 con 0x7f703c0a2350 2024-09-16T06:45:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.259+0000 7f7032ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.259+0000 7f7032ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:21.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.431+0000 7f7040c5d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.431+0000 7f7040c5d700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:21.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.431+0000 7f7040c5d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:21.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:21.431+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7020005930 con 0x7f703c0a2350 2024-09-16T06:45:22.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.015+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v183) v1 ==== 166+0+0 (unknown 2743541878 0 0) 0x7f702c012b70 con 0x7f703c0a2350 2024-09-16T06:45:22.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.015+0000 7f7032ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.015+0000 7f7032ffd700 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-16T06:45:22.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7040c5d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7040c5d700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7040c5d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f70200053f0 con 0x7f703c0a2350 2024-09-16T06:45:22.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7032ffd700 1 -- 172.21.15.189:0/1524170425 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v183) v1 ==== 145+0+0 (unknown 737674240 0 0) 0x7f702c0142f0 con 0x7f703c0a2350 2024-09-16T06:45:22.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7032ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.127+0000 7f7032ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 >> v1:172.21.15.189:6812/3098810674 conn(0x7f701c036220 legacy=0x7f701c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 10 monclient: shutdown 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 >> v1:172.21.15.189:6789/0 conn(0x7f703c0a2350 legacy=0x7f703c1cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 shutdown_connections 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 >> 172.21.15.189:0/1524170425 conn(0x7f703c0a7d70 msgr2=0x7f703c1bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:22.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 shutdown_connections 2024-09-16T06:45:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.131+0000 7f7040c5d700 1 -- 172.21.15.189:0/1524170425 wait complete. 2024-09-16T06:45:22.135 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h2' already exists 2024-09-16T06:45:22.144 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-16T06:45:22.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:45:22.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 10 monclient: build_initial_monmap 2024-09-16T06:45:22.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 10 monclient: monmap: 2024-09-16T06:45:22.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:22.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:22.258753+0000 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:22.258753+0000 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:22.259 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 1 Processor -- start 2024-09-16T06:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 1 -- start start 2024-09-16T06:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.255+0000 7f664b59e700 10 monclient: init 2024-09-16T06:45:22.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f664c099c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f664c09ada0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f664c09b1b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0998c0 con 0x7f664c09ada0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c09bf60 con 0x7f664c09b1b0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c09d410 con 0x7f664c099c80 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient(hunting): authenticate will time out at 1274.772264s 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6641d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f664c09ada0 legacy=0x7f664c137320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f664c09b1b0 0x7f664c13aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:35998/0 (socket says 172.21.15.189:35998) 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664ad9d700 1 -- 172.21.15.189:0/113374295 learned_addr learned my addr 172.21.15.189:0/113374295 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:22.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640259932 0 0) 0x7f664c09bf60 con 0x7f664c09b1b0 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): my global_id is 174343 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66380046e0 con 0x7f664c09b1b0 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516774485 0 0) 0x7f664c09d410 con 0x7f664c099c80 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient(hunting): my global_id is 125666 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c09bf60 con 0x7f664c099c80 2024-09-16T06:45:22.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113497464 0 0) 0x7f66380046e0 con 0x7f664c09b1b0 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 >> v1:172.21.15.76:6790/0 conn(0x7f664c099c80 legacy=0x7f664c09a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 >> v1:172.21.15.76:6789/0 conn(0x7f664c09ada0 legacy=0x7f664c137320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 monclient: found mon.noname-c 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c09c160 con 0x7f664c09b1b0 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: _finish_auth 0 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:22.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6644001990 con 0x7f664c09b1b0 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 5 monclient: authenticate success, global_id 174343 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: dump: 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6644001c70 con 0x7f664c09b1b0 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 1 -- 172.21.15.189:0/113374295 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6644002000 con 0x7f664c09b1b0 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 4 set_mon_vals no callback set 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f6649d9b700 10 monclient: dump: 2024-09-16T06:45:22.276 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:22.277 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: shutdown 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- 172.21.15.189:0/113374295 >> v1:172.21.15.189:6789/0 conn(0x7f664c09b1b0 legacy=0x7f664c13aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- 172.21.15.189:0/113374295 shutdown_connections 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- 172.21.15.189:0/113374295 >> 172.21.15.189:0/113374295 conn(0x7f664c09fe70 msgr2=0x7f664c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- 172.21.15.189:0/113374295 shutdown_connections 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- 172.21.15.189:0/113374295 wait complete. 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: build_initial_monmap 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 10 monclient: monmap: 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:22.278 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 Processor -- start 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.259+0000 7f664b59e700 1 -- start start 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: init 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:22.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f664c099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f664c09ada0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f664c09b1b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c091020 con 0x7f664c09b1b0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c090a60 con 0x7f664c099c80 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c090780 con 0x7f664c09ada0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient(hunting): authenticate will time out at 1274.776936s 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664ad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f664c09b1b0 legacy=0x7f664c1ce070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f664c099c80 0x7f664c096590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36012/0 (socket says 172.21.15.189:36012) 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664a59c700 1 -- 172.21.15.189:0/4131937081 learned_addr learned my addr 172.21.15.189:0/4131937081 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274195663 0 0) 0x7f664c090a60 con 0x7f664c099c80 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:22.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): my global_id is 174346 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f662c0035d0 con 0x7f664c099c80 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857061762 0 0) 0x7f664c090780 con 0x7f664c09ada0 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): my global_id is 125669 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c090a60 con 0x7f664c09ada0 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187997567 0 0) 0x7f662c0035d0 con 0x7f664c099c80 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c090780 con 0x7f664c099c80 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f663c001910 con 0x7f664c099c80 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:22.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): dump: 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147257406 0 0) 0x7f664c090a60 con 0x7f664c09ada0 2024-09-16T06:45:22.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f662c0035d0 con 0x7f664c09ada0 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f66340041a0 con 0x7f664c09ada0 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient(hunting): dump: 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:22.283 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 843107931 0 0) 0x7f664c090780 con 0x7f664c099c80 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 >> v1:172.21.15.76:6790/0 conn(0x7f664c09ada0 legacy=0x7f664c1ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 >> v1:172.21.15.76:6789/0 conn(0x7f664c09b1b0 legacy=0x7f664c1ce070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 monclient: found mon.b 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c090d40 con 0x7f664c099c80 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 5 monclient: authenticate success, global_id 174346 2024-09-16T06:45:22.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _renew_subs 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c08f350 con 0x7f664c099c80 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _renew_subs 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f663c001ca0 con 0x7f664c099c80 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c08eef0 con 0x7f664c099c80 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f663c003b10 con 0x7f664c099c80 2024-09-16T06:45:22.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f664959a700 4 set_mon_vals no callback set 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.263+0000 7f66437fe700 10 monclient: dump: 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:22.286 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f664b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f664b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f664b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c0705b0 con 0x7f664c099c80 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f663c003d70 con 0x7f664c099c80 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(183..183 src has 151..183) v4 ==== 7319+0+0 (unknown 1673475103 0 0) 0x7f663c044160 con 0x7f664c099c80 2024-09-16T06:45:22.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.267+0000 7f6641d9b700 10 monclient: get_auth_request con 0x7f662c0362c0 auth_method 0 2024-09-16T06:45:22.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.271+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f663c014630 con 0x7f664c099c80 2024-09-16T06:45:22.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.271+0000 7f66437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:22.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.271+0000 7f66437fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:22.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.447+0000 7f664b59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.447+0000 7f664b59e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:22.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.447+0000 7f664b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:22.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:22.447+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f664c0677c0 con 0x7f664c099c80 2024-09-16T06:45:23.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.023+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v184) v1 ==== 166+0+0 (unknown 2624985424 0 0) 0x7f663c012cd0 con 0x7f664c099c80 2024-09-16T06:45:23.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.023+0000 7f66437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.023+0000 7f66437fe700 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-16T06:45:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.131+0000 7f664b59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.131+0000 7f664b59e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.131+0000 7f664b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.131+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f664c096d60 con 0x7f664c099c80 2024-09-16T06:45:23.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f66437fe700 1 -- 172.21.15.189:0/4131937081 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v184) v1 ==== 145+0+0 (unknown 2604452895 0 0) 0x7f663c014450 con 0x7f664c099c80 2024-09-16T06:45:23.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f66437fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.137 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f66437fe700 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 >> v1:172.21.15.189:6812/3098810674 conn(0x7f662c0362c0 legacy=0x7f662c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f664b59e700 10 monclient: shutdown 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f664b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.135+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 >> v1:172.21.15.189:6789/0 conn(0x7f664c099c80 legacy=0x7f664c096590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.139+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 shutdown_connections 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.139+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 >> 172.21.15.189:0/4131937081 conn(0x7f664c09fe70 msgr2=0x7f664c06d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.139+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 shutdown_connections 2024-09-16T06:45:23.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.139+0000 7f664b59e700 1 -- 172.21.15.189:0/4131937081 wait complete. 2024-09-16T06:45:23.141 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h3' already exists 2024-09-16T06:45:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:23.151 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-16T06:45:23.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: get_monmap_and_config 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: build_initial_monmap 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: monmap: 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:23.264770+0000 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:23.264770+0000 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:23.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:23.266 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:23.266 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 Processor -- start 2024-09-16T06:45:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 -- start start 2024-09-16T06:45:23.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: init 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-a con 0x7f07f40a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-b con 0x7f07f409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-c con 0x7f07f40a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:23.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f409a040 con 0x7f07f40a2350 2024-09-16T06:45:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40a33d0 con 0x7f07f40a3ca0 2024-09-16T06:45:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f4099d60 con 0x7f07f409aa20 2024-09-16T06:45:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient(hunting): authenticate will time out at 1275.777884s 2024-09-16T06:45:23.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f8948700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f07f40a2350 legacy=0x7f07f40a2730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f9149700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f07f40a3ca0 0x7f07f413dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36026/0 (socket says 172.21.15.189:36026) 2024-09-16T06:45:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f9149700 1 -- 172.21.15.189:0/4248889077 learned_addr learned my addr 172.21.15.189:0/4248889077 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948253067 0 0) 0x7f07f40a33d0 con 0x7f07f40a3ca0 2024-09-16T06:45:23.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): my global_id is 174349 2024-09-16T06:45:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc0035d0 con 0x7f07f40a3ca0 2024-09-16T06:45:23.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725944990 0 0) 0x7f07f4099d60 con 0x7f07f409aa20 2024-09-16T06:45:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:23.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient(hunting): my global_id is 125672 2024-09-16T06:45:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f40a33d0 con 0x7f07f409aa20 2024-09-16T06:45:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584034797 0 0) 0x7f07dc0035d0 con 0x7f07f40a3ca0 2024-09-16T06:45:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 >> v1:172.21.15.76:6790/0 conn(0x7f07f409aa20 legacy=0x7f07f413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:23.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 >> v1:172.21.15.76:6789/0 conn(0x7f07f40a2350 legacy=0x7f07f40a2730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 monclient: found mon.noname-c 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40a3920 con 0x7f07f40a3ca0 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:23.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 5 monclient: authenticate success, global_id 174349 2024-09-16T06:45:23.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f07ec001990 con 0x7f07f40a3ca0 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: dump: 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:23.283 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:23.284 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07ec001c70 con 0x7f07f40a3ca0 2024-09-16T06:45:23.284 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 4 set_mon_vals no callback set 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 1 -- 172.21.15.189:0/4248889077 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f07ec002000 con 0x7f07f40a3ca0 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07f37fe700 10 monclient: dump: 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:23.285 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 10 monclient: shutdown 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.263+0000 7f07fabac700 1 -- 172.21.15.189:0/4248889077 >> v1:172.21.15.189:6789/0 conn(0x7f07f40a3ca0 legacy=0x7f07f413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/4248889077 shutdown_connections 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/4248889077 >> 172.21.15.189:0/4248889077 conn(0x7f07f40a7d70 msgr2=0x7f07f40a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/4248889077 shutdown_connections 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/4248889077 wait complete. 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: build_initial_monmap 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: monmap: 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:23.287 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 Processor -- start 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- start start 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: init 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-a con 0x7f07f409aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-b con 0x7f07f40a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:23.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient(hunting): picked mon.noname-c con 0x7f07f40a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:23.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f4121f90 con 0x7f07f409aa20 2024-09-16T06:45:23.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f409afd0 con 0x7f07f40a2350 2024-09-16T06:45:23.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f409b520 con 0x7f07f40a3ca0 2024-09-16T06:45:23.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient(hunting): authenticate will time out at 1275.782190s 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f8948700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f07f409aa20 legacy=0x7f07f41c9980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f07f40a2350 0x7f07f41345d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36038/0 (socket says 172.21.15.189:36038) 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f3fff700 1 -- 172.21.15.189:0/3816526678 learned_addr learned my addr 172.21.15.189:0/3816526678 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238344883 0 0) 0x7f07f409afd0 con 0x7f07f40a2350 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): my global_id is 174352 2024-09-16T06:45:23.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e00035d0 con 0x7f07f40a2350 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 672799915 0 0) 0x7f07e00035d0 con 0x7f07f40a2350 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07f409afd0 con 0x7f07f40a2350 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816981123 0 0) 0x7f07f409b520 con 0x7f07f40a3ca0 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): my global_id is 125675 2024-09-16T06:45:23.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e00035d0 con 0x7f07f40a3ca0 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f07e4001910 con 0x7f07f40a2350 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): dump: 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:23.295 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3587386378 0 0) 0x7f07f409afd0 con 0x7f07f40a2350 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 >> v1:172.21.15.76:6790/0 conn(0x7f07f40a3ca0 legacy=0x7f07f41d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 >> v1:172.21.15.76:6789/0 conn(0x7f07f409aa20 legacy=0x7f07f41c9980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 monclient: found mon.b 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4121cb0 con 0x7f07f40a2350 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient: _finish_auth 0 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:45:23.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 5 monclient: authenticate success, global_id 174352 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _renew_subs 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f4120580 con 0x7f07f40a2350 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _renew_subs 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f4134ec0 con 0x7f07f40a2350 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.267+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07e4001c90 con 0x7f07f40a2350 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:23.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f07e4003b10 con 0x7f07f40a2350 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 10 monclient: dump: 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:23.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07fabac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f2ffd700 4 set_mon_vals no callback set 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f07e4003d30 con 0x7f07f40a2350 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(184..184 src has 151..184) v4 ==== 7529+0+0 (unknown 3799624972 0 0) 0x7f07e40441d0 con 0x7f07f40a2350 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07fabac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07fabac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f406beb0 con 0x7f07f40a2350 2024-09-16T06:45:23.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.271+0000 7f07f8948700 10 monclient: get_auth_request con 0x7f07e0036200 auth_method 0 2024-09-16T06:45:23.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.275+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f07e4014900 con 0x7f07f40a2350 2024-09-16T06:45:23.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.275+0000 7f07f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:23.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.275+0000 7f07f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:23.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.451+0000 7f07fabac700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.451+0000 7f07fabac700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:23.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.451+0000 7f07fabac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:23.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:23.451+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f07f4121750 con 0x7f07f40a2350 2024-09-16T06:45:24.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.035+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v185) v1 ==== 166+0+0 (unknown 3235861011 0 0) 0x7f07f4121750 con 0x7f07f40a2350 2024-09-16T06:45:24.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.035+0000 7f07f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.035+0000 7f07f17fa700 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-16T06:45:24.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07fabac700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07fabac700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07fabac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f07f4121750 con 0x7f07f40a2350 2024-09-16T06:45:24.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07f17fa700 1 -- 172.21.15.189:0/3816526678 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v185) v1 ==== 145+0+0 (unknown 3947653689 0 0) 0x7f07f4121750 con 0x7f07f40a2350 2024-09-16T06:45:24.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07f17fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.147+0000 7f07f17fa700 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 >> v1:172.21.15.189:6812/3098810674 conn(0x7f07e0036200 legacy=0x7f07e00386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 10 monclient: shutdown 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 >> v1:172.21.15.189:6789/0 conn(0x7f07f40a2350 legacy=0x7f07f41345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 shutdown_connections 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 >> 172.21.15.189:0/3816526678 conn(0x7f07f40a7d70 msgr2=0x7f07f4135380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 shutdown_connections 2024-09-16T06:45:24.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.151+0000 7f07fabac700 1 -- 172.21.15.189:0/3816526678 wait complete. 2024-09-16T06:45:24.155 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h4' already exists 2024-09-16T06:45:24.168 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:24.169 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 10 monclient: get_monmap_and_config 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 10 monclient: build_initial_monmap 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 10 monclient: monmap: 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:24.274788+0000 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:24.274788+0000 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:24.274 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:24.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 1 Processor -- start 2024-09-16T06:45:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 1 -- start start 2024-09-16T06:45:24.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.271+0000 7f607cff0700 10 monclient: init 2024-09-16T06:45:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-c con 0x7f607809d830 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-a con 0x7f607809dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:24.285 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-b con 0x7f6078137dd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607809a790 con 0x7f607809dc10 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078071510 con 0x7f607809d830 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078132040 con 0x7f6078137dd0 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient(hunting): authenticate will time out at 1276.789117s 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60777fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f607809d830 0x7f6078131930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36044/0 (socket says 172.21.15.189:36044) 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60777fe700 1 -- 172.21.15.189:0/1686602373 learned_addr learned my addr 172.21.15.189:0/1686602373 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f6076ffd700 1 -- 172.21.15.189:0/1686602373 >> v1:172.21.15.76:6789/0 conn(0x7f607809dc10 legacy=0x7f60781366b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061540963 0 0) 0x7f6068001110 con 0x7f607809d830 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:24.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): my global_id is 174355 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60600035d0 con 0x7f607809d830 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270873279 0 0) 0x7f6078132040 con 0x7f6078137dd0 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient(hunting): my global_id is 125678 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068001110 con 0x7f6078137dd0 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718908664 0 0) 0x7f60600035d0 con 0x7f607809d830 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 >> v1:172.21.15.76:6790/0 conn(0x7f6078137dd0 legacy=0x7f607813a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 >> v1:172.21.15.76:6789/0 conn(0x7f607809dc10 legacy=0x7f60781366b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:24.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 monclient: found mon.noname-c 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607809e060 con 0x7f607809d830 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 5 monclient: authenticate success, global_id 174355 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6068001600 con 0x7f607809d830 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: dump: 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:24.288 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6068001820 con 0x7f607809d830 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:24.289 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 1 -- 172.21.15.189:0/1686602373 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6068001bb0 con 0x7f607809d830 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 4 set_mon_vals no callback set 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f60767fc700 10 monclient: dump: 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:24.290 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 10 monclient: shutdown 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- 172.21.15.189:0/1686602373 >> v1:172.21.15.189:6789/0 conn(0x7f607809d830 legacy=0x7f6078131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- 172.21.15.189:0/1686602373 shutdown_connections 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- 172.21.15.189:0/1686602373 >> 172.21.15.189:0/1686602373 conn(0x7f60780a4320 msgr2=0x7f60780a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- 172.21.15.189:0/1686602373 shutdown_connections 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.275+0000 7f607cff0700 1 -- 172.21.15.189:0/1686602373 wait complete. 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: build_initial_monmap 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: monmap: 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:24.291 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 Processor -- start 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- start start 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: init 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:24.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-c con 0x7f607809d830 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-a con 0x7f607809dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient(hunting): picked mon.noname-b con 0x7f6078137dd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078134d50 con 0x7f607809dc10 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078135a20 con 0x7f6078137dd0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607809ed60 con 0x7f607809d830 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient(hunting): authenticate will time out at 1276.794288s 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f6076ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f607809dc10 legacy=0x7f6078134640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f6077fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6078137dd0 0x7f60781cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36056/0 (socket says 172.21.15.189:36056) 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f6077fff700 1 -- 172.21.15.189:0/2001327369 learned_addr learned my addr 172.21.15.189:0/2001327369 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965979212 0 0) 0x7f6078135a20 con 0x7f6078137dd0 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:24.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): my global_id is 174358 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60640035d0 con 0x7f6078137dd0 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662449320 0 0) 0x7f60640035d0 con 0x7f6078137dd0 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6078135a20 con 0x7f6078137dd0 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6070001d00 con 0x7f6078137dd0 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): dump: 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.294 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547376578 0 0) 0x7f607809ed60 con 0x7f607809d830 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:24.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient(hunting): my global_id is 125681 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60640035d0 con 0x7f607809d830 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2870055065 0 0) 0x7f6078135a20 con 0x7f6078137dd0 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 >> v1:172.21.15.76:6790/0 conn(0x7f607809d830 legacy=0x7f6078133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 >> v1:172.21.15.76:6789/0 conn(0x7f607809dc10 legacy=0x7f6078134640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 monclient: found mon.b 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607809e7f0 con 0x7f6078137dd0 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: _finish_auth 0 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:24.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 5 monclient: authenticate success, global_id 174358 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _renew_subs 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607811a510 con 0x7f6078137dd0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _renew_subs 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6078136310 con 0x7f6078137dd0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60700020b0 con 0x7f6078137dd0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6070003f90 con 0x7f6078137dd0 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f6075ffb700 4 set_mon_vals no callback set 2024-09-16T06:45:24.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.279+0000 7f605ffff700 10 monclient: dump: 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:24.298 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6070004170 con 0x7f6078137dd0 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f607cff0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f607cff0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f607cff0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6058005100 con 0x7f6078137dd0 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(185..185 src has 151..185) v4 ==== 7739+0+0 (unknown 2065633807 0 0) 0x7f60700447a0 con 0x7f6078137dd0 2024-09-16T06:45:24.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.283+0000 7f60777fe700 10 monclient: get_auth_request con 0x7f6064036180 auth_method 0 2024-09-16T06:45:24.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.287+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6070011700 con 0x7f6078137dd0 2024-09-16T06:45:24.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.287+0000 7f605ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:24.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.287+0000 7f605ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:24.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.459+0000 7f607cff0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.459+0000 7f607cff0700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:24.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.459+0000 7f607cff0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:24.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:24.459+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f60580053b0 con 0x7f6078137dd0 2024-09-16T06:45:25.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.043+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v186) v1 ==== 166+0+0 (unknown 3560518588 0 0) 0x7f60700131d0 con 0x7f6078137dd0 2024-09-16T06:45:25.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.043+0000 7f605ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.043+0000 7f605ffff700 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-16T06:45:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f607cff0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f607cff0700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f607cff0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6058000bc0 con 0x7f6078137dd0 2024-09-16T06:45:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f605ffff700 1 -- 172.21.15.189:0/2001327369 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v186) v1 ==== 145+0+0 (unknown 943781703 0 0) 0x7f6070014950 con 0x7f6078137dd0 2024-09-16T06:45:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f605ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.158 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.155+0000 7f605ffff700 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6064036180 legacy=0x7f6064038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 10 monclient: shutdown 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 >> v1:172.21.15.189:6789/0 conn(0x7f6078137dd0 legacy=0x7f60781cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 shutdown_connections 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 >> 172.21.15.189:0/2001327369 conn(0x7f60780a4320 msgr2=0x7f607809f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 shutdown_connections 2024-09-16T06:45:25.161 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.159+0000 7f607cff0700 1 -- 172.21.15.189:0/2001327369 wait complete. 2024-09-16T06:45:25.162 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h5' already exists 2024-09-16T06:45:25.171 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:25.172 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: get_monmap_and_config 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: build_initial_monmap 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: monmap: 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:25.281631+0000 2024-09-16T06:45:25.281 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:25.281631+0000 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:25.282 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 1 Processor -- start 2024-09-16T06:45:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 1 -- start start 2024-09-16T06:45:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: init 2024-09-16T06:45:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:25.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient(hunting): picked mon.noname-a con 0x7f365409bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient(hunting): picked mon.noname-b con 0x7f365409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient(hunting): picked mon.noname-c con 0x7f3654138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365409cca0 con 0x7f365409bb00 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365409f750 con 0x7f3654138da0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3654097620 con 0x7f365409c5f0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient(hunting): authenticate will time out at 1277.794705s 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365acf0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3654138da0 0x7f365413b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36068/0 (socket says 172.21.15.189:36068) 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365acf0700 1 -- 172.21.15.189:0/3646417902 learned_addr learned my addr 172.21.15.189:0/3646417902 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365a4ef700 1 -- 172.21.15.189:0/3646417902 >> v1:172.21.15.76:6789/0 conn(0x7f365409bb00 legacy=0x7f365409bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471497243 0 0) 0x7f365409f750 con 0x7f3654138da0 2024-09-16T06:45:25.293 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient(hunting): _negotiate 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient(hunting): my global_id is 174361 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36400035d0 con 0x7f3654138da0 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182484514 0 0) 0x7f36400035d0 con 0x7f3654138da0 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 >> v1:172.21.15.76:6790/0 conn(0x7f365409c5f0 legacy=0x7f3654137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 >> v1:172.21.15.76:6789/0 conn(0x7f365409bb00 legacy=0x7f365409bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: _finish_hunting 0 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 monclient: found mon.noname-c 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654097900 con 0x7f3654138da0 2024-09-16T06:45:25.294 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: _finish_auth 0 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: _check_auth_tickets 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 5 monclient: authenticate success, global_id 174361 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f364c001990 con 0x7f3654138da0 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: dump: 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:25.295 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f364c001c30 con 0x7f3654138da0 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 1 -- 172.21.15.189:0/3646417902 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f364c002000 con 0x7f3654138da0 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f365c753700 4 set_mon_vals no callback set 2024-09-16T06:45:25.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.279+0000 7f36594ed700 10 monclient: dump: 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: shutdown 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- 172.21.15.189:0/3646417902 >> v1:172.21.15.189:6789/0 conn(0x7f3654138da0 legacy=0x7f365413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- 172.21.15.189:0/3646417902 shutdown_connections 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- 172.21.15.189:0/3646417902 >> 172.21.15.189:0/3646417902 conn(0x7f36540a03e0 msgr2=0x7f36540a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- 172.21.15.189:0/3646417902 shutdown_connections 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- 172.21.15.189:0/3646417902 wait complete. 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: build_initial_monmap 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: monmap: 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:25.298 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 Processor -- start 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- start start 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: init 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient(hunting): picked mon.noname-b con 0x7f365409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient(hunting): picked mon.noname-c con 0x7f365409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient(hunting): picked mon.noname-a con 0x7f3654138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:25.299 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365408fce0 con 0x7f3654138da0 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3654091ba0 con 0x7f365409bb00 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3654099460 con 0x7f365409c5f0 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365c753700 10 monclient(hunting): authenticate will time out at 1277.799086s 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365acf0700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3654138da0 legacy=0x7f36541d23c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365a4ef700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f365409bb00 0x7f36541348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36084/0 (socket says 172.21.15.189:36084) 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f365a4ef700 1 -- 172.21.15.189:0/1355455668 learned_addr learned my addr 172.21.15.189:0/1355455668 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556110845 0 0) 0x7f3654091ba0 con 0x7f365409bb00 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): my global_id is 174364 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36380035d0 con 0x7f365409bb00 2024-09-16T06:45:25.300 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859905355 0 0) 0x7f3654099460 con 0x7f365409c5f0 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 10 monclient(hunting): my global_id is 125687 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.283+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3654091ba0 con 0x7f365409c5f0 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3503503533 0 0) 0x7f36380035d0 con 0x7f365409bb00 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3654099460 con 0x7f365409bb00 2024-09-16T06:45:25.301 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3644004230 con 0x7f365409bb00 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): dump: 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:25.302 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4024848937 0 0) 0x7f3654091ba0 con 0x7f365409c5f0 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36380035d0 con 0x7f365409c5f0 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36500018b0 con 0x7f365409c5f0 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient(hunting): dump: 2024-09-16T06:45:25.303 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1659977453 0 0) 0x7f3654099460 con 0x7f365409bb00 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 >> v1:172.21.15.76:6790/0 conn(0x7f365409c5f0 legacy=0x7f3654134ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 >> v1:172.21.15.76:6789/0 conn(0x7f3654138da0 legacy=0x7f36541d23c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:25.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 monclient: found mon.b 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654071830 con 0x7f365409bb00 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: _finish_auth 0 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 5 monclient: authenticate success, global_id 174364 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _renew_subs 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36540918c0 con 0x7f365409bb00 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _renew_subs 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36540915e0 con 0x7f365409bb00 2024-09-16T06:45:25.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3644004450 con 0x7f365409bb00 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36440047e0 con 0x7f365409bb00 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364bfff700 4 set_mon_vals no callback set 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 10 monclient: dump: 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3644004a40 con 0x7f365409bb00 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c005100 con 0x7f365409bb00 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(186..186 src has 151..186) v4 ==== 8077+0+0 (unknown 389786852 0 0) 0x7f36440443f0 con 0x7f365409bb00 2024-09-16T06:45:25.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.287+0000 7f3659cee700 10 monclient: get_auth_request con 0x7f36380362c0 auth_method 0 2024-09-16T06:45:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.295+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3644014800 con 0x7f365409bb00 2024-09-16T06:45:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.295+0000 7f364affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:25.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.295+0000 7f364affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.467+0000 7f365c753700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.467+0000 7f365c753700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:25.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.467+0000 7f365c753700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:25.471 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:25.467+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f363c005930 con 0x7f365409bb00 2024-09-16T06:45:26.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.059+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v187) v1 ==== 166+0+0 (unknown 4028540618 0 0) 0x7f3644012c70 con 0x7f365409bb00 2024-09-16T06:45:26.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.059+0000 7f364affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.059+0000 7f364affd700 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-16T06:45:26.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f365c753700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f365c753700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f365c753700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.173 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f363c0053f0 con 0x7f365409bb00 2024-09-16T06:45:26.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f364affd700 1 -- 172.21.15.189:0/1355455668 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v187) v1 ==== 145+0+0 (unknown 1778313963 0 0) 0x7f36440145d0 con 0x7f365409bb00 2024-09-16T06:45:26.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f364affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.171+0000 7f364affd700 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-16T06:45:26.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 >> v1:172.21.15.189:6812/3098810674 conn(0x7f36380362c0 legacy=0x7f3638038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:26.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 10 monclient: shutdown 2024-09-16T06:45:26.177 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 >> v1:172.21.15.189:6789/0 conn(0x7f365409bb00 legacy=0x7f36541348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 shutdown_connections 2024-09-16T06:45:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 >> 172.21.15.189:0/1355455668 conn(0x7f36540a03e0 msgr2=0x7f3654137f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 shutdown_connections 2024-09-16T06:45:26.178 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.175+0000 7f365c753700 1 -- 172.21.15.189:0/1355455668 wait complete. 2024-09-16T06:45:26.179 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h6' already exists 2024-09-16T06:45:26.189 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:26.189 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-16T06:45:26.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: get_monmap_and_config 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: build_initial_monmap 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: monmap: 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:26.298275+0000 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:26.298275+0000 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:26.298 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 1 Processor -- start 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 1 -- start start 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: init 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-c con 0x7fa1ac09fce0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:26.306 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-b con 0x7fa1ac0717e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-a con 0x7fa1ac0a07d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac074a10 con 0x7fa1ac0a07d0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac098450 con 0x7fa1ac09fce0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0986f0 con 0x7fa1ac0717e0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b3582700 10 monclient(hunting): authenticate will time out at 1278.811417s 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b2580700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa1ac09fce0 0x7fa1ac098df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36092/0 (socket says 172.21.15.189:36092) 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b2d81700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa1ac0a07d0 legacy=0x7fa1ac13d800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.295+0000 7fa1b2580700 1 -- 172.21.15.189:0/3552222638 learned_addr learned my addr 172.21.15.189:0/3552222638 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159837867 0 0) 0x7fa1ac098450 con 0x7fa1ac09fce0 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _negotiate 2024-09-16T06:45:26.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): my global_id is 174367 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1980035d0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040205059 0 0) 0x7fa1ac0986f0 con 0x7fa1ac0717e0 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _negotiate 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient(hunting): my global_id is 125690 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ac098450 con 0x7fa1ac0717e0 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057119175 0 0) 0x7fa1980035d0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 >> v1:172.21.15.76:6790/0 conn(0x7fa1ac0717e0 legacy=0x7fa1ac0a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 >> v1:172.21.15.76:6789/0 conn(0x7fa1ac0a07d0 legacy=0x7fa1ac13d800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: _finish_hunting 0 2024-09-16T06:45:26.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 monclient: found mon.noname-c 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0746f0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: _finish_auth 0 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: _check_auth_tickets 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 5 monclient: authenticate success, global_id 174367 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa19c001610 con 0x7fa1ac09fce0 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: dump: 2024-09-16T06:45:26.309 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa19c0018f0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:26.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 4 set_mon_vals no callback set 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 1 -- 172.21.15.189:0/3552222638 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa19c001c80 con 0x7fa1ac09fce0 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b157e700 10 monclient: dump: 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:26.311 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: shutdown 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- 172.21.15.189:0/3552222638 >> v1:172.21.15.189:6789/0 conn(0x7fa1ac09fce0 legacy=0x7fa1ac098df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- 172.21.15.189:0/3552222638 shutdown_connections 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- 172.21.15.189:0/3552222638 >> 172.21.15.189:0/3552222638 conn(0x7fa1ac0a0fb0 msgr2=0x7fa1ac0a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- 172.21.15.189:0/3552222638 shutdown_connections 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- 172.21.15.189:0/3552222638 wait complete. 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: build_initial_monmap 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: monmap: 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:26.312 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 Processor -- start 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- start start 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: init 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-c con 0x7fa1ac0717e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:26.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-b con 0x7fa1ac09fce0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient(hunting): picked mon.noname-a con 0x7fa1ac0a07d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac08dc00 con 0x7fa1ac0a07d0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac09b4c0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0930c0 con 0x7fa1ac0717e0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b3582700 10 monclient(hunting): authenticate will time out at 1278.815914s 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b1d7f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa1ac09fce0 0x7fa1ac08c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36102/0 (socket says 172.21.15.189:36102) 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b1d7f700 1 -- 172.21.15.189:0/3875769311 learned_addr learned my addr 172.21.15.189:0/3875769311 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.299+0000 7fa1b2d81700 1 -- 172.21.15.189:0/3875769311 >> v1:172.21.15.76:6789/0 conn(0x7fa1ac0a07d0 legacy=0x7fa1ac08d0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431322923 0 0) 0x7fa1ac09b4c0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:26.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): my global_id is 174370 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1900035d0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703941237 0 0) 0x7fa1ac0930c0 con 0x7fa1ac0717e0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): my global_id is 125693 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ac09b4c0 con 0x7fa1ac0717e0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710441301 0 0) 0x7fa1900035d0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ac0930c0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1a8001910 con 0x7fa1ac09fce0 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:26.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670328450 0 0) 0x7fa1ac0930c0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 >> v1:172.21.15.76:6790/0 conn(0x7fa1ac0717e0 legacy=0x7fa1ac08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 >> v1:172.21.15.76:6789/0 conn(0x7fa1ac0a07d0 legacy=0x7fa1ac08d0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 monclient: found mon.b 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac092de0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 5 monclient: authenticate success, global_id 174370 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _renew_subs 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac0933a0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _renew_subs 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac0927a0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a8001ca0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1a8003b10 con 0x7fa1ac09fce0 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b0d7d700 4 set_mon_vals no callback set 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 10 monclient: dump: 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.318 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa1a8003cf0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(187..187 src has 151..187) v4 ==== 8287+0+0 (unknown 2603263283 0 0) 0x7fa1a8044550 con 0x7fa1ac09fce0 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa194005100 con 0x7fa1ac09fce0 2024-09-16T06:45:26.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.303+0000 7fa1b2580700 10 monclient: get_auth_request con 0x7fa190036180 auth_method 0 2024-09-16T06:45:26.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.311+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa1a80148c0 con 0x7fa1ac09fce0 2024-09-16T06:45:26.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.311+0000 7fa1a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:26.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.311+0000 7fa1a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:26.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.487+0000 7fa1b3582700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.487+0000 7fa1b3582700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:26.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.487+0000 7fa1b3582700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:26.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:26.487+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa194005930 con 0x7fa1ac09fce0 2024-09-16T06:45:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.071+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v188) v1 ==== 166+0+0 (unknown 4260753634 0 0) 0x7fa1a8012d30 con 0x7fa1ac09fce0 2024-09-16T06:45:27.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.071+0000 7fa1a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.071+0000 7fa1a2ffd700 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-16T06:45:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.179+0000 7fa1b3582700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.179+0000 7fa1b3582700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.179+0000 7fa1b3582700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.183 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.179+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa194005b70 con 0x7fa1ac09fce0 2024-09-16T06:45:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1a2ffd700 1 -- 172.21.15.189:0/3875769311 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v188) v1 ==== 145+0+0 (unknown 2612378442 0 0) 0x7fa1a80144b0 con 0x7fa1ac09fce0 2024-09-16T06:45:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1a2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.184 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1a2ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-16T06:45:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa190036180 legacy=0x7fa190038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:27.186 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 10 monclient: shutdown 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 >> v1:172.21.15.189:6789/0 conn(0x7fa1ac09fce0 legacy=0x7fa1ac08c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 shutdown_connections 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 >> 172.21.15.189:0/3875769311 conn(0x7fa1ac0a0fb0 msgr2=0x7fa1ac098cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 shutdown_connections 2024-09-16T06:45:27.187 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.183+0000 7fa1b3582700 1 -- 172.21.15.189:0/3875769311 wait complete. 2024-09-16T06:45:27.188 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h7' already exists 2024-09-16T06:45:27.197 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:27.197 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-16T06:45:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: get_monmap_and_config 2024-09-16T06:45:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: build_initial_monmap 2024-09-16T06:45:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: monmap: 2024-09-16T06:45:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:27.305 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:27.305540+0000 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:27.305540+0000 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:27.306 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.309 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 1 Processor -- start 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 1 -- start start 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: init 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-b con 0x7f8b9409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-c con 0x7f8b9409c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-a con 0x7f8b94138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9409cca0 con 0x7f8b94138da0 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9409f750 con 0x7f8b9409c5f0 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b94097620 con 0x7f8b9409bb00 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:27.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient(hunting): authenticate will time out at 1279.818639s 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b927fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8b9409c5f0 0x7f8b94137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36114/0 (socket says 172.21.15.189:36114) 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b927fc700 1 -- 172.21.15.189:0/797393917 learned_addr learned my addr 172.21.15.189:0/797393917 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b937fe700 1 -- 172.21.15.189:0/797393917 >> v1:172.21.15.76:6789/0 conn(0x7f8b94138da0 legacy=0x7f8b9413b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28008560 0 0) 0x7f8b9409f750 con 0x7f8b9409c5f0 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient(hunting): my global_id is 174373 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b800057e0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162042595 0 0) 0x7f8b800057e0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 >> v1:172.21.15.76:6790/0 conn(0x7f8b9409bb00 legacy=0x7f8b9409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 >> v1:172.21.15.76:6789/0 conn(0x7f8b94138da0 legacy=0x7f8b9413b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:27.311 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 monclient: found mon.noname-c 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b94097900 con 0x7f8b9409c5f0 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 5 monclient: authenticate success, global_id 174373 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b84001610 con 0x7f8b9409c5f0 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: dump: 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:27.312 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b84001930 con 0x7f8b9409c5f0 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:27.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b99a4d700 4 set_mon_vals no callback set 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 1 -- 172.21.15.189:0/797393917 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b84001d00 con 0x7f8b9409c5f0 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.303+0000 7f8b91ffb700 10 monclient: dump: 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:27.314 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: shutdown 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- 172.21.15.189:0/797393917 >> v1:172.21.15.189:6789/0 conn(0x7f8b9409c5f0 legacy=0x7f8b94137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- 172.21.15.189:0/797393917 shutdown_connections 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- 172.21.15.189:0/797393917 >> 172.21.15.189:0/797393917 conn(0x7f8b940a03e0 msgr2=0x7f8b940a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- 172.21.15.189:0/797393917 shutdown_connections 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- 172.21.15.189:0/797393917 wait complete. 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: build_initial_monmap 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: monmap: 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:27.315 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 Processor -- start 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- start start 2024-09-16T06:45:27.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: init 2024-09-16T06:45:27.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-c con 0x7f8b9409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-b con 0x7f8b9409c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient(hunting): picked mon.noname-a con 0x7f8b94138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940916c0 con 0x7f8b94138da0 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9408f870 con 0x7f8b9409c5f0 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b94099380 con 0x7f8b9409bb00 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b99a4d700 10 monclient(hunting): authenticate will time out at 1279.822836s 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b927fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8b9409c5f0 0x7f8b9408e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36116/0 (socket says 172.21.15.189:36116) 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b927fc700 1 -- 172.21.15.189:0/2179892104 learned_addr learned my addr 172.21.15.189:0/2179892104 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:27.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b937fe700 1 -- 172.21.15.189:0/2179892104 >> v1:172.21.15.76:6789/0 conn(0x7f8b94138da0 legacy=0x7f8b941d1fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002891474 0 0) 0x7f8b9408f870 con 0x7f8b9409c5f0 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): my global_id is 174376 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b740035d0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069720972 0 0) 0x7f8b94099380 con 0x7f8b9409bb00 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): my global_id is 125699 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b9408f870 con 0x7f8b9409bb00 2024-09-16T06:45:27.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848627929 0 0) 0x7f8b740035d0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b94099380 con 0x7f8b9409c5f0 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b840015d0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): dump: 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:27.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2390113329 0 0) 0x7f8b94099380 con 0x7f8b9409c5f0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 >> v1:172.21.15.76:6790/0 conn(0x7f8b9409bb00 legacy=0x7f8b9408d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 >> v1:172.21.15.76:6789/0 conn(0x7f8b94138da0 legacy=0x7f8b941d1fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 monclient: found mon.b 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9406fe90 con 0x7f8b9409c5f0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.307+0000 7f8b8bfff700 10 monclient: _finish_auth 0 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:27.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 5 monclient: authenticate success, global_id 174376 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _renew_subs 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b940913e0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _renew_subs 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9408e7f0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b84004ca0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8b84004fb0 con 0x7f8b9409c5f0 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b917fa700 4 set_mon_vals no callback set 2024-09-16T06:45:27.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 10 monclient: dump: 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:27.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b78005100 con 0x7f8b9409c5f0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8b8400fa20 con 0x7f8b9409c5f0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(188..188 src has 151..188) v4 ==== 8497+0+0 (unknown 272613430 0 0) 0x7f8b84044750 con 0x7f8b9409c5f0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.311+0000 7f8b92ffd700 10 monclient: get_auth_request con 0x7f8b74036200 auth_method 0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.319+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8b84014850 con 0x7f8b9409c5f0 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.319+0000 7f8b8bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:27.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.319+0000 7f8b8bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:27.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.487+0000 7f8b99a4d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.487+0000 7f8b99a4d700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:27.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.487+0000 7f8b99a4d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:27.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:27.487+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8b78005930 con 0x7f8b9409c5f0 2024-09-16T06:45:28.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.079+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v189) v1 ==== 166+0+0 (unknown 1346439115 0 0) 0x7f8b84012ef0 con 0x7f8b9409c5f0 2024-09-16T06:45:28.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.079+0000 7f8b8bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.079+0000 7f8b8bfff700 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-16T06:45:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.187+0000 7f8b99a4d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.187+0000 7f8b99a4d700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.187+0000 7f8b99a4d700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.187+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8b780053f0 con 0x7f8b9409c5f0 2024-09-16T06:45:28.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b8bfff700 1 -- 172.21.15.189:0/2179892104 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v189) v1 ==== 145+0+0 (unknown 2824366712 0 0) 0x7f8b84014670 con 0x7f8b9409c5f0 2024-09-16T06:45:28.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b8bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b8bfff700 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-16T06:45:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8b74036200 legacy=0x7f8b740386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 10 monclient: shutdown 2024-09-16T06:45:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 >> v1:172.21.15.189:6789/0 conn(0x7f8b9409c5f0 legacy=0x7f8b9408e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:28.194 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 shutdown_connections 2024-09-16T06:45:28.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 >> 172.21.15.189:0/2179892104 conn(0x7f8b940a03e0 msgr2=0x7f8b94137cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:28.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 shutdown_connections 2024-09-16T06:45:28.195 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.191+0000 7f8b99a4d700 1 -- 172.21.15.189:0/2179892104 wait complete. 2024-09-16T06:45:28.196 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h8' already exists 2024-09-16T06:45:28.205 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:28.205 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 10 monclient: get_monmap_and_config 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 10 monclient: build_initial_monmap 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 10 monclient: monmap: 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:28.310728+0000 2024-09-16T06:45:28.310 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:28.310728+0000 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:28.311 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 1 Processor -- start 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 1 -- start start 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.307+0000 7f8cd780c700 10 monclient: init 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-a con 0x7f8cd00707a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-b con 0x7f8cd0070b80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-c con 0x7f8cd013b620 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:28.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd009f110 con 0x7f8cd00707a0 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00a0670 con 0x7f8cd013b620 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd009f380 con 0x7f8cd0070b80 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient(hunting): authenticate will time out at 1280.824234s 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd55a8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8cd00707a0 legacy=0x7f8cd009ff60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd5da9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8cd013b620 0x7f8cd013da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36128/0 (socket says 172.21.15.189:36128) 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd5da9700 1 -- 172.21.15.189:0/3340340406 learned_addr learned my addr 172.21.15.189:0/3340340406 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113915200 0 0) 0x7f8cd00a0670 con 0x7f8cd013b620 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:28.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient(hunting): my global_id is 174379 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc0035d0 con 0x7f8cd013b620 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368388215 0 0) 0x7f8cbc0035d0 con 0x7f8cd013b620 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 >> v1:172.21.15.76:6790/0 conn(0x7f8cd0070b80 legacy=0x7f8cd0070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 >> v1:172.21.15.76:6789/0 conn(0x7f8cd00707a0 legacy=0x7f8cd009ff60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 monclient: found mon.noname-c 2024-09-16T06:45:28.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd009f660 con 0x7f8cd013b620 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 5 monclient: authenticate success, global_id 174379 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cc8001990 con 0x7f8cd013b620 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: dump: 2024-09-16T06:45:28.318 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc8001c30 con 0x7f8cd013b620 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:28.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 1 -- 172.21.15.189:0/3340340406 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cc8002000 con 0x7f8cd013b620 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 4 set_mon_vals no callback set 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cc3fff700 10 monclient: dump: 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:28.320 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: shutdown 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- 172.21.15.189:0/3340340406 >> v1:172.21.15.189:6789/0 conn(0x7f8cd013b620 legacy=0x7f8cd013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- 172.21.15.189:0/3340340406 shutdown_connections 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- 172.21.15.189:0/3340340406 >> 172.21.15.189:0/3340340406 conn(0x7f8cd00a44a0 msgr2=0x7f8cd00a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- 172.21.15.189:0/3340340406 shutdown_connections 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- 172.21.15.189:0/3340340406 wait complete. 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: build_initial_monmap 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: monmap: 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:28.321 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:28.322 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 Processor -- start 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 1 -- start start 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.311+0000 7f8cd780c700 10 monclient: init 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-a con 0x7f8cd00707a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-c con 0x7f8cd0070b80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:28.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient(hunting): picked mon.noname-b con 0x7f8cd013b620 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd013eac0 con 0x7f8cd00707a0 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd009b0d0 con 0x7f8cd013b620 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd0135940 con 0x7f8cd0070b80 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient(hunting): authenticate will time out at 1280.828908s 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd55a8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8cd00707a0 legacy=0x7f8cd01c99a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd5da9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8cd013b620 0x7f8cd01d1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:36142/0 (socket says 172.21.15.189:36142) 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd5da9700 1 -- 172.21.15.189:0/3017342430 learned_addr learned my addr 172.21.15.189:0/3017342430 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084015818 0 0) 0x7f8cd009b0d0 con 0x7f8cd013b620 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): my global_id is 174382 2024-09-16T06:45:28.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb40035d0 con 0x7f8cd013b620 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618134086 0 0) 0x7f8cd0135940 con 0x7f8cd0070b80 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): my global_id is 125705 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd009b0d0 con 0x7f8cd0070b80 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3860708734 0 0) 0x7f8cb40035d0 con 0x7f8cd013b620 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cd0135940 con 0x7f8cd013b620 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cc8001f40 con 0x7f8cd013b620 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:28.328 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3969409399 0 0) 0x7f8cd0135940 con 0x7f8cd013b620 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 >> v1:172.21.15.76:6790/0 conn(0x7f8cd0070b80 legacy=0x7f8cd01ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:28.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 >> v1:172.21.15.76:6789/0 conn(0x7f8cd00707a0 legacy=0x7f8cd01c99a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 monclient: found mon.b 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0135660 con 0x7f8cd013b620 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 5 monclient: authenticate success, global_id 174382 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _renew_subs 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd0133b90 con 0x7f8cd013b620 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _renew_subs 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0134a00 con 0x7f8cd013b620 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc8005020 con 0x7f8cd013b620 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cc80053b0 con 0x7f8cd013b620 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 10 monclient: dump: 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:28.331 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc37fe700 4 set_mon_vals no callback set 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8cc8005510 con 0x7f8cd013b620 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb0005100 con 0x7f8cd013b620 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(189..189 src has 151..189) v4 ==== 8707+0+0 (unknown 1238435057 0 0) 0x7f8cc8044b90 con 0x7f8cd013b620 2024-09-16T06:45:28.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.315+0000 7f8cd55a8700 10 monclient: get_auth_request con 0x7f8cb40361d0 auth_method 0 2024-09-16T06:45:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.323+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8cc8014e00 con 0x7f8cd013b620 2024-09-16T06:45:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.323+0000 7f8cc1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:28.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.323+0000 7f8cc1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:28.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.507+0000 7f8cd780c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.507+0000 7f8cd780c700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:28.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.507+0000 7f8cd780c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:28.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:28.507+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8cb00053b0 con 0x7f8cd013b620 2024-09-16T06:45:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.087+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v190) v1 ==== 166+0+0 (unknown 1151821412 0 0) 0x7f8cc80149f0 con 0x7f8cd013b620 2024-09-16T06:45:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.087+0000 7f8cc1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.087+0000 7f8cc1ffb700 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-16T06:45:29.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.199+0000 7f8cd780c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.199+0000 7f8cd780c700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.199+0000 7f8cd780c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.199+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8cb0000bc0 con 0x7f8cd013b620 2024-09-16T06:45:29.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cc1ffb700 1 -- 172.21.15.189:0/3017342430 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v190) v1 ==== 145+0+0 (unknown 2069165830 0 0) 0x7f8cc8014bd0 con 0x7f8cd013b620 2024-09-16T06:45:29.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cc1ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cc1ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-16T06:45:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8cb40361d0 legacy=0x7f8cb4038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 10 monclient: shutdown 2024-09-16T06:45:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 >> v1:172.21.15.189:6789/0 conn(0x7f8cd013b620 legacy=0x7f8cd01d1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:29.207 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 shutdown_connections 2024-09-16T06:45:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 >> 172.21.15.189:0/3017342430 conn(0x7f8cd00a44a0 msgr2=0x7f8cd0131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 shutdown_connections 2024-09-16T06:45:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.203+0000 7f8cd780c700 1 -- 172.21.15.189:0/3017342430 wait complete. 2024-09-16T06:45:29.208 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h9' already exists 2024-09-16T06:45:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:29.217 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-16T06:45:29.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: get_monmap_and_config 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: build_initial_monmap 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: monmap: 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:29.326356+0000 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:29.326356+0000 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:29.326 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 1 Processor -- start 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 1 -- start start 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: init 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-c con 0x7fd71c09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-a con 0x7fd71c09f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-b con 0x7fd71c139a50 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:29.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c09ab80 con 0x7fd71c09f690 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c09a690 con 0x7fd71c09eba0 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c072230 con 0x7fd71c139a50 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72322e700 10 monclient(hunting): authenticate will time out at 1281.839625s 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72222c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd71c09eba0 0x7fd71c09ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57924/0 (socket says 172.21.15.189:57924) 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd72222c700 1 -- 172.21.15.189:0/2482546018 learned_addr learned my addr 172.21.15.189:0/2482546018 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.323+0000 7fd721a2b700 1 -- 172.21.15.189:0/2482546018 >> v1:172.21.15.76:6789/0 conn(0x7fd71c09f690 legacy=0x7fd71c138330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067751844 0 0) 0x7fd71c09a690 con 0x7fd71c09eba0 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:29.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient(hunting): my global_id is 174385 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7080035d0 con 0x7fd71c09eba0 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716260791 0 0) 0x7fd7080035d0 con 0x7fd71c09eba0 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 >> v1:172.21.15.76:6790/0 conn(0x7fd71c139a50 legacy=0x7fd71c13be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 >> v1:172.21.15.76:6789/0 conn(0x7fd71c09f690 legacy=0x7fd71c138330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: _finish_hunting 0 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 monclient: found mon.noname-c 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c09be90 con 0x7fd71c09eba0 2024-09-16T06:45:29.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: _finish_auth 0 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: _check_auth_tickets 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd718001610 con 0x7fd71c09eba0 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: dump: 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.333 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 5 monclient: authenticate success, global_id 174385 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd718001930 con 0x7fd71c09eba0 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 1 -- 172.21.15.189:0/2482546018 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd718001cc0 con 0x7fd71c09eba0 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:29.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72122a700 10 monclient: dump: 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 4 set_mon_vals no callback set 2024-09-16T06:45:29.335 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 10 monclient: shutdown 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- 172.21.15.189:0/2482546018 >> v1:172.21.15.189:6789/0 conn(0x7fd71c09eba0 legacy=0x7fd71c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- 172.21.15.189:0/2482546018 shutdown_connections 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- 172.21.15.189:0/2482546018 >> 172.21.15.189:0/2482546018 conn(0x7fd71c09fe70 msgr2=0x7fd71c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- 172.21.15.189:0/2482546018 shutdown_connections 2024-09-16T06:45:29.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- 172.21.15.189:0/2482546018 wait complete. 2024-09-16T06:45:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 10 monclient: build_initial_monmap 2024-09-16T06:45:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 10 monclient: monmap: 2024-09-16T06:45:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 Processor -- start 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 1 -- start start 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.327+0000 7fd72322e700 10 monclient: init 2024-09-16T06:45:29.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-c con 0x7fd71c09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-b con 0x7fd71c09f690 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient(hunting): picked mon.noname-a con 0x7fd71c139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c0966b0 con 0x7fd71c139a50 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c06b950 con 0x7fd71c09f690 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c071560 con 0x7fd71c09eba0 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient(hunting): authenticate will time out at 1281.844317s 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd721a2b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd71c09f690 0x7fd71c1cce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57932/0 (socket says 172.21.15.189:57932) 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd721a2b700 1 -- 172.21.15.189:0/2209912427 learned_addr learned my addr 172.21.15.189:0/2209912427 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:29.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd722a2d700 1 -- 172.21.15.189:0/2209912427 >> v1:172.21.15.76:6789/0 conn(0x7fd71c139a50 legacy=0x7fd71c1d0540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181087969 0 0) 0x7fd71c06b950 con 0x7fd71c09f690 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): my global_id is 174388 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7000035d0 con 0x7fd71c09f690 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438810254 0 0) 0x7fd7000035d0 con 0x7fd71c09f690 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd71c06b950 con 0x7fd71c09f690 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd70c004230 con 0x7fd71c09f690 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:29.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569559177 0 0) 0x7fd71c071560 con 0x7fd71c09eba0 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:29.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient(hunting): my global_id is 125711 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7000035d0 con 0x7fd71c09eba0 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 457334776 0 0) 0x7fd71c06b950 con 0x7fd71c09f690 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 >> v1:172.21.15.76:6790/0 conn(0x7fd71c09eba0 legacy=0x7fd71c1c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 >> v1:172.21.15.76:6789/0 conn(0x7fd71c139a50 legacy=0x7fd71c1d0540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 monclient: found mon.b 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c08f520 con 0x7fd71c09f690 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:29.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 5 monclient: authenticate success, global_id 174388 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _renew_subs 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71c08f800 con 0x7fd71c09f690 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _renew_subs 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c08d430 con 0x7fd71c09f690 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd70c004450 con 0x7fd71c09f690 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd70c0047e0 con 0x7fd71c09f690 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd713fff700 4 set_mon_vals no callback set 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:29.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 10 monclient: dump: 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:29.346 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd70c004940 con 0x7fd71c09f690 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd704005100 con 0x7fd71c09f690 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(190..190 src has 151..190) v4 ==== 8917+0+0 (unknown 2704292983 0 0) 0x7fd70c0446d0 con 0x7fd71c09f690 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.331+0000 7fd72222c700 10 monclient: get_auth_request con 0x7fd700036220 auth_method 0 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.339+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd70c005110 con 0x7fd71c09f690 2024-09-16T06:45:29.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.339+0000 7fd712ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:29.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.339+0000 7fd712ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.515+0000 7fd72322e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.515+0000 7fd72322e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.515+0000 7fd72322e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:29.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:29.515+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd704005930 con 0x7fd71c09f690 2024-09-16T06:45:30.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.095+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v191) v1 ==== 168+0+0 (unknown 707530931 0 0) 0x7fd70c012c00 con 0x7fd71c09f690 2024-09-16T06:45:30.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.095+0000 7fd712ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.095+0000 7fd712ffd700 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-16T06:45:30.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd72322e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd72322e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd72322e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:30.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd7040053f0 con 0x7fd71c09f690 2024-09-16T06:45:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd712ffd700 1 -- 172.21.15.189:0/2209912427 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v191) v1 ==== 147+0+0 (unknown 1722745840 0 0) 0x7fd70c014380 con 0x7fd71c09f690 2024-09-16T06:45:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd712ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.207+0000 7fd712ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd700036220 legacy=0x7fd7000386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 10 monclient: shutdown 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 >> v1:172.21.15.189:6789/0 conn(0x7fd71c09f690 legacy=0x7fd71c1cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 shutdown_connections 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 >> 172.21.15.189:0/2209912427 conn(0x7fd71c09fe70 msgr2=0x7fd71c1383d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 shutdown_connections 2024-09-16T06:45:30.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.211+0000 7fd72322e700 1 -- 172.21.15.189:0/2209912427 wait complete. 2024-09-16T06:45:30.215 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h10' already exists 2024-09-16T06:45:30.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:30.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: get_monmap_and_config 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: build_initial_monmap 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: monmap: 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:30.332608+0000 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:30.332608+0000 2024-09-16T06:45:30.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:30.333 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.333 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:30.333 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:30.333 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:30.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 Processor -- start 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- start start 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: init 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-a con 0x7f364c071e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-c con 0x7f364c06b7c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-b con 0x7f364c137e50 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c09a1f0 con 0x7f364c071e00 2024-09-16T06:45:30.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c09b6e0 con 0x7f364c06b7c0 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c072250 con 0x7f364c137e50 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient(hunting): authenticate will time out at 1282.845467s 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f364c06b7c0 0x7f364c136730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57944/0 (socket says 172.21.15.189:57944) 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364b7fe700 1 -- 172.21.15.189:0/2446167402 learned_addr learned my addr 172.21.15.189:0/2446167402 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364bfff700 1 -- 172.21.15.189:0/2446167402 >> v1:172.21.15.76:6789/0 conn(0x7f364c071e00 legacy=0x7f364c09af20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364346410 0 0) 0x7f364c09b6e0 con 0x7f364c06b7c0 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): my global_id is 174391 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36340035d0 con 0x7f364c06b7c0 2024-09-16T06:45:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236311610 0 0) 0x7f364c072250 con 0x7f364c137e50 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient(hunting): my global_id is 125714 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c09b6e0 con 0x7f364c137e50 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608932492 0 0) 0x7f36340035d0 con 0x7f364c06b7c0 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 >> v1:172.21.15.76:6790/0 conn(0x7f364c137e50 legacy=0x7f364c13a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 >> v1:172.21.15.76:6789/0 conn(0x7f364c071e00 legacy=0x7f364c09af20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 monclient: found mon.noname-c 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c09bb60 con 0x7f364c06b7c0 2024-09-16T06:45:30.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: _finish_auth 0 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 5 monclient: authenticate success, global_id 174391 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f363c001610 con 0x7f364c06b7c0 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: dump: 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:30.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f363c001880 con 0x7f364c06b7c0 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 4 set_mon_vals no callback set 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 1 -- 172.21.15.189:0/2446167402 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f363c001c10 con 0x7f364c06b7c0 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f364affd700 10 monclient: dump: 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:30.342 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 10 monclient: shutdown 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- 172.21.15.189:0/2446167402 >> v1:172.21.15.189:6789/0 conn(0x7f364c06b7c0 legacy=0x7f364c136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- 172.21.15.189:0/2446167402 shutdown_connections 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- 172.21.15.189:0/2446167402 >> 172.21.15.189:0/2446167402 conn(0x7f364c09fe70 msgr2=0x7f364c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:30.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- 172.21.15.189:0/2446167402 shutdown_connections 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.331+0000 7f36510e1700 1 -- 172.21.15.189:0/2446167402 wait complete. 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: build_initial_monmap 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: monmap: 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 Processor -- start 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- start start 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: init 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-c con 0x7f364c06b7c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-a con 0x7f364c071e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient(hunting): picked mon.noname-b con 0x7f364c137e50 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c1342f0 con 0x7f364c071e00 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c071890 con 0x7f364c137e50 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c137be0 con 0x7f364c06b7c0 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient(hunting): authenticate will time out at 1282.849657s 2024-09-16T06:45:30.345 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36508e0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f364c137e50 0x7f364c1d1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57952/0 (socket says 172.21.15.189:57952) 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36508e0700 1 -- 172.21.15.189:0/1378598851 learned_addr learned my addr 172.21.15.189:0/1378598851 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f364b7fe700 1 -- 172.21.15.189:0/1378598851 >> v1:172.21.15.76:6789/0 conn(0x7f364c071e00 legacy=0x7f364c13e660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823560586 0 0) 0x7f364c071890 con 0x7f364c137e50 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): my global_id is 174394 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36380035d0 con 0x7f364c137e50 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990425670 0 0) 0x7f364c137be0 con 0x7f364c06b7c0 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): my global_id is 125717 2024-09-16T06:45:30.346 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c071890 con 0x7f364c06b7c0 2024-09-16T06:45:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714720976 0 0) 0x7f36380035d0 con 0x7f364c137e50 2024-09-16T06:45:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f364c137be0 con 0x7f364c137e50 2024-09-16T06:45:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3644001c50 con 0x7f364c137e50 2024-09-16T06:45:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:30.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:30.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138858065 0 0) 0x7f364c071890 con 0x7f364c06b7c0 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36380035d0 con 0x7f364c06b7c0 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36400041a0 con 0x7f364c06b7c0 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:30.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2023514166 0 0) 0x7f36380035d0 con 0x7f364c06b7c0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 >> v1:172.21.15.189:6789/0 conn(0x7f364c137e50 legacy=0x7f364c1d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 >> v1:172.21.15.76:6789/0 conn(0x7f364c071e00 legacy=0x7f364c13e660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 monclient: found mon.c 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c09a380 con 0x7f364c06b7c0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 5 monclient: authenticate success, global_id 125717 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _renew_subs 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c070f60 con 0x7f364c06b7c0 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _renew_subs 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c13ed70 con 0x7f364c06b7c0 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36400046c0 con 0x7f364c06b7c0 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3640004a50 con 0x7f364c06b7c0 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f3648ff9700 10 monclient: dump: 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:30.352 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.335+0000 7f364a7fc700 4 set_mon_vals no callback set 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f36510e1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f36510e1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f36510e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362c005100 con 0x7f364c06b7c0 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3640004bb0 con 0x7f364c06b7c0 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(191..191 src has 151..191) v4 ==== 9130+0+0 (unknown 40873664 0 0) 0x7f3640044700 con 0x7f364c06b7c0 2024-09-16T06:45:30.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.339+0000 7f364b7fe700 10 monclient: get_auth_request con 0x7f3638036270 auth_method 0 2024-09-16T06:45:30.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.347+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f36400144b0 con 0x7f364c06b7c0 2024-09-16T06:45:30.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.347+0000 7f3648ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:30.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.347+0000 7f3648ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.523+0000 7f36510e1700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.523+0000 7f36510e1700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.523+0000 7f36510e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.523+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f362c005930 con 0x7f364c06b7c0 2024-09-16T06:45:30.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.791+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v192) v1 ==== 168+0+0 (unknown 3177685118 0 0) 0x7f3640012b50 con 0x7f364c06b7c0 2024-09-16T06:45:30.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.791+0000 7f3648ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.791+0000 7f3648ff9700 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-16T06:45:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f36510e1700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f36510e1700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f36510e1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:30.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f362c0053f0 con 0x7f364c06b7c0 2024-09-16T06:45:30.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f3648ff9700 1 -- 172.21.15.189:0/1378598851 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v192) v1 ==== 147+0+0 (unknown 1510049628 0 0) 0x7f36400142d0 con 0x7f364c06b7c0 2024-09-16T06:45:30.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f3648ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:30.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.903+0000 7f3648ff9700 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-16T06:45:30.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3638036270 legacy=0x7f3638038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 10 monclient: shutdown 2024-09-16T06:45:30.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:30.908 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 >> v1:172.21.15.76:6790/0 conn(0x7f364c06b7c0 legacy=0x7f364c1c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 shutdown_connections 2024-09-16T06:45:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 >> 172.21.15.189:0/1378598851 conn(0x7f364c09fe70 msgr2=0x7f364c136930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 shutdown_connections 2024-09-16T06:45:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:30.907+0000 7f36510e1700 1 -- 172.21.15.189:0/1378598851 wait complete. 2024-09-16T06:45:30.909 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h11' already exists 2024-09-16T06:45:30.924 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:30.924 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: get_monmap_and_config 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: monmap: 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:31.028 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:31.028674+0000 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:31.028674+0000 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:31.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 Processor -- start 2024-09-16T06:45:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- start start 2024-09-16T06:45:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: init 2024-09-16T06:45:31.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-b con 0x7f9044132080 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-a con 0x7f9044138460 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-c con 0x7f904413bf70 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904406abd0 con 0x7f9044138460 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904406b170 con 0x7f904413bf70 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9044132b70 con 0x7f9044132080 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient(hunting): authenticate will time out at 1283.541588s 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9048eb1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f904413bf70 0x7f904413e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57954/0 (socket says 172.21.15.189:57954) 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9048eb1700 1 -- 172.21.15.189:0/3701723689 learned_addr learned my addr 172.21.15.189:0/3701723689 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:31.037 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90437fe700 1 -- 172.21.15.189:0/3701723689 >> v1:172.21.15.76:6789/0 conn(0x7f9044138460 legacy=0x7f904413a850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282500714 0 0) 0x7f904406b170 con 0x7f904413bf70 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient(hunting): my global_id is 174397 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90240035d0 con 0x7f904413bf70 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789269628 0 0) 0x7f90240035d0 con 0x7f904413bf70 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 >> v1:172.21.15.76:6790/0 conn(0x7f9044132080 legacy=0x7f9044132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 >> v1:172.21.15.76:6789/0 conn(0x7f9044138460 legacy=0x7f904413a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 monclient: found mon.noname-c 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.038 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440a38b0 con 0x7f904413bf70 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 5 monclient: authenticate success, global_id 174397 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f903c001610 con 0x7f904413bf70 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: dump: 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:31.039 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f903c001930 con 0x7f904413bf70 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 1 -- 172.21.15.189:0/3701723689 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f903c001d00 con 0x7f904413bf70 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 4 set_mon_vals no callback set 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:31.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f9042ffd700 10 monclient: dump: 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: shutdown 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:31.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- 172.21.15.189:0/3701723689 >> v1:172.21.15.189:6789/0 conn(0x7f904413bf70 legacy=0x7f904413e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- 172.21.15.189:0/3701723689 shutdown_connections 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- 172.21.15.189:0/3701723689 >> 172.21.15.189:0/3701723689 conn(0x7f90440a4660 msgr2=0x7f9044099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- 172.21.15.189:0/3701723689 shutdown_connections 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 1 -- 172.21.15.189:0/3701723689 wait complete. 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.027+0000 7f90496b2700 10 monclient: monmap: 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:31.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 Processor -- start 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- start start 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: init 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-a con 0x7f9044132080 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-c con 0x7f9044138460 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient(hunting): picked mon.noname-b con 0x7f904413bf70 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440a0040 con 0x7f9044132080 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904409de60 con 0x7f904413bf70 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904413b900 con 0x7f9044138460 2024-09-16T06:45:31.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient(hunting): authenticate will time out at 1283.545767s 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9043fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f9044132080 legacy=0x7f904409bd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9048eb1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f904413bf70 0x7f90441d2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57968/0 (socket says 172.21.15.189:57968) 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9048eb1700 1 -- 172.21.15.189:0/3085968381 learned_addr learned my addr 172.21.15.189:0/3085968381 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553683959 0 0) 0x7f904409de60 con 0x7f904413bf70 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): my global_id is 174400 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90280035d0 con 0x7f904413bf70 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074125745 0 0) 0x7f904413b900 con 0x7f9044138460 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:31.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): my global_id is 125723 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904409de60 con 0x7f9044138460 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487191724 0 0) 0x7f90280035d0 con 0x7f904413bf70 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904413b900 con 0x7f904413bf70 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f903c0015d0 con 0x7f904413bf70 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:31.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1354595690 0 0) 0x7f904413b900 con 0x7f904413bf70 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 >> v1:172.21.15.76:6790/0 conn(0x7f9044138460 legacy=0x7f904409c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 >> v1:172.21.15.76:6789/0 conn(0x7f9044132080 legacy=0x7f904409bd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:31.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 monclient: found mon.b 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904409fd60 con 0x7f904413bf70 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 5 monclient: authenticate success, global_id 174400 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _renew_subs 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904409d8e0 con 0x7f904413bf70 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _renew_subs 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f904409e030 con 0x7f904413bf70 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f903c004ca0 con 0x7f904413bf70 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f903c004fb0 con 0x7f904413bf70 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9041ffb700 4 set_mon_vals no callback set 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 10 monclient: dump: 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:31.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f90496b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.031+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f903c00fa20 con 0x7f904413bf70 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.035+0000 7f90496b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.035+0000 7f90496b2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.035+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c005100 con 0x7f904413bf70 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.035+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(192..192 src has 151..192) v4 ==== 9343+0+0 (unknown 979106167 0 0) 0x7f903c044ab0 con 0x7f904413bf70 2024-09-16T06:45:31.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.035+0000 7f9043fff700 10 monclient: get_auth_request con 0x7f90280361d0 auth_method 0 2024-09-16T06:45:31.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.039+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f903c014870 con 0x7f904413bf70 2024-09-16T06:45:31.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.039+0000 7f9040ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:31.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.039+0000 7f9040ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:31.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.215+0000 7f90496b2700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:31.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.215+0000 7f90496b2700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:31.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.215+0000 7f90496b2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:31.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:31.215+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f901c0053b0 con 0x7f904413bf70 2024-09-16T06:45:32.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.107+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v193) v1 ==== 168+0+0 (unknown 2305136372 0 0) 0x7f903c012f10 con 0x7f904413bf70 2024-09-16T06:45:32.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.107+0000 7f9040ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.107+0000 7f9040ff9700 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-16T06:45:32.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f90496b2700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f90496b2700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f90496b2700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f901c000bc0 con 0x7f904413bf70 2024-09-16T06:45:32.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f9040ff9700 1 -- 172.21.15.189:0/3085968381 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v193) v1 ==== 147+0+0 (unknown 1871227522 0 0) 0x7f903c014690 con 0x7f904413bf70 2024-09-16T06:45:32.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f9040ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.223+0000 7f9040ff9700 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 >> v1:172.21.15.189:6812/3098810674 conn(0x7f90280361d0 legacy=0x7f9028038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 10 monclient: shutdown 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 >> v1:172.21.15.189:6789/0 conn(0x7f904413bf70 legacy=0x7f90441d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 shutdown_connections 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 >> 172.21.15.189:0/3085968381 conn(0x7f90440a4660 msgr2=0x7f9044136910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 shutdown_connections 2024-09-16T06:45:32.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.227+0000 7f90496b2700 1 -- 172.21.15.189:0/3085968381 wait complete. 2024-09-16T06:45:32.230 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h12' already exists 2024-09-16T06:45:32.238 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:32.238 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-16T06:45:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: get_monmap_and_config 2024-09-16T06:45:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: build_initial_monmap 2024-09-16T06:45:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: monmap: 2024-09-16T06:45:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:32.349 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:32.349842+0000 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:32.349842+0000 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:32.350 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 1 Processor -- start 2024-09-16T06:45:32.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 1 -- start start 2024-09-16T06:45:32.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: init 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-b con 0x7fa80c09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-a con 0x7fa80c09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-c con 0x7fa80c138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c09cca0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c09f750 con 0x7fa80c138da0 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c097620 con 0x7fa80c09bb00 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient(hunting): authenticate will time out at 1284.862945s 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa80c138da0 0x7fa80c13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57978/0 (socket says 172.21.15.189:57978) 2024-09-16T06:45:32.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80ad9d700 1 -- 172.21.15.189:0/1314970831 learned_addr learned my addr 172.21.15.189:0/1314970831 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa809d9b700 1 -- 172.21.15.189:0/1314970831 >> v1:172.21.15.76:6789/0 conn(0x7fa80c09c5f0 legacy=0x7fa80c137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509948694 0 0) 0x7fa80c09f750 con 0x7fa80c138da0 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient(hunting): my global_id is 174403 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f40035d0 con 0x7fa80c138da0 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251264251 0 0) 0x7fa7f40035d0 con 0x7fa80c138da0 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 >> v1:172.21.15.76:6790/0 conn(0x7fa80c09bb00 legacy=0x7fa80c09bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 >> v1:172.21.15.76:6789/0 conn(0x7fa80c09c5f0 legacy=0x7fa80c137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: _finish_hunting 0 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 monclient: found mon.noname-c 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c097900 con 0x7fa80c138da0 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: _finish_auth 0 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: _check_auth_tickets 2024-09-16T06:45:32.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa804001990 con 0x7fa80c138da0 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 5 monclient: authenticate success, global_id 174403 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: dump: 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.358 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa804001cb0 con 0x7fa80c138da0 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 1 -- 172.21.15.189:0/1314970831 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa804002040 con 0x7fa80c138da0 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa80959a700 10 monclient: dump: 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.359 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.347+0000 7fa810ec8700 4 set_mon_vals no callback set 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 10 monclient: shutdown 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- 172.21.15.189:0/1314970831 >> v1:172.21.15.189:6789/0 conn(0x7fa80c138da0 legacy=0x7fa80c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- 172.21.15.189:0/1314970831 shutdown_connections 2024-09-16T06:45:32.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- 172.21.15.189:0/1314970831 >> 172.21.15.189:0/1314970831 conn(0x7fa80c0a03e0 msgr2=0x7fa80c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- 172.21.15.189:0/1314970831 shutdown_connections 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- 172.21.15.189:0/1314970831 wait complete. 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 10 monclient: build_initial_monmap 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 10 monclient: monmap: 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:32.361 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:32.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:32.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 Processor -- start 2024-09-16T06:45:32.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 1 -- start start 2024-09-16T06:45:32.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.351+0000 7fa810ec8700 10 monclient: init 2024-09-16T06:45:32.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:32.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-c con 0x7fa80c09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-b con 0x7fa80c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient(hunting): picked mon.noname-a con 0x7fa80c138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c08ed00 con 0x7fa80c138da0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c0a2400 con 0x7fa80c09c5f0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c08eaa0 con 0x7fa80c09bb00 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient(hunting): authenticate will time out at 1284.868757s 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa80ad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa80c138da0 legacy=0x7fa80c1d24c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:45:32.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa809d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa80c09c5f0 0x7fa80c08e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57986/0 (socket says 172.21.15.189:57986) 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa809d9b700 1 -- 172.21.15.189:0/3678319570 learned_addr learned my addr 172.21.15.189:0/3678319570 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314593455 0 0) 0x7fa80c0a2400 con 0x7fa80c09c5f0 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): my global_id is 174406 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ec0035d0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974543836 0 0) 0x7fa80c08eaa0 con 0x7fa80c09bb00 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): my global_id is 125729 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa80c0a2400 con 0x7fa80c09bb00 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634213801 0 0) 0x7fa7ec0035d0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa80c08eaa0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7fc004230 con 0x7fa80c09c5f0 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): dump: 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131056912 0 0) 0x7fa80c0a2400 con 0x7fa80c09bb00 2024-09-16T06:45:32.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7ec0035d0 con 0x7fa80c09bb00 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa800001830 con 0x7fa80c09bb00 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient(hunting): dump: 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.372 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 108700733 0 0) 0x7fa80c08eaa0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 >> v1:172.21.15.76:6790/0 conn(0x7fa80c09bb00 legacy=0x7fa80c08dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 >> v1:172.21.15.76:6789/0 conn(0x7fa80c138da0 legacy=0x7fa80c1d24c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:32.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 monclient: found mon.b 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c08f390 con 0x7fa80c09c5f0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 5 monclient: authenticate success, global_id 174406 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _renew_subs 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c09a740 con 0x7fa80c09c5f0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _renew_subs 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c0949a0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7fc0044a0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7fc0047b0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 10 monclient: dump: 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.375 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa810ec8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa7fc004910 con 0x7fa80c09c5f0 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.355+0000 7fa808d99700 4 set_mon_vals no callback set 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.359+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(193..193 src has 151..193) v4 ==== 9556+0+0 (unknown 3746259404 0 0) 0x7fa7fc044930 con 0x7fa80c09c5f0 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.359+0000 7fa810ec8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.359+0000 7fa810ec8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.359+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c070ad0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.359+0000 7fa80a59c700 10 monclient: get_auth_request con 0x7fa7ec0361c0 auth_method 0 2024-09-16T06:45:32.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.367+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa7fc004bc0 con 0x7fa80c09c5f0 2024-09-16T06:45:32.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.367+0000 7fa7faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.367+0000 7fa7faffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:32.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.539+0000 7fa810ec8700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.539+0000 7fa810ec8700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:32.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.539+0000 7fa810ec8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:32.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:32.539+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa80c0a2220 con 0x7fa80c09c5f0 2024-09-16T06:45:32.609 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:45:32.609 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:45:32.610 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:45:32.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:45:32.613 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:45:32.613 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:45:32.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:32.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 10 monclient: get_monmap_and_config 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 10 monclient: monmap: 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:32.726558+0000 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:32.726558+0000 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:32.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:32.725 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:32.725 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:32.725 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 1 Processor -- start 2024-09-16T06:45:32.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 1 -- start start 2024-09-16T06:45:32.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.721+0000 7f8e603b2700 10 monclient: init 2024-09-16T06:45:32.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-b con 0x7f8e58135500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-a con 0x7f8e5809a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-c con 0x7f8e5813b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580a67b0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580a7320 con 0x7f8e5813b980 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5809ade0 con 0x7f8e58135500 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient(hunting): authenticate will time out at 1279.860743s 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d94d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8e5809a9d0 0x7f8e5813a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33844/0 (socket says 172.21.15.76:33844) 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5e14e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8e58135500 0x7f8e581358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39142/0 (socket says 172.21.15.76:39142) 2024-09-16T06:45:32.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d94d700 1 -- 172.21.15.76:0/420152322 learned_addr learned my addr 172.21.15.76:0/420152322 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325855321 0 0) 0x7f8e5809ade0 con 0x7f8e58135500 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): my global_id is 125732 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400035d0 con 0x7f8e58135500 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48668307 0 0) 0x7f8e580a7320 con 0x7f8e5813b980 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient(hunting): my global_id is 174409 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5809ade0 con 0x7f8e5813b980 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511267949 0 0) 0x7f8e400035d0 con 0x7f8e58135500 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 >> v1:172.21.15.189:6789/0 conn(0x7f8e5813b980 legacy=0x7f8e5813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 >> v1:172.21.15.76:6789/0 conn(0x7f8e5809a9d0 legacy=0x7f8e5813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: _finish_hunting 0 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 monclient: found mon.noname-b 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58099d10 con 0x7f8e58135500 2024-09-16T06:45:32.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: _finish_auth 0 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: _check_auth_tickets 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e54001630 con 0x7f8e58135500 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 5 monclient: authenticate success, global_id 125732 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: dump: 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.731 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e54001910 con 0x7f8e58135500 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 1 -- 172.21.15.76:0/420152322 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e54001ca0 con 0x7f8e58135500 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 4 set_mon_vals no callback set 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:32.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e5d14c700 10 monclient: dump: 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.733 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 10 monclient: shutdown 2024-09-16T06:45:32.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:32.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- 172.21.15.76:0/420152322 >> v1:172.21.15.76:6790/0 conn(0x7f8e58135500 legacy=0x7f8e581358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- 172.21.15.76:0/420152322 shutdown_connections 2024-09-16T06:45:32.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.725+0000 7f8e603b2700 1 -- 172.21.15.76:0/420152322 >> 172.21.15.76:0/420152322 conn(0x7f8e580a42f0 msgr2=0x7f8e580a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- 172.21.15.76:0/420152322 shutdown_connections 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- 172.21.15.76:0/420152322 wait complete. 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: monmap: 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:32.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 Processor -- start 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- start start 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: init 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-c con 0x7f8e5809a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-b con 0x7f8e58135500 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient(hunting): picked mon.noname-a con 0x7f8e5813b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5813b5e0 con 0x7f8e5813b980 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580a3dd0 con 0x7f8e58135500 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e581309d0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient(hunting): authenticate will time out at 1279.866558s 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e5e14e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8e5809a9d0 0x7f8e5812e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39146/0 (socket says 172.21.15.76:39146) 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e5e14e700 1 -- 172.21.15.76:0/3061252642 learned_addr learned my addr 172.21.15.76:0/3061252642 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273456759 0 0) 0x7f8e581309d0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): my global_id is 125735 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e440035d0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489548070 0 0) 0x7f8e580a3dd0 con 0x7f8e58135500 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:32.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): my global_id is 174412 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e581309d0 con 0x7f8e58135500 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 649262841 0 0) 0x7f8e440035d0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e580a3dd0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e540015f0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): dump: 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148206476 0 0) 0x7f8e580a3dd0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 >> v1:172.21.15.189:6789/0 conn(0x7f8e58135500 legacy=0x7f8e5812ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 >> v1:172.21.15.76:6789/0 conn(0x7f8e5813b980 legacy=0x7f8e5812f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:32.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 monclient: found mon.c 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58133090 con 0x7f8e5809a9d0 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient: _finish_auth 0 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e4effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 5 monclient: authenticate success, global_id 125735 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: _renew_subs 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.729+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e581325e0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 10 monclient: _renew_subs 2024-09-16T06:45:32.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e581d3500 con 0x7f8e5809a9d0 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e54004c30 con 0x7f8e5809a9d0 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e54004fc0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e5c94b700 4 set_mon_vals no callback set 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 10 monclient: dump: 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:32.744 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8e540051a0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(193..193 src has 151..193) v4 ==== 9556+0+0 (unknown 3746259404 0 0) 0x7f8e54044bd0 con 0x7f8e5809a9d0 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e5d94d700 10 monclient: get_auth_request con 0x7f8e440361e0 auth_method 0 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.733+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e38005100 con 0x7f8e5809a9d0 2024-09-16T06:45:32.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.741+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8e54044e60 con 0x7f8e5809a9d0 2024-09-16T06:45:32.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.741+0000 7f8e4effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:32.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.741+0000 7f8e4effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e603b2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e603b2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e603b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e38005930 con 0x7f8e5809a9d0 2024-09-16T06:45:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e4effd700 1 -- 172.21.15.76:0/3061252642 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3000120670) 0x7f8e54014b50 con 0x7f8e5809a9d0 2024-09-16T06:45:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e4effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.981+0000 7f8e4effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:32.984 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8e440361e0 legacy=0x7f8e440386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 10 monclient: shutdown 2024-09-16T06:45:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 >> v1:172.21.15.76:6790/0 conn(0x7f8e5809a9d0 legacy=0x7f8e5812e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 shutdown_connections 2024-09-16T06:45:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 >> 172.21.15.76:0/3061252642 conn(0x7f8e580a42f0 msgr2=0x7f8e5812c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 shutdown_connections 2024-09-16T06:45:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:32.985+0000 7f8e603b2700 1 -- 172.21.15.76:0/3061252642 wait complete. 2024-09-16T06:45:32.998 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:45:32.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:45:32.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.113+0000 7fd2f5f8c700 10 monclient: get_monmap_and_config 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.113+0000 7fd2f5f8c700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.113+0000 7fd2f5f8c700 10 monclient: monmap: 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:33.119493+0000 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:33.119493+0000 2024-09-16T06:45:33.117 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.118 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.118 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.118 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:33.118 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:33.118 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.115+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v194) v1 ==== 168+0+0 (unknown 1161997188 0 0) 0x7fa7fc012c00 con 0x7fa80c09c5f0 2024-09-16T06:45:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.115+0000 7fa7faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.115+0000 7fa7faffd700 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-16T06:45:33.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 1 Processor -- start 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 1 -- start start 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient: init 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-a con 0x7fd2f0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-c con 0x7fd2f0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-b con 0x7fd2f0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f0098990 con 0x7fd2f0097b60 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f0097f40 con 0x7fd2f0099090 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f009d610 con 0x7fd2f0139500 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f5f8c700 10 monclient(hunting): authenticate will time out at 1280.253207s 2024-09-16T06:45:33.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f4f8a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2f0097b60 0x7fd2f009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33862/0 (socket says 172.21.15.76:33862) 2024-09-16T06:45:33.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.117+0000 7fd2f4f8a700 1 -- 172.21.15.76:0/3745833975 learned_addr learned my addr 172.21.15.76:0/3745833975 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170148377 0 0) 0x7fd2f009d610 con 0x7fd2f0139500 2024-09-16T06:45:33.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient(hunting): my global_id is 125738 2024-09-16T06:45:33.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d80035d0 con 0x7fd2f0139500 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1531972835 0 0) 0x7fd2d80035d0 con 0x7fd2f0139500 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 >> v1:172.21.15.189:6789/0 conn(0x7fd2f0099090 legacy=0x7fd2f009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 >> v1:172.21.15.76:6789/0 conn(0x7fd2f0097b60 legacy=0x7fd2f009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 monclient: found mon.noname-b 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f00986b0 con 0x7fd2f0139500 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 5 monclient: authenticate success, global_id 125738 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e8001990 con 0x7fd2f0139500 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: dump: 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2e8001cb0 con 0x7fd2f0139500 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 1 -- 172.21.15.76:0/3745833975 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e8002040 con 0x7fd2f0139500 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 4 set_mon_vals no callback set 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2e77fe700 10 monclient: dump: 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.129 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 10 monclient: shutdown 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/3745833975 >> v1:172.21.15.76:6790/0 conn(0x7fd2f0139500 legacy=0x7fd2f013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/3745833975 shutdown_connections 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/3745833975 >> 172.21.15.76:0/3745833975 conn(0x7fd2f00a0030 msgr2=0x7fd2f00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.121+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/3745833975 shutdown_connections 2024-09-16T06:45:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/3745833975 wait complete. 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient: monmap: 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 Processor -- start 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 -- start start 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient: init 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-a con 0x7fd2f0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-c con 0x7fd2f0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient(hunting): picked mon.noname-b con 0x7fd2f0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f0139330 con 0x7fd2f0097b60 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f00a2ba0 con 0x7fd2f0139500 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f008f0b0 con 0x7fd2f0099090 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f5f8c700 10 monclient(hunting): authenticate will time out at 1280.262333s 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f4f8a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2f0097b60 0x7fd2f0138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33866/0 (socket says 172.21.15.76:33866) 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2f4f8a700 1 -- 172.21.15.76:0/2748220261 learned_addr learned my addr 172.21.15.76:0/2748220261 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143170605 0 0) 0x7fd2f008f0b0 con 0x7fd2f0099090 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 10 monclient(hunting): my global_id is 125741 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.125+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2dc0035d0 con 0x7fd2f0099090 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436671072 0 0) 0x7fd2dc0035d0 con 0x7fd2f0099090 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2f008f0b0 con 0x7fd2f0099090 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e00017e0 con 0x7fd2f0099090 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): dump: 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260837523 0 0) 0x7fd2f00a2ba0 con 0x7fd2f0139500 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): my global_id is 174415 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2dc0035d0 con 0x7fd2f0139500 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328704041 0 0) 0x7fd2f008f0b0 con 0x7fd2f0099090 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.189:6789/0 conn(0x7fd2f0139500 legacy=0x7fd2f01cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6789/0 conn(0x7fd2f0097b60 legacy=0x7fd2f0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 monclient: found mon.c 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f008d050 con 0x7fd2f0099090 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 5 monclient: authenticate success, global_id 125741 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 10 monclient: _renew_subs 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f013c270 con 0x7fd2f0099090 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 10 monclient: _renew_subs 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f008edd0 con 0x7fd2f0099090 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2e00036c0 con 0x7fd2f0099090 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e00039d0 con 0x7fd2f0099090 2024-09-16T06:45:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: dump: 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e6ffd700 4 set_mon_vals no callback set 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd2e0003b30 con 0x7fd2f0099090 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(194..194 src has 151..194) v4 ==== 9769+0+0 (unknown 541791645 0 0) 0x7fd2e0043820 con 0x7fd2f0099090 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2f0096fc0 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2f4f8a700 10 monclient: get_auth_request con 0x7fd2dc036270 auth_method 0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190595400 0 0) 0x7fd2f0096fc0 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient(hunting): my global_id is 184098 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f008f0b0 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3304348417 0 0) 0x7fd2f008f0b0 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2f0096fc0 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e8001950 con 0x7fd2f0096be0 2024-09-16T06:45:33.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 10 monclient: dump: 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 722201787 0 0) 0x7fd2f0096fc0 con 0x7fd2f0096be0 2024-09-16T06:45:33.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.129+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f009fb80 con 0x7fd2f0096be0 2024-09-16T06:45:33.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.133+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd2e8002010 con 0x7fd2f0096be0 2024-09-16T06:45:33.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.133+0000 7fd2e57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.133+0000 7fd2e57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:33.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.133+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6789/0 conn(0x7fd2f0096be0 legacy=0x7fd2f006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.133+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6789/0 conn(0x7fd2f0096be0 legacy=0x7fd2f006dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2f5f8c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2f008f450 con 0x7fd2f002c6b0 2024-09-16T06:45:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572175454 0 0) 0x7fd2f008f450 con 0x7fd2f002c6b0 2024-09-16T06:45:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient(hunting): my global_id is 184101 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f0096fc0 con 0x7fd2f002c6b0 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2242216009 0 0) 0x7fd2f0096fc0 con 0x7fd2f002c6b0 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2f008f450 con 0x7fd2f002c6b0 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2e8003f40 con 0x7fd2f002c6b0 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 10 monclient: dump: 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.147 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 110081294 0 0) 0x7fd2f008f450 con 0x7fd2f002c6b0 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.141+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd2f0099500 con 0x7fd2f002c6b0 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2033571483) 0x7fd2e80041e0 con 0x7fd2f002c6b0 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2e57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2e57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6789/0 conn(0x7fd2f002c6b0 legacy=0x7fd2f006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2e57fa700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6789/0 conn(0x7fd2f002c6b0 legacy=0x7fd2f006e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.148 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:33.149 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.150 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:33.152 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.153 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.154 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.157 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.158 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:33.159 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd2dc036270 legacy=0x7fd2dc038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 10 monclient: shutdown 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 >> v1:172.21.15.76:6790/0 conn(0x7fd2f0099090 legacy=0x7fd2f0138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f578b700 1 -- 172.21.15.76:0/2748220261 reap_dead start 2024-09-16T06:45:33.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f578b700 1 -- 172.21.15.76:0/2748220261 reap_dead start 2024-09-16T06:45:33.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 shutdown_connections 2024-09-16T06:45:33.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 >> 172.21.15.76:0/2748220261 conn(0x7fd2f00a0030 msgr2=0x7fd2f013c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 shutdown_connections 2024-09-16T06:45:33.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.145+0000 7fd2f5f8c700 1 -- 172.21.15.76:0/2748220261 wait complete. 2024-09-16T06:45:33.162 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:33.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:45:33.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:45:33.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.227+0000 7fa810ec8700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.227+0000 7fa810ec8700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.227+0000 7fa810ec8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.227+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa80c066b50 con 0x7fa80c09c5f0 2024-09-16T06:45:33.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.227+0000 7fa7faffd700 1 -- 172.21.15.189:0/3678319570 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v194) v1 ==== 147+0+0 (unknown 1299035270 0 0) 0x7fa7fc014380 con 0x7fa80c09c5f0 2024-09-16T06:45:33.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa7faffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa7faffd700 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa7ec0361c0 legacy=0x7fa7ec038680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 10 monclient: shutdown 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 >> v1:172.21.15.189:6789/0 conn(0x7fa80c09c5f0 legacy=0x7fa80c08e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 shutdown_connections 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 >> 172.21.15.189:0/3678319570 conn(0x7fa80c0a03e0 msgr2=0x7fa80c0a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 shutdown_connections 2024-09-16T06:45:33.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.231+0000 7fa810ec8700 1 -- 172.21.15.189:0/3678319570 wait complete. 2024-09-16T06:45:33.235 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h13' already exists 2024-09-16T06:45:33.245 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:33.245 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: get_monmap_and_config 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: monmap: 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:33.277708+0000 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:33.277708+0000 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:33.276 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 1 Processor -- start 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 1 -- start start 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: init 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-c con 0x7f2c7c09c8e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-a con 0x7f2c7c09f410 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-b con 0x7f2c7c09d3d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c09ed10 con 0x7f2c7c09f410 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c09dc40 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0a1c20 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c83464700 10 monclient(hunting): authenticate will time out at 1280.411572s 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c82c63700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2c7c09d3d0 0x7f2c7c13bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39178/0 (socket says 172.21.15.76:39178) 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.273+0000 7f2c82c63700 1 -- 172.21.15.76:0/1464129816 learned_addr learned my addr 172.21.15.76:0/1464129816 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581289394 0 0) 0x7f2c7c0a1c20 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): my global_id is 125744 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c680035d0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587027252 0 0) 0x7f2c7c09ed10 con 0x7f2c7c09f410 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): my global_id is 184104 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c0a1c20 con 0x7f2c7c09f410 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697895715 0 0) 0x7f2c7c09dc40 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient(hunting): my global_id is 174418 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c09ed10 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787055741 0 0) 0x7f2c680035d0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 >> v1:172.21.15.189:6789/0 conn(0x7f2c7c09c8e0 legacy=0x7f2c7c09ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 >> v1:172.21.15.76:6789/0 conn(0x7f2c7c09f410 legacy=0x7f2c7c138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 monclient: found mon.noname-b 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c09c3e0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c740018c0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: dump: 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 5 monclient: authenticate success, global_id 125744 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c74001b20 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 1 -- 172.21.15.76:0/1464129816 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c74001ef0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 4 set_mon_vals no callback set 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c81460700 10 monclient: dump: 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 10 monclient: shutdown 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- 172.21.15.76:0/1464129816 >> v1:172.21.15.76:6790/0 conn(0x7f2c7c09d3d0 legacy=0x7f2c7c13bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- 172.21.15.76:0/1464129816 shutdown_connections 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- 172.21.15.76:0/1464129816 >> 172.21.15.76:0/1464129816 conn(0x7f2c7c0a2300 msgr2=0x7f2c7c0a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- 172.21.15.76:0/1464129816 shutdown_connections 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- 172.21.15.76:0/1464129816 wait complete. 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 10 monclient: monmap: 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:33.286 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 Processor -- start 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 1 -- start start 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.277+0000 7f2c83464700 10 monclient: init 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-a con 0x7f2c7c09c8e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-c con 0x7f2c7c09d3d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient(hunting): picked mon.noname-b con 0x7f2c7c09f410 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c1b6ba0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c1b4080 con 0x7f2c7c09f410 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c1b52a0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient(hunting): authenticate will time out at 1280.416318s 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c81c61700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2c7c09d3d0 0x7f2c7c1b3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39194/0 (socket says 172.21.15.76:39194) 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c81c61700 1 -- 172.21.15.76:0/1140669293 learned_addr learned my addr 172.21.15.76:0/1140669293 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c82462700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2c7c09c8e0 0x7f2c7c1b3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33894/0 (socket says 172.21.15.76:33894) 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984858442 0 0) 0x7f2c7c1b52a0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): my global_id is 125747 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c600035d0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750571958 0 0) 0x7f2c7c1b6ba0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): my global_id is 184107 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c1b52a0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578795687 0 0) 0x7f2c7c1b4080 con 0x7f2c7c09f410 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): my global_id is 174421 2024-09-16T06:45:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c1b6ba0 con 0x7f2c7c09f410 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38086643 0 0) 0x7f2c600035d0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c7c1b4080 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c70001af0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): dump: 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426815225 0 0) 0x7f2c7c1b52a0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c600035d0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c780018b0 con 0x7f2c7c09c8e0 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): dump: 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108401539 0 0) 0x7f2c7c1b4080 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.189:6789/0 conn(0x7f2c7c09f410 legacy=0x7f2c7c1cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6789/0 conn(0x7f2c7c09c8e0 legacy=0x7f2c7c1b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 monclient: found mon.c 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c099860 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 5 monclient: authenticate success, global_id 125747 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _renew_subs 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c099310 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _renew_subs 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c1b54e0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c70001230 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c70003af0 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c80c5f700 4 set_mon_vals no callback set 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient: dump: 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.299 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c64006590 con 0x7f2c64002830 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507540916 0 0) 0x7f2c64006590 con 0x7f2c64002830 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 10 monclient(hunting): my global_id is 125750 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c1b4080 con 0x7f2c64002830 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.281+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2c70003c50 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(194..194 src has 151..194) v4 ==== 9769+0+0 (unknown 541791645 0 0) 0x7f2c70044960 con 0x7f2c7c09d3d0 2024-09-16T06:45:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c82c63700 10 monclient: get_auth_request con 0x7f2c600361a0 auth_method 0 2024-09-16T06:45:33.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3869499172 0 0) 0x7f2c7c1b4080 con 0x7f2c64002830 2024-09-16T06:45:33.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c64006590 con 0x7f2c64002830 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c78003860 con 0x7f2c64002830 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 10 monclient: dump: 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.302 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2852705287 0 0) 0x7f2c64006590 con 0x7f2c64002830 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c6400a190 con 0x7f2c64002830 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2c78005870 con 0x7f2c64002830 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6790/0 conn(0x7f2c64002830 legacy=0x7f2c64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.285+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6790/0 conn(0x7f2c64002830 legacy=0x7f2c64004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:33.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c64009e30 con 0x7f2c64009150 2024-09-16T06:45:33.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164013628 0 0) 0x7f2c64009e30 con 0x7f2c64009150 2024-09-16T06:45:33.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient(hunting): my global_id is 125753 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64006590 con 0x7f2c64009150 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2305323410 0 0) 0x7f2c64006590 con 0x7f2c64009150 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c64009e30 con 0x7f2c64009150 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2c78003ed0 con 0x7f2c64009150 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient: dump: 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1863169515 0 0) 0x7f2c64009e30 con 0x7f2c64009150 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2c6400c340 con 0x7f2c64009150 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 3971312253) 0x7f2c780041e0 con 0x7f2c64009150 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6790/0 conn(0x7f2c64009150 legacy=0x7f2c640095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c6effd700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6790/0 conn(0x7f2c64009150 legacy=0x7f2c640095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:33.308 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.309 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.310 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:33.311 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:33.312 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.317 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.318 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.319 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.320 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:33.320 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:33.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2c600361a0 legacy=0x7f2c60038660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 10 monclient: shutdown 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 >> v1:172.21.15.76:6790/0 conn(0x7f2c7c09d3d0 legacy=0x7f2c7c1b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c82c63700 1 -- 172.21.15.76:0/1140669293 reap_dead start 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c82c63700 1 -- 172.21.15.76:0/1140669293 reap_dead start 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 shutdown_connections 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 >> 172.21.15.76:0/1140669293 conn(0x7f2c7c0a2300 msgr2=0x7f2c7c070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 shutdown_connections 2024-09-16T06:45:33.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.297+0000 7f2c83464700 1 -- 172.21.15.76:0/1140669293 wait complete. 2024-09-16T06:45:33.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:33.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:45:33.323 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:45:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 10 monclient: get_monmap_and_config 2024-09-16T06:45:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 10 monclient: monmap: 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:33.355026+0000 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:33.355026+0000 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:33.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 1 Processor -- start 2024-09-16T06:45:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 1 -- start start 2024-09-16T06:45:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.351+0000 7f53bb6e9700 10 monclient: init 2024-09-16T06:45:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-a con 0x7f53b409aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-c con 0x7f53b40a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b409a040 con 0x7f53b409aa20 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40a33d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4099d60 con 0x7f53b40a2350 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient(hunting): authenticate will time out at 1285.868711s 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53b9c86700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53b40a3ca0 0x7f53b413dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57988/0 (socket says 172.21.15.189:57988) 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53b9c86700 1 -- 172.21.15.189:0/1170751687 learned_addr learned my addr 172.21.15.189:0/1170751687 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:33.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004873698 0 0) 0x7f53b40a33d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): my global_id is 174424 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a00035d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747320114 0 0) 0x7f53b409a040 con 0x7f53b409aa20 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient(hunting): my global_id is 184110 2024-09-16T06:45:33.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b40a33d0 con 0x7f53b409aa20 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656075276 0 0) 0x7f53a00035d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 >> v1:172.21.15.76:6790/0 conn(0x7f53b40a2350 legacy=0x7f53b40a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 >> v1:172.21.15.76:6789/0 conn(0x7f53b409aa20 legacy=0x7f53b413a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 monclient: found mon.noname-c 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40a3920 con 0x7f53b40a3ca0 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 5 monclient: authenticate success, global_id 174424 2024-09-16T06:45:33.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53ac001990 con 0x7f53b40a3ca0 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: dump: 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ac001bb0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 4 set_mon_vals no callback set 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 1 -- 172.21.15.189:0/1170751687 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53ac001f40 con 0x7f53b40a3ca0 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53a7fff700 10 monclient: dump: 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:33.364 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 10 monclient: shutdown 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- 172.21.15.189:0/1170751687 >> v1:172.21.15.189:6789/0 conn(0x7f53b40a3ca0 legacy=0x7f53b413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- 172.21.15.189:0/1170751687 shutdown_connections 2024-09-16T06:45:33.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- 172.21.15.189:0/1170751687 >> 172.21.15.189:0/1170751687 conn(0x7f53b40a7d70 msgr2=0x7f53b40a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- 172.21.15.189:0/1170751687 shutdown_connections 2024-09-16T06:45:33.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.355+0000 7f53bb6e9700 1 -- 172.21.15.189:0/1170751687 wait complete. 2024-09-16T06:45:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: monmap: 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 Processor -- start 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- start start 2024-09-16T06:45:33.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: init 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-c con 0x7f53b409aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-a con 0x7f53b40a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4121d00 con 0x7f53b40a2350 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b411f9c0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b409b180 con 0x7f53b409aa20 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient(hunting): authenticate will time out at 1285.873941s 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53b9c86700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53b40a3ca0 0x7f53b41d2410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57994/0 (socket says 172.21.15.189:57994) 2024-09-16T06:45:33.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53b9c86700 1 -- 172.21.15.189:0/253804690 learned_addr learned my addr 172.21.15.189:0/253804690 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812666362 0 0) 0x7f53b411f9c0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): my global_id is 174427 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53980035d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853529558 0 0) 0x7f53b409b180 con 0x7f53b409aa20 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): my global_id is 125759 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b411f9c0 con 0x7f53b409aa20 2024-09-16T06:45:33.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300408052 0 0) 0x7f53b4121d00 con 0x7f53b40a2350 2024-09-16T06:45:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): my global_id is 184113 2024-09-16T06:45:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b409b180 con 0x7f53b40a2350 2024-09-16T06:45:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2026742044 0 0) 0x7f53980035d0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53b4121d00 con 0x7f53b40a3ca0 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53ac001950 con 0x7f53b40a3ca0 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.376 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010396097 0 0) 0x7f53b411f9c0 con 0x7f53b409aa20 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53980035d0 con 0x7f53b409aa20 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53b0001830 con 0x7f53b409aa20 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.377 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3618946306 0 0) 0x7f53b4121d00 con 0x7f53b40a3ca0 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 >> v1:172.21.15.76:6790/0 conn(0x7f53b409aa20 legacy=0x7f53b4134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 >> v1:172.21.15.76:6789/0 conn(0x7f53b40a2350 legacy=0x7f53b41cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 monclient: found mon.b 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b406bed0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 5 monclient: authenticate success, global_id 174427 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _renew_subs 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b41350c0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _renew_subs 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b41202c0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ac004f40 con 0x7f53b40a3ca0 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53ac005250 con 0x7f53b40a3ca0 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a5ffb700 10 monclient: dump: 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.359+0000 7f53a77fe700 4 set_mon_vals no callback set 2024-09-16T06:45:33.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53bb6e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53bb6e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53bb6e9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f53ac0053b0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5394005100 con 0x7f53b40a3ca0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(194..194 src has 151..194) v4 ==== 9769+0+0 (unknown 541791645 0 0) 0x7f53ac044f50 con 0x7f53b40a3ca0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.363+0000 7f53b9485700 10 monclient: get_auth_request con 0x7f5398036230 auth_method 0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.371+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f53ac014df0 con 0x7f53b40a3ca0 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.371+0000 7f53a5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.371+0000 7f53a5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:33.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.445+0000 7fc666847700 10 monclient: get_monmap_and_config 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.445+0000 7fc666847700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.445+0000 7fc666847700 10 monclient: monmap: 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:33.450902+0000 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:33.450902+0000 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:33.449 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 Processor -- start 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- start start 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: init 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient(hunting): picked mon.noname-b con 0x7fc660097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient(hunting): picked mon.noname-c con 0x7fc660099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient(hunting): picked mon.noname-a con 0x7fc660139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc660098990 con 0x7fc660139500 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc660097f40 con 0x7fc660099090 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66009d610 con 0x7fc660097b60 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient(hunting): authenticate will time out at 1280.584789s 2024-09-16T06:45:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666046700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc660139500 0x7fc66013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33906/0 (socket says 172.21.15.76:33906) 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666046700 1 -- 172.21.15.76:0/3550604067 learned_addr learned my addr 172.21.15.76:0/3550604067 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206083236 0 0) 0x7fc66009d610 con 0x7fc660097b60 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): my global_id is 125762 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c0035d0 con 0x7fc660097b60 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318497396 0 0) 0x7fc660098990 con 0x7fc660139500 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): my global_id is 184116 2024-09-16T06:45:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66009d610 con 0x7fc660139500 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201312297 0 0) 0x7fc660097f40 con 0x7fc660099090 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient(hunting): my global_id is 174430 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc660098990 con 0x7fc660099090 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046417700 0 0) 0x7fc64c0035d0 con 0x7fc660097b60 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 >> v1:172.21.15.189:6789/0 conn(0x7fc660099090 legacy=0x7fc66009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 >> v1:172.21.15.76:6789/0 conn(0x7fc660139500 legacy=0x7fc66013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 monclient: found mon.noname-b 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600986b0 con 0x7fc660097b60 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 5 monclient: authenticate success, global_id 125762 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc65c001610 con 0x7fc660097b60 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: dump: 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c0018f0 con 0x7fc660097b60 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 1 -- 172.21.15.76:0/3550604067 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc65c001c80 con 0x7fc660097b60 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc664843700 10 monclient: dump: 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 4 set_mon_vals no callback set 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: shutdown 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- 172.21.15.76:0/3550604067 >> v1:172.21.15.76:6790/0 conn(0x7fc660097b60 legacy=0x7fc66009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- 172.21.15.76:0/3550604067 shutdown_connections 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- 172.21.15.76:0/3550604067 >> 172.21.15.76:0/3550604067 conn(0x7fc6600a0030 msgr2=0x7fc6600a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- 172.21.15.76:0/3550604067 shutdown_connections 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 1 -- 172.21.15.76:0/3550604067 wait complete. 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.449+0000 7fc666847700 10 monclient: monmap: 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 Processor -- start 2024-09-16T06:45:33.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- start start 2024-09-16T06:45:33.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: init 2024-09-16T06:45:33.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:33.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient(hunting): picked mon.noname-a con 0x7fc660097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient(hunting): picked mon.noname-c con 0x7fc660099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient(hunting): picked mon.noname-b con 0x7fc660139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006c0c0 con 0x7fc660097b60 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006c280 con 0x7fc660139500 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006d6d0 con 0x7fc660099090 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient(hunting): authenticate will time out at 1280.589286s 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc665845700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc660097b60 0x7fc6601c7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33914/0 (socket says 172.21.15.76:33914) 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc665845700 1 -- 172.21.15.76:0/1023226841 learned_addr learned my addr 172.21.15.76:0/1023226841 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375942812 0 0) 0x7fc6600986b0 con 0x7fc660097b60 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): my global_id is 184119 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6440035d0 con 0x7fc660097b60 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764501338 0 0) 0x7fc6440035d0 con 0x7fc660097b60 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6600986b0 con 0x7fc660097b60 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc65c001530 con 0x7fc660097b60 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): dump: 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.467 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869359471 0 0) 0x7fc66006d6d0 con 0x7fc660099090 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): my global_id is 125765 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6440035d0 con 0x7fc660099090 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466939344 0 0) 0x7fc66006c280 con 0x7fc660139500 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient(hunting): my global_id is 174433 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc66006d6d0 con 0x7fc660139500 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 38694698 0 0) 0x7fc6600986b0 con 0x7fc660097b60 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.76:6790/0 conn(0x7fc660099090 legacy=0x7fc6601c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6789/0 conn(0x7fc660139500 legacy=0x7fc6601d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 monclient: found mon.a 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6601c8cc0 con 0x7fc660097b60 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 5 monclient: authenticate success, global_id 184119 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _renew_subs 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:33.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66009a540 con 0x7fc660097b60 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _renew_subs 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc660139be0 con 0x7fc660097b60 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c0039d0 con 0x7fc660097b60 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc65c003d60 con 0x7fc660097b60 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc657fff700 4 set_mon_vals no callback set 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 10 monclient: dump: 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.471 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc65c003f40 con 0x7fc660097b60 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.453+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc648006590 con 0x7fc648002830 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(194..194 src has 151..194) v4 ==== 9769+0+0 (unknown 541791645 0 0) 0x7fc65c044c20 con 0x7fc660097b60 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc666046700 10 monclient: get_auth_request con 0x7fc64403a7c0 auth_method 0 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687218432 0 0) 0x7fc648006590 con 0x7fc648002830 2024-09-16T06:45:33.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient(hunting): my global_id is 174436 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6600986b0 con 0x7fc648002830 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2136777506 0 0) 0x7fc6600986b0 con 0x7fc648002830 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc648006590 con 0x7fc648002830 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc650001800 con 0x7fc648002830 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient: dump: 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.473 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1320661242 0 0) 0x7fc648006590 con 0x7fc648002830 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc64800a190 con 0x7fc648002830 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc650001aa0 con 0x7fc648002830 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6789/0 conn(0x7fc648002830 legacy=0x7fc648004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.457+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6789/0 conn(0x7fc648002830 legacy=0x7fc648004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc648009e30 con 0x7fc648009150 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338676534 0 0) 0x7fc648009e30 con 0x7fc648009150 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient(hunting): my global_id is 174439 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc648006590 con 0x7fc648009150 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3872193650 0 0) 0x7fc648006590 con 0x7fc648009150 2024-09-16T06:45:33.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc648009e30 con 0x7fc648009150 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc650001d80 con 0x7fc648009150 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient: dump: 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.477 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2969954201 0 0) 0x7fc648009e30 con 0x7fc648009150 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc64800c340 con 0x7fc648009150 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 3478610641) 0x7fc650005be0 con 0x7fc648009150 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6789/0 conn(0x7fc648009150 legacy=0x7fc6480095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc6567fc700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6789/0 conn(0x7fc648009150 legacy=0x7fc6480095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:33.478 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:33.479 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:33.480 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.481 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.484 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.487 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.488 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:33.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:33.490 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:33.491 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc64403a7c0 legacy=0x7fc64403cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 10 monclient: shutdown 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 >> v1:172.21.15.76:6789/0 conn(0x7fc660097b60 legacy=0x7fc6601c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666046700 1 -- 172.21.15.76:0/1023226841 reap_dead start 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666046700 1 -- 172.21.15.76:0/1023226841 reap_dead start 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 shutdown_connections 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 >> 172.21.15.76:0/1023226841 conn(0x7fc6600a0030 msgr2=0x7fc6600a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 shutdown_connections 2024-09-16T06:45:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.469+0000 7fc666847700 1 -- 172.21.15.76:0/1023226841 wait complete. 2024-09-16T06:45:33.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:33.493 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:45:33.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:45:33.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.543+0000 7f53bb6e9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.543+0000 7f53bb6e9700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:33.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.543+0000 7f53bb6e9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:33.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:33.543+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5394005930 con 0x7f53b40a3ca0 2024-09-16T06:45:33.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: get_monmap_and_config 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: monmap: 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:33.613516+0000 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:33.613516+0000 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.629 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.630 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:33.630 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:33.630 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 1 Processor -- start 2024-09-16T06:45:33.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 1 -- start start 2024-09-16T06:45:33.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: init 2024-09-16T06:45:33.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:33.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-c con 0x7fa12409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-a con 0x7fa1240709d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-b con 0x7fa124070de0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12409f140 con 0x7fa1240709d0 2024-09-16T06:45:33.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124072530 con 0x7fa12409ed60 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124097f00 con 0x7fa124070de0 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a944700 10 monclient(hunting): authenticate will time out at 1280.746903s 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a143700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa124070de0 0x7fa12413de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39244/0 (socket says 172.21.15.76:39244) 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa12a143700 1 -- 172.21.15.76:0/1895944702 learned_addr learned my addr 172.21.15.76:0/1895944702 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa129141700 1 --1- 172.21.15.76:0/1895944702 >> v1:172.21.15.76:6789/0 conn(0x7fa1240709d0 0x7fa12413a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33926/0 (socket says 172.21.15.76:33926) 2024-09-16T06:45:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036208623 0 0) 0x7fa124097f00 con 0x7fa124070de0 2024-09-16T06:45:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): my global_id is 125768 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1100035d0 con 0x7fa124070de0 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280886413 0 0) 0x7fa12409f140 con 0x7fa1240709d0 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): my global_id is 184122 2024-09-16T06:45:33.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa124097f00 con 0x7fa1240709d0 2024-09-16T06:45:33.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725076534 0 0) 0x7fa124072530 con 0x7fa12409ed60 2024-09-16T06:45:33.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient(hunting): my global_id is 174442 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12409f140 con 0x7fa12409ed60 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570111282 0 0) 0x7fa1100035d0 con 0x7fa124070de0 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 >> v1:172.21.15.189:6789/0 conn(0x7fa12409ed60 legacy=0x7fa1240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 >> v1:172.21.15.76:6789/0 conn(0x7fa1240709d0 legacy=0x7fa12413a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 monclient: found mon.noname-b 2024-09-16T06:45:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12409a320 con 0x7fa124070de0 2024-09-16T06:45:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.609+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa11c001610 con 0x7fa124070de0 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: dump: 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.643 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 5 monclient: authenticate success, global_id 125768 2024-09-16T06:45:33.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:33.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa11c0018f0 con 0x7fa124070de0 2024-09-16T06:45:33.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 1 -- 172.21.15.76:0/1895944702 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa11c001c80 con 0x7fa124070de0 2024-09-16T06:45:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 4 set_mon_vals no callback set 2024-09-16T06:45:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa128940700 10 monclient: dump: 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.646 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: shutdown 2024-09-16T06:45:33.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- 172.21.15.76:0/1895944702 >> v1:172.21.15.76:6790/0 conn(0x7fa124070de0 legacy=0x7fa12413de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- 172.21.15.76:0/1895944702 shutdown_connections 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- 172.21.15.76:0/1895944702 >> 172.21.15.76:0/1895944702 conn(0x7fa1240a0030 msgr2=0x7fa1240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- 172.21.15.76:0/1895944702 shutdown_connections 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- 172.21.15.76:0/1895944702 wait complete. 2024-09-16T06:45:33.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: build_initial_monmap 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: monmap: 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:33.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:33.650 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:33.650 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:33.650 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:33.650 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:33.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 Processor -- start 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- start start 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: init 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:33.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-c con 0x7fa1240709d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-b con 0x7fa124070de0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient(hunting): picked mon.noname-a con 0x7fa12409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1240908e0 con 0x7fa12409ed60 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12409a540 con 0x7fa124070de0 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1240a26c0 con 0x7fa1240709d0 2024-09-16T06:45:33.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa12a944700 10 monclient(hunting): authenticate will time out at 1280.751430s 2024-09-16T06:45:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa129942700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa1240709d0 0x7fa1241c9c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:39248/0 (socket says 172.21.15.76:39248) 2024-09-16T06:45:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa129942700 1 -- 172.21.15.76:0/3408508756 learned_addr learned my addr 172.21.15.76:0/3408508756 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.613+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930217527 0 0) 0x7fa1240908e0 con 0x7fa12409ed60 2024-09-16T06:45:33.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): my global_id is 184125 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1080035d0 con 0x7fa12409ed60 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255351183 0 0) 0x7fa12409a540 con 0x7fa124070de0 2024-09-16T06:45:33.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): my global_id is 174445 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1240908e0 con 0x7fa124070de0 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252853621 0 0) 0x7fa1240a26c0 con 0x7fa1240709d0 2024-09-16T06:45:33.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): my global_id is 125771 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12409a540 con 0x7fa1240709d0 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323318135 0 0) 0x7fa12409a540 con 0x7fa1240709d0 2024-09-16T06:45:33.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1240a26c0 con 0x7fa1240709d0 2024-09-16T06:45:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1140018b0 con 0x7fa1240709d0 2024-09-16T06:45:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): dump: 2024-09-16T06:45:33.657 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:33.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 750446420 0 0) 0x7fa1240908e0 con 0x7fa124070de0 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa12409a540 con 0x7fa124070de0 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158984821 0 0) 0x7fa1080035d0 con 0x7fa12409ed60 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1240908e0 con 0x7fa12409ed60 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa120001960 con 0x7fa124070de0 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): dump: 2024-09-16T06:45:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.661 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.662 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa11c001430 con 0x7fa12409ed60 2024-09-16T06:45:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient(hunting): dump: 2024-09-16T06:45:33.663 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.664 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.664 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.664 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.664 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.665 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.665 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.665 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 783328291 0 0) 0x7fa1240a26c0 con 0x7fa1240709d0 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 >> v1:172.21.15.189:6789/0 conn(0x7fa124070de0 legacy=0x7fa1240966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 >> v1:172.21.15.76:6789/0 conn(0x7fa12409ed60 legacy=0x7fa1241d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 monclient: found mon.c 2024-09-16T06:45:33.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa124091180 con 0x7fa1240709d0 2024-09-16T06:45:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:33.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 5 monclient: authenticate success, global_id 125771 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _renew_subs 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12408f2d0 con 0x7fa1240709d0 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _renew_subs 2024-09-16T06:45:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa124096e00 con 0x7fa1240709d0 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1140036c0 con 0x7fa1240709d0 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa114003a50 con 0x7fa1240709d0 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11bfff700 4 set_mon_vals no callback set 2024-09-16T06:45:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 10 monclient: dump: 2024-09-16T06:45:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:33.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:33.672 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa114003c30 con 0x7fa1240709d0 2024-09-16T06:45:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10c005100 con 0x7fa1240709d0 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(194..194 src has 151..194) v4 ==== 9769+0+0 (unknown 541791645 0 0) 0x7fa114044a90 con 0x7fa1240709d0 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.617+0000 7fa129141700 10 monclient: get_auth_request con 0x7fa1080366f0 auth_method 0 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.625+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa114014810 con 0x7fa1240709d0 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.625+0000 7fa11a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.625+0000 7fa11a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.861+0000 7fa12a944700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:45:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.861+0000 7fa12a944700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:45:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.861+0000 7fa12a944700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:33.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.861+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa10c005bc0 con 0x7fa1240709d0 2024-09-16T06:45:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.865+0000 7fa11a7fc700 1 -- 172.21.15.76:0/3408508756 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fa114012c80 con 0x7fa1240709d0 2024-09-16T06:45:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.865+0000 7fa11a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:45:33.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.865+0000 7fa11a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=13,logm=50} crc {auth=1874712226,config=1097430493,health=1608197807,kv=1001081567,logm=1008926324}) 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=12177877,config=1097430493,health=3353365297,kv=2879498132,logm=2335481344}) 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=13,logm=50} crc {auth=1874712226,config=1097430493,health=1608197807,kv=1001081567,logm=1008926324}) 2024-09-16T06:45:33.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.865+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=654116656,config=1097430493,health=3353365297,kv=1001081567,logm=596449861}) 2024-09-16T06:45:33.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796316982}) 2024-09-16T06:45:33.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427675543}) 2024-09-16T06:45:33.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796316982}) 2024-09-16T06:45:33.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441178590}) 2024-09-16T06:45:33.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa1080366f0 legacy=0x7fa108038bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 10 monclient: shutdown 2024-09-16T06:45:33.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:33.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 >> v1:172.21.15.76:6790/0 conn(0x7fa1240709d0 legacy=0x7fa1241c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:33.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 shutdown_connections 2024-09-16T06:45:33.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 >> 172.21.15.76:0/3408508756 conn(0x7fa1240a0030 msgr2=0x7fa1240a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:33.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 shutdown_connections 2024-09-16T06:45:33.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:33.869+0000 7fa12a944700 1 -- 172.21.15.76:0/3408508756 wait complete. 2024-09-16T06:45:33.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335875672}) 2024-09-16T06:45:33.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321997289}) 2024-09-16T06:45:33.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335875672}) 2024-09-16T06:45:33.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.869+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37062153}) 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018083383}) 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612218278}) 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018083383}) 2024-09-16T06:45:33.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240517765}) 2024-09-16T06:45:33.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317594002}) 2024-09-16T06:45:33.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838350881}) 2024-09-16T06:45:33.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.873+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317594002}) 2024-09-16T06:45:33.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=356463537}) 2024-09-16T06:45:33.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514798639}) 2024-09-16T06:45:33.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484802494}) 2024-09-16T06:45:33.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514798639}) 2024-09-16T06:45:33.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=305810361}) 2024-09-16T06:45:33.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057265218}) 2024-09-16T06:45:33.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4196829533}) 2024-09-16T06:45:33.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057265218}) 2024-09-16T06:45:33.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.877+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813532572}) 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255420981}) 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089125386}) 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255420981}) 2024-09-16T06:45:33.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26949929}) 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687949095}) 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375752786}) 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687949095}) 2024-09-16T06:45:33.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=276280292}) 2024-09-16T06:45:33.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757334347}) 2024-09-16T06:45:33.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581432561}) 2024-09-16T06:45:33.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757334347}) 2024-09-16T06:45:33.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.881+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936378873}) 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467014937}) 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990353666}) 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467014937}) 2024-09-16T06:45:33.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883649805}) 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598608874}) 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616758990}) 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598608874}) 2024-09-16T06:45:33.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526903595}) 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687430789}) 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945257253}) 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687430789}) 2024-09-16T06:45:33.889 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.885+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161879373}) 2024-09-16T06:45:33.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278889417}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239482699}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278889417}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135999370}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458784576}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815076885}) 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458784576}) 2024-09-16T06:45:33.892 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52788305}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=742640289}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755285230}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=742640289}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316980356}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278144903}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270415356}) 2024-09-16T06:45:33.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278144903}) 2024-09-16T06:45:33.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.889+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668004386}) 2024-09-16T06:45:33.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1796104294}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334838820}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1796104294}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491048132}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429380342}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955131067}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429380342}) 2024-09-16T06:45:33.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349546118}) 2024-09-16T06:45:33.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301965235}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481186926}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301965235}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376654680}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281521169}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56556949}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281521169}) 2024-09-16T06:45:33.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.893+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725089910}) 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254827895}) 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940481496}) 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254827895}) 2024-09-16T06:45:33.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208725770}) 2024-09-16T06:45:33.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516565884}) 2024-09-16T06:45:33.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276221022}) 2024-09-16T06:45:33.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516565884}) 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=132050486}) 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306810499}) 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329062908}) 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306810499}) 2024-09-16T06:45:33.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053380122}) 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352883505}) 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2192599245}) 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352883505}) 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.897+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140191458}) 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568617508}) 2024-09-16T06:45:33.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945044332}) 2024-09-16T06:45:33.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568617508}) 2024-09-16T06:45:33.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2067320507}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959345062}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441225476}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959345062}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874869854}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937699554}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481870476}) 2024-09-16T06:45:33.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937699554}) 2024-09-16T06:45:33.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543297961}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188801760}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508653684}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188801760}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487745489}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74060878}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672442073}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74060878}) 2024-09-16T06:45:33.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.901+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565651843}) 2024-09-16T06:45:33.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319440315}) 2024-09-16T06:45:33.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639787560}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319440315}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447880211}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839349653}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1691592872}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839349653}) 2024-09-16T06:45:33.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718956516}) 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609218709}) 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230043858}) 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609218709}) 2024-09-16T06:45:33.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879402072}) 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=2,mgr=6,mgrstat=12,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=39} crc {logm=76764192,mdsmap=3140514009,mgr=2389209194,mgrstat=1054088828,mon_config_key=3158179092,monmap=968649221,osd_metadata=2101314684,osd_pg_creating=933646749,osd_snap=2033388227,osdmap=3921184514}) 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=34} crc {logm=1137711487,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=717412433,mon_config_key=4229390698,monmap=1887442722,osd_metadata=3249661422,osd_pg_creating=933646749,osd_snap=2033388227,osdmap=736005266}) 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=2,mgr=6,mgrstat=12,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=39} crc {logm=76764192,mdsmap=3140514009,mgr=2389209194,mgrstat=1054088828,mon_config_key=3158179092,monmap=968649221,osd_metadata=2101314684,osd_pg_creating=933646749,osd_snap=2033388227,osdmap=3921184514}) 2024-09-16T06:45:33.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.905+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=33} crc {logm=1137711487,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=353350544,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=933646749,osd_snap=2033388227,osdmap=336425983}) 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=41} crc {osdmap=125446202}) 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=3157970466}) 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=41} crc {osdmap=125446202}) 2024-09-16T06:45:33.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:45:33.913+0000 7f2941141700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=2863029366}) 2024-09-16T06:45:34.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.147+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v195) v1 ==== 168+0+0 (unknown 3781295072 0 0) 0x7f53ac013260 con 0x7f53b40a3ca0 2024-09-16T06:45:34.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.151+0000 7f53a5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.151+0000 7f53a5ffb700 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-16T06:45:34.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53bb6e9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53bb6e9700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53bb6e9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f53940053f0 con 0x7f53b40a3ca0 2024-09-16T06:45:34.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53a5ffb700 1 -- 172.21.15.189:0/253804690 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v195) v1 ==== 147+0+0 (unknown 77263166 0 0) 0x7f53ac0149e0 con 0x7f53b40a3ca0 2024-09-16T06:45:34.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53a5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.259+0000 7f53a5ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-16T06:45:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5398036230 legacy=0x7f53980386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 10 monclient: shutdown 2024-09-16T06:45:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:34.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 >> v1:172.21.15.189:6789/0 conn(0x7f53b40a3ca0 legacy=0x7f53b41d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 shutdown_connections 2024-09-16T06:45:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 >> 172.21.15.189:0/253804690 conn(0x7f53b40a7d70 msgr2=0x7f53b41354f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 shutdown_connections 2024-09-16T06:45:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.263+0000 7f53bb6e9700 1 -- 172.21.15.189:0/253804690 wait complete. 2024-09-16T06:45:34.267 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h14' already exists 2024-09-16T06:45:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:34.281 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 10 monclient: get_monmap_and_config 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 10 monclient: build_initial_monmap 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 10 monclient: monmap: 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:34.394711+0000 2024-09-16T06:45:34.394 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:34.394711+0000 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 1 Processor -- start 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 1 -- start start 2024-09-16T06:45:34.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.391+0000 7fe24afaa700 10 monclient: init 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-a con 0x7fe24409b3e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-c con 0x7fe2440a1210 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe244071830 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440a0750 con 0x7fe24409b3e0 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440a1690 con 0x7fe2440a1210 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24409ce00 con 0x7fe244071830 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:34.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient(hunting): authenticate will time out at 1286.908485s 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe243fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2440a1210 0x7fe244137670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58004/0 (socket says 172.21.15.189:58004) 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe243fff700 1 -- 172.21.15.189:0/2882633369 learned_addr learned my addr 172.21.15.189:0/2882633369 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136257331 0 0) 0x7fe2440a1690 con 0x7fe2440a1210 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): my global_id is 174448 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22c0035d0 con 0x7fe2440a1210 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125126807 0 0) 0x7fe24409ce00 con 0x7fe244071830 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): my global_id is 125774 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2440a1690 con 0x7fe244071830 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957672934 0 0) 0x7fe2440a0750 con 0x7fe24409b3e0 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient(hunting): my global_id is 184128 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24409ce00 con 0x7fe24409b3e0 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420039854 0 0) 0x7fe22c0035d0 con 0x7fe2440a1210 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 >> v1:172.21.15.76:6790/0 conn(0x7fe244071830 legacy=0x7fe24413ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 >> v1:172.21.15.76:6789/0 conn(0x7fe24409b3e0 legacy=0x7fe24409b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 monclient: found mon.noname-c 2024-09-16T06:45:34.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24409cf90 con 0x7fe2440a1210 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 5 monclient: authenticate success, global_id 174448 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe238001610 con 0x7fe2440a1210 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: dump: 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:34.402 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe238001880 con 0x7fe2440a1210 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 1 -- 172.21.15.189:0/2882633369 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe238001c10 con 0x7fe2440a1210 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 4 set_mon_vals no callback set 2024-09-16T06:45:34.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe2437fe700 10 monclient: dump: 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 10 monclient: shutdown 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- 172.21.15.189:0/2882633369 >> v1:172.21.15.189:6789/0 conn(0x7fe2440a1210 legacy=0x7fe244137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- 172.21.15.189:0/2882633369 shutdown_connections 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- 172.21.15.189:0/2882633369 >> 172.21.15.189:0/2882633369 conn(0x7fe2440a44a0 msgr2=0x7fe2440a0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- 172.21.15.189:0/2882633369 shutdown_connections 2024-09-16T06:45:34.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.395+0000 7fe24afaa700 1 -- 172.21.15.189:0/2882633369 wait complete. 2024-09-16T06:45:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: build_initial_monmap 2024-09-16T06:45:34.408 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: monmap: 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 Processor -- start 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- start start 2024-09-16T06:45:34.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: init 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-c con 0x7fe244071830 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-a con 0x7fe24409b3e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe2440a1210 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440a2ff0 con 0x7fe24409b3e0 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440a2b20 con 0x7fe2440a1210 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24409b0d0 con 0x7fe244071830 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient(hunting): authenticate will time out at 1286.914125s 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe249547700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2440a1210 0x7fe2441d2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58008/0 (socket says 172.21.15.189:58008) 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe249547700 1 -- 172.21.15.189:0/470984568 learned_addr learned my addr 172.21.15.189:0/470984568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:34.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363086843 0 0) 0x7fe2440a2b20 con 0x7fe2440a1210 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): my global_id is 174451 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2300035d0 con 0x7fe2440a1210 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884168061 0 0) 0x7fe24409b0d0 con 0x7fe244071830 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): my global_id is 125777 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2440a2b20 con 0x7fe244071830 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341462090 0 0) 0x7fe2440a2ff0 con 0x7fe24409b3e0 2024-09-16T06:45:34.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): my global_id is 184131 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24409b0d0 con 0x7fe24409b3e0 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072038213 0 0) 0x7fe2300035d0 con 0x7fe2440a1210 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2440a2ff0 con 0x7fe2440a1210 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe23c001c30 con 0x7fe2440a1210 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): dump: 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:34.412 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037170408 0 0) 0x7fe2440a2ff0 con 0x7fe2440a1210 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 >> v1:172.21.15.76:6790/0 conn(0x7fe244071830 legacy=0x7fe2441b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 >> v1:172.21.15.76:6789/0 conn(0x7fe24409b3e0 legacy=0x7fe2441cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: _finish_hunting 0 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 monclient: found mon.b 2024-09-16T06:45:34.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2441a58b0 con 0x7fe2440a1210 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: _finish_auth 0 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: _check_auth_tickets 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 5 monclient: authenticate success, global_id 174451 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _renew_subs 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2441a3ea0 con 0x7fe2440a1210 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _renew_subs 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2441a3550 con 0x7fe2440a1210 2024-09-16T06:45:34.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe23c003a20 con 0x7fe2440a1210 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe23c003db0 con 0x7fe2440a1210 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe242ffd700 4 set_mon_vals no callback set 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.399+0000 7fe2417fa700 10 monclient: dump: 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:34.415 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe23c003fd0 con 0x7fe2440a1210 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe24afaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe24afaa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(195..195 src has 151..195) v4 ==== 9982+0+0 (unknown 1858956614 0 0) 0x7fe23c044e70 con 0x7fe2440a1210 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe24afaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe224005100 con 0x7fe2440a1210 2024-09-16T06:45:34.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.403+0000 7fe248d46700 10 monclient: get_auth_request con 0x7fe2300362c0 auth_method 0 2024-09-16T06:45:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.411+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe23c014b00 con 0x7fe2440a1210 2024-09-16T06:45:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.411+0000 7fe2417fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:34.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.411+0000 7fe2417fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:34.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.583+0000 7fe24afaa700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.583+0000 7fe24afaa700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:34.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.583+0000 7fe24afaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:34.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:34.583+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe224005930 con 0x7fe2440a1210 2024-09-16T06:45:34.891 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:45:34.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:35.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.001+0000 7f1190e1b700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.001+0000 7f1190e1b700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.001+0000 7f1190e1b700 10 monclient: monmap: 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:35.007470+0000 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:35.007470+0000 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.005 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.006 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.006 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 Processor -- start 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- start start 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient: init 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-c con 0x7f118c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-b con 0x7f118c099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-a con 0x7f118c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c098990 con 0x7f118c139500 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c097f40 con 0x7f118c097b60 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c09d610 con 0x7f118c099090 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient(hunting): authenticate will time out at 1282.141202s 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1182ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f118c099090 0x7f118c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35128/0 (socket says 172.21.15.76:35128) 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1182ffd700 1 -- 172.21.15.76:0/307439124 learned_addr learned my addr 172.21.15.76:0/307439124 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385496501 0 0) 0x7f118c09d610 con 0x7f118c099090 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): my global_id is 125780 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11780035d0 con 0x7f118c099090 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734253002 0 0) 0x7f118c098990 con 0x7f118c139500 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): my global_id is 184134 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c09d610 con 0x7f118c139500 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491148217 0 0) 0x7f118c097f40 con 0x7f118c097b60 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient(hunting): my global_id is 174454 2024-09-16T06:45:35.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c098990 con 0x7f118c097b60 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681587323 0 0) 0x7f11780035d0 con 0x7f118c099090 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 >> v1:172.21.15.189:6789/0 conn(0x7f118c097b60 legacy=0x7f118c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 >> v1:172.21.15.76:6789/0 conn(0x7f118c139500 legacy=0x7f118c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 monclient: found mon.noname-b 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0986b0 con 0x7f118c099090 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 5 monclient: authenticate success, global_id 125780 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1174001610 con 0x7f118c099090 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: dump: 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.013 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1174001830 con 0x7f118c099090 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 1 -- 172.21.15.76:0/307439124 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1174001bc0 con 0x7f118c099090 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 4 set_mon_vals no callback set 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f118affd700 10 monclient: dump: 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 10 monclient: shutdown 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- 172.21.15.76:0/307439124 >> v1:172.21.15.76:6790/0 conn(0x7f118c099090 legacy=0x7f118c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- 172.21.15.76:0/307439124 shutdown_connections 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- 172.21.15.76:0/307439124 >> 172.21.15.76:0/307439124 conn(0x7f118c0a0030 msgr2=0x7f118c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- 172.21.15.76:0/307439124 shutdown_connections 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.005+0000 7f1190e1b700 1 -- 172.21.15.76:0/307439124 wait complete. 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient: monmap: 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.017 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.017 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.017 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 1 Processor -- start 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 1 -- start start 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient: init 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-c con 0x7f118c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-a con 0x7f118c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient(hunting): picked mon.noname-b con 0x7f118c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c070400 con 0x7f118c099090 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c06d3e0 con 0x7f118c139500 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0a2d30 con 0x7f118c097b60 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1190e1b700 10 monclient(hunting): authenticate will time out at 1282.146648s 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f118b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f118c097b60 0x7f118c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35134/0 (socket says 172.21.15.76:35134) 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1182ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f118c099090 0x7f118c06ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42942/0 (socket says 172.21.15.76:42942) 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f118b7fe700 1 -- 172.21.15.76:0/1646914372 learned_addr learned my addr 172.21.15.76:0/1646914372 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152085792 0 0) 0x7f118c0a2d30 con 0x7f118c097b60 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): my global_id is 125783 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c0035d0 con 0x7f118c097b60 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925702965 0 0) 0x7f118c070400 con 0x7f118c099090 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): my global_id is 184137 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c0a2d30 con 0x7f118c099090 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469156980 0 0) 0x7f118c06d3e0 con 0x7f118c139500 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): my global_id is 174457 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c070400 con 0x7f118c139500 2024-09-16T06:45:35.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3331469260 0 0) 0x7f118c0a2d30 con 0x7f118c099090 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f118c06d3e0 con 0x7f118c099090 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1174001b40 con 0x7f118c099090 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.025 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1622594442 0 0) 0x7f116c0035d0 con 0x7f118c097b60 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f118c0a2d30 con 0x7f118c097b60 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f117c001990 con 0x7f118c097b60 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.009+0000 7f1188ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.026 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906944439 0 0) 0x7f118c070400 con 0x7f118c139500 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f116c0035d0 con 0x7f118c139500 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f11840019b0 con 0x7f118c139500 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.027 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3234955283 0 0) 0x7f118c0a2d30 con 0x7f118c097b60 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 >> v1:172.21.15.189:6789/0 conn(0x7f118c139500 legacy=0x7f118c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 >> v1:172.21.15.76:6789/0 conn(0x7f118c099090 legacy=0x7f118c06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 monclient: found mon.c 2024-09-16T06:45:35.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c06fb60 con 0x7f118c097b60 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 5 monclient: authenticate success, global_id 125783 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _renew_subs 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c06fe40 con 0x7f118c097b60 2024-09-16T06:45:35.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _renew_subs 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f117c0036c0 con 0x7f118c097b60 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c06ed20 con 0x7f118c097b60 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f117c003a50 con 0x7f118c097b60 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 10 monclient: dump: 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.030 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1189ffb700 4 set_mon_vals no callback set 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1170005100 con 0x7f118c097b60 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f117c003bb0 con 0x7f118c097b60 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(195..195 src has 151..195) v4 ==== 9982+0+0 (unknown 1858956614 0 0) 0x7f117c044b50 con 0x7f118c097b60 2024-09-16T06:45:35.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.013+0000 7f1182ffd700 10 monclient: get_auth_request con 0x7f116c036530 auth_method 0 2024-09-16T06:45:35.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.021+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f117c014800 con 0x7f118c097b60 2024-09-16T06:45:35.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.021+0000 7f1188ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.021+0000 7f1188ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:35.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.163+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v196) v1 ==== 168+0+0 (unknown 1220657531 0 0) 0x7fe23c012f70 con 0x7fe2440a1210 2024-09-16T06:45:35.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.163+0000 7fe2417fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.166 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.163+0000 7fe2417fa700 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-16T06:45:35.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1190e1b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:35.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1190e1b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:35.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1190e1b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1170005930 con 0x7f118c097b60 2024-09-16T06:45:35.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1188ff9700 1 -- 172.21.15.76:0/1646914372 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3336371665) 0x7f117c012c70 con 0x7f118c097b60 2024-09-16T06:45:35.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1188ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:35.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.253+0000 7f1188ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:35.257 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 >> v1:172.21.15.189:6812/3098810674 conn(0x7f116c036530 legacy=0x7f116c0389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 10 monclient: shutdown 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 >> v1:172.21.15.76:6790/0 conn(0x7f118c097b60 legacy=0x7f118c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 shutdown_connections 2024-09-16T06:45:35.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 >> 172.21.15.76:0/1646914372 conn(0x7f118c0a0030 msgr2=0x7f118c0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 shutdown_connections 2024-09-16T06:45:35.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.257+0000 7f1190e1b700 1 -- 172.21.15.76:0/1646914372 wait complete. 2024-09-16T06:45:35.272 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:45:35.272 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:45:35.272 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:45:35.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:45:35.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.271+0000 7fe24afaa700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.271+0000 7fe24afaa700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.271+0000 7fe24afaa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.271+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe224005b70 con 0x7fe2440a1210 2024-09-16T06:45:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe2417fa700 1 -- 172.21.15.189:0/470984568 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v196) v1 ==== 147+0+0 (unknown 1960435944 0 0) 0x7fe23c0146f0 con 0x7fe2440a1210 2024-09-16T06:45:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe2417fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe2417fa700 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-16T06:45:35.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe2300362c0 legacy=0x7fe230038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 10 monclient: shutdown 2024-09-16T06:45:35.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 >> v1:172.21.15.189:6789/0 conn(0x7fe2440a1210 legacy=0x7fe2441d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 shutdown_connections 2024-09-16T06:45:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 >> 172.21.15.189:0/470984568 conn(0x7fe2440a44a0 msgr2=0x7fe244070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 shutdown_connections 2024-09-16T06:45:35.280 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.275+0000 7fe24afaa700 1 -- 172.21.15.189:0/470984568 wait complete. 2024-09-16T06:45:35.281 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h15' already exists 2024-09-16T06:45:35.290 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:35.290 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-16T06:45:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: monmap: 2024-09-16T06:45:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:35.384019+0000 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:35.384019+0000 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 1 Processor -- start 2024-09-16T06:45:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 1 -- start start 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: init 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-a con 0x7f1d40132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-c con 0x7f1d4006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-b con 0x7f1d4006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4006c280 con 0x7f1d40132330 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4006bfa0 con 0x7f1d4006d480 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4009ace0 con 0x7f1d4006cf50 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient(hunting): authenticate will time out at 1282.518006s 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d453c0700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1d40132330 0x7f1d40132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42956/0 (socket says 172.21.15.76:42956) 2024-09-16T06:45:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d45bc1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1d4006cf50 0x7f1d4013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35140/0 (socket says 172.21.15.76:35140) 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d453c0700 1 -- 172.21.15.76:0/17069160 learned_addr learned my addr 172.21.15.76:0/17069160 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862163153 0 0) 0x7f1d4006c280 con 0x7f1d40132330 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): my global_id is 184140 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c0035d0 con 0x7f1d40132330 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885083834 0 0) 0x7f1d4009ace0 con 0x7f1d4006cf50 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): my global_id is 125786 2024-09-16T06:45:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4006c280 con 0x7f1d4006cf50 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081185129 0 0) 0x7f1d4006bfa0 con 0x7f1d4006d480 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient(hunting): my global_id is 174460 2024-09-16T06:45:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4009ace0 con 0x7f1d4006d480 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037025884 0 0) 0x7f1d2c0035d0 con 0x7f1d40132330 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 >> v1:172.21.15.76:6790/0 conn(0x7f1d4006cf50 legacy=0x7f1d4013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 >> v1:172.21.15.189:6789/0 conn(0x7f1d4006d480 legacy=0x7f1d4013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 monclient: found mon.noname-a 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4009ae70 con 0x7f1d40132330 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 5 monclient: authenticate success, global_id 184140 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d3c001990 con 0x7f1d40132330 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: dump: 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d3c001c70 con 0x7f1d40132330 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d463c2700 4 set_mon_vals no callback set 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 1 -- 172.21.15.76:0/17069160 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d3c001e60 con 0x7f1d40132330 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.381+0000 7f1d33fff700 10 monclient: dump: 2024-09-16T06:45:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: shutdown 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- 172.21.15.76:0/17069160 >> v1:172.21.15.76:6789/0 conn(0x7f1d40132330 legacy=0x7f1d40132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- 172.21.15.76:0/17069160 shutdown_connections 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- 172.21.15.76:0/17069160 >> 172.21.15.76:0/17069160 conn(0x7f1d400a41c0 msgr2=0x7f1d400a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- 172.21.15.76:0/17069160 shutdown_connections 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- 172.21.15.76:0/17069160 wait complete. 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: monmap: 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.397 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 Processor -- start 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- start start 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: init 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-a con 0x7f1d4006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-c con 0x7f1d4006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient(hunting): picked mon.noname-b con 0x7f1d40132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40130000 con 0x7f1d4006cf50 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4009e8b0 con 0x7f1d40132330 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4012c680 con 0x7f1d4006d480 2024-09-16T06:45:35.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d463c2700 10 monclient(hunting): authenticate will time out at 1282.522738s 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d44bbf700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1d4006d480 0x7f1d401ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35142/0 (socket says 172.21.15.76:35142) 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d44bbf700 1 -- 172.21.15.76:0/1305168922 learned_addr learned my addr 172.21.15.76:0/1305168922 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486537358 0 0) 0x7f1d40130000 con 0x7f1d4006cf50 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): my global_id is 184143 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d240035d0 con 0x7f1d4006cf50 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619355795 0 0) 0x7f1d4012c680 con 0x7f1d4006d480 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): my global_id is 125789 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40130000 con 0x7f1d4006d480 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152298207 0 0) 0x7f1d4009e8b0 con 0x7f1d40132330 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): my global_id is 174463 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4012c680 con 0x7f1d40132330 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068155892 0 0) 0x7f1d240035d0 con 0x7f1d4006cf50 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d4009e8b0 con 0x7f1d4006cf50 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d3c001380 con 0x7f1d4006cf50 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907850554 0 0) 0x7f1d40130000 con 0x7f1d4006d480 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d240035d0 con 0x7f1d4006d480 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d34001970 con 0x7f1d4006d480 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.385+0000 7f1d31ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.402 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3074126047 0 0) 0x7f1d4012c680 con 0x7f1d40132330 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d40130000 con 0x7f1d40132330 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d38001870 con 0x7f1d40132330 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4094508631 0 0) 0x7f1d4009e8b0 con 0x7f1d4006cf50 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6790/0 conn(0x7f1d4006d480 legacy=0x7f1d401ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.189:6789/0 conn(0x7f1d40132330 legacy=0x7f1d401d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 monclient: found mon.a 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4012f9b0 con 0x7f1d4006cf50 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 5 monclient: authenticate success, global_id 184143 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 10 monclient: _renew_subs 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4012fc90 con 0x7f1d4006cf50 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 10 monclient: _renew_subs 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4012e870 con 0x7f1d4006cf50 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d3c004ec0 con 0x7f1d4006cf50 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d3c005250 con 0x7f1d4006cf50 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d337fe700 4 set_mon_vals no callback set 2024-09-16T06:45:35.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: dump: 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1d20006590 con 0x7f1d20002830 2024-09-16T06:45:35.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1d3c0053b0 con 0x7f1d4006cf50 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7f1d3c045030 con 0x7f1d4006cf50 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d45bc1700 10 monclient: get_auth_request con 0x7f1d24036500 auth_method 0 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557298752 0 0) 0x7f1d20006590 con 0x7f1d20002830 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient(hunting): my global_id is 184146 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4009e8b0 con 0x7f1d20002830 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2609267515 0 0) 0x7f1d4009e8b0 con 0x7f1d20002830 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d20006590 con 0x7f1d20002830 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d34003b90 con 0x7f1d20002830 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: dump: 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3926744263 0 0) 0x7f1d20006590 con 0x7f1d20002830 2024-09-16T06:45:35.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d2000a190 con 0x7f1d20002830 2024-09-16T06:45:35.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1d34003f90 con 0x7f1d20002830 2024-09-16T06:45:35.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:35.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6789/0 conn(0x7f1d20002830 legacy=0x7f1d20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.389+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6789/0 conn(0x7f1d20002830 legacy=0x7f1d20004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d463c2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1d20009e30 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798088437 0 0) 0x7f1d20009e30 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient(hunting): my global_id is 184155 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20006590 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 386347386 0 0) 0x7f1d20006590 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d20009e30 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1d34004510 con 0x7f1d20009150 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient: dump: 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.414 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3639391375 0 0) 0x7f1d20009e30 con 0x7f1d20009150 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1d2000c340 con 0x7f1d20009150 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3198629127) 0x7f1d34005970 con 0x7f1d20009150 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.401+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6789/0 conn(0x7f1d20009150 legacy=0x7f1d200095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d31ffb700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6789/0 conn(0x7f1d20009150 legacy=0x7f1d200095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.415 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.416 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:35.417 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:35.418 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:35.419 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.421 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.423 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.424 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:35.426 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1d24036500 legacy=0x7f1d240389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 10 monclient: shutdown 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 >> v1:172.21.15.76:6789/0 conn(0x7f1d4006cf50 legacy=0x7f1d40129180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d45bc1700 1 -- 172.21.15.76:0/1305168922 reap_dead start 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 shutdown_connections 2024-09-16T06:45:35.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 >> 172.21.15.76:0/1305168922 conn(0x7f1d400a41c0 msgr2=0x7f1d4009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 shutdown_connections 2024-09-16T06:45:35.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.405+0000 7f1d463c2700 1 -- 172.21.15.76:0/1305168922 wait complete. 2024-09-16T06:45:35.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:35.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:45:35.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: monmap: 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:35.396795+0000 2024-09-16T06:45:35.435 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:35.396795+0000 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 1 Processor -- start 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 1 -- start start 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: init 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-b con 0x7f2cc409e6d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-c con 0x7f2cc409f9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-a con 0x7f2cc409f1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc409ff50 con 0x7f2cc409f1c0 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc409b8f0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc409e030 con 0x7f2cc409e6d0 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 10 monclient(hunting): authenticate will time out at 1287.910946s 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2cc409f9d0 0x7f2cc4136aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58014/0 (socket says 172.21.15.189:58014) 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbbfff700 1 -- 172.21.15.189:0/1120409494 learned_addr learned my addr 172.21.15.189:0/1120409494 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074601150 0 0) 0x7f2cc409b8f0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): my global_id is 174466 2024-09-16T06:45:35.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac0035d0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965887067 0 0) 0x7f2cc409e030 con 0x7f2cc409e6d0 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): my global_id is 125792 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc409b8f0 con 0x7f2cc409e6d0 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001550148 0 0) 0x7f2cc409ff50 con 0x7f2cc409f1c0 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient(hunting): my global_id is 184149 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc409e030 con 0x7f2cc409f1c0 2024-09-16T06:45:35.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116933651 0 0) 0x7f2cac0035d0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 >> v1:172.21.15.76:6790/0 conn(0x7f2cc409e6d0 legacy=0x7f2cc409eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 >> v1:172.21.15.76:6789/0 conn(0x7f2cc409f1c0 legacy=0x7f2cc413a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 monclient: found mon.noname-c 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40a1150 con 0x7f2cc409f9d0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 5 monclient: authenticate success, global_id 174466 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2ccac18700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cb4001610 con 0x7f2cc409f9d0 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.395+0000 7f2cbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: dump: 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb40018f0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 1 -- 172.21.15.189:0/1120409494 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cb4001c80 con 0x7f2cc409f9d0 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 4 set_mon_vals no callback set 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbb7fe700 10 monclient: dump: 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.441 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: shutdown 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/1120409494 >> v1:172.21.15.189:6789/0 conn(0x7f2cc409f9d0 legacy=0x7f2cc4136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/1120409494 shutdown_connections 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/1120409494 >> 172.21.15.189:0/1120409494 conn(0x7f2cc40a4410 msgr2=0x7f2cc40a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/1120409494 shutdown_connections 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/1120409494 wait complete. 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: monmap: 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:35.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 Processor -- start 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- start start 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: init 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:35.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-a con 0x7f2cc409e6d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-b con 0x7f2cc409f1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient(hunting): picked mon.noname-c con 0x7f2cc409f9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4131c70 con 0x7f2cc409e6d0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40a26b0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4070220 con 0x7f2cc409f9d0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient(hunting): authenticate will time out at 1287.914554s 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2cc409f1c0 0x7f2cc406cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:58024/0 (socket says 172.21.15.189:58024) 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cbbfff700 1 -- 172.21.15.189:0/2990554818 learned_addr learned my addr 172.21.15.189:0/2990554818 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185851093 0 0) 0x7f2cc4070220 con 0x7f2cc409f9d0 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): my global_id is 125795 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb00035d0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497395281 0 0) 0x7f2cc40a26b0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): my global_id is 174469 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc4070220 con 0x7f2cc409f1c0 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196669171 0 0) 0x7f2cc4131c70 con 0x7f2cc409e6d0 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): my global_id is 184152 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc40a26b0 con 0x7f2cc409e6d0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743172148 0 0) 0x7f2cb00035d0 con 0x7f2cc409f9d0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc4131c70 con 0x7f2cc409f9d0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715901135 0 0) 0x7f2cc4070220 con 0x7f2cc409f1c0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb00035d0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cbc001c30 con 0x7f2cc409f9d0 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): dump: 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.446 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cb40015d0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): dump: 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2064394718 0 0) 0x7f2cc40a26b0 con 0x7f2cc409e6d0 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc4070220 con 0x7f2cc409e6d0 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cc00015e0 con 0x7f2cc409e6d0 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient(hunting): dump: 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 633715138 0 0) 0x7f2cb00035d0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 >> v1:172.21.15.76:6790/0 conn(0x7f2cc409f9d0 legacy=0x7f2cc41d6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 >> v1:172.21.15.76:6789/0 conn(0x7f2cc409e6d0 legacy=0x7f2cc406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 monclient: found mon.b 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4070500 con 0x7f2cc409f1c0 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2cb97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 5 monclient: authenticate success, global_id 174469 2024-09-16T06:45:35.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: _renew_subs 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.399+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc406d4e0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 10 monclient: _renew_subs 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40a22e0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb4004ca0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cb4004fb0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 10 monclient: dump: 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:35.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cba7fc700 4 set_mon_vals no callback set 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc406f040 con 0x7f2cc409f1c0 2024-09-16T06:45:35.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2cb4005110 con 0x7f2cc409f1c0 2024-09-16T06:45:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7f2cb4044db0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.403+0000 7f2cc89b4700 10 monclient: get_auth_request con 0x7f2cb00365a0 auth_method 0 2024-09-16T06:45:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.407+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2cb4014800 con 0x7f2cc409f1c0 2024-09-16T06:45:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.407+0000 7f2cb97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.407+0000 7f2cb97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: monmap: 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:35.556304+0000 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:35.556304+0000 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.554 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 1 Processor -- start 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 1 -- start start 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: init 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-a con 0x7f42d40964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-c con 0x7f42d409b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-b con 0x7f42d409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d4098560 con 0x7f42d40964f0 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d4099ec0 con 0x7f42d409b1c0 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d409b5f0 con 0x7f42d409ed60 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient(hunting): authenticate will time out at 1282.690068s 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d359e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f42d409ed60 0x7f42d413af90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35158/0 (socket says 172.21.15.76:35158) 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d359e700 1 -- 172.21.15.76:0/4160106631 learned_addr learned my addr 172.21.15.76:0/4160106631 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172161743 0 0) 0x7f42d409b5f0 con 0x7f42d409ed60 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient(hunting): my global_id is 125798 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c00057e0 con 0x7f42d409ed60 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675776474 0 0) 0x7f42c00057e0 con 0x7f42d409ed60 2024-09-16T06:45:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 >> v1:172.21.15.189:6789/0 conn(0x7f42d409b1c0 legacy=0x7f42d4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 >> v1:172.21.15.76:6789/0 conn(0x7f42d40964f0 legacy=0x7f42d40968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 monclient: found mon.noname-b 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4099be0 con 0x7f42d409ed60 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42cc001990 con 0x7f42d409ed60 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: dump: 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 5 monclient: authenticate success, global_id 125798 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42cc001cb0 con 0x7f42d409ed60 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 1 -- 172.21.15.76:0/4160106631 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42cc002080 con 0x7f42d409ed60 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d899a700 4 set_mon_vals no callback set 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.553+0000 7f42d259c700 10 monclient: dump: 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.568 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: shutdown 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- 172.21.15.76:0/4160106631 >> v1:172.21.15.76:6790/0 conn(0x7f42d409ed60 legacy=0x7f42d413af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- 172.21.15.76:0/4160106631 shutdown_connections 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- 172.21.15.76:0/4160106631 >> 172.21.15.76:0/4160106631 conn(0x7f42d40a0030 msgr2=0x7f42d40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- 172.21.15.76:0/4160106631 shutdown_connections 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- 172.21.15.76:0/4160106631 wait complete. 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: monmap: 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 Processor -- start 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- start start 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: init 2024-09-16T06:45:35.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-a con 0x7f42d40964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-b con 0x7f42d409b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient(hunting): picked mon.noname-c con 0x7f42d409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d408fd20 con 0x7f42d40964f0 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d41c64d0 con 0x7f42d409b1c0 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d408fa40 con 0x7f42d409ed60 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d899a700 10 monclient(hunting): authenticate will time out at 1282.694847s 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d359e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f42d409ed60 0x7f42d41ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35174/0 (socket says 172.21.15.76:35174) 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d2d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f42d40964f0 0x7f42d4095550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42990/0 (socket says 172.21.15.76:42990) 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42d359e700 1 -- 172.21.15.76:0/4280390076 learned_addr learned my addr 172.21.15.76:0/4280390076 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134879217 0 0) 0x7f42d408fa40 con 0x7f42d409ed60 2024-09-16T06:45:35.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): my global_id is 125801 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b40035d0 con 0x7f42d409ed60 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62888378 0 0) 0x7f42d41c64d0 con 0x7f42d409b1c0 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): my global_id is 174475 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d408fa40 con 0x7f42d409b1c0 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115001092 0 0) 0x7f42d408fd20 con 0x7f42d40964f0 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): my global_id is 184161 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d41c64d0 con 0x7f42d40964f0 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340225861 0 0) 0x7f42b40035d0 con 0x7f42d409ed60 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42d408fd20 con 0x7f42d409ed60 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42cc001950 con 0x7f42d409ed60 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): dump: 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.557+0000 7f42cbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2101700082 0 0) 0x7f42d408fd20 con 0x7f42d409ed60 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.189:6789/0 conn(0x7f42d409b1c0 legacy=0x7f42d41ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6789/0 conn(0x7f42d40964f0 legacy=0x7f42d4095550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 monclient: found mon.c 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d408f760 con 0x7f42d409ed60 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 5 monclient: authenticate success, global_id 125801 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 10 monclient: _renew_subs 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d408e2f0 con 0x7f42d409ed60 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 10 monclient: _renew_subs 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d408df40 con 0x7f42d409ed60 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42cc001530 con 0x7f42d409ed60 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42cc0051c0 con 0x7f42d409ed60 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d1d9b700 4 set_mon_vals no callback set 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: dump: 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f42cc005320 con 0x7f42d409ed60 2024-09-16T06:45:35.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f42b8006590 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7f42cc045130 con 0x7f42d409ed60 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42d2d9d700 10 monclient: get_auth_request con 0x7f42b4036110 auth_method 0 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191632923 0 0) 0x7f42b8006590 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient(hunting): my global_id is 125804 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d408fd20 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1540349024 0 0) 0x7f42d408fd20 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b8006590 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42bc0017f0 con 0x7f42b8002830 2024-09-16T06:45:35.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: dump: 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 774340022 0 0) 0x7f42b8006590 con 0x7f42b8002830 2024-09-16T06:45:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42b800a190 con 0x7f42b8002830 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f42bc001ca0 con 0x7f42b8002830 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6790/0 conn(0x7f42b8002830 legacy=0x7f42b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.561+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6790/0 conn(0x7f42b8002830 legacy=0x7f42b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f42b8009e10 con 0x7f42b8009150 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215337343 0 0) 0x7f42b8009e10 con 0x7f42b8009150 2024-09-16T06:45:35.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient(hunting): my global_id is 125807 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b8006590 con 0x7f42b8009150 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2188100878 0 0) 0x7f42b8006590 con 0x7f42b8009150 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b8009e10 con 0x7f42b8009150 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f42bc003b10 con 0x7f42b8009150 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient: dump: 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1449533456 0 0) 0x7f42b8009e10 con 0x7f42b8009150 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f42b800c340 con 0x7f42b8009150 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 2039381842) 0x7f42bc003930 con 0x7f42b8009150 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6790/0 conn(0x7f42b8009150 legacy=0x7f42b80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42cbfff700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6790/0 conn(0x7f42b8009150 legacy=0x7f42b80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.584 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.585 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.587 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.588 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.589 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.590 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.591 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.592 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:35.593 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:35.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.189:6812/3098810674 conn(0x7f42b4036110 legacy=0x7f42b40385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 10 monclient: shutdown 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 >> v1:172.21.15.76:6790/0 conn(0x7f42d409ed60 legacy=0x7f42d41ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d359e700 1 -- 172.21.15.76:0/4280390076 reap_dead start 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 shutdown_connections 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 >> 172.21.15.76:0/4280390076 conn(0x7f42d40a0030 msgr2=0x7f42d40a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 shutdown_connections 2024-09-16T06:45:35.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.573+0000 7f42d899a700 1 -- 172.21.15.76:0/4280390076 wait complete. 2024-09-16T06:45:35.595 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:35.595 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:45:35.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:45:35.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.583+0000 7f2ccac18700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.583+0000 7f2ccac18700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:35.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.583+0000 7f2ccac18700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:35.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:35.583+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2cc4066ed0 con 0x7f2cc409f1c0 2024-09-16T06:45:35.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: monmap: 2024-09-16T06:45:35.717 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.717 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:35.719989+0000 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:35.719989+0000 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.718 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 1 Processor -- start 2024-09-16T06:45:35.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 1 -- start start 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: init 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-c con 0x7f7130132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-a con 0x7f713006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-b con 0x7f713006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713006c280 con 0x7f713006d480 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713006bfa0 con 0x7f7130132330 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713009ace0 con 0x7f713006cf50 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient(hunting): authenticate will time out at 1282.854356s 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f713006d480 0x7f713013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43000/0 (socket says 172.21.15.76:43000) 2024-09-16T06:45:35.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712f7fe700 1 -- 172.21.15.76:0/3821207635 learned_addr learned my addr 172.21.15.76:0/3821207635 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524894919 0 0) 0x7f713009ace0 con 0x7f713006cf50 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): my global_id is 125810 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71180035d0 con 0x7f713006cf50 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939425068 0 0) 0x7f713006bfa0 con 0x7f7130132330 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): my global_id is 174478 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713009ace0 con 0x7f7130132330 2024-09-16T06:45:35.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517115126 0 0) 0x7f713006c280 con 0x7f713006d480 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient(hunting): my global_id is 184164 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713006bfa0 con 0x7f713006d480 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662461065 0 0) 0x7f71180035d0 con 0x7f713006cf50 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 >> v1:172.21.15.189:6789/0 conn(0x7f7130132330 legacy=0x7f7130132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 >> v1:172.21.15.76:6789/0 conn(0x7f713006d480 legacy=0x7f713013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 monclient: found mon.noname-b 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713009ae70 con 0x7f713006cf50 2024-09-16T06:45:35.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7128001990 con 0x7f713006cf50 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: dump: 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 5 monclient: authenticate success, global_id 125810 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7128001c70 con 0x7f713006cf50 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 1 -- 172.21.15.76:0/3821207635 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7128001e60 con 0x7f713006cf50 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f71350e2700 4 set_mon_vals no callback set 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.717+0000 7f712effd700 10 monclient: dump: 2024-09-16T06:45:35.725 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.726 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: shutdown 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/3821207635 >> v1:172.21.15.76:6790/0 conn(0x7f713006cf50 legacy=0x7f713013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/3821207635 shutdown_connections 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/3821207635 >> 172.21.15.76:0/3821207635 conn(0x7f71300a41c0 msgr2=0x7f71300a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/3821207635 shutdown_connections 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/3821207635 wait complete. 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: monmap: 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:35.732 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 Processor -- start 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- start start 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: init 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-a con 0x7f713006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-b con 0x7f713006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient(hunting): picked mon.noname-c con 0x7f7130132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713013e060 con 0x7f713006cf50 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71301393b0 con 0x7f713006d480 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7130136220 con 0x7f7130132330 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient(hunting): authenticate will time out at 1282.857729s 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f713006cf50 0x7f7130135400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43002/0 (socket says 172.21.15.76:43002) 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712ffff700 1 -- 172.21.15.76:0/2519971893 learned_addr learned my addr 172.21.15.76:0/2519971893 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693534851 0 0) 0x7f713013e060 con 0x7f713006cf50 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): my global_id is 184167 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c0035d0 con 0x7f713006cf50 2024-09-16T06:45:35.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927053151 0 0) 0x7f7130136220 con 0x7f7130132330 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): my global_id is 125813 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713013e060 con 0x7f7130132330 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032270971 0 0) 0x7f713013e060 con 0x7f7130132330 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7130136220 con 0x7f7130132330 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f71280013a0 con 0x7f7130132330 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): dump: 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489416611 0 0) 0x7f71301393b0 con 0x7f713006d480 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): my global_id is 174481 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713013e060 con 0x7f713006d480 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3169725941 0 0) 0x7f711c0035d0 con 0x7f713006cf50 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71301393b0 con 0x7f713006cf50 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7120001870 con 0x7f713006cf50 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient(hunting): dump: 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2053068190 0 0) 0x7f7130136220 con 0x7f7130132330 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6789/0 conn(0x7f713006d480 legacy=0x7f7130135b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.76:6789/0 conn(0x7f713006cf50 legacy=0x7f7130135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 monclient: found mon.c 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130131710 con 0x7f7130132330 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 5 monclient: authenticate success, global_id 125813 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _renew_subs 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71301311e0 con 0x7f7130132330 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _renew_subs 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71301390d0 con 0x7f7130132330 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7128003cd0 con 0x7f7130132330 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7128004060 con 0x7f7130132330 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712e7fc700 4 set_mon_vals no callback set 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.721+0000 7f712cff9700 10 monclient: dump: 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7128004240 con 0x7f7130132330 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7f7128043ef0 con 0x7f7130132330 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712ffff700 10 monclient: get_auth_request con 0x7f711c036490 auth_method 0 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f71350e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7110006590 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328883819 0 0) 0x7f7110006590 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient(hunting): my global_id is 174484 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7130136220 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4266301850 0 0) 0x7f7130136220 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7110006590 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7124001740 con 0x7f7110002830 2024-09-16T06:45:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 10 monclient: dump: 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2039055684 0 0) 0x7f7110006590 con 0x7f7110002830 2024-09-16T06:45:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.725+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f711000a190 con 0x7f7110002830 2024-09-16T06:45:35.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.729+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7124001b10 con 0x7f7110002830 2024-09-16T06:45:35.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.729+0000 7f712cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.729+0000 7f712cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:35.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.729+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6789/0 conn(0x7f7110002830 legacy=0x7f7110004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.729+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6789/0 conn(0x7f7110002830 legacy=0x7f7110004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f71350e2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:35.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7110009e30 con 0x7f7110009150 2024-09-16T06:45:35.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472674631 0 0) 0x7f7110009e30 con 0x7f7110009150 2024-09-16T06:45:35.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 10 monclient(hunting): my global_id is 174487 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.737+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7110006590 con 0x7f7110009150 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 531616876 0 0) 0x7f7110006590 con 0x7f7110009150 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7110009e30 con 0x7f7110009150 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7124004ab0 con 0x7f7110009150 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 10 monclient: dump: 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.745 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1769839437 0 0) 0x7f7110009e30 con 0x7f7110009150 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f711000c340 con 0x7f7110009150 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2599312739) 0x7f7124004de0 con 0x7f7110009150 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6789/0 conn(0x7f7110009150 legacy=0x7f71100095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f712cff9700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6789/0 conn(0x7f7110009150 legacy=0x7f71100095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 66, 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:45:35.747 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.748 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.749 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:35.750 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:35.751 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:35.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:35.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:35.758 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.189:6812/3098810674 conn(0x7f711c036490 legacy=0x7f711c038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 10 monclient: shutdown 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 >> v1:172.21.15.76:6790/0 conn(0x7f7130132330 legacy=0x7f71301d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71348e1700 1 -- 172.21.15.76:0/2519971893 reap_dead start 2024-09-16T06:45:35.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71348e1700 1 -- 172.21.15.76:0/2519971893 reap_dead start 2024-09-16T06:45:35.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 shutdown_connections 2024-09-16T06:45:35.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 >> 172.21.15.76:0/2519971893 conn(0x7f71300a41c0 msgr2=0x7f71300a6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 shutdown_connections 2024-09-16T06:45:35.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.741+0000 7f71350e2700 1 -- 172.21.15.76:0/2519971893 wait complete. 2024-09-16T06:45:35.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:45:35.761 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:45:35.761 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:45:35.761 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:45:35.761 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:45:35.761 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:45:35.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:45:35.804 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:45:35.805 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:45:35.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:35.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: get_monmap_and_config 2024-09-16T06:45:35.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: monmap: 2024-09-16T06:45:35.922 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:35.925031+0000 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:35.925031+0000 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:35.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 Processor -- start 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 -- start start 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: init 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:35.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient(hunting): picked mon.noname-c con 0x7fe91009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient(hunting): picked mon.noname-a con 0x7fe91009b6a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient(hunting): picked mon.noname-b con 0x7fe910138030 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91009f610 con 0x7fe91009b6a0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91009bfb0 con 0x7fe91009f230 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91009a880 con 0x7fe910138030 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient(hunting): authenticate will time out at 1283.058372s 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe916bb5700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe910138030 legacy=0x7fe91013a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe915bb3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe91009b6a0 0x7fe91009a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43032/0 (socket says 172.21.15.76:43032) 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe915bb3700 1 -- 172.21.15.76:0/3663833254 learned_addr learned my addr 172.21.15.76:0/3663833254 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436424907 0 0) 0x7fe91009f610 con 0x7fe91009b6a0 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient(hunting): my global_id is 184170 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc0035d0 con 0x7fe91009b6a0 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581026572 0 0) 0x7fe8fc0035d0 con 0x7fe91009b6a0 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 >> v1:172.21.15.76:6790/0 conn(0x7fe910138030 legacy=0x7fe91013a420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 >> v1:172.21.15.189:6789/0 conn(0x7fe91009f230 legacy=0x7fe910099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 monclient: found mon.noname-a 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91009d3c0 con 0x7fe91009b6a0 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 5 monclient: authenticate success, global_id 184170 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe90c001610 con 0x7fe91009b6a0 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: dump: 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.929 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe90c001930 con 0x7fe91009b6a0 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 1 -- 172.21.15.76:0/3663833254 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe90c001cc0 con 0x7fe91009b6a0 2024-09-16T06:45:35.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 4 set_mon_vals no callback set 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe9153b2700 10 monclient: dump: 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 10 monclient: shutdown 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.921+0000 7fe918618700 1 -- 172.21.15.76:0/3663833254 >> v1:172.21.15.76:6789/0 conn(0x7fe91009b6a0 legacy=0x7fe91009a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/3663833254 shutdown_connections 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/3663833254 >> 172.21.15.76:0/3663833254 conn(0x7fe9100a3e50 msgr2=0x7fe9100a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/3663833254 shutdown_connections 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/3663833254 wait complete. 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: build_initial_monmap 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: monmap: 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 Processor -- start 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- start start 2024-09-16T06:45:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: init 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient(hunting): picked mon.noname-c con 0x7fe91009b6a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient(hunting): picked mon.noname-a con 0x7fe91009f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient(hunting): picked mon.noname-b con 0x7fe910138030 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe910123ec0 con 0x7fe91009f230 2024-09-16T06:45:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100a1e80 con 0x7fe910138030 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe910135320 con 0x7fe91009b6a0 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient(hunting): authenticate will time out at 1283.061752s 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe9163b4700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe91009b6a0 legacy=0x7fe9101344e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe915bb3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe91009f230 0x7fe9101cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43040/0 (socket says 172.21.15.76:43040) 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe915bb3700 1 -- 172.21.15.76:0/2607501788 learned_addr learned my addr 172.21.15.76:0/2607501788 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156060335 0 0) 0x7fe910123ec0 con 0x7fe91009f230 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): my global_id is 184173 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f40035d0 con 0x7fe91009f230 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478169504 0 0) 0x7fe9100a1e80 con 0x7fe910138030 2024-09-16T06:45:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): my global_id is 174493 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe910123ec0 con 0x7fe910138030 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3046539660 0 0) 0x7fe8f40035d0 con 0x7fe91009f230 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9100a1e80 con 0x7fe91009f230 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe90c0015d0 con 0x7fe91009f230 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012335811 0 0) 0x7fe910123ec0 con 0x7fe910138030 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f40035d0 con 0x7fe910138030 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe9080043e0 con 0x7fe910138030 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4274138610 0 0) 0x7fe9100a1e80 con 0x7fe91009f230 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 >> v1:172.21.15.76:6790/0 conn(0x7fe91009b6a0 legacy=0x7fe9101344e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 >> v1:172.21.15.189:6789/0 conn(0x7fe910138030 legacy=0x7fe9101d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 monclient: found mon.a 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe910123c50 con 0x7fe91009f230 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 5 monclient: authenticate success, global_id 184173 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _renew_subs 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9100724b0 con 0x7fe91009f230 2024-09-16T06:45:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _renew_subs 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe910134bf0 con 0x7fe91009f230 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe90c004ca0 con 0x7fe91009f230 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe90c0010a0 con 0x7fe91009f230 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe914bb1700 4 set_mon_vals no callback set 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.925+0000 7fe906ffd700 10 monclient: dump: 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe918618700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe918618700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe918618700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe90c00f720 con 0x7fe91009f230 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8005100 con 0x7fe91009f230 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7fe90c044c30 con 0x7fe91009f230 2024-09-16T06:45:35.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.929+0000 7fe9163b4700 10 monclient: get_auth_request con 0x7fe8f40362c0 auth_method 0 2024-09-16T06:45:35.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.937+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe90c014680 con 0x7fe91009f230 2024-09-16T06:45:35.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.937+0000 7fe906ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:35.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:35.937+0000 7fe906ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe8f8005930 con 0x7fe91009f230 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe906ffd700 1 -- 172.21.15.76:0/2607501788 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2116920046) 0x7fe90c012d20 con 0x7fe91009f230 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe906ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe906ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:36.179 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe8f40362c0 legacy=0x7fe8f4038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 10 monclient: shutdown 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 >> v1:172.21.15.76:6789/0 conn(0x7fe91009f230 legacy=0x7fe9101cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 shutdown_connections 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 >> 172.21.15.76:0/2607501788 conn(0x7fe9100a3e50 msgr2=0x7fe910137870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 shutdown_connections 2024-09-16T06:45:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:36.177+0000 7fe918618700 1 -- 172.21.15.76:0/2607501788 wait complete. 2024-09-16T06:45:38.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:38.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:38.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:38.399+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:38.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:38.399+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:38.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:38.399+0000 7f2cabfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:39.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: get_monmap_and_config 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: build_initial_monmap 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: monmap: 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:39.316373+0000 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:39.316373+0000 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 Processor -- start 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- start start 2024-09-16T06:45:39.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: init 2024-09-16T06:45:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-b con 0x7fc818097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-c con 0x7fc818099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-a con 0x7fc818139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818098990 con 0x7fc818139500 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc818097f40 con 0x7fc818099090 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81809d610 con 0x7fc818097b60 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient(hunting): authenticate will time out at 1286.449626s 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81cd88700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fc818097b60 legacy=0x7fc81809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81d589700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc818139500 0x7fc81813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43054/0 (socket says 172.21.15.76:43054) 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81d589700 1 -- 172.21.15.76:0/3959931896 learned_addr learned my addr 172.21.15.76:0/3959931896 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702843512 0 0) 0x7fc818098990 con 0x7fc818139500 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): my global_id is 184176 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f80035d0 con 0x7fc818139500 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016007993 0 0) 0x7fc818097f40 con 0x7fc818099090 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:39.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient(hunting): my global_id is 174496 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818098990 con 0x7fc818099090 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724542754 0 0) 0x7fc7f80035d0 con 0x7fc818139500 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 >> v1:172.21.15.76:6790/0 conn(0x7fc818097b60 legacy=0x7fc81809ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 >> v1:172.21.15.189:6789/0 conn(0x7fc818099090 legacy=0x7fc81809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 monclient: found mon.noname-a 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180986b0 con 0x7fc818139500 2024-09-16T06:45:39.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 5 monclient: authenticate success, global_id 184176 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc810001990 con 0x7fc818139500 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: dump: 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:39.321 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc810001c70 con 0x7fc818139500 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 1 -- 172.21.15.76:0/3959931896 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc810002000 con 0x7fc818139500 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 4 set_mon_vals no callback set 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc8177fe700 10 monclient: dump: 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:39.322 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 10 monclient: shutdown 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3959931896 >> v1:172.21.15.76:6789/0 conn(0x7fc818139500 legacy=0x7fc81813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3959931896 shutdown_connections 2024-09-16T06:45:39.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3959931896 >> 172.21.15.76:0/3959931896 conn(0x7fc8180a0030 msgr2=0x7fc8180a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3959931896 shutdown_connections 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.313+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3959931896 wait complete. 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: build_initial_monmap 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: monmap: 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:39.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 Processor -- start 2024-09-16T06:45:39.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- start start 2024-09-16T06:45:39.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: init 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-b con 0x7fc818097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-a con 0x7fc818099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient(hunting): picked mon.noname-c con 0x7fc818139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81806dff0 con 0x7fc818099090 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81806d2d0 con 0x7fc818097b60 2024-09-16T06:45:39.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8180a3040 con 0x7fc818139500 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient(hunting): authenticate will time out at 1286.453146s 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81d589700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fc818139500 legacy=0x7fc8181cfef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc817fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc818099090 0x7fc81806cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43062/0 (socket says 172.21.15.76:43062) 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc817fff700 1 -- 172.21.15.76:0/3942372534 learned_addr learned my addr 172.21.15.76:0/3942372534 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683767541 0 0) 0x7fc81806dff0 con 0x7fc818099090 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): my global_id is 184179 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc0035d0 con 0x7fc818099090 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467220705 0 0) 0x7fc81806d2d0 con 0x7fc818097b60 2024-09-16T06:45:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _negotiate 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): my global_id is 174499 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc81806dff0 con 0x7fc818097b60 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2440755407 0 0) 0x7fc7fc0035d0 con 0x7fc818099090 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc81806d2d0 con 0x7fc818099090 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc800001910 con 0x7fc818099090 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): dump: 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2982946064 0 0) 0x7fc81806d2d0 con 0x7fc818099090 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 >> v1:172.21.15.76:6790/0 conn(0x7fc818139500 legacy=0x7fc8181cfef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 >> v1:172.21.15.189:6789/0 conn(0x7fc818097b60 legacy=0x7fc81806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: _finish_hunting 0 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 monclient: found mon.a 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180a2b10 con 0x7fc818099090 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: _finish_auth 0 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: _check_auth_tickets 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 5 monclient: authenticate success, global_id 184179 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _renew_subs 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81809a5a0 con 0x7fc818099090 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _renew_subs 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc818070070 con 0x7fc818099090 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc800001ca0 con 0x7fc818099090 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc800003b10 con 0x7fc818099090 2024-09-16T06:45:39.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc816ffd700 4 set_mon_vals no callback set 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.317+0000 7fc8157fa700 10 monclient: dump: 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:39.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc800003c70 con 0x7fc818099090 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc81dd8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc81dd8a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc81dd8a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f0005100 con 0x7fc818099090 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7fc800044c40 con 0x7fc818099090 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.321+0000 7fc81cd88700 10 monclient: get_auth_request con 0x7fc7fc035ff0 auth_method 0 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.329+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc800014800 con 0x7fc818099090 2024-09-16T06:45:39.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.329+0000 7fc8157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:39.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.329+0000 7fc8157fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:39.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:39.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:39.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:39.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:39.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:39.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.556+0000 7fc81dd8a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:39.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.556+0000 7fc81dd8a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:39.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.556+0000 7fc81dd8a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:39.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.556+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc7f0005930 con 0x7fc818099090 2024-09-16T06:45:39.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc8157fa700 1 -- 172.21.15.76:0/3942372534 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 784428239) 0x7fc80001cdf0 con 0x7fc818099090 2024-09-16T06:45:39.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc8157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:39.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc8157fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:39.562 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:39.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7fc035ff0 legacy=0x7fc7fc0384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:39.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 10 monclient: shutdown 2024-09-16T06:45:39.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:39.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 >> v1:172.21.15.76:6789/0 conn(0x7fc818099090 legacy=0x7fc81806cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:39.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 shutdown_connections 2024-09-16T06:45:39.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 >> 172.21.15.76:0/3942372534 conn(0x7fc8180a0030 msgr2=0x7fc8180a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:39.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 shutdown_connections 2024-09-16T06:45:39.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:39.560+0000 7fc81dd8a700 1 -- 172.21.15.76:0/3942372534 wait complete. 2024-09-16T06:45:40.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:40.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:40.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:40.399+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:40.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:40.399+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:41.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:41.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:41.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:41.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:41.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:41.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:42.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:42.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:42.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:42.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:42.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:42.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:42.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:45:42.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: get_monmap_and_config 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: build_initial_monmap 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: monmap: 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:42.697509+0000 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:42.697509+0000 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:42.695 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:42.696 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 1 Processor -- start 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 1 -- start start 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: init 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:42.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-c con 0x7f53480973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-a con 0x7f5348097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-b con 0x7f53481400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348099350 con 0x7f5348097eb0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534809f690 con 0x7f53480973c0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348099bf0 con 0x7f53481400d0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534d2fc700 10 monclient(hunting): authenticate will time out at 1289.830919s 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f534cafb700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f53481424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f53477fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5348097eb0 0x7f534813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43070/0 (socket says 172.21.15.76:43070) 2024-09-16T06:45:42.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.692+0000 7f53477fe700 1 -- 172.21.15.76:0/2082968055 learned_addr learned my addr 172.21.15.76:0/2082968055 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:42.896+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f53481424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:43.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:43.300+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f53481424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:43.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:43.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:43.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:43.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:43.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:43.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:44.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:44.104+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f53481424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:44.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:44.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:44.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:44.399+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:44.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:44.399+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:45.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:45.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:45.399+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:45.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:45.399+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:45.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): tick 2024-09-16T06:45:45.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f53481424c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.189:6789/0 conn(0x7f53480973c0 legacy=0x7f53480977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6789/0 conn(0x7f5348097eb0 legacy=0x7f534813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f53481400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f5334006f00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f533400ac30 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348098360 con 0x7f5334006f00 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348099bf0 con 0x7f533400ac30 2024-09-16T06:45:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5334010970 con 0x7f53481400d0 2024-09-16T06:45:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5345ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.692+0000 7f5347fff700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:45.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:45.896+0000 7f5347fff700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:46.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:46.300+0000 7f5347fff700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:46.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:46.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:46.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:46.399+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:46.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:46.399+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:47.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:47.104+0000 7f5347fff700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:47.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:47.399+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:47.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:47.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:47.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:47.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:48.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:48.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:48.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:48.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:48.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:48.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:48.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:48.403+0000 7f2cabfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): tick 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.189:6789/0 conn(0x7f533400ac30 legacy=0x7f533400d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6789/0 conn(0x7f5334006f00 legacy=0x7f5334009390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f5334010c10 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f5334012e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:48.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f53481400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:48.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5334011b80 con 0x7f5334012e00 2024-09-16T06:45:48.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5334010970 con 0x7f5334010c10 2024-09-16T06:45:48.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400b0b0 con 0x7f53481400d0 2024-09-16T06:45:48.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f5345ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:48.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.692+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:48.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:48.896+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:49.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:49.300+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:49.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:49.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:49.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:49.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:49.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:50.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:50.104+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:50.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:50.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:50.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:50.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:51.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:51.403+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:51.403+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:51.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:51.403+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): tick 2024-09-16T06:45:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.189:6789/0 conn(0x7f5334010c10 legacy=0x7f5334010ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6789/0 conn(0x7f5334012e00 legacy=0x7f53340151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f533400d8d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f533400dd50 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f53481400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400c250 con 0x7f533400d8d0 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400b0b0 con 0x7f533400dd50 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400d5f0 con 0x7f53481400d0 2024-09-16T06:45:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.692+0000 7f5345ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.696+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:51.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:51.900+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:52.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:52.304+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:52.402+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:52.402+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:52.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:52.402+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:53.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:53.108+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:53.402+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:53.402+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:53.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:53.402+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:54.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:54.402+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:54.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:54.402+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:54.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:54.402+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): tick 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.189:6789/0 conn(0x7f533400dd50 legacy=0x7f533400e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6789/0 conn(0x7f533400d8d0 legacy=0x7f5334009310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f5334011200 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f533400a310 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f53481400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5334010ff0 con 0x7f5334011200 2024-09-16T06:45:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400d5f0 con 0x7f533400a310 2024-09-16T06:45:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533400a910 con 0x7f53481400d0 2024-09-16T06:45:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f5345ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.696+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:54.900+0000 7f534cafb700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368393886 0 0) 0x7f5334010ff0 con 0x7f5334011200 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): my global_id is 184182 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53300035d0 con 0x7f5334011200 2024-09-16T06:45:55.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968295976 0 0) 0x7f533400d5f0 con 0x7f533400a310 2024-09-16T06:45:55.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient(hunting): my global_id is 174502 2024-09-16T06:45:55.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5334010ff0 con 0x7f533400a310 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395325524 0 0) 0x7f5334010ff0 con 0x7f533400a310 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6790/0 conn(0x7f53481400d0 legacy=0x7f5334005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.76:6789/0 conn(0x7f5334011200 legacy=0x7f5334009c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 monclient: found mon.noname-c 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533400e130 con 0x7f533400a310 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5338001910 con 0x7f533400a310 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f534d2fc700 5 monclient: authenticate success, global_id 174502 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f534d2fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f5346ffd700 10 monclient: dump: 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.232+0000 7f534d2fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5338001b90 con 0x7f533400a310 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 4 set_mon_vals no callback set 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 1 -- 172.21.15.76:0/2082968055 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5338001f20 con 0x7f533400a310 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f5346ffd700 10 monclient: dump: 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.241 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 10 monclient: shutdown 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- 172.21.15.76:0/2082968055 >> v1:172.21.15.189:6789/0 conn(0x7f533400a310 legacy=0x7f53340151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- 172.21.15.76:0/2082968055 shutdown_connections 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- 172.21.15.76:0/2082968055 >> 172.21.15.76:0/2082968055 conn(0x7f53480a0030 msgr2=0x7f53480a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- 172.21.15.76:0/2082968055 shutdown_connections 2024-09-16T06:45:55.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- 172.21.15.76:0/2082968055 wait complete. 2024-09-16T06:45:55.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 10 monclient: monmap: 2024-09-16T06:45:55.244 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:55.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 Processor -- start 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 1 -- start start 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.236+0000 7f534d2fc700 10 monclient: init 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-b con 0x7f533400a310 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-c con 0x7f5334011200 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient(hunting): picked mon.noname-a con 0x7f53481400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348072660 con 0x7f53481400d0 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534813e490 con 0x7f533400a310 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534806cc90 con 0x7f5334011200 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534d2fc700 10 monclient(hunting): authenticate will time out at 1302.376887s 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f53477fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f5334011200 legacy=0x7f53481d36a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534cafb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f53481400d0 0x7f53481d6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50438/0 (socket says 172.21.15.76:50438) 2024-09-16T06:45:55.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.240+0000 7f534cafb700 1 -- 172.21.15.76:0/1689220870 learned_addr learned my addr 172.21.15.76:0/1689220870 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814807757 0 0) 0x7f5348072660 con 0x7f53481400d0 2024-09-16T06:45:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): my global_id is 184185 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533400a910 con 0x7f53481400d0 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802544049 0 0) 0x7f534813e490 con 0x7f533400a310 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): my global_id is 174505 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5348072660 con 0x7f533400a310 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2498593450 0 0) 0x7f533400a910 con 0x7f53481400d0 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f534813e490 con 0x7f53481400d0 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5340004390 con 0x7f53481400d0 2024-09-16T06:45:55.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:55.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003661148 0 0) 0x7f5348072660 con 0x7f533400a310 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f533400a910 con 0x7f533400a310 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f533c0016d0 con 0x7f533400a310 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient(hunting): dump: 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3464331422 0 0) 0x7f533400a910 con 0x7f533400a310 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 >> v1:172.21.15.76:6790/0 conn(0x7f5334011200 legacy=0x7f53481d36a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 >> v1:172.21.15.76:6789/0 conn(0x7f53481400d0 legacy=0x7f53481d6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 monclient: found mon.b 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534809a540 con 0x7f533400a310 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 5 monclient: authenticate success, global_id 174505 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 10 monclient: _renew_subs 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53481387a0 con 0x7f533400a310 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 10 monclient: _renew_subs 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f534806c6d0 con 0x7f533400a310 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f533c001ba0 con 0x7f533400a310 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f533c0036a0 con 0x7f533400a310 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f5344ff9700 10 monclient: dump: 2024-09-16T06:45:55.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.244+0000 7f53467fc700 4 set_mon_vals no callback set 2024-09-16T06:45:55.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f533c003800 con 0x7f533400a310 2024-09-16T06:45:55.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(196..196 src has 151..196) v4 ==== 10195+0+0 (unknown 196388032 0 0) 0x7f533c0447d0 con 0x7f533400a310 2024-09-16T06:45:55.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f534d2fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f534d2fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f534d2fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5348094fc0 con 0x7f533400a310 2024-09-16T06:45:55.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.248+0000 7f53477fe700 10 monclient: get_auth_request con 0x7f5334039600 auth_method 0 2024-09-16T06:45:55.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.256+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f533c003ab0 con 0x7f533400a310 2024-09-16T06:45:55.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.256+0000 7f5344ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.256+0000 7f5344ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:55.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.298+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v197) v1 ==== 168+0+0 (unknown 2093217777 0 0) 0x7f2cb4014800 con 0x7f2cc409f1c0 2024-09-16T06:45:55.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.298+0000 7f2cb97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.298+0000 7f2cb97fa700 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-16T06:45:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.402+0000 7f2cabfff700 10 monclient: tick 2024-09-16T06:45:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.402+0000 7f2cabfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.402+0000 7f2cabfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2ccac18700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2ccac18700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2ccac18700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2cc409b2f0 con 0x7f2cc409f1c0 2024-09-16T06:45:55.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2cb97fa700 1 -- 172.21.15.189:0/2990554818 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v197) v1 ==== 147+0+0 (unknown 1095814454 0 0) 0x7f2cb4014800 con 0x7f2cc409f1c0 2024-09-16T06:45:55.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2cb97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.410+0000 7f2cb97fa700 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2cb00365a0 legacy=0x7f2cb0038a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 10 monclient: shutdown 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 >> v1:172.21.15.189:6789/0 conn(0x7f2cc409f1c0 legacy=0x7f2cc406cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 shutdown_connections 2024-09-16T06:45:55.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 >> 172.21.15.189:0/2990554818 conn(0x7f2cc40a4410 msgr2=0x7f2cc4099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 shutdown_connections 2024-09-16T06:45:55.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.414+0000 7f2ccac18700 1 -- 172.21.15.189:0/2990554818 wait complete. 2024-09-16T06:45:55.418 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h16' already exists 2024-09-16T06:45:55.428 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:55.428 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-16T06:45:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f534d2fc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:45:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f534d2fc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f534d2fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5348094fc0 con 0x7f533400a310 2024-09-16T06:45:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f5344ff9700 1 -- 172.21.15.76:0/1689220870 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2479792048) 0x7f533c032570 con 0x7f533400a310 2024-09-16T06:45:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f5344ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:45:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.492+0000 7f5344ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:55.497 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5334039600 legacy=0x7f533403bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 10 monclient: shutdown 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 >> v1:172.21.15.189:6789/0 conn(0x7f533400a310 legacy=0x7f5348137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 shutdown_connections 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 >> 172.21.15.76:0/1689220870 conn(0x7f53480a0030 msgr2=0x7f534809c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 shutdown_connections 2024-09-16T06:45:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.496+0000 7f534d2fc700 1 -- 172.21.15.76:0/1689220870 wait complete. 2024-09-16T06:45:55.510 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:45:55.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:45:55.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:45:55.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.530+0000 7ff1a2601700 10 monclient: get_monmap_and_config 2024-09-16T06:45:55.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.530+0000 7ff1a2601700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.530+0000 7ff1a2601700 10 monclient: monmap: 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:55.535567+0000 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:55.535567+0000 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:55.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.530+0000 7ff1a2601700 1 Processor -- start 2024-09-16T06:45:55.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- start start 2024-09-16T06:45:55.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: init 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-c con 0x7ff19c099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-b con 0x7ff19c09ada0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-a con 0x7ff19c09b1b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0998c0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c09bf60 con 0x7ff19c099c80 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c09d410 con 0x7ff19c09ada0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient(hunting): authenticate will time out at 1308.048855s 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a0dfe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff19c09ada0 legacy=0x7ff19c137320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a15ff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff19c099c80 0x7ff19c09a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49776/0 (socket says 172.21.15.189:49776) 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a15ff700 1 -- 172.21.15.189:0/63697240 learned_addr learned my addr 172.21.15.189:0/63697240 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815989164 0 0) 0x7ff19c09bf60 con 0x7ff19c099c80 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient(hunting): my global_id is 174508 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1880035d0 con 0x7ff19c099c80 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805747337 0 0) 0x7ff1880035d0 con 0x7ff19c099c80 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 >> v1:172.21.15.76:6790/0 conn(0x7ff19c09ada0 legacy=0x7ff19c137320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 >> v1:172.21.15.76:6789/0 conn(0x7ff19c09b1b0 legacy=0x7ff19c13aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 monclient: found mon.noname-c 2024-09-16T06:45:55.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c09c160 con 0x7ff19c099c80 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 5 monclient: authenticate success, global_id 174508 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff198001610 con 0x7ff19c099c80 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: dump: 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff198001930 con 0x7ff19c099c80 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 1 -- 172.21.15.189:0/63697240 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff198001d00 con 0x7ff19c099c80 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 4 set_mon_vals no callback set 2024-09-16T06:45:55.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff193fff700 10 monclient: dump: 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: shutdown 2024-09-16T06:45:55.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- 172.21.15.189:0/63697240 >> v1:172.21.15.189:6789/0 conn(0x7ff19c099c80 legacy=0x7ff19c09a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- 172.21.15.189:0/63697240 shutdown_connections 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- 172.21.15.189:0/63697240 >> 172.21.15.189:0/63697240 conn(0x7ff19c09fe70 msgr2=0x7ff19c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- 172.21.15.189:0/63697240 shutdown_connections 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- 172.21.15.189:0/63697240 wait complete. 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: monmap: 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.549 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 Processor -- start 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 1 -- start start 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.534+0000 7ff1a2601700 10 monclient: init 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-c con 0x7ff19c099c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-a con 0x7ff19c09ada0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient(hunting): picked mon.noname-b con 0x7ff19c09b1b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c06c290 con 0x7ff19c09ada0 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c06bbd0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c098840 con 0x7ff19c099c80 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient(hunting): authenticate will time out at 1308.052264s 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a15ff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff19c099c80 legacy=0x7ff19c13b9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a1e00700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff19c09b1b0 0x7ff19c1d2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49778/0 (socket says 172.21.15.189:49778) 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a1e00700 1 -- 172.21.15.189:0/1568232517 learned_addr learned my addr 172.21.15.189:0/1568232517 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786082153 0 0) 0x7ff19c06bbd0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): my global_id is 174511 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1800035d0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965031228 0 0) 0x7ff19c06c290 con 0x7ff19c09ada0 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): my global_id is 184191 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c06bbd0 con 0x7ff19c09ada0 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901122890 0 0) 0x7ff1800035d0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c06c290 con 0x7ff19c09b1b0 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff194001c70 con 0x7ff19c09b1b0 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.552 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3093321600 0 0) 0x7ff19c06bbd0 con 0x7ff19c09ada0 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1800035d0 con 0x7ff19c09ada0 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff18c0041f0 con 0x7ff19c09ada0 2024-09-16T06:45:55.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189018424 0 0) 0x7ff19c06c290 con 0x7ff19c09b1b0 2024-09-16T06:45:55.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 >> v1:172.21.15.76:6790/0 conn(0x7ff19c099c80 legacy=0x7ff19c13b9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 >> v1:172.21.15.76:6789/0 conn(0x7ff19c09ada0 legacy=0x7ff19c13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 monclient: found mon.b 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c098410 con 0x7ff19c09b1b0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 5 monclient: authenticate success, global_id 174511 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _renew_subs 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c08f580 con 0x7ff19c09b1b0 2024-09-16T06:45:55.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _renew_subs 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c06b6c0 con 0x7ff19c09b1b0 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff194002000 con 0x7ff19c09b1b0 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff194003e90 con 0x7ff19c09b1b0 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 10 monclient: dump: 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1937fe700 4 set_mon_vals no callback set 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff194004070 con 0x7ff19c09b1b0 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(197..197 src has 151..197) v4 ==== 10664+0+0 (unknown 4246258420 0 0) 0x7ff194045180 con 0x7ff19c09b1b0 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17c005100 con 0x7ff19c09b1b0 2024-09-16T06:45:55.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.538+0000 7ff1a15ff700 10 monclient: get_auth_request con 0x7ff1800362c0 auth_method 0 2024-09-16T06:45:55.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.546+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff194014930 con 0x7ff19c09b1b0 2024-09-16T06:45:55.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.546+0000 7ff191ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.546+0000 7ff191ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:55.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.620+0000 7ff22ab57700 10 monclient: get_monmap_and_config 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.620+0000 7ff22ab57700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.620+0000 7ff22ab57700 10 monclient: monmap: 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:55.627358+0000 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:55.627358+0000 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:55.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.620+0000 7ff22ab57700 1 Processor -- start 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- start start 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient: init 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-c con 0x7ff224097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-a con 0x7ff224099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-b con 0x7ff224139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff224098990 con 0x7ff224099090 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff224097f40 con 0x7ff224097b60 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22409d610 con 0x7ff224139500 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient(hunting): authenticate will time out at 1302.760552s 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22a356700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff224139500 legacy=0x7ff22413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff229354700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff224099090 0x7ff22409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50454/0 (socket says 172.21.15.76:50454) 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff229354700 1 -- 172.21.15.76:0/2790045884 learned_addr learned my addr 172.21.15.76:0/2790045884 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946752283 0 0) 0x7ff224098990 con 0x7ff224099090 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): my global_id is 184194 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2100035d0 con 0x7ff224099090 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243342142 0 0) 0x7ff224097f40 con 0x7ff224097b60 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient(hunting): my global_id is 174514 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224098990 con 0x7ff224097b60 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502033003 0 0) 0x7ff2100035d0 con 0x7ff224099090 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 >> v1:172.21.15.76:6790/0 conn(0x7ff224139500 legacy=0x7ff22413b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 >> v1:172.21.15.189:6789/0 conn(0x7ff224097b60 legacy=0x7ff22409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 monclient: found mon.noname-a 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240986b0 con 0x7ff224099090 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 5 monclient: authenticate success, global_id 184194 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff214001610 con 0x7ff224099090 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: dump: 2024-09-16T06:45:55.631 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2140018f0 con 0x7ff224099090 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 1 -- 172.21.15.76:0/2790045884 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff214001c80 con 0x7ff224099090 2024-09-16T06:45:55.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff228b53700 10 monclient: dump: 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 4 set_mon_vals no callback set 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 10 monclient: shutdown 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- 172.21.15.76:0/2790045884 >> v1:172.21.15.76:6789/0 conn(0x7ff224099090 legacy=0x7ff22409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- 172.21.15.76:0/2790045884 shutdown_connections 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- 172.21.15.76:0/2790045884 >> 172.21.15.76:0/2790045884 conn(0x7ff2240a0030 msgr2=0x7ff2240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- 172.21.15.76:0/2790045884 shutdown_connections 2024-09-16T06:45:55.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.624+0000 7ff22ab57700 1 -- 172.21.15.76:0/2790045884 wait complete. 2024-09-16T06:45:55.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: monmap: 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 Processor -- start 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- start start 2024-09-16T06:45:55.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: init 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-a con 0x7ff224097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-b con 0x7ff224099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient(hunting): picked mon.noname-c con 0x7ff224139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22413c180 con 0x7ff224097b60 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff224139330 con 0x7ff224099090 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240a2d80 con 0x7ff224139500 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient(hunting): authenticate will time out at 1302.764984s 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22a356700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff224139500 legacy=0x7ff2241d1f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff229b55700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff224097b60 0x7ff224138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50468/0 (socket says 172.21.15.76:50468) 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff229b55700 1 -- 172.21.15.76:0/3880691421 learned_addr learned my addr 172.21.15.76:0/3880691421 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:55.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841660453 0 0) 0x7ff22413c180 con 0x7ff224097b60 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): my global_id is 184197 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2080035d0 con 0x7ff224097b60 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 138899573 0 0) 0x7ff2080035d0 con 0x7ff224097b60 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff22413c180 con 0x7ff224097b60 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff220001910 con 0x7ff224097b60 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): dump: 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.643 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250395537 0 0) 0x7ff224139330 con 0x7ff224099090 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): my global_id is 174517 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2080035d0 con 0x7ff224099090 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488652649 0 0) 0x7ff22413c180 con 0x7ff224097b60 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6790/0 conn(0x7ff224139500 legacy=0x7ff2241d1f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.189:6789/0 conn(0x7ff224099090 legacy=0x7ff224138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 monclient: found mon.a 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22409a540 con 0x7ff224097b60 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 5 monclient: authenticate success, global_id 184197 2024-09-16T06:45:55.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _renew_subs 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff224091310 con 0x7ff224097b60 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _renew_subs 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22408fbe0 con 0x7ff224097b60 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff220001ce0 con 0x7ff224097b60 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff220003b90 con 0x7ff224097b60 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient: dump: 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21bfff700 4 set_mon_vals no callback set 2024-09-16T06:45:55.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff220003d70 con 0x7ff224097b60 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff20c006590 con 0x7ff20c002830 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(197..197 src has 151..197) v4 ==== 10664+0+0 (unknown 4246258420 0 0) 0x7ff220044f60 con 0x7ff224097b60 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff22a356700 10 monclient: get_auth_request con 0x7ff208036220 auth_method 0 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749738736 0 0) 0x7ff20c006590 con 0x7ff20c002830 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 10 monclient(hunting): my global_id is 184200 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.628+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22413c180 con 0x7ff20c002830 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3898891747 0 0) 0x7ff22413c180 con 0x7ff20c002830 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff20c006590 con 0x7ff20c002830 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff214001cc0 con 0x7ff20c002830 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 10 monclient: dump: 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3412714553 0 0) 0x7ff20c006590 con 0x7ff20c002830 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff20c00a190 con 0x7ff20c002830 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff2140015e0 con 0x7ff20c002830 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6789/0 conn(0x7ff20c002830 legacy=0x7ff20c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.632+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6789/0 conn(0x7ff20c002830 legacy=0x7ff20c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.640+0000 7ff22ab57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.640+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff20c009e50 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531943972 0 0) 0x7ff20c009e50 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient(hunting): my global_id is 184203 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff20c006590 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4243434482 0 0) 0x7ff20c006590 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff20c009e50 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff214004d80 con 0x7ff20c009170 2024-09-16T06:45:55.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient: dump: 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4057860855 0 0) 0x7ff20c009e50 con 0x7ff20c009170 2024-09-16T06:45:55.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff20c00c340 con 0x7ff20c009170 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 2055931817) 0x7ff214005fd0 con 0x7ff20c009170 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6789/0 conn(0x7ff20c009170 legacy=0x7ff20c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff21a7fc700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6789/0 conn(0x7ff20c009170 legacy=0x7ff20c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:45:55.655 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 70, 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.665 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:55.667 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff208036220 legacy=0x7ff2080386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 10 monclient: shutdown 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 >> v1:172.21.15.76:6789/0 conn(0x7ff224097b60 legacy=0x7ff224138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22a356700 1 -- 172.21.15.76:0/3880691421 reap_dead start 2024-09-16T06:45:55.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 shutdown_connections 2024-09-16T06:45:55.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 >> 172.21.15.76:0/3880691421 conn(0x7ff2240a0030 msgr2=0x7ff2240a1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 shutdown_connections 2024-09-16T06:45:55.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.644+0000 7ff22ab57700 1 -- 172.21.15.76:0/3880691421 wait complete. 2024-09-16T06:45:55.669 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:45:55.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:45:55.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:45:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.714+0000 7ff1a2601700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.714+0000 7ff1a2601700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.714+0000 7ff1a2601700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:55.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:55.714+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff17c005930 con 0x7ff19c09b1b0 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: get_monmap_and_config 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: monmap: 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:45:55.790832+0000 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:45:55.790832+0000 2024-09-16T06:45:55.788 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:55.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:55.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 1 Processor -- start 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 1 -- start start 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: init 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.784+0000 7f202e7b2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-c con 0x7f202001f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-a con 0x7f2020025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-b con 0x7f2020025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202001f8a0 con 0x7f2020025410 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20200233f0 con 0x7f202001f4c0 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2020020050 con 0x7f2020025860 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient(hunting): authenticate will time out at 1302.924094s 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202dfb1700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2020025860 legacy=0x7f20200c2320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202cfaf700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2020025410 0x7f20200bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50484/0 (socket says 172.21.15.76:50484) 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202cfaf700 1 -- 172.21.15.76:0/2392603073 learned_addr learned my addr 172.21.15.76:0/2392603073 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299203061 0 0) 0x7f202001f8a0 con 0x7f2020025410 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): my global_id is 184206 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20140035d0 con 0x7f2020025410 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201245666 0 0) 0x7f20200233f0 con 0x7f202001f4c0 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient(hunting): my global_id is 174520 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202001f8a0 con 0x7f202001f4c0 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586620233 0 0) 0x7f20140035d0 con 0x7f2020025410 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 >> v1:172.21.15.76:6790/0 conn(0x7f2020025860 legacy=0x7f20200c2320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 >> v1:172.21.15.189:6789/0 conn(0x7f202001f4c0 legacy=0x7f2020024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 monclient: found mon.noname-a 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2020021a90 con 0x7f2020025410 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2028069400 con 0x7f2020025410 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 5 monclient: authenticate success, global_id 184206 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: dump: 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f202804ec20 con 0x7f2020025410 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 1 -- 172.21.15.76:0/2392603073 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20280008d0 con 0x7f2020025410 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f201bfff700 10 monclient: dump: 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.803 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 4 set_mon_vals no callback set 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient: shutdown 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- 172.21.15.76:0/2392603073 >> v1:172.21.15.76:6789/0 conn(0x7f2020025410 legacy=0x7f20200bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- 172.21.15.76:0/2392603073 shutdown_connections 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- 172.21.15.76:0/2392603073 >> 172.21.15.76:0/2392603073 conn(0x7f2020005b40 msgr2=0x7f2020005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- 172.21.15.76:0/2392603073 shutdown_connections 2024-09-16T06:45:55.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- 172.21.15.76:0/2392603073 wait complete. 2024-09-16T06:45:55.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient: build_initial_monmap 2024-09-16T06:45:55.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient: monmap: 2024-09-16T06:45:55.805 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:45:55.805 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:55.805 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:45:55.808 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 Processor -- start 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 1 -- start start 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.788+0000 7f202e7b2700 10 monclient: init 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-a con 0x7f202001f4c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:55.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-c con 0x7f2020025410 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient(hunting): picked mon.noname-b con 0x7f2020025860 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2020028de0 con 0x7f202001f4c0 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20201483e0 con 0x7f2020025860 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20200204a0 con 0x7f2020025410 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient(hunting): authenticate will time out at 1302.928861s 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202cfaf700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2020025410 legacy=0x7f2020153000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202d7b0700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f202001f4c0 0x7f2020147cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50500/0 (socket says 172.21.15.76:50500) 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202d7b0700 1 -- 172.21.15.76:0/1271687509 learned_addr learned my addr 172.21.15.76:0/1271687509 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851030026 0 0) 0x7f2020028de0 con 0x7f202001f4c0 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): my global_id is 184209 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f200c0035d0 con 0x7f202001f4c0 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603416393 0 0) 0x7f20201483e0 con 0x7f2020025860 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): my global_id is 174523 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2020028de0 con 0x7f2020025860 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751206975 0 0) 0x7f200c0035d0 con 0x7f202001f4c0 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20201483e0 con 0x7f202001f4c0 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2024001950 con 0x7f202001f4c0 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:55.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553740957 0 0) 0x7f2020028de0 con 0x7f2020025860 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f200c0035d0 con 0x7f2020025860 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f201c004060 con 0x7f2020025860 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient(hunting): dump: 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.813 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572400309 0 0) 0x7f20201483e0 con 0x7f202001f4c0 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.76:6790/0 conn(0x7f2020025410 legacy=0x7f2020153000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6789/0 conn(0x7f2020025860 legacy=0x7f2020156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 monclient: found mon.a 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202012cb50 con 0x7f202001f4c0 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:55.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 5 monclient: authenticate success, global_id 184209 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _renew_subs 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202012f630 con 0x7f202001f4c0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _renew_subs 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f202012e0e0 con 0x7f202001f4c0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2024001ce0 con 0x7f202001f4c0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2024003b10 con 0x7f202001f4c0 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:45:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f201b7fe700 4 set_mon_vals no callback set 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 10 monclient: dump: 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2008006590 con 0x7f2008002830 2024-09-16T06:45:55.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.792+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2024003d70 con 0x7f202001f4c0 2024-09-16T06:45:55.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(197..197 src has 151..197) v4 ==== 10664+0+0 (unknown 4246258420 0 0) 0x7f2024044e70 con 0x7f202001f4c0 2024-09-16T06:45:55.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f202dfb1700 10 monclient: get_auth_request con 0x7f200c0362a0 auth_method 0 2024-09-16T06:45:55.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262943262 0 0) 0x7f2008006590 con 0x7f2008002830 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient(hunting): my global_id is 174526 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20201483e0 con 0x7f2008002830 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2329080119 0 0) 0x7f20201483e0 con 0x7f2008002830 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2008006590 con 0x7f2008002830 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2028069400 con 0x7f2008002830 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient: dump: 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.819 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1866936372 0 0) 0x7f2008006590 con 0x7f2008002830 2024-09-16T06:45:55.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f200800a190 con 0x7f2008002830 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f202804ef50 con 0x7f2008002830 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6789/0 conn(0x7f2008002830 legacy=0x7f2008004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.796+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6789/0 conn(0x7f2008002830 legacy=0x7f2008004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:45:55.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f200800c480 con 0x7f2008009150 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072860837 0 0) 0x7f200800c480 con 0x7f2008009150 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient(hunting): my global_id is 174529 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008006590 con 0x7f2008009150 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 905039550 0 0) 0x7f2008006590 con 0x7f2008009150 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f200800c480 con 0x7f2008009150 2024-09-16T06:45:55.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f202806a4b0 con 0x7f2008009150 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient: dump: 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:55.824 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 733018344 0 0) 0x7f200800c480 con 0x7f2008009150 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2008009cf0 con 0x7f2008009150 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 2637616037) 0x7f2028066de0 con 0x7f2008009150 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6789/0 conn(0x7f2008009150 legacy=0x7f20080095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f2019ffb700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6789/0 conn(0x7f2008009150 legacy=0x7f20080095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 70, 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:45:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:45:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:45:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:45:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:45:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:45:55.839 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.189:6812/3098810674 conn(0x7f200c0362a0 legacy=0x7f200c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 10 monclient: shutdown 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 >> v1:172.21.15.76:6789/0 conn(0x7f202001f4c0 legacy=0x7f2020147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202dfb1700 1 -- 172.21.15.76:0/1271687509 reap_dead start 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 shutdown_connections 2024-09-16T06:45:55.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 >> 172.21.15.76:0/1271687509 conn(0x7f2020005b40 msgr2=0x7f202001b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:55.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 shutdown_connections 2024-09-16T06:45:55.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:45:55.808+0000 7f202e7b2700 1 -- 172.21.15.76:0/1271687509 wait complete. 2024-09-16T06:45:55.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:45:55.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:45:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.314+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v198) v1 ==== 168+0+0 (unknown 2967784065 0 0) 0x7ff194012fd0 con 0x7ff19c09b1b0 2024-09-16T06:45:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.314+0000 7ff191ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.314+0000 7ff191ffb700 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-16T06:45:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.422+0000 7ff1a2601700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.422+0000 7ff1a2601700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.422+0000 7ff1a2601700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.422+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff17c005b70 con 0x7ff19c09b1b0 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff191ffb700 1 -- 172.21.15.189:0/1568232517 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v198) v1 ==== 147+0+0 (unknown 1672860466 0 0) 0x7ff194014750 con 0x7ff19c09b1b0 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff191ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff191ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff1800362c0 legacy=0x7ff180038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 10 monclient: shutdown 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:56.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 >> v1:172.21.15.189:6789/0 conn(0x7ff19c09b1b0 legacy=0x7ff19c1d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 shutdown_connections 2024-09-16T06:45:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 >> 172.21.15.189:0/1568232517 conn(0x7ff19c09fe70 msgr2=0x7ff19c09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.426+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 shutdown_connections 2024-09-16T06:45:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.430+0000 7ff1a2601700 1 -- 172.21.15.189:0/1568232517 wait complete. 2024-09-16T06:45:56.434 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h17' already exists 2024-09-16T06:45:56.442 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:56.442 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.542+0000 7f3f14497700 10 monclient: get_monmap_and_config 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.542+0000 7f3f14497700 10 monclient: build_initial_monmap 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.542+0000 7f3f14497700 10 monclient: monmap: 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:56.547 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:56.547593+0000 2024-09-16T06:45:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:56.547593+0000 2024-09-16T06:45:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:56.548 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:56.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:56.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 Processor -- start 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- start start 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: init 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c09bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-c con 0x7f3f0c09c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-a con 0x7f3f0c138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c09cca0 con 0x7f3f0c138da0 2024-09-16T06:45:56.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c09f750 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c097620 con 0x7f3f0c09bb00 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient(hunting): authenticate will time out at 1309.060637s 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11a32700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3f0c09c5f0 0x7f3f0c137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49782/0 (socket says 172.21.15.189:49782) 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11a32700 1 -- 172.21.15.189:0/1113275503 learned_addr learned my addr 172.21.15.189:0/1113275503 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f12233700 1 -- 172.21.15.189:0/1113275503 >> v1:172.21.15.76:6790/0 conn(0x7f3f0c09bb00 legacy=0x7f3f0c09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54101211 0 0) 0x7f3f0c09f750 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient(hunting): _negotiate 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient(hunting): my global_id is 174532 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef80057e0 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395769687 0 0) 0x7f3ef80057e0 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 >> v1:172.21.15.76:6790/0 conn(0x7f3f0c09bb00 legacy=0x7f3f0c09bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 >> v1:172.21.15.76:6789/0 conn(0x7f3f0c138da0 legacy=0x7f3f0c13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: _finish_hunting 0 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 monclient: found mon.noname-c 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c097900 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: _finish_auth 0 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: _check_auth_tickets 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:56.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 5 monclient: authenticate success, global_id 174532 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3efc001610 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: dump: 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:56.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3efc001930 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 1 -- 172.21.15.189:0/1113275503 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3efc001cc0 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 4 set_mon_vals no callback set 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:56.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f11231700 10 monclient: dump: 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: shutdown 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:56.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- 172.21.15.189:0/1113275503 >> v1:172.21.15.189:6789/0 conn(0x7f3f0c09c5f0 legacy=0x7f3f0c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- 172.21.15.189:0/1113275503 shutdown_connections 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- 172.21.15.189:0/1113275503 >> 172.21.15.189:0/1113275503 conn(0x7f3f0c0a03e0 msgr2=0x7f3f0c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- 172.21.15.189:0/1113275503 shutdown_connections 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 1 -- 172.21.15.189:0/1113275503 wait complete. 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: build_initial_monmap 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.546+0000 7f3f14497700 10 monclient: monmap: 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:56.558 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:56.559 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:56.559 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 Processor -- start 2024-09-16T06:45:56.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- start start 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: init 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-a con 0x7f3f0c09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-c con 0x7f3f0c09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c0a0980 con 0x7f3f0c09bb00 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c132be0 con 0x7f3f0c138da0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c09a6e0 con 0x7f3f0c09c5f0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient(hunting): authenticate will time out at 1309.066208s 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f11a32700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f3f0c09c5f0 legacy=0x7f3f0c1ca860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:56.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f12a34700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3f0c138da0 0x7f3f0c1d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49798/0 (socket says 172.21.15.189:49798) 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f12a34700 1 -- 172.21.15.189:0/2916553535 learned_addr learned my addr 172.21.15.189:0/2916553535 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025943296 0 0) 0x7f3f0c132be0 con 0x7f3f0c138da0 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): my global_id is 174535 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef00035d0 con 0x7f3f0c138da0 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803940720 0 0) 0x7f3f0c0a0980 con 0x7f3f0c09bb00 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): my global_id is 184215 2024-09-16T06:45:56.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f0c132be0 con 0x7f3f0c09bb00 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418103733 0 0) 0x7f3ef00035d0 con 0x7f3f0c138da0 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f0c0a0980 con 0x7f3f0c138da0 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f04001c40 con 0x7f3f0c138da0 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): dump: 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:56.565 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:56.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 387383256 0 0) 0x7f3f0c0a0980 con 0x7f3f0c138da0 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 >> v1:172.21.15.76:6790/0 conn(0x7f3f0c09c5f0 legacy=0x7f3f0c1ca860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 >> v1:172.21.15.76:6789/0 conn(0x7f3f0c09bb00 legacy=0x7f3f0c1ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 monclient: found mon.b 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c13c1b0 con 0x7f3f0c138da0 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:56.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 5 monclient: authenticate success, global_id 174535 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _renew_subs 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c132900 con 0x7f3f0c138da0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _renew_subs 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c1387d0 con 0x7f3f0c138da0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f04001fd0 con 0x7f3f0c138da0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3f04003e90 con 0x7f3f0c138da0 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f10a30700 4 set_mon_vals no callback set 2024-09-16T06:45:56.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.550+0000 7f3f02ffd700 10 monclient: dump: 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:56.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3f04004070 con 0x7f3f0c138da0 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f14497700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f14497700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f14497700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef4005100 con 0x7f3f0c138da0 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(198..198 src has 151..198) v4 ==== 10877+0+0 (unknown 928220554 0 0) 0x7f3f040452d0 con 0x7f3f0c138da0 2024-09-16T06:45:56.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.554+0000 7f3f12233700 10 monclient: get_auth_request con 0x7f3ef0035fa0 auth_method 0 2024-09-16T06:45:56.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.558+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3f04014970 con 0x7f3f0c138da0 2024-09-16T06:45:56.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.558+0000 7f3f02ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:56.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.558+0000 7f3f02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:56.736 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.734+0000 7f3f14497700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.734+0000 7f3f14497700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:56.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.734+0000 7f3f14497700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:56.737 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:56.734+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3ef4005930 con 0x7f3f0c138da0 2024-09-16T06:45:57.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.326+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v199) v1 ==== 168+0+0 (unknown 828973512 0 0) 0x7f3f04013010 con 0x7f3f0c138da0 2024-09-16T06:45:57.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.326+0000 7f3f02ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.326+0000 7f3f02ffd700 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-16T06:45:57.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f14497700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f14497700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f14497700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3ef40053f0 con 0x7f3f0c138da0 2024-09-16T06:45:57.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f02ffd700 1 -- 172.21.15.189:0/2916553535 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v199) v1 ==== 147+0+0 (unknown 3535685190 0 0) 0x7f3f04014790 con 0x7f3f0c138da0 2024-09-16T06:45:57.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f02ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.438+0000 7f3f02ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-16T06:45:57.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3ef0035fa0 legacy=0x7f3ef0038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 10 monclient: shutdown 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 >> v1:172.21.15.189:6789/0 conn(0x7f3f0c138da0 legacy=0x7f3f0c1d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 shutdown_connections 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 >> 172.21.15.189:0/2916553535 conn(0x7f3f0c0a03e0 msgr2=0x7f3f0c0a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 shutdown_connections 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.442+0000 7f3f14497700 1 -- 172.21.15.189:0/2916553535 wait complete. 2024-09-16T06:45:57.447 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h18' already exists 2024-09-16T06:45:57.460 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:57.460 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: build_initial_monmap 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: monmap: 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:57.566409+0000 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:57.566409+0000 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:57.566 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:57.567 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:57.567 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 1 Processor -- start 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 1 -- start start 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: init 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:45:57.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb60c099c80 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb60c135410 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb60c138f20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c0998c0 con 0x7fb60c138f20 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c09bf60 con 0x7fb60c099c80 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c09d410 con 0x7fb60c135410 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60b59e700 10 monclient(hunting): authenticate will time out at 1310.079452s 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb609d9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fb60c135410 legacy=0x7fb60c137800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb60c099c80 0x7fb60c09eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49814/0 (socket says 172.21.15.189:49814) 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.562+0000 7fb60a59c700 1 -- 172.21.15.189:0/1288035512 learned_addr learned my addr 172.21.15.189:0/1288035512 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141101419 0 0) 0x7fb60c09bf60 con 0x7fb60c099c80 2024-09-16T06:45:57.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient(hunting): _negotiate 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient(hunting): my global_id is 174538 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f40035d0 con 0x7fb60c099c80 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136051509 0 0) 0x7fb5f40035d0 con 0x7fb60c099c80 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 >> v1:172.21.15.76:6790/0 conn(0x7fb60c135410 legacy=0x7fb60c137800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 >> v1:172.21.15.76:6789/0 conn(0x7fb60c138f20 legacy=0x7fb60c13b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: _finish_hunting 0 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 monclient: found mon.noname-c 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c09c160 con 0x7fb60c099c80 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: _finish_auth 0 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: _check_auth_tickets 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 5 monclient: authenticate success, global_id 174538 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb600001610 con 0x7fb60c099c80 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: dump: 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.578 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb600001930 con 0x7fb60c099c80 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 1 -- 172.21.15.189:0/1288035512 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb600001d00 con 0x7fb60c099c80 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 4 set_mon_vals no callback set 2024-09-16T06:45:57.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60959a700 10 monclient: dump: 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: shutdown 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- 172.21.15.189:0/1288035512 >> v1:172.21.15.189:6789/0 conn(0x7fb60c099c80 legacy=0x7fb60c09eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- 172.21.15.189:0/1288035512 shutdown_connections 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- 172.21.15.189:0/1288035512 >> 172.21.15.189:0/1288035512 conn(0x7fb60c09fe70 msgr2=0x7fb60c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- 172.21.15.189:0/1288035512 shutdown_connections 2024-09-16T06:45:57.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- 172.21.15.189:0/1288035512 wait complete. 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: build_initial_monmap 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: monmap: 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:57.582 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 Processor -- start 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- start start 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: init 2024-09-16T06:45:57.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb60c099c80 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb60c135410 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb60c138f20 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c06cf60 con 0x7fb60c135410 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c0986a0 con 0x7fb60c138f20 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c06e3f0 con 0x7fb60c099c80 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60b59e700 10 monclient(hunting): authenticate will time out at 1310.083214s 2024-09-16T06:45:57.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60a59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fb60c099c80 legacy=0x7fb60c070e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb60c138f20 0x7fb60c1d0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49818/0 (socket says 172.21.15.189:49818) 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb60ad9d700 1 -- 172.21.15.189:0/1667711376 learned_addr learned my addr 172.21.15.189:0/1667711376 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967034645 0 0) 0x7fb60c0986a0 con 0x7fb60c138f20 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 10 monclient(hunting): my global_id is 174541 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.566+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ec0035d0 con 0x7fb60c138f20 2024-09-16T06:45:57.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747933644 0 0) 0x7fb60c06cf60 con 0x7fb60c135410 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): my global_id is 184221 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb60c0986a0 con 0x7fb60c135410 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3128249176 0 0) 0x7fb5ec0035d0 con 0x7fb60c138f20 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb60c06cf60 con 0x7fb60c138f20 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb604001bb0 con 0x7fb60c138f20 2024-09-16T06:45:57.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): dump: 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:57.587 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952438359 0 0) 0x7fb60c06cf60 con 0x7fb60c138f20 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 >> v1:172.21.15.76:6790/0 conn(0x7fb60c099c80 legacy=0x7fb60c070e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 >> v1:172.21.15.76:6789/0 conn(0x7fb60c135410 legacy=0x7fb60c1cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: _finish_hunting 0 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 monclient: found mon.b 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:57.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0715a0 con 0x7fb60c138f20 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: _finish_auth 0 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: _check_auth_tickets 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 5 monclient: authenticate success, global_id 174541 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _renew_subs 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c09a380 con 0x7fb60c138f20 2024-09-16T06:45:57.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _renew_subs 2024-09-16T06:45:57.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c06d600 con 0x7fb60c138f20 2024-09-16T06:45:57.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb604003a40 con 0x7fb60c138f20 2024-09-16T06:45:57.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb604003dd0 con 0x7fb60c138f20 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5fbfff700 4 set_mon_vals no callback set 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 10 monclient: dump: 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:57.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:57.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f0005100 con 0x7fb60c138f20 2024-09-16T06:45:57.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb604003fb0 con 0x7fb60c138f20 2024-09-16T06:45:57.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(199..199 src has 151..199) v4 ==== 11090+0+0 (unknown 2451257066 0 0) 0x7fb604045320 con 0x7fb60c138f20 2024-09-16T06:45:57.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.570+0000 7fb60a59c700 10 monclient: get_auth_request con 0x7fb5ec036180 auth_method 0 2024-09-16T06:45:57.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.578+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb604014ba0 con 0x7fb60c138f20 2024-09-16T06:45:57.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.578+0000 7fb5faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:57.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.578+0000 7fb5faffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:57.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.750+0000 7fb60b59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.750+0000 7fb60b59e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:57.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.750+0000 7fb60b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:57.757 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:57.750+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb5f00053b0 con 0x7fb60c138f20 2024-09-16T06:45:58.732 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.730+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v200) v1 ==== 168+0+0 (unknown 1405555845 0 0) 0x7fb604013010 con 0x7fb60c138f20 2024-09-16T06:45:58.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.730+0000 7fb5faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:58.733 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.730+0000 7fb5faffd700 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-16T06:45:58.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.838+0000 7fb60b59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.838+0000 7fb60b59e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.838+0000 7fb60b59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.838+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb5f0000bc0 con 0x7fb60c138f20 2024-09-16T06:45:58.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb5faffd700 1 -- 172.21.15.189:0/1667711376 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v200) v1 ==== 147+0+0 (unknown 129058868 0 0) 0x7fb604014970 con 0x7fb60c138f20 2024-09-16T06:45:58.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb5faffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:58.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb5faffd700 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-16T06:45:58.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb5ec036180 legacy=0x7fb5ec038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:58.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 10 monclient: shutdown 2024-09-16T06:45:58.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:58.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 >> v1:172.21.15.189:6789/0 conn(0x7fb60c138f20 legacy=0x7fb60c1d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:58.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 shutdown_connections 2024-09-16T06:45:58.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 >> 172.21.15.189:0/1667711376 conn(0x7fb60c09fe70 msgr2=0x7fb60c134c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:58.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 shutdown_connections 2024-09-16T06:45:58.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.842+0000 7fb60b59e700 1 -- 172.21.15.189:0/1667711376 wait complete. 2024-09-16T06:45:58.849 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h19' already exists 2024-09-16T06:45:58.858 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:58.858 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.962+0000 7fb856060700 10 monclient: get_monmap_and_config 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.962+0000 7fb856060700 10 monclient: build_initial_monmap 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.962+0000 7fb856060700 10 monclient: monmap: 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:58.967 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:58.967639+0000 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:58.967639+0000 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:58.968 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 Processor -- start 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- start start 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient: init 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient(hunting): picked mon.noname-c con 0x7fb8500a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient(hunting): picked mon.noname-a con 0x7fb85009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient(hunting): picked mon.noname-b con 0x7fb8500a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85009a040 con 0x7fb85009aa20 2024-09-16T06:45:58.973 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8500a33d0 con 0x7fb8500a2350 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb850099d60 con 0x7fb8500a3ca0 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient(hunting): authenticate will time out at 1311.481221s 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8500a2350 0x7fb8500a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56982/0 (socket says 172.21.15.189:56982) 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84f7fe700 1 -- 172.21.15.189:0/3580525766 learned_addr learned my addr 172.21.15.189:0/3580525766 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84ffff700 1 -- 172.21.15.189:0/3580525766 >> v1:172.21.15.76:6790/0 conn(0x7fb8500a3ca0 legacy=0x7fb85013dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872074845 0 0) 0x7fb8500a33d0 con 0x7fb8500a2350 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:45:58.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient(hunting): my global_id is 174544 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8380035d0 con 0x7fb8500a2350 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730465150 0 0) 0x7fb8380035d0 con 0x7fb8500a2350 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 >> v1:172.21.15.76:6790/0 conn(0x7fb8500a3ca0 legacy=0x7fb85013dde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 >> v1:172.21.15.76:6789/0 conn(0x7fb85009aa20 legacy=0x7fb85013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500a3920 con 0x7fb8500a2350 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:45:58.975 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb840001610 con 0x7fb8500a2350 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: dump: 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:58.976 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 5 monclient: authenticate success, global_id 174544 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb840001930 con 0x7fb8500a2350 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 1 -- 172.21.15.189:0/3580525766 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb840001d00 con 0x7fb8500a2350 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb84e7fc700 10 monclient: dump: 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:58.977 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 4 set_mon_vals no callback set 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 10 monclient: shutdown 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- 172.21.15.189:0/3580525766 >> v1:172.21.15.189:6789/0 conn(0x7fb8500a2350 legacy=0x7fb8500a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- 172.21.15.189:0/3580525766 shutdown_connections 2024-09-16T06:45:58.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- 172.21.15.189:0/3580525766 >> 172.21.15.189:0/3580525766 conn(0x7fb8500a7d70 msgr2=0x7fb8500a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- 172.21.15.189:0/3580525766 shutdown_connections 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.966+0000 7fb856060700 1 -- 172.21.15.189:0/3580525766 wait complete. 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: build_initial_monmap 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: monmap: 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:58.979 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 Processor -- start 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- start start 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: init 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient(hunting): picked mon.noname-a con 0x7fb85009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient(hunting): picked mon.noname-c con 0x7fb8500a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient(hunting): picked mon.noname-b con 0x7fb8500a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8501ca9f0 con 0x7fb85009aa20 2024-09-16T06:45:58.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85009b640 con 0x7fb8500a3ca0 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85006bed0 con 0x7fb8500a2350 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient(hunting): authenticate will time out at 1311.486410s 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb84effd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fb8500a2350 legacy=0x7fb8501cedf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb84ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8500a3ca0 0x7fb8501d2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56988/0 (socket says 172.21.15.189:56988) 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb84ffff700 1 -- 172.21.15.189:0/1577089670 learned_addr learned my addr 172.21.15.189:0/1577089670 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968852337 0 0) 0x7fb85009b640 con 0x7fb8500a3ca0 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:58.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): my global_id is 174547 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c0035d0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217736989 0 0) 0x7fb8501ca9f0 con 0x7fb85009aa20 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _negotiate 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): my global_id is 184227 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85009b640 con 0x7fb85009aa20 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810609610 0 0) 0x7fb83c0035d0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8501ca9f0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb8480044f0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): dump: 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:58.986 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4058759919 0 0) 0x7fb8501ca9f0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 >> v1:172.21.15.76:6790/0 conn(0x7fb8500a2350 legacy=0x7fb8501cedf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 >> v1:172.21.15.76:6789/0 conn(0x7fb85009aa20 legacy=0x7fb8501354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: _finish_hunting 0 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 monclient: found mon.b 2024-09-16T06:45:58.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850122ea0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: _finish_auth 0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: _check_auth_tickets 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 5 monclient: authenticate success, global_id 174547 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _renew_subs 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb850121490 con 0x7fb8500a3ca0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _renew_subs 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb850136150 con 0x7fb8500a3ca0 2024-09-16T06:45:58.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8480047d0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb848004b60 con 0x7fb8500a3ca0 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb837fff700 10 monclient: dump: 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:58.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.970+0000 7fb84dffb700 4 set_mon_vals no callback set 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb848004cc0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb856060700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb856060700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(200..200 src has 151..200) v4 ==== 11303+0+0 (unknown 2283413469 0 0) 0x7fb8480452d0 con 0x7fb8500a3ca0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb856060700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb830005100 con 0x7fb8500a3ca0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.974+0000 7fb84f7fe700 10 monclient: get_auth_request con 0x7fb83c036180 auth_method 0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.978+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb848014a40 con 0x7fb8500a3ca0 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.978+0000 7fb837fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:58.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:58.978+0000 7fb837fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:45:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.150+0000 7fb856060700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.150+0000 7fb856060700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.150+0000 7fb856060700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.150+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb830005930 con 0x7fb8500a3ca0 2024-09-16T06:45:59.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.734+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v201) v1 ==== 168+0+0 (unknown 2132418898 0 0) 0x7fb848012eb0 con 0x7fb8500a3ca0 2024-09-16T06:45:59.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.738+0000 7fb837fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.738+0000 7fb837fff700 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-16T06:45:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb856060700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb856060700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb856060700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb830005b70 con 0x7fb8500a3ca0 2024-09-16T06:45:59.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb837fff700 1 -- 172.21.15.189:0/1577089670 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v201) v1 ==== 147+0+0 (unknown 1145503140 0 0) 0x7fb848014630 con 0x7fb8500a3ca0 2024-09-16T06:45:59.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb837fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:45:59.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.846+0000 7fb837fff700 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-16T06:45:59.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb83c036180 legacy=0x7fb83c038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:59.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 10 monclient: shutdown 2024-09-16T06:45:59.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:59.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 >> v1:172.21.15.189:6789/0 conn(0x7fb8500a3ca0 legacy=0x7fb8501d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:59.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 shutdown_connections 2024-09-16T06:45:59.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 >> 172.21.15.189:0/1577089670 conn(0x7fb8500a7d70 msgr2=0x7fb8501bd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:59.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 shutdown_connections 2024-09-16T06:45:59.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.850+0000 7fb856060700 1 -- 172.21.15.189:0/1577089670 wait complete. 2024-09-16T06:45:59.855 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h20' already exists 2024-09-16T06:45:59.864 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:45:59.864 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-16T06:45:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: get_monmap_and_config 2024-09-16T06:45:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: build_initial_monmap 2024-09-16T06:45:59.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: monmap: 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:45:59.973039+0000 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:45:59.973039+0000 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:45:59.973 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 1 Processor -- start 2024-09-16T06:45:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 1 -- start start 2024-09-16T06:45:59.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: init 2024-09-16T06:45:59.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-a con 0x7f340809d830 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-c con 0x7f340809dc10 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-b con 0x7f3408137dd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340809a790 con 0x7f340809d830 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3408071510 con 0x7f340809dc10 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3408132040 con 0x7f3408137dd0 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f34067fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f340809dc10 0x7f34081366b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56992/0 (socket says 172.21.15.189:56992) 2024-09-16T06:45:59.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f34067fc700 1 -- 172.21.15.189:0/967907258 learned_addr learned my addr 172.21.15.189:0/967907258 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3407fff700 10 monclient(hunting): authenticate will time out at 1312.487404s 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f34077fe700 1 -- 172.21.15.189:0/967907258 >> v1:172.21.15.76:6790/0 conn(0x7f3408137dd0 legacy=0x7f340813a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186405038 0 0) 0x7f33fc001110 con 0x7f340809dc10 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 10 monclient(hunting): my global_id is 174550 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.970+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f00034f0 con 0x7f340809dc10 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267163284 0 0) 0x7f33f00034f0 con 0x7f340809dc10 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 >> v1:172.21.15.76:6790/0 conn(0x7f3408137dd0 legacy=0x7f340813a1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:45:59.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 >> v1:172.21.15.76:6789/0 conn(0x7f340809d830 legacy=0x7f3408131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 monclient: found mon.noname-c 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340809e060 con 0x7f340809dc10 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: _finish_auth 0 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33fc001600 con 0x7f340809dc10 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 5 monclient: authenticate success, global_id 174550 2024-09-16T06:45:59.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: dump: 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:59.981 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33fc001920 con 0x7f340809dc10 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 1 -- 172.21.15.189:0/967907258 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33fc001cf0 con 0x7f340809dc10 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3405ffb700 10 monclient: dump: 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.982 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 4 set_mon_vals no callback set 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 10 monclient: shutdown 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 1 -- 172.21.15.189:0/967907258 >> v1:172.21.15.189:6789/0 conn(0x7f340809dc10 legacy=0x7f34081366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 1 -- 172.21.15.189:0/967907258 shutdown_connections 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 1 -- 172.21.15.189:0/967907258 >> 172.21.15.189:0/967907258 conn(0x7f34080a4320 msgr2=0x7f34080a0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:45:59.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 1 -- 172.21.15.189:0/967907258 shutdown_connections 2024-09-16T06:45:59.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 1 -- 172.21.15.189:0/967907258 wait complete. 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 10 monclient: build_initial_monmap 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.974+0000 7f3407fff700 10 monclient: monmap: 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:45:59.989 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 Processor -- start 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- start start 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: init 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-b con 0x7f340809d830 addr v1:172.21.15.189:6789/0 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-a con 0x7f340809dc10 addr v1:172.21.15.76:6789/0 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient(hunting): picked mon.noname-c con 0x7f3408137dd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340811b5b0 con 0x7f340809dc10 2024-09-16T06:45:59.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34080a32c0 con 0x7f340809d830 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34080a00c0 con 0x7f3408137dd0 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient(hunting): authenticate will time out at 1312.493466s 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3406ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f340809d830 0x7f340806e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56994/0 (socket says 172.21.15.189:56994) 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3406ffd700 1 -- 172.21.15.189:0/2677405082 learned_addr learned my addr 172.21.15.189:0/2677405082 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f34077fe700 1 -- 172.21.15.189:0/2677405082 >> v1:172.21.15.76:6790/0 conn(0x7f3408137dd0 legacy=0x7f340806f500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270773267 0 0) 0x7f34080a32c0 con 0x7f340809d830 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): my global_id is 174553 2024-09-16T06:45:59.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f40035d0 con 0x7f340809d830 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1878754201 0 0) 0x7f33f40035d0 con 0x7f340809d830 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34080a32c0 con 0x7f340809d830 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33f80019e0 con 0x7f340809d830 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): dump: 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.992 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920460176 0 0) 0x7f340811b5b0 con 0x7f340809dc10 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient(hunting): my global_id is 184233 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f40035d0 con 0x7f340809dc10 2024-09-16T06:45:59.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70167114 0 0) 0x7f34080a32c0 con 0x7f340809d830 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 >> v1:172.21.15.76:6790/0 conn(0x7f3408137dd0 legacy=0x7f340806f500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 >> v1:172.21.15.76:6789/0 conn(0x7f340809dc10 legacy=0x7f340806edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 monclient: found mon.b 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3408070cd0 con 0x7f340809d830 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: _finish_auth 0 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 5 monclient: authenticate success, global_id 174553 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _renew_subs 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340811b2d0 con 0x7f340809d830 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _renew_subs 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34081cefb0 con 0x7f340809d830 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33f8001d00 con 0x7f340809d830 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33f8003b90 con 0x7f340809d830 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f34057fa700 4 set_mon_vals no callback set 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.978+0000 7f33ef7fe700 10 monclient: dump: 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:45:59.995 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:45:59.996 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f33f8003d70 con 0x7f340809d830 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f3407fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f3407fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f3407fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e8005100 con 0x7f340809d830 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(201..201 src has 151..201) v4 ==== 11516+0+0 (unknown 4003803858 0 0) 0x7f33f80452c0 con 0x7f340809d830 2024-09-16T06:45:59.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.982+0000 7f34067fc700 10 monclient: get_auth_request con 0x7f33f40361d0 auth_method 0 2024-09-16T06:45:59.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.986+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f33f8014720 con 0x7f340809d830 2024-09-16T06:45:59.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.986+0000 7f33ef7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:45:59.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:45:59.986+0000 7f33ef7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:00.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.162+0000 7f3407fff700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.162+0000 7f3407fff700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.162+0000 7f3407fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:00.167 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.162+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f33e8005930 con 0x7f340809d830 2024-09-16T06:46:00.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.750+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v202) v1 ==== 168+0+0 (unknown 3007126562 0 0) 0x7f33f8014540 con 0x7f340809d830 2024-09-16T06:46:00.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.750+0000 7f33ef7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.753 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.750+0000 7f33ef7fe700 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-16T06:46:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f3407fff700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f3407fff700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f3407fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:00.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f33e80053f0 con 0x7f340809d830 2024-09-16T06:46:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f33ef7fe700 1 -- 172.21.15.189:0/2677405082 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v202) v1 ==== 147+0+0 (unknown 1722016672 0 0) 0x7f33f8014b80 con 0x7f340809d830 2024-09-16T06:46:00.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f33ef7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:00.868 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.862+0000 7f33ef7fe700 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 >> v1:172.21.15.189:6812/3098810674 conn(0x7f33f40361d0 legacy=0x7f33f4038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 10 monclient: shutdown 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 >> v1:172.21.15.189:6789/0 conn(0x7f340809d830 legacy=0x7f340806e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 shutdown_connections 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 >> 172.21.15.189:0/2677405082 conn(0x7f34080a4320 msgr2=0x7f34080a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:00.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 shutdown_connections 2024-09-16T06:46:00.871 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.866+0000 7f3407fff700 1 -- 172.21.15.189:0/2677405082 wait complete. 2024-09-16T06:46:00.871 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h21' already exists 2024-09-16T06:46:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:00.881 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 10 monclient: build_initial_monmap 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 10 monclient: monmap: 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:00.990606+0000 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:00.990606+0000 2024-09-16T06:46:00.990 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 1 Processor -- start 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 1 -- start start 2024-09-16T06:46:00.991 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.986+0000 7f300f59e700 10 monclient: init 2024-09-16T06:46:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:00.994 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f30101321f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f301006bac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f301013bc00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301006a340 con 0x7f30101321f0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301006bf00 con 0x7f301013bc00 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30100a3930 con 0x7f301006bac0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient(hunting): authenticate will time out at 1313.504247s 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300dd9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f301006bac0 legacy=0x7f301013a4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f301013bc00 0x7f301013dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:56998/0 (socket says 172.21.15.189:56998) 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300ed9d700 1 -- 172.21.15.189:0/3803421310 learned_addr learned my addr 172.21.15.189:0/3803421310 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054214117 0 0) 0x7f301006bf00 con 0x7f301013bc00 2024-09-16T06:46:00.995 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient(hunting): my global_id is 174556 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc0035d0 con 0x7f301013bc00 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243828700 0 0) 0x7f2ffc0035d0 con 0x7f301013bc00 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 >> v1:172.21.15.76:6790/0 conn(0x7f301006bac0 legacy=0x7f301013a4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 >> v1:172.21.15.76:6789/0 conn(0x7f30101321f0 legacy=0x7f30101325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 monclient: found mon.noname-c 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:00.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301006a820 con 0x7f301013bc00 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: _finish_auth 0 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 5 monclient: authenticate success, global_id 174556 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3008001990 con 0x7f301013bc00 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: dump: 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:00.997 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3008001c80 con 0x7f301013bc00 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 1 -- 172.21.15.189:0/3803421310 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3008002090 con 0x7f301013bc00 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:00.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 4 set_mon_vals no callback set 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300d59a700 10 monclient: dump: 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:00.999 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: shutdown 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- 172.21.15.189:0/3803421310 >> v1:172.21.15.189:6789/0 conn(0x7f301013bc00 legacy=0x7f301013dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- 172.21.15.189:0/3803421310 shutdown_connections 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- 172.21.15.189:0/3803421310 >> 172.21.15.189:0/3803421310 conn(0x7f30100a4280 msgr2=0x7f30100a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- 172.21.15.189:0/3803421310 shutdown_connections 2024-09-16T06:46:01.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 1 -- 172.21.15.189:0/3803421310 wait complete. 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: build_initial_monmap 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.990+0000 7f300f59e700 10 monclient: monmap: 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:01.004 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 Processor -- start 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- start start 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: init 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f301006bac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f30101321f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f301013bc00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30101378b0 con 0x7f301006bac0 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3010137cd0 con 0x7f30101321f0 2024-09-16T06:46:01.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f30101321f0 0x7f301009f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57006/0 (socket says 172.21.15.189:57006) 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300dd9b700 1 -- 172.21.15.189:0/2355291568 learned_addr learned my addr 172.21.15.189:0/2355291568 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301009ff90 con 0x7f301013bc00 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient(hunting): authenticate will time out at 1313.510080s 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300ed9d700 1 -- 172.21.15.189:0/2355291568 >> v1:172.21.15.76:6790/0 conn(0x7f301013bc00 legacy=0x7f30101d1dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759148034 0 0) 0x7f3010137cd0 con 0x7f30101321f0 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): my global_id is 174559 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff00035d0 con 0x7f30101321f0 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 296766415 0 0) 0x7f2ff00035d0 con 0x7f30101321f0 2024-09-16T06:46:01.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3010137cd0 con 0x7f30101321f0 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3000004280 con 0x7f30101321f0 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:01.007 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669308361 0 0) 0x7f30101378b0 con 0x7f301006bac0 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient(hunting): my global_id is 184236 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff00035d0 con 0x7f301006bac0 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 33785105 0 0) 0x7f3010137cd0 con 0x7f30101321f0 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 >> v1:172.21.15.76:6790/0 conn(0x7f301013bc00 legacy=0x7f30101d1dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 >> v1:172.21.15.76:6789/0 conn(0x7f301006bac0 legacy=0x7f301009f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:01.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 monclient: found mon.b 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100a18f0 con 0x7f30101321f0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 5 monclient: authenticate success, global_id 174559 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _renew_subs 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301013b7c0 con 0x7f30101321f0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _renew_subs 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30100a1670 con 0x7f30101321f0 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30000044a0 con 0x7f30101321f0 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3000004830 con 0x7f30101321f0 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 10 monclient: dump: 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:01.010 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.994+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3000004990 con 0x7f30101321f0 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300cd99700 4 set_mon_vals no callback set 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(202..202 src has 151..202) v4 ==== 11729+0+0 (unknown 3093786371 0 0) 0x7f3000045220 con 0x7f30101321f0 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300e59c700 10 monclient: get_auth_request con 0x7f2ff00362e0 auth_method 0 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:00.998+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4005100 con 0x7f30101321f0 2024-09-16T06:46:01.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.006+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f30000147e0 con 0x7f30101321f0 2024-09-16T06:46:01.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.006+0000 7f3006ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:01.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.006+0000 7f3006ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.174+0000 7f300f59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.174+0000 7f300f59e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.174+0000 7f300f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.179 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.174+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ff4005930 con 0x7f30101321f0 2024-09-16T06:46:01.755 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.750+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v203) v1 ==== 168+0+0 (unknown 2269063848 0 0) 0x7f3000012c50 con 0x7f30101321f0 2024-09-16T06:46:01.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.750+0000 7f3006ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.756 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.750+0000 7f3006ffd700 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-16T06:46:01.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f300f59e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f300f59e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f300f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:01.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2ff4005b70 con 0x7f30101321f0 2024-09-16T06:46:01.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f3006ffd700 1 -- 172.21.15.189:0/2355291568 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v203) v1 ==== 147+0+0 (unknown 1395314302 0 0) 0x7f30000143d0 con 0x7f30101321f0 2024-09-16T06:46:01.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f3006ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:01.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.862+0000 7f3006ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-16T06:46:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2ff00362e0 legacy=0x7f2ff00387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:01.869 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 10 monclient: shutdown 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 >> v1:172.21.15.189:6789/0 conn(0x7f30101321f0 legacy=0x7f301009f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 shutdown_connections 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 >> 172.21.15.189:0/2355291568 conn(0x7f30100a4280 msgr2=0x7f301009d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 shutdown_connections 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.866+0000 7f300f59e700 1 -- 172.21.15.189:0/2355291568 wait complete. 2024-09-16T06:46:01.870 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h22' already exists 2024-09-16T06:46:01.880 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:01.880 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: get_monmap_and_config 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: build_initial_monmap 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: monmap: 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:01.993 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:01.993480+0000 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:01.993480+0000 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:01.994 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 1 Processor -- start 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 1 -- start start 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: init 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-a con 0x7f41cc09a9f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-b con 0x7f41cc09a230 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:02.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-c con 0x7f41cc09b4e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc09bca0 con 0x7f41cc09a9f0 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc09bef0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc09f770 con 0x7f41cc09a230 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 10 monclient(hunting): authenticate will time out at 1314.506899s 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41cb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41cc09b4e0 0x7f41cc13bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57022/0 (socket says 172.21.15.189:57022) 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41cb7fe700 1 -- 172.21.15.189:0/3686314415 learned_addr learned my addr 172.21.15.189:0/3686314415 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41ca7fc700 1 -- 172.21.15.189:0/3686314415 >> v1:172.21.15.76:6790/0 conn(0x7f41cc09a230 legacy=0x7f41cc1388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811156340 0 0) 0x7f41cc09bef0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:02.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): my global_id is 174562 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b80035d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078800215 0 0) 0x7f41cc09bca0 con 0x7f41cc09a9f0 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient(hunting): my global_id is 184239 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41cc09bef0 con 0x7f41cc09a9f0 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2637182772 0 0) 0x7f41b80035d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 >> v1:172.21.15.76:6790/0 conn(0x7f41cc09a230 legacy=0x7f41cc1388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 >> v1:172.21.15.76:6789/0 conn(0x7f41cc09a9f0 legacy=0x7f41cc09add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:02.006 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc09c1d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41d19ab700 5 monclient: authenticate success, global_id 174562 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41c4001990 con 0x7f41cc09b4e0 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.990+0000 7f41c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: dump: 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:02.007 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41c4001c70 con 0x7f41cc09b4e0 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:02.008 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 1 -- 172.21.15.189:0/3686314415 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41c4002000 con 0x7f41cc09b4e0 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 4 set_mon_vals no callback set 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41c9ffb700 10 monclient: dump: 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:02.009 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: shutdown 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- 172.21.15.189:0/3686314415 >> v1:172.21.15.189:6789/0 conn(0x7f41cc09b4e0 legacy=0x7f41cc13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- 172.21.15.189:0/3686314415 shutdown_connections 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- 172.21.15.189:0/3686314415 >> 172.21.15.189:0/3686314415 conn(0x7f41cc0a03e0 msgr2=0x7f41cc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- 172.21.15.189:0/3686314415 shutdown_connections 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- 172.21.15.189:0/3686314415 wait complete. 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: build_initial_monmap 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: monmap: 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:02.010 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 Processor -- start 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- start start 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: init 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:02.011 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-a con 0x7f41cc09a230 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-c con 0x7f41cc09a9f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient(hunting): picked mon.noname-b con 0x7f41cc09b4e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc12c4d0 con 0x7f41cc09a230 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc1c6700 con 0x7f41cc09b4e0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc0993e0 con 0x7f41cc09a9f0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.994+0000 7f41d19ab700 10 monclient(hunting): authenticate will time out at 1314.511946s 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41ca7fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f41cc09a9f0 legacy=0x7f41cc1caae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41cb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41cc09b4e0 0x7f41cc1ce210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57034/0 (socket says 172.21.15.189:57034) 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41cb7fe700 1 -- 172.21.15.189:0/4228466552 learned_addr learned my addr 172.21.15.189:0/4228466552 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753398305 0 0) 0x7f41cc1c6700 con 0x7f41cc09b4e0 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:02.012 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): my global_id is 174565 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac0035d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122776293 0 0) 0x7f41cc12c4d0 con 0x7f41cc09a230 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): my global_id is 184242 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41cc1c6700 con 0x7f41cc09a230 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2097460491 0 0) 0x7f41ac0035d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41cc12c4d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 650684782 0 0) 0x7f41cc1c6700 con 0x7f41cc09a230 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ac0035d0 con 0x7f41cc09a230 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41c4001950 con 0x7f41cc09b4e0 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): dump: 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:02.014 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41bc0018b0 con 0x7f41cc09a230 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient(hunting): dump: 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:02.015 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 532601910 0 0) 0x7f41cc12c4d0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 >> v1:172.21.15.76:6790/0 conn(0x7f41cc09a9f0 legacy=0x7f41cc1caae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 >> v1:172.21.15.76:6789/0 conn(0x7f41cc09a230 legacy=0x7f41cc133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 monclient: found mon.b 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc12e2b0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 5 monclient: authenticate success, global_id 174565 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _renew_subs 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.016 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc097510 con 0x7f41cc09b4e0 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _renew_subs 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc12bf50 con 0x7f41cc09b4e0 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41c4005020 con 0x7f41cc09b4e0 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41c4005330 con 0x7f41cc09b4e0 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c97fa700 4 set_mon_vals no callback set 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 10 monclient: dump: 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:02.017 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f41c400fda0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b0005100 con 0x7f41cc09b4e0 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:01.998+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(203..203 src has 151..203) v4 ==== 11942+0+0 (unknown 3884717390 0 0) 0x7f41c4045860 con 0x7f41cc09b4e0 2024-09-16T06:46:02.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.002+0000 7f41caffd700 10 monclient: get_auth_request con 0x7f41ac0362d0 auth_method 0 2024-09-16T06:46:02.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.006+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f41c4011810 con 0x7f41cc09b4e0 2024-09-16T06:46:02.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.006+0000 7f41c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:02.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.006+0000 7f41c3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.182+0000 7f41d19ab700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.182+0000 7f41d19ab700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.182+0000 7f41d19ab700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.185 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.182+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f41b0005930 con 0x7f41cc09b4e0 2024-09-16T06:46:02.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.766+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v204) v1 ==== 168+0+0 (unknown 782176307 0 0) 0x7f41c4013280 con 0x7f41cc09b4e0 2024-09-16T06:46:02.771 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.766+0000 7f41c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.766+0000 7f41c3fff700 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-16T06:46:02.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.878+0000 7f41d19ab700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.878+0000 7f41d19ab700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.882 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.878+0000 7f41d19ab700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:02.883 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.878+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f41b00053f0 con 0x7f41cc09b4e0 2024-09-16T06:46:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41c3fff700 1 -- 172.21.15.189:0/4228466552 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v204) v1 ==== 147+0+0 (unknown 594141096 0 0) 0x7f41c4014a00 con 0x7f41cc09b4e0 2024-09-16T06:46:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41c3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:02.884 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41c3fff700 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 >> v1:172.21.15.189:6812/3098810674 conn(0x7f41ac0362d0 legacy=0x7f41ac038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 10 monclient: shutdown 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 >> v1:172.21.15.189:6789/0 conn(0x7f41cc09b4e0 legacy=0x7f41cc1ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 shutdown_connections 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 >> 172.21.15.189:0/4228466552 conn(0x7f41cc0a03e0 msgr2=0x7f41cc06da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 shutdown_connections 2024-09-16T06:46:02.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:02.882+0000 7f41d19ab700 1 -- 172.21.15.189:0/4228466552 wait complete. 2024-09-16T06:46:02.888 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h23' already exists 2024-09-16T06:46:02.901 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:02.903 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-16T06:46:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: get_monmap_and_config 2024-09-16T06:46:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: build_initial_monmap 2024-09-16T06:46:03.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: monmap: 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:03.013954+0000 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:03.013954+0000 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:03.014 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 1 Processor -- start 2024-09-16T06:46:03.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 1 -- start start 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: init 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-a con 0x7ff1a00a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-b con 0x7ff1a009aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-c con 0x7ff1a00a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:03.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a009a040 con 0x7ff1a00a2350 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a00a33d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a0099d60 con 0x7ff1a009aa20 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff1a4e19700 10 monclient(hunting): authenticate will time out at 1315.527329s 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19dd9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff1a009aa20 legacy=0x7ff1a013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff1a00a3ca0 0x7ff1a013dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57044/0 (socket says 172.21.15.189:57044) 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19ed9d700 1 -- 172.21.15.189:0/3356272123 learned_addr learned my addr 172.21.15.189:0/3356272123 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482652289 0 0) 0x7ff1a00a33d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.010+0000 7ff19d59a700 10 monclient(hunting): my global_id is 174568 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff18c0035d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.021 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148201861 0 0) 0x7ff1a009a040 con 0x7ff1a00a2350 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient(hunting): my global_id is 184245 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a00a33d0 con 0x7ff1a00a2350 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795425386 0 0) 0x7ff18c0035d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 >> v1:172.21.15.76:6790/0 conn(0x7ff1a009aa20 legacy=0x7ff1a013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 >> v1:172.21.15.76:6789/0 conn(0x7ff1a00a2350 legacy=0x7ff1a00a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 monclient: found mon.noname-c 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00a3920 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: _finish_auth 0 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 5 monclient: authenticate success, global_id 174568 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff198001610 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: dump: 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.023 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:03.024 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1980018f0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 4 set_mon_vals no callback set 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 1 -- 172.21.15.189:0/3356272123 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff198001c80 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:03.025 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:03.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff19d59a700 10 monclient: dump: 2024-09-16T06:46:03.026 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:03.026 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:03.026 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.026 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:03.027 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 10 monclient: shutdown 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- 172.21.15.189:0/3356272123 >> v1:172.21.15.189:6789/0 conn(0x7ff1a00a3ca0 legacy=0x7ff1a013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- 172.21.15.189:0/3356272123 shutdown_connections 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- 172.21.15.189:0/3356272123 >> 172.21.15.189:0/3356272123 conn(0x7ff1a00a7d70 msgr2=0x7ff1a00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:03.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- 172.21.15.189:0/3356272123 shutdown_connections 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- 172.21.15.189:0/3356272123 wait complete. 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 10 monclient: build_initial_monmap 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 10 monclient: monmap: 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:03.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:03.030 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:03.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 Processor -- start 2024-09-16T06:46:03.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 1 -- start start 2024-09-16T06:46:03.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.014+0000 7ff1a4e19700 10 monclient: init 2024-09-16T06:46:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-a con 0x7ff1a009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-c con 0x7ff1a00a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:03.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient(hunting): picked mon.noname-b con 0x7ff1a00a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a009b060 con 0x7ff1a009aa20 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a0121fb0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a009b590 con 0x7ff1a00a2350 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient(hunting): authenticate will time out at 1315.532816s 2024-09-16T06:46:03.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff19dd9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7ff1a00a2350 legacy=0x7ff1a01cec90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:03.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff19ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff1a00a3ca0 0x7ff1a01d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57048/0 (socket says 172.21.15.189:57048) 2024-09-16T06:46:03.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff19ed9d700 1 -- 172.21.15.189:0/1254959252 learned_addr learned my addr 172.21.15.189:0/1254959252 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:03.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250346855 0 0) 0x7ff1a0121fb0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:03.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:03.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:03.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): my global_id is 174571 2024-09-16T06:46:03.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1800035d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043358546 0 0) 0x7ff1a009b060 con 0x7ff1a009aa20 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): my global_id is 184248 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a0121fb0 con 0x7ff1a009aa20 2024-09-16T06:46:03.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895417629 0 0) 0x7ff1800035d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a009b060 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1980015d0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:03.047 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:03.048 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:03.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 302952064 0 0) 0x7ff1a009b060 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 >> v1:172.21.15.76:6790/0 conn(0x7ff1a00a2350 legacy=0x7ff1a01cec90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:03.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 >> v1:172.21.15.76:6789/0 conn(0x7ff1a009aa20 legacy=0x7ff1a01345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:03.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:03.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 monclient: found mon.b 2024-09-16T06:46:03.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a006bed0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:03.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 5 monclient: authenticate success, global_id 174571 2024-09-16T06:46:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _renew_subs 2024-09-16T06:46:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0121cd0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _renew_subs 2024-09-16T06:46:03.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a0134ec0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff198004ca0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff198005030 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff196ffd700 10 monclient: dump: 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:03.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.018+0000 7ff19cd99700 4 set_mon_vals no callback set 2024-09-16T06:46:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff1a4e19700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff1a4e19700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff1a4e19700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.056 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184005100 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff198005190 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(204..204 src has 151..204) v4 ==== 12155+0+0 (unknown 4134202546 0 0) 0x7ff1980455a0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.022+0000 7ff19e59c700 10 monclient: get_auth_request con 0x7ff180035fa0 auth_method 0 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.026+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff198014ad0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.026+0000 7ff196ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:03.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.026+0000 7ff196ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:03.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.198+0000 7ff1a4e19700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.198+0000 7ff1a4e19700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.198+0000 7ff1a4e19700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.198+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff184005930 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.778+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v205) v1 ==== 168+0+0 (unknown 2327593047 0 0) 0x7ff198012f40 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.782+0000 7ff196ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.784 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.782+0000 7ff196ffd700 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-16T06:46:03.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.890+0000 7ff1a4e19700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.890+0000 7ff1a4e19700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.890+0000 7ff1a4e19700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:03.895 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.890+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff1840053f0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff196ffd700 1 -- 172.21.15.189:0/1254959252 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v205) v1 ==== 147+0+0 (unknown 1788779024 0 0) 0x7ff1980146c0 con 0x7ff1a00a3ca0 2024-09-16T06:46:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff196ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:03.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff196ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-16T06:46:03.901 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff180035fa0 legacy=0x7ff180038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff1a4e19700 10 monclient: shutdown 2024-09-16T06:46:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff1a4e19700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.894+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 >> v1:172.21.15.189:6789/0 conn(0x7ff1a00a3ca0 legacy=0x7ff1a01d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.898+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 shutdown_connections 2024-09-16T06:46:03.902 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.898+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 >> 172.21.15.189:0/1254959252 conn(0x7ff1a00a7d70 msgr2=0x7ff1a00a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:03.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.898+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 shutdown_connections 2024-09-16T06:46:03.903 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:03.898+0000 7ff1a4e19700 1 -- 172.21.15.189:0/1254959252 wait complete. 2024-09-16T06:46:03.903 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h24' already exists 2024-09-16T06:46:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:03.911 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: get_monmap_and_config 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: build_initial_monmap 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: monmap: 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:04.020707+0000 2024-09-16T06:46:04.020 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:04.020707+0000 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:04.021 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 1 Processor -- start 2024-09-16T06:46:04.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 1 -- start start 2024-09-16T06:46:04.022 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: init 2024-09-16T06:46:04.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:04.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:04.026 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-a con 0x7f6ef81354d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-b con 0x7f6ef81358b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-c con 0x7f6ef8135d40 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef8135110 con 0x7f6ef81354d0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef809f3e0 con 0x7f6ef8135d40 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef809f6c0 con 0x7f6ef81358b0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 10 monclient(hunting): authenticate will time out at 1316.535047s 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef6ffd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6ef81358b0 legacy=0x7f6ef813a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ef8135d40 0x7f6ef813dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57058/0 (socket says 172.21.15.189:57058) 2024-09-16T06:46:04.027 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef7fff700 1 -- 172.21.15.189:0/2528217031 learned_addr learned my addr 172.21.15.189:0/2528217031 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592260291 0 0) 0x7f6ef809f3e0 con 0x7f6ef8135d40 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient(hunting): my global_id is 174574 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee00035d0 con 0x7f6ef8135d40 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787428273 0 0) 0x7f6ee00035d0 con 0x7f6ef8135d40 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 >> v1:172.21.15.76:6790/0 conn(0x7f6ef81358b0 legacy=0x7f6ef813a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:04.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 >> v1:172.21.15.76:6789/0 conn(0x7f6ef81354d0 legacy=0x7f6ef809ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 monclient: found mon.noname-c 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef809f9a0 con 0x7f6ef8135d40 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 5 monclient: authenticate success, global_id 174574 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ef0001990 con 0x7f6ef8135d40 2024-09-16T06:46:04.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6efdd30700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.018+0000 7f6ef67fc700 10 monclient: dump: 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:04.030 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef0001cb0 con 0x7f6ef8135d40 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 4 set_mon_vals no callback set 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 1 -- 172.21.15.189:0/2528217031 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ef0002040 con 0x7f6ef8135d40 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef67fc700 10 monclient: dump: 2024-09-16T06:46:04.031 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: shutdown 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- 172.21.15.189:0/2528217031 >> v1:172.21.15.189:6789/0 conn(0x7f6ef8135d40 legacy=0x7f6ef813dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:04.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- 172.21.15.189:0/2528217031 shutdown_connections 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- 172.21.15.189:0/2528217031 >> 172.21.15.189:0/2528217031 conn(0x7f6ef80a0760 msgr2=0x7f6ef80a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- 172.21.15.189:0/2528217031 shutdown_connections 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- 172.21.15.189:0/2528217031 wait complete. 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: build_initial_monmap 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: monmap: 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:04.033 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 Processor -- start 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- start start 2024-09-16T06:46:04.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: init 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-c con 0x7f6ef81354d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-b con 0x7f6ef81358b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient(hunting): picked mon.noname-a con 0x7f6ef8135d40 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef81398e0 con 0x7f6ef8135d40 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef813eb20 con 0x7f6ef81358b0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef8132f80 con 0x7f6ef81354d0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6efdd30700 10 monclient(hunting): authenticate will time out at 1316.539238s 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef77fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6ef81354d0 legacy=0x7f6ef81ca4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:04.039 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ef81358b0 0x7f6ef81cf4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57070/0 (socket says 172.21.15.189:57070) 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6ef6ffd700 1 -- 172.21.15.189:0/4291720429 learned_addr learned my addr 172.21.15.189:0/4291720429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521443404 0 0) 0x7f6ef813eb20 con 0x7f6ef81358b0 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 10 monclient(hunting): my global_id is 174577 2024-09-16T06:46:04.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.022+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee40035d0 con 0x7f6ef81358b0 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035054190 0 0) 0x7f6ef81398e0 con 0x7f6ef8135d40 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): my global_id is 184254 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef813eb20 con 0x7f6ef8135d40 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1845163388 0 0) 0x7f6ee40035d0 con 0x7f6ef81358b0 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ef81398e0 con 0x7f6ef81358b0 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6eec004280 con 0x7f6ef81358b0 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): dump: 2024-09-16T06:46:04.041 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676943244 0 0) 0x7f6ef813eb20 con 0x7f6ef8135d40 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ee40035d0 con 0x7f6ef8135d40 2024-09-16T06:46:04.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ef00017b0 con 0x7f6ef8135d40 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient(hunting): dump: 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:04.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107055470 0 0) 0x7f6ef81398e0 con 0x7f6ef81358b0 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 >> v1:172.21.15.76:6790/0 conn(0x7f6ef81354d0 legacy=0x7f6ef81ca4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 >> v1:172.21.15.76:6789/0 conn(0x7f6ef8135d40 legacy=0x7f6ef81d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 monclient: found mon.b 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef80a3730 con 0x7f6ef81358b0 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: _finish_auth 0 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 5 monclient: authenticate success, global_id 174577 2024-09-16T06:46:04.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _renew_subs 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef806b150 con 0x7f6ef81358b0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _renew_subs 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef813bc40 con 0x7f6ef81358b0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eec0044a0 con 0x7f6ef81358b0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6eec0047b0 con 0x7f6ef81358b0 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6ef5ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:04.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 10 monclient: dump: 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed8005100 con 0x7f6ef81358b0 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6eec004990 con 0x7f6ef81358b0 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(205..205 src has 151..205) v4 ==== 12368+0+0 (unknown 2722591501 0 0) 0x7f6eec045460 con 0x7f6ef81358b0 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.026+0000 7f6ef77fe700 10 monclient: get_auth_request con 0x7f6ee4036310 auth_method 0 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.034+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6eec0051b0 con 0x7f6ef81358b0 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.034+0000 7f6edffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:04.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.034+0000 7f6edffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:04.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.206+0000 7f6efdd30700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.206+0000 7f6efdd30700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.206+0000 7f6efdd30700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.206+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6ed8005930 con 0x7f6ef81358b0 2024-09-16T06:46:04.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.790+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v206) v1 ==== 168+0+0 (unknown 1184595239 0 0) 0x7f6eec012c10 con 0x7f6ef81358b0 2024-09-16T06:46:04.794 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.790+0000 7f6edffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.795 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.790+0000 7f6edffff700 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-16T06:46:04.905 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6efdd30700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6efdd30700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6efdd30700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:04.906 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6ed80053f0 con 0x7f6ef81358b0 2024-09-16T06:46:04.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6edffff700 1 -- 172.21.15.189:0/4291720429 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v206) v1 ==== 147+0+0 (unknown 1216054292 0 0) 0x7f6eec014390 con 0x7f6ef81358b0 2024-09-16T06:46:04.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6edffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:04.907 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.902+0000 7f6edffff700 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6ee4036310 legacy=0x7f6ee40387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 10 monclient: shutdown 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 >> v1:172.21.15.189:6789/0 conn(0x7f6ef81358b0 legacy=0x7f6ef81cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 shutdown_connections 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 >> 172.21.15.189:0/4291720429 conn(0x7f6ef80a0760 msgr2=0x7f6ef8132260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 shutdown_connections 2024-09-16T06:46:04.910 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:04.906+0000 7f6efdd30700 1 -- 172.21.15.189:0/4291720429 wait complete. 2024-09-16T06:46:04.911 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h25' already exists 2024-09-16T06:46:04.920 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:04.920 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-16T06:46:05.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: get_monmap_and_config 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: build_initial_monmap 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: monmap: 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:05.029368+0000 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:05.029368+0000 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:05.029 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 1 Processor -- start 2024-09-16T06:46:05.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 1 -- start start 2024-09-16T06:46:05.030 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: init 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-a con 0x7f3e600a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-b con 0x7f3e6009aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-c con 0x7f3e600a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6009a040 con 0x7f3e600a2350 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e600a33d0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e60099d60 con 0x7f3e6009aa20 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:05.032 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 10 monclient(hunting): authenticate will time out at 1317.542986s 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e66476700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e600a3ca0 0x7f3e6013dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57076/0 (socket says 172.21.15.189:57076) 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e66476700 1 -- 172.21.15.189:0/1209958945 learned_addr learned my addr 172.21.15.189:0/1209958945 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e65474700 1 -- 172.21.15.189:0/1209958945 >> v1:172.21.15.76:6790/0 conn(0x7f3e6009aa20 legacy=0x7f3e6013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140455807 0 0) 0x7f3e600a33d0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient(hunting): _negotiate 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient(hunting): my global_id is 174580 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c0035d0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718369718 0 0) 0x7f3e4c0035d0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 >> v1:172.21.15.76:6790/0 conn(0x7f3e6009aa20 legacy=0x7f3e6013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 >> v1:172.21.15.76:6789/0 conn(0x7f3e600a2350 legacy=0x7f3e600a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:05.033 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: _finish_hunting 0 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 monclient: found mon.noname-c 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600a3920 con 0x7f3e600a3ca0 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: _finish_auth 0 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: _check_auth_tickets 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 5 monclient: authenticate success, global_id 174580 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e58001990 con 0x7f3e600a3ca0 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:05.034 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e64c73700 10 monclient: dump: 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.026+0000 7f3e67ed9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e58001cb0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:05.035 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 1 -- 172.21.15.189:0/1209958945 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e58002040 con 0x7f3e600a3ca0 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 4 set_mon_vals no callback set 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e64c73700 10 monclient: dump: 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:05.036 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:05.037 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:05.037 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 10 monclient: shutdown 2024-09-16T06:46:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- 172.21.15.189:0/1209958945 >> v1:172.21.15.189:6789/0 conn(0x7f3e600a3ca0 legacy=0x7f3e6013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- 172.21.15.189:0/1209958945 shutdown_connections 2024-09-16T06:46:05.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- 172.21.15.189:0/1209958945 >> 172.21.15.189:0/1209958945 conn(0x7f3e600a7d70 msgr2=0x7f3e600a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- 172.21.15.189:0/1209958945 shutdown_connections 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- 172.21.15.189:0/1209958945 wait complete. 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 10 monclient: build_initial_monmap 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 10 monclient: monmap: 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:05.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 Processor -- start 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 1 -- start start 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.030+0000 7f3e67ed9700 10 monclient: init 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-a con 0x7f3e6009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-b con 0x7f3e600a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient(hunting): picked mon.noname-c con 0x7f3e600a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6011fc00 con 0x7f3e6009aa20 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e60121730 con 0x7f3e600a2350 2024-09-16T06:46:05.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e60121fd0 con 0x7f3e600a3ca0 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e65474700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3e600a2350 0x7f3e60134610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57084/0 (socket says 172.21.15.189:57084) 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e65474700 1 -- 172.21.15.189:0/2459744224 learned_addr learned my addr 172.21.15.189:0/2459744224 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e66476700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f3e600a3ca0 legacy=0x7f3e601d1d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient(hunting): authenticate will time out at 1317.549151s 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113270711 0 0) 0x7f3e60121730 con 0x7f3e600a2350 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): my global_id is 174583 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e440035d0 con 0x7f3e600a2350 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98864506 0 0) 0x7f3e6011fc00 con 0x7f3e6009aa20 2024-09-16T06:46:05.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): my global_id is 184260 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60121730 con 0x7f3e6009aa20 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116489873 0 0) 0x7f3e440035d0 con 0x7f3e600a2350 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6011fc00 con 0x7f3e600a2350 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e50004230 con 0x7f3e600a2350 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): dump: 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:05.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2855517868 0 0) 0x7f3e6011fc00 con 0x7f3e600a2350 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 >> v1:172.21.15.76:6790/0 conn(0x7f3e600a3ca0 legacy=0x7f3e601d1d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 >> v1:172.21.15.76:6789/0 conn(0x7f3e6009aa20 legacy=0x7f3e601c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:05.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 monclient: found mon.b 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6009b010 con 0x7f3e600a2350 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 5 monclient: authenticate success, global_id 174583 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _renew_subs 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6006bed0 con 0x7f3e600a2350 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _renew_subs 2024-09-16T06:46:05.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.034+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6011f710 con 0x7f3e600a2350 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e50004510 con 0x7f3e600a2350 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e500048a0 con 0x7f3e600a2350 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e57fff700 4 set_mon_vals no callback set 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 10 monclient: dump: 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.048 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3e50004a00 con 0x7f3e600a2350 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(206..206 src has 151..206) v4 ==== 12581+0+0 (unknown 152673947 0 0) 0x7f3e50045590 con 0x7f3e600a2350 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e67ed9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e67ed9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e67ed9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e48005100 con 0x7f3e600a2350 2024-09-16T06:46:05.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.038+0000 7f3e65c75700 10 monclient: get_auth_request con 0x7f3e44036040 auth_method 0 2024-09-16T06:46:05.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.046+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3e50004cb0 con 0x7f3e600a2350 2024-09-16T06:46:05.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.046+0000 7f3e567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:05.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.046+0000 7f3e567fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:05.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.218+0000 7f3e67ed9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.218+0000 7f3e67ed9700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.218+0000 7f3e67ed9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.218+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e48005930 con 0x7f3e600a2350 2024-09-16T06:46:05.800 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.798+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v207) v1 ==== 168+0+0 (unknown 1922649005 0 0) 0x7f3e50012cf0 con 0x7f3e600a2350 2024-09-16T06:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.798+0000 7f3e567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.801 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.798+0000 7f3e567fc700 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-16T06:46:05.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e67ed9700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.913 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e67ed9700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e67ed9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:05.914 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3e48005b70 con 0x7f3e600a2350 2024-09-16T06:46:05.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e567fc700 1 -- 172.21.15.189:0/2459744224 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v207) v1 ==== 147+0+0 (unknown 2113055178 0 0) 0x7f3e50014470 con 0x7f3e600a2350 2024-09-16T06:46:05.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e567fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:05.915 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.910+0000 7f3e567fc700 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-16T06:46:05.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3e44036040 legacy=0x7f3e44038500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:05.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 10 monclient: shutdown 2024-09-16T06:46:05.919 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 >> v1:172.21.15.189:6789/0 conn(0x7f3e600a2350 legacy=0x7f3e60134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 shutdown_connections 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 >> 172.21.15.189:0/2459744224 conn(0x7f3e600a7d70 msgr2=0x7f3e60135410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 shutdown_connections 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:05.914+0000 7f3e67ed9700 1 -- 172.21.15.189:0/2459744224 wait complete. 2024-09-16T06:46:05.920 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h26' already exists 2024-09-16T06:46:05.932 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:05.932 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-16T06:46:06.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: get_monmap_and_config 2024-09-16T06:46:06.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: build_initial_monmap 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: monmap: 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:06.041954+0000 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:06.041954+0000 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:06.042 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 1 Processor -- start 2024-09-16T06:46:06.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 1 -- start start 2024-09-16T06:46:06.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: init 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.038+0000 7f1563710700 10 monclient(hunting): picked mon.noname-a con 0x7f155c133380 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient(hunting): picked mon.noname-b con 0x7f155c09f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient(hunting): picked mon.noname-c con 0x7f155c0a1190 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c132ec0 con 0x7f155c133380 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c09e4c0 con 0x7f155c0a1190 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c09e7a0 con 0x7f155c09f760 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient(hunting): authenticate will time out at 1318.556223s 2024-09-16T06:46:06.047 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1560cab700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f155c09f760 legacy=0x7f155c138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1561cad700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f155c0a1190 0x7f155c13b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57086/0 (socket says 172.21.15.189:57086) 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1561cad700 1 -- 172.21.15.189:0/730016009 learned_addr learned my addr 172.21.15.189:0/730016009 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397729410 0 0) 0x7f155c09e4c0 con 0x7f155c0a1190 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): my global_id is 174586 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15480035d0 con 0x7f155c0a1190 2024-09-16T06:46:06.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555269275 0 0) 0x7f155c132ec0 con 0x7f155c133380 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient(hunting): my global_id is 184263 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f155c09e4c0 con 0x7f155c133380 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277975171 0 0) 0x7f15480035d0 con 0x7f155c0a1190 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 >> v1:172.21.15.76:6790/0 conn(0x7f155c09f760 legacy=0x7f155c138240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 >> v1:172.21.15.76:6789/0 conn(0x7f155c133380 legacy=0x7f155c133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 monclient: found mon.noname-c 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0a1cb0 con 0x7f155c0a1190 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: _finish_auth 0 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1554001990 con 0x7f155c0a1190 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 5 monclient: authenticate success, global_id 174586 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: dump: 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.050 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1554001c70 con 0x7f155c0a1190 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 1 -- 172.21.15.189:0/730016009 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1554002000 con 0x7f155c0a1190 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 4 set_mon_vals no callback set 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:06.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f154ffff700 10 monclient: dump: 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient: shutdown 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:06.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- 172.21.15.189:0/730016009 >> v1:172.21.15.189:6789/0 conn(0x7f155c0a1190 legacy=0x7f155c13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- 172.21.15.189:0/730016009 shutdown_connections 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- 172.21.15.189:0/730016009 >> 172.21.15.189:0/730016009 conn(0x7f155c0a21f0 msgr2=0x7f155c0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- 172.21.15.189:0/730016009 shutdown_connections 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 1 -- 172.21.15.189:0/730016009 wait complete. 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient: build_initial_monmap 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.042+0000 7f1563710700 10 monclient: monmap: 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:06.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:06.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:06.054 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:06.054 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 Processor -- start 2024-09-16T06:46:06.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- start start 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: init 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient(hunting): picked mon.noname-c con 0x7f155c09f760 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient(hunting): picked mon.noname-a con 0x7f155c0a1190 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient(hunting): picked mon.noname-b con 0x7f155c133380 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c06f4b0 con 0x7f155c0a1190 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c06ed60 con 0x7f155c133380 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c09c700 con 0x7f155c09f760 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient(hunting): authenticate will time out at 1318.560612s 2024-09-16T06:46:06.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f15614ac700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f155c09f760 legacy=0x7f155c12bb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1561cad700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f155c133380 0x7f155c1d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57102/0 (socket says 172.21.15.189:57102) 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1561cad700 1 -- 172.21.15.189:0/4149737178 learned_addr learned my addr 172.21.15.189:0/4149737178 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426055848 0 0) 0x7f155c06ed60 con 0x7f155c133380 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): my global_id is 174589 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15400035d0 con 0x7f155c133380 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449866330 0 0) 0x7f15400035d0 con 0x7f155c133380 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f155c06ed60 con 0x7f155c133380 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894343507 0 0) 0x7f155c06f4b0 con 0x7f155c0a1190 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:06.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): my global_id is 184266 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15400035d0 con 0x7f155c0a1190 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1554001950 con 0x7f155c133380 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): dump: 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:06.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2456946244 0 0) 0x7f155c06ed60 con 0x7f155c133380 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 >> v1:172.21.15.76:6790/0 conn(0x7f155c09f760 legacy=0x7f155c12bb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 >> v1:172.21.15.76:6789/0 conn(0x7f155c0a1190 legacy=0x7f155c12c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 monclient: found mon.b 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c070620 con 0x7f155c133380 2024-09-16T06:46:06.061 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 5 monclient: authenticate success, global_id 174589 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _renew_subs 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c12c920 con 0x7f155c133380 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _renew_subs 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c0a5020 con 0x7f155c133380 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1554005090 con 0x7f155c133380 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1554005420 con 0x7f155c133380 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:06.062 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154effd700 4 set_mon_vals no callback set 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 10 monclient: dump: 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:06.063 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c005100 con 0x7f155c133380 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.046+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1554005600 con 0x7f155c133380 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.050+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(207..207 src has 151..207) v4 ==== 12794+0+0 (unknown 3297923000 0 0) 0x7f1554045c00 con 0x7f155c133380 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.050+0000 7f15614ac700 10 monclient: get_auth_request con 0x7f1540036180 auth_method 0 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.054+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1554014c40 con 0x7f155c133380 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.054+0000 7f154dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:06.064 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.054+0000 7f154dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.226+0000 7f1563710700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.226+0000 7f1563710700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.226+0000 7f1563710700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.226+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f153c0053b0 con 0x7f155c133380 2024-09-16T06:46:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.810+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v208) v1 ==== 168+0+0 (unknown 2210863037 0 0) 0x7f15540132e0 con 0x7f155c133380 2024-09-16T06:46:06.812 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.810+0000 7f154dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.810+0000 7f154dffb700 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-16T06:46:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.918+0000 7f1563710700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.918+0000 7f1563710700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.918+0000 7f1563710700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:06.923 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.918+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f153c000bc0 con 0x7f155c133380 2024-09-16T06:46:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f154dffb700 1 -- 172.21.15.189:0/4149737178 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v208) v1 ==== 147+0+0 (unknown 3884576257 0 0) 0x7f1554014a60 con 0x7f155c133380 2024-09-16T06:46:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f154dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:06.925 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f154dffb700 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-16T06:46:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1540036180 legacy=0x7f1540038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 10 monclient: shutdown 2024-09-16T06:46:06.927 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 >> v1:172.21.15.189:6789/0 conn(0x7f155c133380 legacy=0x7f155c1d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 shutdown_connections 2024-09-16T06:46:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 >> 172.21.15.189:0/4149737178 conn(0x7f155c0a21f0 msgr2=0x7f155c12ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 shutdown_connections 2024-09-16T06:46:06.928 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:06.922+0000 7f1563710700 1 -- 172.21.15.189:0/4149737178 wait complete. 2024-09-16T06:46:06.929 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h27' already exists 2024-09-16T06:46:06.938 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:06.939 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-16T06:46:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: get_monmap_and_config 2024-09-16T06:46:07.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: build_initial_monmap 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: monmap: 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:07.044981+0000 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:07.044981+0000 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:07.045 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 Processor -- start 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- start start 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: init 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient(hunting): picked mon.noname-c con 0x7fee2c06a8d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient(hunting): picked mon.noname-b con 0x7fee2c0a6090 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient(hunting): picked mon.noname-a con 0x7fee2c13bab0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c06acb0 con 0x7fee2c13bab0 2024-09-16T06:46:07.049 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c099d60 con 0x7fee2c06a8d0 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c09a040 con 0x7fee2c0a6090 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient(hunting): authenticate will time out at 1319.558066s 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee2affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fee2c06a8d0 0x7fee2c0a5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57108/0 (socket says 172.21.15.189:57108) 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee2a7fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fee2c0a6090 legacy=0x7fee2c13a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee2affd700 1 -- 172.21.15.189:0/2501080242 learned_addr learned my addr 172.21.15.189:0/2501080242 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612823178 0 0) 0x7fee2c099d60 con 0x7fee2c06a8d0 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient(hunting): my global_id is 174592 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee180035d0 con 0x7fee2c06a8d0 2024-09-16T06:46:07.050 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704924187 0 0) 0x7fee180035d0 con 0x7fee2c06a8d0 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 >> v1:172.21.15.76:6790/0 conn(0x7fee2c0a6090 legacy=0x7fee2c13a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 >> v1:172.21.15.76:6789/0 conn(0x7fee2c13bab0 legacy=0x7fee2c13dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c09a320 con 0x7fee2c06a8d0 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:07.051 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 5 monclient: authenticate success, global_id 174592 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee1c001610 con 0x7fee2c06a8d0 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: dump: 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.052 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee1c001930 con 0x7fee2c06a8d0 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 1 -- 172.21.15.189:0/2501080242 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee1c001d00 con 0x7fee2c06a8d0 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 4 set_mon_vals no callback set 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee29ffb700 10 monclient: dump: 2024-09-16T06:46:07.053 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.054 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 10 monclient: shutdown 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- 172.21.15.189:0/2501080242 >> v1:172.21.15.189:6789/0 conn(0x7fee2c06a8d0 legacy=0x7fee2c0a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- 172.21.15.189:0/2501080242 shutdown_connections 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.042+0000 7fee31640700 1 -- 172.21.15.189:0/2501080242 >> 172.21.15.189:0/2501080242 conn(0x7fee2c0a7bf0 msgr2=0x7fee2c0a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- 172.21.15.189:0/2501080242 shutdown_connections 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- 172.21.15.189:0/2501080242 wait complete. 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: build_initial_monmap 2024-09-16T06:46:07.055 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: monmap: 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:07.056 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 Processor -- start 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- start start 2024-09-16T06:46:07.057 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: init 2024-09-16T06:46:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:07.065 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): picked mon.noname-c con 0x7fee2c06a8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): picked mon.noname-a con 0x7fee2c0a6090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): picked mon.noname-b con 0x7fee2c13bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0a5750 con 0x7fee2c0a6090 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c06bd10 con 0x7fee2c13bab0 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c131ba0 con 0x7fee2c06a8d0 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:07.066 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): authenticate will time out at 1319.561482s 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2affd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fee2c06a8d0 legacy=0x7fee2c1393f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 0x7fee2c13a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57116/0 (socket says 172.21.15.189:57116) 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 -- 172.21.15.189:0/1608094330 learned_addr learned my addr 172.21.15.189:0/1608094330 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597312597 0 0) 0x7fee2c06bd10 con 0x7fee2c13bab0 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): my global_id is 174595 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c0035d0 con 0x7fee2c13bab0 2024-09-16T06:46:07.067 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775911370 0 0) 0x7fee0c0035d0 con 0x7fee2c13bab0 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee2c06bd10 con 0x7fee2c13bab0 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee24001c70 con 0x7fee2c13bab0 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): dump: 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.068 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678535465 0 0) 0x7fee2c0a5750 con 0x7fee2c0a6090 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): my global_id is 184269 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c0035d0 con 0x7fee2c0a6090 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 217157573 0 0) 0x7fee2c06bd10 con 0x7fee2c13bab0 2024-09-16T06:46:07.069 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.76:6790/0 conn(0x7fee2c06a8d0 legacy=0x7fee2c1393f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.76:6789/0 conn(0x7fee2c0a6090 legacy=0x7fee2c139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 monclient: found mon.b 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c131e80 con 0x7fee2c13bab0 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:07.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 5 monclient: authenticate success, global_id 174595 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _renew_subs 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c1321c0 con 0x7fee2c13bab0 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _renew_subs 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c130590 con 0x7fee2c13bab0 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee24002040 con 0x7fee2c13bab0 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:07.071 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 legacy=0x7fee2c13a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee24003f10 con 0x7fee2c13bab0 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 legacy=0x7fee2c13a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 --1- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 0x7fee2c13a210 :-1 s=OPENED pgs=698 cs=1 l=1).handle_message read tag failed 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2b7fe700 1 --1- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 0x7fee2c13a210 :-1 s=OPENED pgs=698 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: dump: 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.072 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee2c13bab0 legacy=0x7fee2c13a210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee297fa700 4 set_mon_vals no callback set 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): picked mon.c con 0x7fee2c06a8d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): picked mon.b con 0x7fee0c005240 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:07.073 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): picked mon.a con 0x7fee2c0a6090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fee2c06bd10 con 0x7fee2c0a6090 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fee2c131ba0 con 0x7fee0c005240 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fee2c0a5750 con 0x7fee2c06a8d0 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee237fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee2affd700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.76:6790/0 conn(0x7fee2c06a8d0 legacy=0x7fee0c0048a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:07.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.046+0000 7fee31640700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338023341 0 0) 0x7fee2c131ba0 con 0x7fee0c005240 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fee0c0198d0 con 0x7fee0c005240 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631471969 0 0) 0x7fee2c06bd10 con 0x7fee2c0a6090 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fee2c131ba0 con 0x7fee2c0a6090 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3445210346 0 0) 0x7fee0c0198d0 con 0x7fee0c005240 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.76:6790/0 conn(0x7fee2c06a8d0 legacy=0x7fee0c0048a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.76:6789/0 conn(0x7fee2c0a6090 legacy=0x7fee0c016d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 monclient: found mon.b 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fee0c01bab0 con 0x7fee0c005240 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10005100 con 0x7fee0c005240 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c131ba0 con 0x7fee0c005240 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:07.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fee14003f80 con 0x7fee0c005240 2024-09-16T06:46:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: dump: 2024-09-16T06:46:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.078 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.079 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee140041e0 con 0x7fee0c005240 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee28ff9700 4 set_mon_vals no callback set 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fee1400ece0 con 0x7fee0c005240 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 6 ==== osd_map(208..208 src has 151..208) v4 ==== 13007+0+0 (unknown 919035517 0 0) 0x7fee14044ce0 con 0x7fee0c005240 2024-09-16T06:46:07.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.050+0000 7fee2affd700 10 monclient: get_auth_request con 0x7fee0c04a620 auth_method 0 2024-09-16T06:46:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.054+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fee14013b90 con 0x7fee0c005240 2024-09-16T06:46:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.054+0000 7fee237fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.054+0000 7fee237fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.058+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fee140139b0 con 0x7fee0c005240 2024-09-16T06:46:07.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.058+0000 7fee237fe700 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T06:46:07.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.226+0000 7fee31640700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.226+0000 7fee31640700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.226+0000 7fee31640700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.226+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fee10000e80 con 0x7fee0c005240 2024-09-16T06:46:07.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.722+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v209) v1 ==== 168+0+0 (unknown 38468852 0 0) 0x7fee140139b0 con 0x7fee0c005240 2024-09-16T06:46:07.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.722+0000 7fee237fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.725 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.722+0000 7fee237fe700 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-16T06:46:07.835 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.830+0000 7fee31640700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.830+0000 7fee31640700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.830+0000 7fee31640700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.836 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.830+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fee10001b40 con 0x7fee0c005240 2024-09-16T06:46:07.837 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.834+0000 7fee237fe700 1 -- 172.21.15.189:0/1608094330 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v209) v1 ==== 147+0+0 (unknown 1451391861 0 0) 0x7fee140146f0 con 0x7fee0c005240 2024-09-16T06:46:07.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.834+0000 7fee237fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:07.838 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.834+0000 7fee237fe700 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6812/3098810674 conn(0x7fee0c04a620 legacy=0x7fee0c04cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 10 monclient: shutdown 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 >> v1:172.21.15.189:6789/0 conn(0x7fee0c005240 legacy=0x7fee0c00aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 shutdown_connections 2024-09-16T06:46:07.840 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 >> 172.21.15.189:0/1608094330 conn(0x7fee2c0a7bf0 msgr2=0x7fee2c13dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:07.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 shutdown_connections 2024-09-16T06:46:07.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.838+0000 7fee31640700 1 -- 172.21.15.189:0/1608094330 wait complete. 2024-09-16T06:46:07.841 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h28' already exists 2024-09-16T06:46:07.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:07.856 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-16T06:46:07.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: get_monmap_and_config 2024-09-16T06:46:07.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: build_initial_monmap 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: monmap: 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:07.962846+0000 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:07.962846+0000 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:07.963 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 1 Processor -- start 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 1 -- start start 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: init 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-c con 0x7fabb009eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-b con 0x7fabb009f690 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:07.966 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-a con 0x7fabb0139a50 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb009ab80 con 0x7fabb0139a50 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb009a690 con 0x7fabb009eba0 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0072230 con 0x7fabb009f690 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.958+0000 7fabb5a79700 10 monclient(hunting): authenticate will time out at 1320.475859s 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaffff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fabb009f690 legacy=0x7fabb0138330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb4a77700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fabb009eba0 0x7fabb009ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57146/0 (socket says 172.21.15.189:57146) 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb4a77700 1 -- 172.21.15.189:0/4286782960 learned_addr learned my addr 172.21.15.189:0/4286782960 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5888254 0 0) 0x7fabb009a690 con 0x7fabb009eba0 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.967 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient(hunting): my global_id is 174598 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab980035d0 con 0x7fabb009eba0 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573414768 0 0) 0x7fab980035d0 con 0x7fabb009eba0 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 >> v1:172.21.15.76:6790/0 conn(0x7fabb009f690 legacy=0x7fabb0138330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 >> v1:172.21.15.76:6789/0 conn(0x7fabb0139a50 legacy=0x7fabb013be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 monclient: found mon.noname-c 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb009be90 con 0x7fabb009eba0 2024-09-16T06:46:07.968 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 5 monclient: authenticate success, global_id 174598 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba4001610 con 0x7fabb009eba0 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: dump: 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.969 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faba4001930 con 0x7fabb009eba0 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 1 -- 172.21.15.189:0/4286782960 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba4001cc0 con 0x7fabb009eba0 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 4 set_mon_vals no callback set 2024-09-16T06:46:07.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabaf7fe700 10 monclient: dump: 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.971 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: shutdown 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- 172.21.15.189:0/4286782960 >> v1:172.21.15.189:6789/0 conn(0x7fabb009eba0 legacy=0x7fabb009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- 172.21.15.189:0/4286782960 shutdown_connections 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- 172.21.15.189:0/4286782960 >> 172.21.15.189:0/4286782960 conn(0x7fabb009fe70 msgr2=0x7fabb00a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- 172.21.15.189:0/4286782960 shutdown_connections 2024-09-16T06:46:07.972 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- 172.21.15.189:0/4286782960 wait complete. 2024-09-16T06:46:07.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: build_initial_monmap 2024-09-16T06:46:07.976 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: monmap: 2024-09-16T06:46:07.976 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:07.976 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:07.976 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 Processor -- start 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- start start 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: init 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:07.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-c con 0x7fabb009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-a con 0x7fabb009f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient(hunting): picked mon.noname-b con 0x7fabb0139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0092240 con 0x7fabb009f690 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0092800 con 0x7fabb0139a50 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb0092ae0 con 0x7fabb009eba0 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5a79700 10 monclient(hunting): authenticate will time out at 1320.479768s 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb4a77700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fabb009eba0 legacy=0x7fabb008f3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:07.978 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5278700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fabb0139a50 0x7fabb01d1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57148/0 (socket says 172.21.15.189:57148) 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.962+0000 7fabb5278700 1 -- 172.21.15.189:0/771633165 learned_addr learned my addr 172.21.15.189:0/771633165 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324426626 0 0) 0x7fabb0092800 con 0x7fabb0139a50 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): my global_id is 174601 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab9c0035d0 con 0x7fabb0139a50 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019489347 0 0) 0x7fabb0092240 con 0x7fabb009f690 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:07.979 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): my global_id is 184275 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0092800 con 0x7fabb009f690 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811712569 0 0) 0x7fab9c0035d0 con 0x7fabb0139a50 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb0092240 con 0x7fabb0139a50 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba80018f0 con 0x7fabb0139a50 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): dump: 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.980 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 598948364 0 0) 0x7fabb0092240 con 0x7fabb0139a50 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 >> v1:172.21.15.76:6790/0 conn(0x7fabb009eba0 legacy=0x7fabb008f3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 >> v1:172.21.15.76:6789/0 conn(0x7fabb009f690 legacy=0x7fabb008faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:07.981 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 monclient: found mon.b 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0092520 con 0x7fabb0139a50 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 5 monclient: authenticate success, global_id 174601 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _renew_subs 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb0090900 con 0x7fabb0139a50 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _renew_subs 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb00910d0 con 0x7fabb0139a50 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faba80036c0 con 0x7fabb0139a50 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faba80039d0 con 0x7fabb0139a50 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabae7fc700 4 set_mon_vals no callback set 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 10 monclient: dump: 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:07.983 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90005100 con 0x7fabb0139a50 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faba8003b30 con 0x7fabb0139a50 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(209..209 src has 151..209) v4 ==== 13220+0+0 (unknown 1380832251 0 0) 0x7faba8045760 con 0x7fabb0139a50 2024-09-16T06:46:07.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.966+0000 7fabb4a77700 10 monclient: get_auth_request con 0x7fab9c0361d0 auth_method 0 2024-09-16T06:46:07.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.974+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7faba8004330 con 0x7fabb0139a50 2024-09-16T06:46:07.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.974+0000 7fabad7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:07.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:07.974+0000 7fabad7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.150+0000 7fabb5a79700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.150+0000 7fabb5a79700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.150+0000 7fabb5a79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:08.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.150+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fab900053b0 con 0x7fabb0139a50 2024-09-16T06:46:08.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.818+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v210) v1 ==== 168+0+0 (unknown 3463299460 0 0) 0x7faba8012c00 con 0x7fabb0139a50 2024-09-16T06:46:08.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.818+0000 7fabad7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.822 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.818+0000 7fabad7fa700 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-16T06:46:08.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.930+0000 7fabb5a79700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.930+0000 7fabb5a79700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.930+0000 7fabb5a79700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:08.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.930+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fab90000bc0 con 0x7fabb0139a50 2024-09-16T06:46:08.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabad7fa700 1 -- 172.21.15.189:0/771633165 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v210) v1 ==== 147+0+0 (unknown 1952933233 0 0) 0x7faba8014380 con 0x7fabb0139a50 2024-09-16T06:46:08.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabad7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:08.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabad7fa700 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-16T06:46:08.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 >> v1:172.21.15.189:6812/3098810674 conn(0x7fab9c0361d0 legacy=0x7fab9c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 10 monclient: shutdown 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 >> v1:172.21.15.189:6789/0 conn(0x7fabb0139a50 legacy=0x7fabb01d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 shutdown_connections 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 >> 172.21.15.189:0/771633165 conn(0x7fabb009fe70 msgr2=0x7fabb0135390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 shutdown_connections 2024-09-16T06:46:08.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:08.934+0000 7fabb5a79700 1 -- 172.21.15.189:0/771633165 wait complete. 2024-09-16T06:46:08.940 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h29' already exists 2024-09-16T06:46:08.950 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:08.950 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-16T06:46:09.058 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: get_monmap_and_config 2024-09-16T06:46:09.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: build_initial_monmap 2024-09-16T06:46:09.059 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: monmap: 2024-09-16T06:46:09.059 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:09.059 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:09.059 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:09.059003+0000 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:09.059003+0000 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:09.060 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:09.061 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.075 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 1 Processor -- start 2024-09-16T06:46:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 1 -- start start 2024-09-16T06:46:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: init 2024-09-16T06:46:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:09.076 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-a con 0x7f2e6409eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-c con 0x7f2e6409beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-b con 0x7f2e6409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e64099ed0 con 0x7f2e6409eba0 2024-09-16T06:46:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6409d2e0 con 0x7f2e6409beb0 2024-09-16T06:46:09.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6409ef80 con 0x7f2e6409c2e0 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.054+0000 7f2e699da700 10 monclient(hunting): authenticate will time out at 1321.571925s 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e691d9700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2e6409c2e0 legacy=0x7f2e6413ac60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e63fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2e6409beb0 0x7f2e64137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51748/0 (socket says 172.21.15.189:51748) 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e63fff700 1 -- 172.21.15.189:0/2651551480 learned_addr learned my addr 172.21.15.189:0/2651551480 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:09.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032242305 0 0) 0x7f2e6409d2e0 con 0x7f2e6409beb0 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): my global_id is 174604 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e540035d0 con 0x7f2e6409beb0 2024-09-16T06:46:09.079 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010436669 0 0) 0x7f2e64099ed0 con 0x7f2e6409eba0 2024-09-16T06:46:09.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:09.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:09.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:09.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient(hunting): my global_id is 184278 2024-09-16T06:46:09.080 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6409d2e0 con 0x7f2e6409eba0 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002495026 0 0) 0x7f2e540035d0 con 0x7f2e6409beb0 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 >> v1:172.21.15.76:6790/0 conn(0x7f2e6409c2e0 legacy=0x7f2e6413ac60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 >> v1:172.21.15.76:6789/0 conn(0x7f2e6409eba0 legacy=0x7f2e6409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 monclient: found mon.noname-c 2024-09-16T06:46:09.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:09.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6409cc70 con 0x7f2e6409beb0 2024-09-16T06:46:09.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:09.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:09.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 5 monclient: authenticate success, global_id 174604 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e58001610 con 0x7f2e6409beb0 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: dump: 2024-09-16T06:46:09.083 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:09.084 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:09.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e58001830 con 0x7f2e6409beb0 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 1 -- 172.21.15.189:0/2651551480 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e58001b40 con 0x7f2e6409beb0 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e637fe700 10 monclient: dump: 2024-09-16T06:46:09.086 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:09.087 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:09.087 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.087 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.087 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:09.087 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:09.088 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:09.088 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:09.088 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:09.088 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 4 set_mon_vals no callback set 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 10 monclient: shutdown 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 -- 172.21.15.189:0/2651551480 >> v1:172.21.15.189:6789/0 conn(0x7f2e6409beb0 legacy=0x7f2e64137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 -- 172.21.15.189:0/2651551480 shutdown_connections 2024-09-16T06:46:09.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 -- 172.21.15.189:0/2651551480 >> 172.21.15.189:0/2651551480 conn(0x7f2e6409fe70 msgr2=0x7f2e640a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:09.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 -- 172.21.15.189:0/2651551480 shutdown_connections 2024-09-16T06:46:09.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 -- 172.21.15.189:0/2651551480 wait complete. 2024-09-16T06:46:09.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 10 monclient: build_initial_monmap 2024-09-16T06:46:09.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 10 monclient: monmap: 2024-09-16T06:46:09.090 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:09.091 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:09.092 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:09.092 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:09.092 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.058+0000 7f2e699da700 1 Processor -- start 2024-09-16T06:46:09.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- start start 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: init 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-a con 0x7f2e6409beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-c con 0x7f2e6409c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:09.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient(hunting): picked mon.noname-b con 0x7f2e6409eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6409a3f0 con 0x7f2e6409beb0 2024-09-16T06:46:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e641304e0 con 0x7f2e6409eba0 2024-09-16T06:46:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6412f8f0 con 0x7f2e6409c2e0 2024-09-16T06:46:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:09.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient(hunting): authenticate will time out at 1321.576848s 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e63fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f2e6409c2e0 legacy=0x7f2e641326d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e691d9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2e6409eba0 0x7f2e641cfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51756/0 (socket says 172.21.15.189:51756) 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e691d9700 1 -- 172.21.15.189:0/3974791569 learned_addr learned my addr 172.21.15.189:0/3974791569 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016201817 0 0) 0x7f2e641304e0 con 0x7f2e6409eba0 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:09.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:09.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:09.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): my global_id is 174607 2024-09-16T06:46:09.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e480035d0 con 0x7f2e6409eba0 2024-09-16T06:46:09.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150188166 0 0) 0x7f2e6409d2e0 con 0x7f2e6409beb0 2024-09-16T06:46:09.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): my global_id is 184281 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e641304e0 con 0x7f2e6409beb0 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2795827560 0 0) 0x7f2e480035d0 con 0x7f2e6409eba0 2024-09-16T06:46:09.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6409d2e0 con 0x7f2e6409eba0 2024-09-16T06:46:09.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e5c0044f0 con 0x7f2e6409eba0 2024-09-16T06:46:09.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:09.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:09.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): dump: 2024-09-16T06:46:09.098 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:09.099 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:09.100 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:09.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2954065260 0 0) 0x7f2e6409d2e0 con 0x7f2e6409eba0 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 >> v1:172.21.15.76:6790/0 conn(0x7f2e6409c2e0 legacy=0x7f2e641326d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 >> v1:172.21.15.76:6789/0 conn(0x7f2e6409beb0 legacy=0x7f2e641c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 monclient: found mon.b 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:09.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6412fbd0 con 0x7f2e6409eba0 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 5 monclient: authenticate success, global_id 174607 2024-09-16T06:46:09.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _renew_subs 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6412feb0 con 0x7f2e6409eba0 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _renew_subs 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e64132de0 con 0x7f2e6409eba0 2024-09-16T06:46:09.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e5c0047d0 con 0x7f2e6409eba0 2024-09-16T06:46:09.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:09.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e5c004b60 con 0x7f2e6409eba0 2024-09-16T06:46:09.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e62ffd700 4 set_mon_vals no callback set 2024-09-16T06:46:09.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:09.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 10 monclient: dump: 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:09.105 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:09.106 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:09.106 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:09.106 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:09.106 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:09.106 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2e5c004d40 con 0x7f2e6409eba0 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(210..210 src has 151..210) v4 ==== 13433+0+0 (unknown 201659258 0 0) 0x7f2e5c045c00 con 0x7f2e6409eba0 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e689d8700 10 monclient: get_auth_request con 0x7f2e48036180 auth_method 0 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.062+0000 7f2e699da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:09.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.066+0000 7f2e699da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.066+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e44005100 con 0x7f2e6409eba0 2024-09-16T06:46:09.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.070+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2e5c014b10 con 0x7f2e6409eba0 2024-09-16T06:46:09.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.070+0000 7f2e617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:09.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.070+0000 7f2e617fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.242+0000 7f2e699da700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.242+0000 7f2e699da700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.242+0000 7f2e699da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.242+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2e44005930 con 0x7f2e6409eba0 2024-09-16T06:46:09.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.830+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v211) v1 ==== 168+0+0 (unknown 1202377734 0 0) 0x7f2e5c012f80 con 0x7f2e6409eba0 2024-09-16T06:46:09.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.830+0000 7f2e617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.833 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.830+0000 7f2e617fa700 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-16T06:46:09.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.938+0000 7f2e699da700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.938+0000 7f2e699da700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.938+0000 7f2e699da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:09.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.938+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f2e44005b70 con 0x7f2e6409eba0 2024-09-16T06:46:09.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.942+0000 7f2e617fa700 1 -- 172.21.15.189:0/3974791569 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v211) v1 ==== 147+0+0 (unknown 337466768 0 0) 0x7f2e5c014700 con 0x7f2e6409eba0 2024-09-16T06:46:09.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.942+0000 7f2e617fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:09.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.942+0000 7f2e617fa700 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-16T06:46:09.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2e48036180 legacy=0x7f2e48038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 10 monclient: shutdown 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 >> v1:172.21.15.189:6789/0 conn(0x7f2e6409eba0 legacy=0x7f2e641cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 shutdown_connections 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 >> 172.21.15.189:0/3974791569 conn(0x7f2e6409fe70 msgr2=0x7f2e6406f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 shutdown_connections 2024-09-16T06:46:09.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:09.946+0000 7f2e699da700 1 -- 172.21.15.189:0/3974791569 wait complete. 2024-09-16T06:46:09.950 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h30' already exists 2024-09-16T06:46:09.964 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:09.964 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-16T06:46:10.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: get_monmap_and_config 2024-09-16T06:46:10.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: build_initial_monmap 2024-09-16T06:46:10.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: monmap: 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:10.074996+0000 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:10.074996+0000 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:10.075 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 1 Processor -- start 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 1 -- start start 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: init 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-b con 0x7f31b009aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:10.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-a con 0x7f31b00a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b009a040 con 0x7f31b00a3ca0 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b00a33d0 con 0x7f31b00a2350 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b0099d60 con 0x7f31b009aa20 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.070+0000 7f31b58a4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): authenticate will time out at 1322.588009s 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31aeffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31b00a2350 0x7f31b00a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51762/0 (socket says 172.21.15.189:51762) 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31ae7fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f31b009aa20 legacy=0x7f31b013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31aeffd700 1 -- 172.21.15.189:0/3663024724 learned_addr learned my addr 172.21.15.189:0/3663024724 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066848126 0 0) 0x7f31b00a33d0 con 0x7f31b00a2350 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:10.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): my global_id is 174610 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31980035d0 con 0x7f31b00a2350 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997765663 0 0) 0x7f31b009a040 con 0x7f31b00a3ca0 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient(hunting): my global_id is 184284 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b00a33d0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862691652 0 0) 0x7f31980035d0 con 0x7f31b00a2350 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 >> v1:172.21.15.76:6790/0 conn(0x7f31b009aa20 legacy=0x7f31b013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 >> v1:172.21.15.76:6789/0 conn(0x7f31b00a3ca0 legacy=0x7f31b013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b00a3920 con 0x7f31b00a2350 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:10.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31a0001610 con 0x7f31b00a2350 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 5 monclient: authenticate success, global_id 174610 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: dump: 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:10.086 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31a0001830 con 0x7f31b00a2350 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 1 -- 172.21.15.189:0/3663024724 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31a0001bc0 con 0x7f31b00a2350 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 4 set_mon_vals no callback set 2024-09-16T06:46:10.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31adffb700 10 monclient: dump: 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:10.088 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: shutdown 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- 172.21.15.189:0/3663024724 >> v1:172.21.15.189:6789/0 conn(0x7f31b00a2350 legacy=0x7f31b00a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- 172.21.15.189:0/3663024724 shutdown_connections 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- 172.21.15.189:0/3663024724 >> 172.21.15.189:0/3663024724 conn(0x7f31b00a7d70 msgr2=0x7f31b00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- 172.21.15.189:0/3663024724 shutdown_connections 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- 172.21.15.189:0/3663024724 wait complete. 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: build_initial_monmap 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: monmap: 2024-09-16T06:46:10.089 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:10.090 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 Processor -- start 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- start start 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: init 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-a con 0x7f31b009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): picked mon.noname-b con 0x7f31b00a3ca0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b01361b0 con 0x7f31b009aa20 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b006bed0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b0122db0 con 0x7f31b00a2350 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.074+0000 7f31b58a4700 10 monclient(hunting): authenticate will time out at 1322.591988s 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31ae7fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f31b00a2350 legacy=0x7f31b01ceb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31af7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f31b00a3ca0 0x7f31b01d2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51766/0 (socket says 172.21.15.189:51766) 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31af7fe700 1 -- 172.21.15.189:0/1640616035 learned_addr learned my addr 172.21.15.189:0/1640616035 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577783754 0 0) 0x7f31b006bed0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): my global_id is 174613 2024-09-16T06:46:10.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c0035d0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066899134 0 0) 0x7f31b01361b0 con 0x7f31b009aa20 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): my global_id is 184287 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b006bed0 con 0x7f31b009aa20 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566059095 0 0) 0x7f319c0035d0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b01361b0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31a8001d00 con 0x7f31b00a3ca0 2024-09-16T06:46:10.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): dump: 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:10.094 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1177820898 0 0) 0x7f31b01361b0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 >> v1:172.21.15.76:6790/0 conn(0x7f31b00a2350 legacy=0x7f31b01ceb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 >> v1:172.21.15.76:6789/0 conn(0x7f31b009aa20 legacy=0x7f31b01353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 monclient: found mon.b 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b009b610 con 0x7f31b00a3ca0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:10.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 5 monclient: authenticate success, global_id 174613 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _renew_subs 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b0122ad0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _renew_subs 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b0135b00 con 0x7f31b00a3ca0 2024-09-16T06:46:10.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31a8001fe0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f31a8003e90 con 0x7f31b00a3ca0 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 10 monclient: dump: 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:10.098 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f31a8004030 con 0x7f31b00a3ca0 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31ad7fa700 4 set_mon_vals no callback set 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(211..211 src has 151..211) v4 ==== 13646+0+0 (unknown 27205116 0 0) 0x7f31a8045d50 con 0x7f31b00a3ca0 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:10.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b01228d0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.078+0000 7f31aeffd700 10 monclient: get_auth_request con 0x7f319c0362c0 auth_method 0 2024-09-16T06:46:10.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.086+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f31a80042e0 con 0x7f31b00a3ca0 2024-09-16T06:46:10.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.086+0000 7f3197fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:10.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.086+0000 7f3197fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:10.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.258+0000 7f31b58a4700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.258+0000 7f31b58a4700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.258+0000 7f31b58a4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.258+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f31b0135d50 con 0x7f31b00a3ca0 2024-09-16T06:46:10.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.842+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v212) v1 ==== 168+0+0 (unknown 3993693853 0 0) 0x7f31a8013080 con 0x7f31b00a3ca0 2024-09-16T06:46:10.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.842+0000 7f3197fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.842+0000 7f3197fff700 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-16T06:46:10.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f31b58a4700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f31b58a4700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f31b58a4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:10.953 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f31b0066480 con 0x7f31b00a3ca0 2024-09-16T06:46:10.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f3197fff700 1 -- 172.21.15.189:0/1640616035 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v212) v1 ==== 147+0+0 (unknown 1683899462 0 0) 0x7f31a8014800 con 0x7f31b00a3ca0 2024-09-16T06:46:10.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f3197fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:10.956 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.950+0000 7f3197fff700 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 >> v1:172.21.15.189:6812/3098810674 conn(0x7f319c0362c0 legacy=0x7f319c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 10 monclient: shutdown 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 >> v1:172.21.15.189:6789/0 conn(0x7f31b00a3ca0 legacy=0x7f31b01d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 shutdown_connections 2024-09-16T06:46:10.957 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 >> 172.21.15.189:0/1640616035 conn(0x7f31b00a7d70 msgr2=0x7f31b00a03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:10.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 shutdown_connections 2024-09-16T06:46:10.958 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:10.954+0000 7f31b58a4700 1 -- 172.21.15.189:0/1640616035 wait complete. 2024-09-16T06:46:10.959 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h31' already exists 2024-09-16T06:46:10.968 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:10.968 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: get_monmap_and_config 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: build_initial_monmap 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: monmap: 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:11.078 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:11.078527+0000 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:11.078527+0000 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:11.079 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 1 Processor -- start 2024-09-16T06:46:11.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 1 -- start start 2024-09-16T06:46:11.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: init 2024-09-16T06:46:11.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:11.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-c con 0x7f7c8806fbd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-b con 0x7f7c88070c20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-a con 0x7f7c88071710 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88097190 con 0x7f7c88071710 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880976c0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880720d0 con 0x7f7c88070c20 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.074+0000 7f7c8d62b700 10 monclient(hunting): authenticate will time out at 1323.591838s 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c87fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7c8806fbd0 0x7f7c8806ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51768/0 (socket says 172.21.15.189:51768) 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c87fff700 1 -- 172.21.15.189:0/3504856728 learned_addr learned my addr 172.21.15.189:0/3504856728 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:11.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c877fe700 1 -- 172.21.15.189:0/3504856728 >> v1:172.21.15.76:6790/0 conn(0x7f7c88070c20 legacy=0x7f7c8813a770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493973180 0 0) 0x7f7c880976c0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient(hunting): my global_id is 174616 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c740035d0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646309640 0 0) 0x7f7c740035d0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 >> v1:172.21.15.76:6790/0 conn(0x7f7c88070c20 legacy=0x7f7c8813a770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 >> v1:172.21.15.76:6789/0 conn(0x7f7c88071710 legacy=0x7f7c8813dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 monclient: found mon.noname-c 2024-09-16T06:46:11.084 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8809ee10 con 0x7f7c8806fbd0 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 5 monclient: authenticate success, global_id 174616 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c70001610 con 0x7f7c8806fbd0 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: dump: 2024-09-16T06:46:11.085 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70001900 con 0x7f7c8806fbd0 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:11.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 1 -- 172.21.15.189:0/3504856728 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c70001d10 con 0x7f7c8806fbd0 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 4 set_mon_vals no callback set 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c86ffd700 10 monclient: dump: 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.087 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 10 monclient: shutdown 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- 172.21.15.189:0/3504856728 >> v1:172.21.15.189:6789/0 conn(0x7f7c8806fbd0 legacy=0x7f7c8806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:11.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- 172.21.15.189:0/3504856728 shutdown_connections 2024-09-16T06:46:11.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- 172.21.15.189:0/3504856728 >> 172.21.15.189:0/3504856728 conn(0x7f7c8809fe70 msgr2=0x7f7c880a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:11.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- 172.21.15.189:0/3504856728 shutdown_connections 2024-09-16T06:46:11.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- 172.21.15.189:0/3504856728 wait complete. 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 10 monclient: build_initial_monmap 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 10 monclient: monmap: 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:11.092 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 Processor -- start 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.078+0000 7f7c8d62b700 1 -- start start 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: init 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-b con 0x7f7c8806fbd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:11.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-a con 0x7f7c88070c20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient(hunting): picked mon.noname-c con 0x7f7c88071710 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8809a380 con 0x7f7c88070c20 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880a25f0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880a20a0 con 0x7f7c88071710 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient(hunting): authenticate will time out at 1323.596939s 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c87fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7c8806fbd0 0x7f7c88096510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51782/0 (socket says 172.21.15.189:51782) 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c87fff700 1 -- 172.21.15.189:0/1447915096 learned_addr learned my addr 172.21.15.189:0/1447915096 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8ce2a700 1 -- 172.21.15.189:0/1447915096 >> v1:172.21.15.76:6790/0 conn(0x7f7c88071710 legacy=0x7f7c881d2350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871568988 0 0) 0x7f7c880a25f0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:11.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): my global_id is 174619 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c680035d0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406502152 0 0) 0x7f7c680035d0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c880a25f0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c700015d0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.095 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1291154089 0 0) 0x7f7c880a25f0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 >> v1:172.21.15.76:6790/0 conn(0x7f7c88071710 legacy=0x7f7c881d2350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 >> v1:172.21.15.76:6789/0 conn(0x7f7c88070c20 legacy=0x7f7c881cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 monclient: found mon.b 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:11.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88094fe0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 5 monclient: authenticate success, global_id 174619 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _renew_subs 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c88093360 con 0x7f7c8806fbd0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _renew_subs 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88092c10 con 0x7f7c8806fbd0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70004d70 con 0x7f7c8806fbd0 2024-09-16T06:46:11.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c70005100 con 0x7f7c8806fbd0 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c867fc700 4 set_mon_vals no callback set 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 10 monclient: dump: 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:11.098 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7c70005320 con 0x7f7c8806fbd0 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(212..212 src has 151..212) v4 ==== 13859+0+0 (unknown 2522866170 0 0) 0x7f7c70045ef0 con 0x7f7c8806fbd0 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.082+0000 7f7c877fe700 10 monclient: get_auth_request con 0x7f7c680360b0 auth_method 0 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.086+0000 7f7c8d62b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.086+0000 7f7c8d62b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.086+0000 7f7c8d62b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.086+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c005100 con 0x7f7c8806fbd0 2024-09-16T06:46:11.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.090+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7c70014d20 con 0x7f7c8806fbd0 2024-09-16T06:46:11.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.090+0000 7f7c84ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:11.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.090+0000 7f7c84ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.266+0000 7f7c8d62b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.266+0000 7f7c8d62b700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.266+0000 7f7c8d62b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.266+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7c6c005930 con 0x7f7c8806fbd0 2024-09-16T06:46:11.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.846+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v213) v1 ==== 168+0+0 (unknown 3658012695 0 0) 0x7f7c70013190 con 0x7f7c8806fbd0 2024-09-16T06:46:11.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.846+0000 7f7c84ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.846+0000 7f7c84ff9700 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-16T06:46:11.959 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.954+0000 7f7c8d62b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.954+0000 7f7c8d62b700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.954+0000 7f7c8d62b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:11.960 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.954+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7c6c005b70 con 0x7f7c8806fbd0 2024-09-16T06:46:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c84ff9700 1 -- 172.21.15.189:0/1447915096 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v213) v1 ==== 147+0+0 (unknown 1373071768 0 0) 0x7f7c70014910 con 0x7f7c8806fbd0 2024-09-16T06:46:11.961 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c84ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T06:46:11.962 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c84ff9700 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7c680360b0 legacy=0x7f7c68038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c8d62b700 10 monclient: shutdown 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c8d62b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.958+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 >> v1:172.21.15.189:6789/0 conn(0x7f7c8806fbd0 legacy=0x7f7c88096510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.962+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 shutdown_connections 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.962+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 >> 172.21.15.189:0/1447915096 conn(0x7f7c8809fe70 msgr2=0x7f7c8808afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.962+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 shutdown_connections 2024-09-16T06:46:11.964 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:11.962+0000 7f7c8d62b700 1 -- 172.21.15.189:0/1447915096 wait complete. 2024-09-16T06:46:11.965 INFO:tasks.workunit.client.0.smithi189.stderr:bucket 'h32' already exists 2024-09-16T06:46:11.979 INFO:tasks.workunit.client.0.smithi189.stderr:++ seq 1 32 2024-09-16T06:46:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:11.980 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h1 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 10 monclient: get_monmap_and_config 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 10 monclient: build_initial_monmap 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 10 monclient: monmap: 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:12.086557+0000 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:12.086557+0000 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:12.086 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 1 Processor -- start 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 1 -- start start 2024-09-16T06:46:12.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.082+0000 7f15f7ec4700 10 monclient: init 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-c con 0x7f15f00a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-b con 0x7f15f009aa20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-a con 0x7f15f00a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f009a040 con 0x7f15f00a3ca0 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00a33d0 con 0x7f15f00a2350 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0099d60 con 0x7f15f009aa20 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient(hunting): authenticate will time out at 1324.600799s 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f545f700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f15f009aa20 legacy=0x7f15f013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f5c60700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15f00a2350 0x7f15f00a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51796/0 (socket says 172.21.15.189:51796) 2024-09-16T06:46:12.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f5c60700 1 -- 172.21.15.189:0/244184020 learned_addr learned my addr 172.21.15.189:0/244184020 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700297782 0 0) 0x7f15f00a33d0 con 0x7f15f00a2350 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient(hunting): _negotiate 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient(hunting): my global_id is 174622 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15dc0035d0 con 0x7f15f00a2350 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074082209 0 0) 0x7f15dc0035d0 con 0x7f15f00a2350 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 >> v1:172.21.15.76:6790/0 conn(0x7f15f009aa20 legacy=0x7f15f013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 >> v1:172.21.15.76:6789/0 conn(0x7f15f00a3ca0 legacy=0x7f15f013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: _finish_hunting 0 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 monclient: found mon.noname-c 2024-09-16T06:46:12.092 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00a3920 con 0x7f15f00a2350 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: _finish_auth 0 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: _check_auth_tickets 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 5 monclient: authenticate success, global_id 174622 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15ec001610 con 0x7f15f00a2350 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: dump: 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:12.093 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15ec001930 con 0x7f15f00a2350 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 4 set_mon_vals no callback set 2024-09-16T06:46:12.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 1 -- 172.21.15.189:0/244184020 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15ec001cc0 con 0x7f15f00a2350 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f4c5e700 10 monclient: dump: 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:12.095 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: shutdown 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- 172.21.15.189:0/244184020 >> v1:172.21.15.189:6789/0 conn(0x7f15f00a2350 legacy=0x7f15f00a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- 172.21.15.189:0/244184020 shutdown_connections 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- 172.21.15.189:0/244184020 >> 172.21.15.189:0/244184020 conn(0x7f15f00a7d70 msgr2=0x7f15f00a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- 172.21.15.189:0/244184020 shutdown_connections 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- 172.21.15.189:0/244184020 wait complete. 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: build_initial_monmap 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: monmap: 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:12.096 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:12.097 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:12.097 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:12.097 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:12.097 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:12.097 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 Processor -- start 2024-09-16T06:46:12.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 1 -- start start 2024-09-16T06:46:12.099 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.086+0000 7f15f7ec4700 10 monclient: init 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-a con 0x7f15f009aa20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-b con 0x7f15f00a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient(hunting): picked mon.noname-c con 0x7f15f00a3ca0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f009ae60 con 0x7f15f009aa20 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0125ca0 con 0x7f15f00a2350 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0123bb0 con 0x7f15f00a3ca0 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient(hunting): authenticate will time out at 1324.604173s 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f545f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f15f00a2350 0x7f15f0135590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51806/0 (socket says 172.21.15.189:51806) 2024-09-16T06:46:12.100 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f545f700 1 -- 172.21.15.189:0/3330586542 learned_addr learned my addr 172.21.15.189:0/3330586542 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f6461700 1 -- 172.21.15.189:0/3330586542 >> v1:172.21.15.76:6790/0 conn(0x7f15f00a3ca0 legacy=0x7f15f01d1af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763382802 0 0) 0x7f15f0125ca0 con 0x7f15f00a2350 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): my global_id is 174625 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d40035d0 con 0x7f15f00a2350 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3913904489 0 0) 0x7f15d40035d0 con 0x7f15f00a2350 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f0125ca0 con 0x7f15f00a2350 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15e0004280 con 0x7f15f00a2350 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:12.101 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): dump: 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970931003 0 0) 0x7f15f009ae60 con 0x7f15f009aa20 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:12.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient(hunting): my global_id is 184296 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d40035d0 con 0x7f15f009aa20 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3513573577 0 0) 0x7f15f0125ca0 con 0x7f15f00a2350 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 >> v1:172.21.15.76:6790/0 conn(0x7f15f00a3ca0 legacy=0x7f15f01d1af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 >> v1:172.21.15.76:6789/0 conn(0x7f15f009aa20 legacy=0x7f15f01c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 monclient: found mon.b 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f006bed0 con 0x7f15f00a2350 2024-09-16T06:46:12.103 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 5 monclient: authenticate success, global_id 174625 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _renew_subs 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f009b2b0 con 0x7f15f00a2350 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _renew_subs 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f0135ca0 con 0x7f15f00a2350 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e0004530 con 0x7f15f00a2350 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15e00048c0 con 0x7f15f00a2350 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e7fff700 4 set_mon_vals no callback set 2024-09-16T06:46:12.104 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 10 monclient: dump: 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:12.105 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f15e0004aa0 con 0x7f15f00a2350 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d8005100 con 0x7f15f00a2350 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(213..213 src has 151..213) v4 ==== 14072+0+0 (unknown 657283935 0 0) 0x7f15e0045cb0 con 0x7f15f00a2350 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.090+0000 7f15f5c60700 10 monclient: get_auth_request con 0x7f15d4036180 auth_method 0 2024-09-16T06:46:12.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.098+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f15e00052c0 con 0x7f15f00a2350 2024-09-16T06:46:12.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.098+0000 7f15e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:12.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.098+0000 7f15e67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:12.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.274+0000 7f15f7ec4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.274+0000 7f15f7ec4700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.274+0000 7f15f7ec4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.274+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f15d80053b0 con 0x7f15f00a2350 2024-09-16T06:46:12.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.862+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v214) v1 ==== 118+0+0 (unknown 3981619514 0 0) 0x7f15e0012d80 con 0x7f15f00a2350 2024-09-16T06:46:12.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.862+0000 7f15e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.862+0000 7f15e67fc700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-16T06:46:12.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.974+0000 7f15f7ec4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.974+0000 7f15f7ec4700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.974+0000 7f15f7ec4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:12.980 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.974+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f15d8005650 con 0x7f15f00a2350 2024-09-16T06:46:12.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.978+0000 7f15e67fc700 1 -- 172.21.15.189:0/3330586542 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v214) v1 ==== 118+0+0 (unknown 3521241855 0 0) 0x7f15e0014500 con 0x7f15f00a2350 2024-09-16T06:46:12.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.978+0000 7f15e67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T06:46:12.982 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.978+0000 7f15e67fc700 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-16T06:46:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 >> v1:172.21.15.189:6812/3098810674 conn(0x7f15d4036180 legacy=0x7f15d4038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 10 monclient: shutdown 2024-09-16T06:46:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 >> v1:172.21.15.189:6789/0 conn(0x7f15f00a2350 legacy=0x7f15f0135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:12.984 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 shutdown_connections 2024-09-16T06:46:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 >> 172.21.15.189:0/3330586542 conn(0x7f15f00a7d70 msgr2=0x7f15f00a6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 shutdown_connections 2024-09-16T06:46:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:12.982+0000 7f15f7ec4700 1 -- 172.21.15.189:0/3330586542 wait complete. 2024-09-16T06:46:12.985 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h1' does not appear in the crush map 2024-09-16T06:46:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:13.001 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h2 2024-09-16T06:46:13.108 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: get_monmap_and_config 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: build_initial_monmap 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: monmap: 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:13.109266+0000 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:13.109266+0000 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:13.109 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:13.110 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 1 Processor -- start 2024-09-16T06:46:13.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 1 -- start start 2024-09-16T06:46:13.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: init 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c0a1280 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c09ed70 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c09f1b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0665a0 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0a1660 con 0x7f1a2c0a1280 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09f710 con 0x7f1a2c09ed70 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a313e3700 10 monclient(hunting): authenticate will time out at 1325.623544s 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a2affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a2c0a1280 0x7f1a2c09e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51820/0 (socket says 172.21.15.189:51820) 2024-09-16T06:46:13.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a2affd700 1 -- 172.21.15.189:0/1430136622 learned_addr learned my addr 172.21.15.189:0/1430136622 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.106+0000 7f1a2a7fc700 1 -- 172.21.15.189:0/1430136622 >> v1:172.21.15.76:6790/0 conn(0x7f1a2c09ed70 legacy=0x7f1a2c138280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348242028 0 0) 0x7f1a2c0a1660 con 0x7f1a2c0a1280 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): my global_id is 174628 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a140035d0 con 0x7f1a2c0a1280 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794311781 0 0) 0x7f1a2c0665a0 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient(hunting): my global_id is 184299 2024-09-16T06:46:13.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c0a1660 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526106109 0 0) 0x7f1a140035d0 con 0x7f1a2c0a1280 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 >> v1:172.21.15.76:6790/0 conn(0x7f1a2c09ed70 legacy=0x7f1a2c138280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 >> v1:172.21.15.76:6789/0 conn(0x7f1a2c09f1b0 legacy=0x7f1a2c13b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c09f9f0 con 0x7f1a2c0a1280 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a1c001610 con 0x7f1a2c0a1280 2024-09-16T06:46:13.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 5 monclient: authenticate success, global_id 174628 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: dump: 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:13.117 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a1c001880 con 0x7f1a2c0a1280 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 1 -- 172.21.15.189:0/1430136622 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a1c001c10 con 0x7f1a2c0a1280 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 4 set_mon_vals no callback set 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a29ffb700 10 monclient: dump: 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: shutdown 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- 172.21.15.189:0/1430136622 >> v1:172.21.15.189:6789/0 conn(0x7f1a2c0a1280 legacy=0x7f1a2c09e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- 172.21.15.189:0/1430136622 shutdown_connections 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- 172.21.15.189:0/1430136622 >> 172.21.15.189:0/1430136622 conn(0x7f1a2c0a0760 msgr2=0x7f1a2c0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- 172.21.15.189:0/1430136622 shutdown_connections 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- 172.21.15.189:0/1430136622 wait complete. 2024-09-16T06:46:13.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: build_initial_monmap 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: monmap: 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:13.120 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 Processor -- start 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- start start 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: init 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a2c09ed70 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c09f1b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c0a1280 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c12c980 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c12ee80 con 0x7f1a2c09ed70 2024-09-16T06:46:13.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c06b0f0 con 0x7f1a2c0a1280 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a313e3700 10 monclient(hunting): authenticate will time out at 1325.627056s 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a2b7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f1a2c0a1280 legacy=0x7f1a2c1d2030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a2affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1a2c09ed70 0x7f1a2c12bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51836/0 (socket says 172.21.15.189:51836) 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a2affd700 1 -- 172.21.15.189:0/3792822039 learned_addr learned my addr 172.21.15.189:0/3792822039 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183513411 0 0) 0x7f1a2c12ee80 con 0x7f1a2c09ed70 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): my global_id is 174631 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a180035d0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3075646947 0 0) 0x7f1a180035d0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.125 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2c12ee80 con 0x7f1a2c09ed70 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a1c001690 con 0x7f1a2c09ed70 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): dump: 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:13.126 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336928027 0 0) 0x7f1a2c12c980 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 10 monclient(hunting): my global_id is 184302 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.110+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a180035d0 con 0x7f1a2c09f1b0 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 682499740 0 0) 0x7f1a2c12ee80 con 0x7f1a2c09ed70 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 >> v1:172.21.15.76:6790/0 conn(0x7f1a2c0a1280 legacy=0x7f1a2c1d2030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 >> v1:172.21.15.76:6789/0 conn(0x7f1a2c09f1b0 legacy=0x7f1a2c12c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 monclient: found mon.b 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:13.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0a2df0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 5 monclient: authenticate success, global_id 174631 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _renew_subs 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c06fdb0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _renew_subs 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c06f680 con 0x7f1a2c09ed70 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a1c004cd0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:13.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1a1c005060 con 0x7f1a2c09ed70 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a297fa700 4 set_mon_vals no callback set 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 10 monclient: dump: 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:13.129 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c005100 con 0x7f1a2c09ed70 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1a1c005240 con 0x7f1a2c09ed70 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(214..214 src has 151..214) v4 ==== 13862+0+0 (unknown 2246408064 0 0) 0x7f1a1c045e30 con 0x7f1a2c09ed70 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.114+0000 7f1a2a7fc700 10 monclient: get_auth_request con 0x7f1a18036220 auth_method 0 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.122+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1a1c006710 con 0x7f1a2c09ed70 2024-09-16T06:46:13.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.122+0000 7f1a137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:13.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.122+0000 7f1a137fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:13.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.294+0000 7f1a313e3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.294+0000 7f1a313e3700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.294+0000 7f1a313e3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.294+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f1a0c0053b0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.874+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v215) v1 ==== 118+0+0 (unknown 291163445 0 0) 0x7f1a1c013090 con 0x7f1a2c09ed70 2024-09-16T06:46:13.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.874+0000 7f1a137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.874+0000 7f1a137fe700 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-16T06:46:13.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.982+0000 7f1a313e3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.982+0000 7f1a313e3700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.982+0000 7f1a313e3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:13.988 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.982+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f1a0c005650 con 0x7f1a2c09ed70 2024-09-16T06:46:13.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.986+0000 7f1a137fe700 1 -- 172.21.15.189:0/3792822039 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v215) v1 ==== 118+0+0 (unknown 343894350 0 0) 0x7f1a1c0149f0 con 0x7f1a2c09ed70 2024-09-16T06:46:13.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.986+0000 7f1a137fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T06:46:13.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.986+0000 7f1a137fe700 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-16T06:46:13.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1a18036220 legacy=0x7f1a180386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:13.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 10 monclient: shutdown 2024-09-16T06:46:13.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 >> v1:172.21.15.189:6789/0 conn(0x7f1a2c09ed70 legacy=0x7f1a2c12bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 shutdown_connections 2024-09-16T06:46:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 >> 172.21.15.189:0/3792822039 conn(0x7f1a2c0a0760 msgr2=0x7f1a2c132ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 shutdown_connections 2024-09-16T06:46:13.993 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:13.990+0000 7f1a313e3700 1 -- 172.21.15.189:0/3792822039 wait complete. 2024-09-16T06:46:13.994 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h2' does not appear in the crush map 2024-09-16T06:46:14.003 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:14.003 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h3 2024-09-16T06:46:14.106 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 10 monclient: get_monmap_and_config 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 10 monclient: build_initial_monmap 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 10 monclient: monmap: 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:14.107261+0000 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:14.107261+0000 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:14.107 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:14.108 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:14.108 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 1 Processor -- start 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 1 -- start start 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.102+0000 7fe1ce211700 10 monclient: init 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-b con 0x7fe1c809eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-a con 0x7fe1c809f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-c con 0x7fe1c8139a50 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:14.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c809ab80 con 0x7fe1c809f690 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c809a690 con 0x7fe1c8139a50 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c8072230 con 0x7fe1c809eba0 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient(hunting): authenticate will time out at 1326.620436s 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1cd20f700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fe1c809eba0 legacy=0x7fe1c809ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1cda10700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1c8139a50 0x7fe1c813be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51838/0 (socket says 172.21.15.189:51838) 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1cda10700 1 -- 172.21.15.189:0/2571061413 learned_addr learned my addr 172.21.15.189:0/2571061413 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502008565 0 0) 0x7fe1c809a690 con 0x7fe1c8139a50 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): my global_id is 174634 2024-09-16T06:46:14.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b40035d0 con 0x7fe1c8139a50 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043609897 0 0) 0x7fe1c809ab80 con 0x7fe1c809f690 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient(hunting): my global_id is 184305 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c809a690 con 0x7fe1c809f690 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757168099 0 0) 0x7fe1b40035d0 con 0x7fe1c8139a50 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 >> v1:172.21.15.76:6790/0 conn(0x7fe1c809eba0 legacy=0x7fe1c809ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:14.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 >> v1:172.21.15.76:6789/0 conn(0x7fe1c809f690 legacy=0x7fe1c8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 monclient: found mon.noname-c 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c809be90 con 0x7fe1c8139a50 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: _finish_auth 0 2024-09-16T06:46:14.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1c0001990 con 0x7fe1c8139a50 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 5 monclient: authenticate success, global_id 174634 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: dump: 2024-09-16T06:46:14.115 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:14.116 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c0001c70 con 0x7fe1c8139a50 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 1 -- 172.21.15.189:0/2571061413 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1c0002000 con 0x7fe1c8139a50 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:14.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 4 set_mon_vals no callback set 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1bffff700 10 monclient: dump: 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:14.118 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: shutdown 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- 172.21.15.189:0/2571061413 >> v1:172.21.15.189:6789/0 conn(0x7fe1c8139a50 legacy=0x7fe1c813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:14.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- 172.21.15.189:0/2571061413 shutdown_connections 2024-09-16T06:46:14.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- 172.21.15.189:0/2571061413 >> 172.21.15.189:0/2571061413 conn(0x7fe1c809fe70 msgr2=0x7fe1c80a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:14.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- 172.21.15.189:0/2571061413 shutdown_connections 2024-09-16T06:46:14.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 1 -- 172.21.15.189:0/2571061413 wait complete. 2024-09-16T06:46:14.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: build_initial_monmap 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.106+0000 7fe1ce211700 10 monclient: monmap: 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:14.128 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 Processor -- start 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- start start 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: init 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:14.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-b con 0x7fe1c809eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-a con 0x7fe1c809f690 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient(hunting): picked mon.noname-c con 0x7fe1c8139a50 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c8090690 con 0x7fe1c809f690 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c8091090 con 0x7fe1c809eba0 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80929f0 con 0x7fe1c8139a50 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1cd20f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe1c809eba0 0x7fe1c808ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51846/0 (socket says 172.21.15.189:51846) 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1cd20f700 1 -- 172.21.15.189:0/137106393 learned_addr learned my addr 172.21.15.189:0/137106393 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient(hunting): authenticate will time out at 1326.625664s 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1cda10700 1 -- 172.21.15.189:0/137106393 >> v1:172.21.15.76:6790/0 conn(0x7fe1c8139a50 legacy=0x7fe1c81cfc20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:14.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949713665 0 0) 0x7fe1c8091090 con 0x7fe1c809eba0 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): my global_id is 174637 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ac0035d0 con 0x7fe1c809eba0 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942318056 0 0) 0x7fe1c8090690 con 0x7fe1c809f690 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): my global_id is 184308 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c8091090 con 0x7fe1c809f690 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3472637540 0 0) 0x7fe1ac0035d0 con 0x7fe1c809eba0 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c8090690 con 0x7fe1c809eba0 2024-09-16T06:46:14.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1c4004230 con 0x7fe1c809eba0 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): dump: 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:14.132 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3656444316 0 0) 0x7fe1c8090690 con 0x7fe1c809eba0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 >> v1:172.21.15.76:6790/0 conn(0x7fe1c8139a50 legacy=0x7fe1c81cfc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 >> v1:172.21.15.76:6789/0 conn(0x7fe1c809f690 legacy=0x7fe1c808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 monclient: found mon.b 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c806b750 con 0x7fe1c809eba0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 5 monclient: authenticate success, global_id 174637 2024-09-16T06:46:14.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _renew_subs 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c809a3e0 con 0x7fe1c809eba0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _renew_subs 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.110+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c8091e20 con 0x7fe1c809eba0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1ce211700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1ce211700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1ce211700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c80713c0 con 0x7fe1c809eba0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c4004450 con 0x7fe1c809eba0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1c40047e0 con 0x7fe1c809eba0 2024-09-16T06:46:14.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bf7fe700 4 set_mon_vals no callback set 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 10 monclient: dump: 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:14.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe1c40049c0 con 0x7fe1c809eba0 2024-09-16T06:46:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(215..215 src has 151..215) v4 ==== 13652+0+0 (unknown 768854249 0 0) 0x7fe1c40458a0 con 0x7fe1c809eba0 2024-09-16T06:46:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.114+0000 7fe1cca0e700 10 monclient: get_auth_request con 0x7fe1ac0361d0 auth_method 0 2024-09-16T06:46:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.122+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe1c40144d0 con 0x7fe1c809eba0 2024-09-16T06:46:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.122+0000 7fe1bdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:14.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.122+0000 7fe1bdffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:14.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.294+0000 7fe1ce211700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:14.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.294+0000 7fe1ce211700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:14.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.294+0000 7fe1ce211700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.294+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7fe1c80677c0 con 0x7fe1c809eba0 2024-09-16T06:46:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.882+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v216) v1 ==== 118+0+0 (unknown 1940167445 0 0) 0x7fe1c4012b70 con 0x7fe1c809eba0 2024-09-16T06:46:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.882+0000 7fe1bdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.882+0000 7fe1bdffb700 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-16T06:46:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.994+0000 7fe1ce211700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.994+0000 7fe1ce211700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.994+0000 7fe1ce211700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:14.998 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.994+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7fe1c808fad0 con 0x7fe1c809eba0 2024-09-16T06:46:15.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1bdffb700 1 -- 172.21.15.189:0/137106393 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v216) v1 ==== 118+0+0 (unknown 1080936637 0 0) 0x7fe1c40142f0 con 0x7fe1c809eba0 2024-09-16T06:46:15.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1bdffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T06:46:15.000 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1bdffb700 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-16T06:46:15.002 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe1ac0361d0 legacy=0x7fe1ac038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 10 monclient: shutdown 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 >> v1:172.21.15.189:6789/0 conn(0x7fe1c809eba0 legacy=0x7fe1c808ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 shutdown_connections 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 >> 172.21.15.189:0/137106393 conn(0x7fe1c809fe70 msgr2=0x7fe1c806bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 shutdown_connections 2024-09-16T06:46:15.003 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:14.998+0000 7fe1ce211700 1 -- 172.21.15.189:0/137106393 wait complete. 2024-09-16T06:46:15.004 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h3' does not appear in the crush map 2024-09-16T06:46:15.013 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:15.013 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h4 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: get_monmap_and_config 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: build_initial_monmap 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: monmap: 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:15.113499+0000 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:15.113499+0000 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:15.113 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:15.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:15.114 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:15.114 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 1 Processor -- start 2024-09-16T06:46:15.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 1 -- start start 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: init 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-b con 0x7f593009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-c con 0x7f593009c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-a con 0x7f5930138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593009cca0 con 0x7f5930138da0 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593009f750 con 0x7f593009c5f0 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930097620 con 0x7f593009bb00 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 10 monclient(hunting): authenticate will time out at 1327.626569s 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f593009c5f0 0x7f5930137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51854/0 (socket says 172.21.15.189:51854) 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592effd700 1 -- 172.21.15.189:0/20932192 learned_addr learned my addr 172.21.15.189:0/20932192 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:15.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592f7fe700 1 -- 172.21.15.189:0/20932192 >> v1:172.21.15.76:6790/0 conn(0x7f593009bb00 legacy=0x7f593009bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006410753 0 0) 0x7f593009f750 con 0x7f593009c5f0 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): my global_id is 174640 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59180035d0 con 0x7f593009c5f0 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911501354 0 0) 0x7f593009cca0 con 0x7f5930138da0 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient(hunting): my global_id is 184311 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f593009f750 con 0x7f5930138da0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497485982 0 0) 0x7f59180035d0 con 0x7f593009c5f0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 >> v1:172.21.15.76:6790/0 conn(0x7f593009bb00 legacy=0x7f593009bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 >> v1:172.21.15.76:6789/0 conn(0x7f5930138da0 legacy=0x7f593013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930097900 con 0x7f593009c5f0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:15.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 5 monclient: authenticate success, global_id 174640 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5920001990 con 0x7f593009c5f0 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f5935ff1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.110+0000 7f592e7fc700 10 monclient: dump: 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.121 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5920001c70 con 0x7f593009c5f0 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 1 -- 172.21.15.189:0/20932192 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5920002000 con 0x7f593009c5f0 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 4 set_mon_vals no callback set 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592e7fc700 10 monclient: dump: 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: shutdown 2024-09-16T06:46:15.123 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:15.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- 172.21.15.189:0/20932192 >> v1:172.21.15.189:6789/0 conn(0x7f593009c5f0 legacy=0x7f5930137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- 172.21.15.189:0/20932192 shutdown_connections 2024-09-16T06:46:15.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- 172.21.15.189:0/20932192 >> 172.21.15.189:0/20932192 conn(0x7f59300a03e0 msgr2=0x7f59300a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:15.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- 172.21.15.189:0/20932192 shutdown_connections 2024-09-16T06:46:15.124 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- 172.21.15.189:0/20932192 wait complete. 2024-09-16T06:46:15.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: build_initial_monmap 2024-09-16T06:46:15.127 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: monmap: 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:15.128 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 Processor -- start 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- start start 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: init 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-c con 0x7f593009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-a con 0x7f593009c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient(hunting): picked mon.noname-b con 0x7f5930138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930071490 con 0x7f593009c5f0 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59300a2390 con 0x7f5930138da0 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5930094600 con 0x7f593009bb00 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:15.129 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5935ff1700 10 monclient(hunting): authenticate will time out at 1327.631028s 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592f7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f593009bb00 legacy=0x7f59301ca1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5930138da0 0x7f59301d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51870/0 (socket says 172.21.15.189:51870) 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f592ffff700 1 -- 172.21.15.189:0/3816397862 learned_addr learned my addr 172.21.15.189:0/3816397862 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702415781 0 0) 0x7f59300a2390 con 0x7f5930138da0 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): my global_id is 174643 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f591c0035d0 con 0x7f5930138da0 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335076344 0 0) 0x7f5930097900 con 0x7f593009c5f0 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 10 monclient(hunting): my global_id is 184314 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.114+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59300a2390 con 0x7f593009c5f0 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535582819 0 0) 0x7f591c0035d0 con 0x7f5930138da0 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5930097900 con 0x7f5930138da0 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59280018b0 con 0x7f5930138da0 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): dump: 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.131 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580143721 0 0) 0x7f59300a2390 con 0x7f593009c5f0 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f591c0035d0 con 0x7f593009c5f0 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f59200017b0 con 0x7f593009c5f0 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient(hunting): dump: 2024-09-16T06:46:15.132 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093233189 0 0) 0x7f5930097900 con 0x7f5930138da0 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 >> v1:172.21.15.76:6790/0 conn(0x7f593009bb00 legacy=0x7f59301ca1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 >> v1:172.21.15.76:6789/0 conn(0x7f593009c5f0 legacy=0x7f59301cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:15.133 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 monclient: found mon.b 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59300948e0 con 0x7f5930138da0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 5 monclient: authenticate success, global_id 174643 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _renew_subs 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593009a6e0 con 0x7f5930138da0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _renew_subs 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5928001c60 con 0x7f5930138da0 2024-09-16T06:46:15.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5930093d80 con 0x7f5930138da0 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5928003b10 con 0x7f5930138da0 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 10 monclient: dump: 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.135 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f592dffb700 4 set_mon_vals no callback set 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5928003d30 con 0x7f5930138da0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(216..216 src has 151..216) v4 ==== 13442+0+0 (unknown 3537954340 0 0) 0x7f59280458f0 con 0x7f5930138da0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f592f7fe700 10 monclient: get_auth_request con 0x7f591c0362c0 auth_method 0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.118+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5910005100 con 0x7f5930138da0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.122+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5928014800 con 0x7f5930138da0 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.122+0000 7f5917fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.122+0000 7f5917fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:15.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.298+0000 7f5935ff1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:15.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.298+0000 7f5935ff1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:15.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.298+0000 7f5935ff1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:15.302 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.298+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7f59100053b0 con 0x7f5930138da0 2024-09-16T06:46:15.842 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:46:15.843 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:46:15.843 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:46:15.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:46:15.845 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:46:15.846 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:46:15.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.886+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v217) v1 ==== 118+0+0 (unknown 225563191 0 0) 0x7f5928012c70 con 0x7f5930138da0 2024-09-16T06:46:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.886+0000 7f5917fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:15.889 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:15.886+0000 7f5917fff700 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-16T06:46:15.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: build_initial_monmap 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: monmap: 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:15.960195+0000 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:15.960195+0000 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 Processor -- start 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- start start 2024-09-16T06:46:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: init 2024-09-16T06:46:15.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:15.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:15.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f9e0c0964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:15.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f9e0c09b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f9e0c137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c098560 con 0x7f9e0c0964f0 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c099ec0 con 0x7f9e0c137e70 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c09b5f0 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient(hunting): authenticate will time out at 1323.093417s 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9e0c0964f0 0x7f9e0c0968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49468/0 (socket says 172.21.15.76:49468) 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0a7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9e0c09b1c0 0x7f9e0c136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33940/0 (socket says 172.21.15.76:33940) 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0affd700 1 -- 172.21.15.76:0/740444914 learned_addr learned my addr 172.21.15.76:0/740444914 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192064980 0 0) 0x7f9e0c098560 con 0x7f9e0c0964f0 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): my global_id is 184317 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df40057e0 con 0x7f9e0c0964f0 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754141001 0 0) 0x7f9e0c099ec0 con 0x7f9e0c137e70 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient(hunting): my global_id is 174646 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c098560 con 0x7f9e0c137e70 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408184780 0 0) 0x7f9df40057e0 con 0x7f9e0c0964f0 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 >> v1:172.21.15.76:6790/0 conn(0x7f9e0c09b1c0 legacy=0x7f9e0c136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 >> v1:172.21.15.189:6789/0 conn(0x7f9e0c137e70 legacy=0x7f9e0c13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 monclient: found mon.noname-a 2024-09-16T06:46:15.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c099be0 con 0x7f9e0c0964f0 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e00001610 con 0x7f9e0c0964f0 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: dump: 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.967 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 5 monclient: authenticate success, global_id 184317 2024-09-16T06:46:15.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:15.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e000018f0 con 0x7f9e0c0964f0 2024-09-16T06:46:15.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:15.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 1 -- 172.21.15.76:0/740444914 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9e00001c80 con 0x7f9e0c0964f0 2024-09-16T06:46:15.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 4 set_mon_vals no callback set 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e09ffb700 10 monclient: dump: 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.970 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 10 monclient: shutdown 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- 172.21.15.76:0/740444914 >> v1:172.21.15.76:6789/0 conn(0x7f9e0c0964f0 legacy=0x7f9e0c0968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- 172.21.15.76:0/740444914 shutdown_connections 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- 172.21.15.76:0/740444914 >> 172.21.15.76:0/740444914 conn(0x7f9e0c0a0030 msgr2=0x7f9e0c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- 172.21.15.76:0/740444914 shutdown_connections 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.956+0000 7f9e0bfff700 1 -- 172.21.15.76:0/740444914 wait complete. 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: build_initial_monmap 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: monmap: 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:15.971 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 Processor -- start 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- start start 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: init 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:15.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f9e0c0964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f9e0c09b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f9e0c137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c093a80 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9e0c09b1c0 0x7f9e0c1cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49484/0 (socket says 172.21.15.76:49484) 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0a7fc700 1 -- 172.21.15.76:0/643027537 learned_addr learned my addr 172.21.15.76:0/643027537 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c098230 con 0x7f9e0c0964f0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c09a540 con 0x7f9e0c137e70 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient(hunting): authenticate will time out at 1323.097949s 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932364004 0 0) 0x7f9e0c093a80 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): my global_id is 184320 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df80035d0 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766908368 0 0) 0x7f9e0c098230 con 0x7f9e0c0964f0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): my global_id is 174649 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c093a80 con 0x7f9e0c0964f0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653588853 0 0) 0x7f9df80035d0 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e0c098230 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9dfc001860 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): dump: 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1205998304 0 0) 0x7f9e0c098230 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 >> v1:172.21.15.76:6790/0 conn(0x7f9e0c137e70 legacy=0x7f9e0c1d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 >> v1:172.21.15.189:6789/0 conn(0x7f9e0c0964f0 legacy=0x7f9e0c1ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 monclient: found mon.a 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c097de0 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 5 monclient: authenticate success, global_id 184320 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _renew_subs 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:15.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c091e00 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _renew_subs 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c091730 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dfc004890 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9dfc004c20 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9df37fe700 10 monclient: dump: 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:15.977 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.960+0000 7f9e097fa700 4 set_mon_vals no callback set 2024-09-16T06:46:15.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9dfc004e00 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9e0bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7f9dfc045820 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9e0bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9e0bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dec005100 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.964+0000 7f9e0affd700 10 monclient: get_auth_request con 0x7f9df8035ff0 auth_method 0 2024-09-16T06:46:15.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.968+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9dfc014800 con 0x7f9e0c09b1c0 2024-09-16T06:46:15.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.968+0000 7f9df37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:15.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:15.968+0000 7f9df37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:16.004 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.002+0000 7f5935ff1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.002+0000 7f5935ff1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.002+0000 7f5935ff1700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:16.005 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.002+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7f59100023c0 con 0x7f5930138da0 2024-09-16T06:46:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.006+0000 7f5917fff700 1 -- 172.21.15.189:0/3816397862 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v217) v1 ==== 118+0+0 (unknown 2282808934 0 0) 0x7f59280145d0 con 0x7f5930138da0 2024-09-16T06:46:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.006+0000 7f5917fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T06:46:16.010 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.006+0000 7f5917fff700 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 >> v1:172.21.15.189:6812/3098810674 conn(0x7f591c0362c0 legacy=0x7f591c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 10 monclient: shutdown 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 >> v1:172.21.15.189:6789/0 conn(0x7f5930138da0 legacy=0x7f59301d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 shutdown_connections 2024-09-16T06:46:16.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 >> 172.21.15.189:0/3816397862 conn(0x7f59300a03e0 msgr2=0x7f5930070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 shutdown_connections 2024-09-16T06:46:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.010+0000 7f5935ff1700 1 -- 172.21.15.189:0/3816397862 wait complete. 2024-09-16T06:46:16.014 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h4' does not appear in the crush map 2024-09-16T06:46:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:16.024 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h5 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.122+0000 7f00d3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.122+0000 7f00d3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.122+0000 7f00d3fff700 10 monclient: monmap: 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:16.128274+0000 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:16.128274+0000 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.128 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:16.129 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:16.129 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 Processor -- start 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- start start 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: init 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f00d406fbd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f00d4070c20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f00d4071710 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d4097190 con 0x7f00d406fbd0 2024-09-16T06:46:16.140 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40976c0 con 0x7f00d4071710 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40720d0 con 0x7f00d4070c20 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient(hunting): authenticate will time out at 1328.641043s 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00d4071710 0x7f00d413dea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51872/0 (socket says 172.21.15.189:51872) 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d37fe700 1 -- 172.21.15.189:0/567846647 learned_addr learned my addr 172.21.15.189:0/567846647 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14247485 0 0) 0x7f00d40976c0 con 0x7f00d4071710 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): my global_id is 174652 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c00035d0 con 0x7f00d4071710 2024-09-16T06:46:16.141 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926857260 0 0) 0x7f00d40720d0 con 0x7f00d4070c20 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): my global_id is 194099 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d40976c0 con 0x7f00d4070c20 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705135508 0 0) 0x7f00d4097190 con 0x7f00d406fbd0 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient(hunting): my global_id is 184323 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d40720d0 con 0x7f00d406fbd0 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269593579 0 0) 0x7f00c00035d0 con 0x7f00d4071710 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 >> v1:172.21.15.76:6790/0 conn(0x7f00d4070c20 legacy=0x7f00d413a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.142 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 >> v1:172.21.15.76:6789/0 conn(0x7f00d406fbd0 legacy=0x7f00d406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d409ee10 con 0x7f00d4071710 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 5 monclient: authenticate success, global_id 174652 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00cc001610 con 0x7f00d4071710 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:16.143 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: dump: 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.144 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00cc0018f0 con 0x7f00d4071710 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 4 set_mon_vals no callback set 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 1 -- 172.21.15.189:0/567846647 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00cc001c80 con 0x7f00d4071710 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d1ffb700 10 monclient: dump: 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.145 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: shutdown 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- 172.21.15.189:0/567846647 >> v1:172.21.15.189:6789/0 conn(0x7f00d4071710 legacy=0x7f00d413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- 172.21.15.189:0/567846647 shutdown_connections 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- 172.21.15.189:0/567846647 >> 172.21.15.189:0/567846647 conn(0x7f00d409fe70 msgr2=0x7f00d40a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- 172.21.15.189:0/567846647 shutdown_connections 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 1 -- 172.21.15.189:0/567846647 wait complete. 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.146 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.126+0000 7f00d3fff700 10 monclient: monmap: 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 Processor -- start 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- start start 2024-09-16T06:46:16.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: init 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f00d406fbd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f00d4070c20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f00d4071710 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40a2b80 con 0x7f00d4070c20 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40a26e0 con 0x7f00d4071710 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d409a380 con 0x7f00d406fbd0 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient(hunting): authenticate will time out at 1328.645482s 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00d4071710 0x7f00d41d2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51880/0 (socket says 172.21.15.189:51880) 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d37fe700 1 -- 172.21.15.189:0/2475176034 learned_addr learned my addr 172.21.15.189:0/2475176034 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:16.148 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d27fc700 1 --1- 172.21.15.189:0/2475176034 >> v1:172.21.15.76:6789/0 conn(0x7f00d4070c20 0x7f00d41cedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:34728/0 (socket says 172.21.15.189:34728) 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152234860 0 0) 0x7f00d409a380 con 0x7f00d406fbd0 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): my global_id is 194102 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b40035d0 con 0x7f00d406fbd0 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379835728 0 0) 0x7f00d40a26e0 con 0x7f00d4071710 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): my global_id is 174655 2024-09-16T06:46:16.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d409a380 con 0x7f00d4071710 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282185745 0 0) 0x7f00d40a2b80 con 0x7f00d4070c20 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): my global_id is 184326 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d40a26e0 con 0x7f00d4070c20 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064293309 0 0) 0x7f00b40035d0 con 0x7f00d406fbd0 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d40a2b80 con 0x7f00d406fbd0 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00c4001c30 con 0x7f00d406fbd0 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:16.150 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729985236 0 0) 0x7f00d409a380 con 0x7f00d4071710 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00b40035d0 con 0x7f00d4071710 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00cc001430 con 0x7f00d4071710 2024-09-16T06:46:16.151 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.152 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646561767 0 0) 0x7f00d40a26e0 con 0x7f00d4070c20 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d409a380 con 0x7f00d4070c20 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00bc0017e0 con 0x7f00d4070c20 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.153 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778200742 0 0) 0x7f00d40a2b80 con 0x7f00d406fbd0 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 >> v1:172.21.15.189:6789/0 conn(0x7f00d4071710 legacy=0x7f00d41d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 >> v1:172.21.15.76:6789/0 conn(0x7f00d4070c20 legacy=0x7f00d41cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 monclient: found mon.c 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4091d10 con 0x7f00d406fbd0 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.154 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 5 monclient: authenticate success, global_id 194102 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _renew_subs 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d4091ff0 con 0x7f00d406fbd0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _renew_subs 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d4096c60 con 0x7f00d406fbd0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00c4003a40 con 0x7f00d406fbd0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00c4003d50 con 0x7f00d406fbd0 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.155 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00d17fa700 4 set_mon_vals no callback set 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.130+0000 7f00cb7fe700 10 monclient: dump: 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f00c400e7c0 con 0x7f00d406fbd0 2024-09-16T06:46:16.156 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7f00c4044980 con 0x7f00d406fbd0 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00d3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00d3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b8005100 con 0x7f00d406fbd0 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.134+0000 7f00d27fc700 10 monclient: get_auth_request con 0x7f00b4036560 auth_method 0 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.142+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f00c40136d0 con 0x7f00d406fbd0 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.142+0000 7f00cb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.157 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.142+0000 7f00cb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.200+0000 7f9e0bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.200+0000 7f9e0bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.200+0000 7f9e0bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.200+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9dec005930 con 0x7f9e0c09b1c0 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.204+0000 7f9df37fe700 1 -- 172.21.15.76:0/643027537 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3822127366) 0x7f9dfc01cdf0 con 0x7f9e0c09b1c0 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.204+0000 7f9df37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:16.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.204+0000 7f9df37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:16.206 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:16.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9df8035ff0 legacy=0x7f9df80384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 10 monclient: shutdown 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 >> v1:172.21.15.76:6789/0 conn(0x7f9e0c09b1c0 legacy=0x7f9e0c1cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 shutdown_connections 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 >> 172.21.15.76:0/643027537 conn(0x7f9e0c0a0030 msgr2=0x7f9e0c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 shutdown_connections 2024-09-16T06:46:16.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.208+0000 7f9e0bfff700 1 -- 172.21.15.76:0/643027537 wait complete. 2024-09-16T06:46:16.225 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:46:16.225 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:46:16.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:46:16.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.318+0000 7f00d3fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:16.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.318+0000 7f00d3fff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:16.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.318+0000 7f00d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:16.318+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f00b80053b0 con 0x7f00d406fbd0 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.332+0000 7f6202d37700 10 monclient: get_monmap_and_config 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.332+0000 7f6202d37700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.332+0000 7f6202d37700 10 monclient: monmap: 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.336 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:16.339326+0000 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:16.339326+0000 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:16.337 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 Processor -- start 2024-09-16T06:46:16.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- start start 2024-09-16T06:46:16.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient: init 2024-09-16T06:46:16.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:16.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-a con 0x7f61fc12cb10 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-c con 0x7f61fc09ace0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-b con 0x7f61fc0a0320 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc072660 con 0x7f61fc12cb10 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc12d600 con 0x7f61fc09ace0 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc073bd0 con 0x7f61fc0a0320 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient(hunting): authenticate will time out at 1323.473744s 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6200ad3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f61fc12cb10 0x7f61fc12cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49486/0 (socket says 172.21.15.76:49486) 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6200ad3700 1 -- 172.21.15.76:0/3636538772 learned_addr learned my addr 172.21.15.76:0/3636538772 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f62012d4700 1 --1- 172.21.15.76:0/3636538772 >> v1:172.21.15.76:6790/0 conn(0x7f61fc0a0320 0x7f61fc13b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33954/0 (socket says 172.21.15.76:33954) 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020896827 0 0) 0x7f61fc073bd0 con 0x7f61fc0a0320 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): my global_id is 194105 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e40035d0 con 0x7f61fc0a0320 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634835091 0 0) 0x7f61fc072660 con 0x7f61fc12cb10 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): my global_id is 184329 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc073bd0 con 0x7f61fc12cb10 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668277081 0 0) 0x7f61fc12d600 con 0x7f61fc09ace0 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient(hunting): my global_id is 174658 2024-09-16T06:46:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc072660 con 0x7f61fc09ace0 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487150425 0 0) 0x7f61e40035d0 con 0x7f61fc0a0320 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 >> v1:172.21.15.189:6789/0 conn(0x7f61fc09ace0 legacy=0x7f61fc138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 >> v1:172.21.15.76:6789/0 conn(0x7f61fc12cb10 legacy=0x7f61fc12cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 monclient: found mon.noname-b 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc09dd80 con 0x7f61fc0a0320 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 5 monclient: authenticate success, global_id 194105 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61f40019d0 con 0x7f61fc0a0320 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: dump: 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.352 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61f4001c60 con 0x7f61fc0a0320 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 1 -- 172.21.15.76:0/3636538772 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61f4001ff0 con 0x7f61fc0a0320 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 4 set_mon_vals no callback set 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f61fb7fe700 10 monclient: dump: 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 10 monclient: shutdown 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- 172.21.15.76:0/3636538772 >> v1:172.21.15.76:6790/0 conn(0x7f61fc0a0320 legacy=0x7f61fc13b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- 172.21.15.76:0/3636538772 shutdown_connections 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- 172.21.15.76:0/3636538772 >> 172.21.15.76:0/3636538772 conn(0x7f61fc0a0b50 msgr2=0x7f61fc0a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- 172.21.15.76:0/3636538772 shutdown_connections 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.336+0000 7f6202d37700 1 -- 172.21.15.76:0/3636538772 wait complete. 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: monmap: 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:16.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 Processor -- start 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- start start 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: init 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-c con 0x7f61fc09ace0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-b con 0x7f61fc0a0320 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient(hunting): picked mon.noname-a con 0x7f61fc12cb10 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc137bd0 con 0x7f61fc12cb10 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0a3c10 con 0x7f61fc0a0320 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61fc0a36c0 con 0x7f61fc09ace0 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient(hunting): authenticate will time out at 1323.477680s 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6200ad3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f61fc09ace0 0x7f61fc12be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33960/0 (socket says 172.21.15.76:33960) 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6200ad3700 1 -- 172.21.15.76:0/434979732 learned_addr learned my addr 172.21.15.76:0/434979732 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279185365 0 0) 0x7f61fc0a36c0 con 0x7f61fc09ace0 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): my global_id is 194108 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e80035d0 con 0x7f61fc09ace0 2024-09-16T06:46:16.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437987509 0 0) 0x7f61fc137bd0 con 0x7f61fc12cb10 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): my global_id is 184332 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc0a36c0 con 0x7f61fc12cb10 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030019814 0 0) 0x7f61e80035d0 con 0x7f61fc09ace0 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61fc137bd0 con 0x7f61fc09ace0 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61ec001870 con 0x7f61fc09ace0 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): dump: 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.358 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3033707224 0 0) 0x7f61fc0a36c0 con 0x7f61fc12cb10 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61e80035d0 con 0x7f61fc12cb10 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61f4001990 con 0x7f61fc12cb10 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): dump: 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211725240 0 0) 0x7f61fc0a3c10 con 0x7f61fc0a0320 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient(hunting): my global_id is 174661 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc0a36c0 con 0x7f61fc0a0320 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2102937947 0 0) 0x7f61fc137bd0 con 0x7f61fc09ace0 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.189:6789/0 conn(0x7f61fc0a0320 legacy=0x7f61fc1cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6789/0 conn(0x7f61fc12cb10 legacy=0x7f61fc1d2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 monclient: found mon.c 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc09b240 con 0x7f61fc09ace0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 5 monclient: authenticate success, global_id 194108 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _renew_subs 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc137eb0 con 0x7f61fc09ace0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _renew_subs 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc136430 con 0x7f61fc09ace0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ec004890 con 0x7f61fc09ace0 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61ec004ba0 con 0x7f61fc09ace0 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61faffd700 4 set_mon_vals no callback set 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f61f97fa700 10 monclient: dump: 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.340+0000 7f6202d37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61dc006590 con 0x7f61dc002830 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f61ec00f610 con 0x7f61fc09ace0 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7f61ec045810 con 0x7f61fc09ace0 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859632064 0 0) 0x7f61dc006590 con 0x7f61dc002830 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f62012d4700 10 monclient: get_auth_request con 0x7f61e8036490 auth_method 0 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient(hunting): my global_id is 184335 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc137bd0 con 0x7f61dc002830 2024-09-16T06:46:16.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3234561384 0 0) 0x7f61fc137bd0 con 0x7f61dc002830 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61dc006590 con 0x7f61dc002830 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61f0001760 con 0x7f61dc002830 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient: dump: 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 935587962 0 0) 0x7f61dc006590 con 0x7f61dc002830 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61dc00a190 con 0x7f61dc002830 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f61f0001b50 con 0x7f61dc002830 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6789/0 conn(0x7f61dc002830 legacy=0x7f61dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.344+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6789/0 conn(0x7f61dc002830 legacy=0x7f61dc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.352+0000 7f6202d37700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.352+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f61dc009e30 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866025208 0 0) 0x7f61dc009e30 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient(hunting): my global_id is 184338 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61dc006590 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 541342390 0 0) 0x7f61dc006590 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61dc009e30 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f61f0003a30 con 0x7f61dc009150 2024-09-16T06:46:16.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient: dump: 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2104811970 0 0) 0x7f61dc009e30 con 0x7f61dc009150 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f61dc00c340 con 0x7f61dc009150 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2283748817) 0x7f61f0003c40 con 0x7f61dc009150 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6789/0 conn(0x7f61dc009150 legacy=0x7f61dc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f61f97fa700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6789/0 conn(0x7f61dc009150 legacy=0x7f61dc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:16.370 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.371 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:16.372 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:16.373 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:16.374 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.375 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.376 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.380 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.381 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:16.382 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.189:6812/3098810674 conn(0x7f61e8036490 legacy=0x7f61e8038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 10 monclient: shutdown 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 >> v1:172.21.15.76:6790/0 conn(0x7f61fc09ace0 legacy=0x7f61fc12be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f62012d4700 1 -- 172.21.15.76:0/434979732 reap_dead start 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f62012d4700 1 -- 172.21.15.76:0/434979732 reap_dead start 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 shutdown_connections 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 >> 172.21.15.76:0/434979732 conn(0x7f61fc0a0b50 msgr2=0x7f61fc13c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 shutdown_connections 2024-09-16T06:46:16.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.356+0000 7f6202d37700 1 -- 172.21.15.76:0/434979732 wait complete. 2024-09-16T06:46:16.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:16.385 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:46:16.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.496+0000 7ff3bad5b700 10 monclient: get_monmap_and_config 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.496+0000 7ff3bad5b700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.496+0000 7ff3bad5b700 10 monclient: monmap: 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:16.503683+0000 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:16.503683+0000 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.502 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.502 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:16.502 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:16.502 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 Processor -- start 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- start start 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient: init 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-c con 0x7ff3b4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-a con 0x7ff3b4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4098990 con 0x7ff3b4099090 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4097f40 con 0x7ff3b4097b60 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b409d610 con 0x7ff3b4139500 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient(hunting): authenticate will time out at 1323.637599s 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3ba55a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff3b4139500 0x7ff3b413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:33964/0 (socket says 172.21.15.76:33964) 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b9d59700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3b4097b60 0x7ff3b409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:38482/0 (socket says 172.21.15.76:38482) 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b9558700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff3b4099090 0x7ff3b409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49520/0 (socket says 172.21.15.76:49520) 2024-09-16T06:46:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3ba55a700 1 -- 172.21.15.76:0/2025306822 learned_addr learned my addr 172.21.15.76:0/2025306822 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139621820 0 0) 0x7ff3b4097f40 con 0x7ff3b4097b60 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): my global_id is 174664 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c0035d0 con 0x7ff3b4097b60 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867181593 0 0) 0x7ff3b409d610 con 0x7ff3b4139500 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): my global_id is 194111 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4097f40 con 0x7ff3b4139500 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680750957 0 0) 0x7ff3b4098990 con 0x7ff3b4099090 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient(hunting): my global_id is 184341 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b409d610 con 0x7ff3b4099090 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199208778 0 0) 0x7ff3b4097f40 con 0x7ff3b4139500 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 >> v1:172.21.15.189:6789/0 conn(0x7ff3b4097b60 legacy=0x7ff3b409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 >> v1:172.21.15.76:6789/0 conn(0x7ff3b4099090 legacy=0x7ff3b409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 monclient: found mon.noname-b 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b40986b0 con 0x7ff3b4139500 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3ac0015f0 con 0x7ff3b4139500 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: dump: 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 5 monclient: authenticate success, global_id 194111 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ac001890 con 0x7ff3b4139500 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 1 -- 172.21.15.76:0/2025306822 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3ac001c60 con 0x7ff3b4139500 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 4 set_mon_vals no callback set 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3b8d57700 10 monclient: dump: 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.518 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 10 monclient: shutdown 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- 172.21.15.76:0/2025306822 >> v1:172.21.15.76:6790/0 conn(0x7ff3b4139500 legacy=0x7ff3b413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- 172.21.15.76:0/2025306822 shutdown_connections 2024-09-16T06:46:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- 172.21.15.76:0/2025306822 >> 172.21.15.76:0/2025306822 conn(0x7ff3b40a0030 msgr2=0x7ff3b40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- 172.21.15.76:0/2025306822 shutdown_connections 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.500+0000 7ff3bad5b700 1 -- 172.21.15.76:0/2025306822 wait complete. 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: monmap: 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:16.520 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 Processor -- start 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- start start 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: init 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-a con 0x7ff3b4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient(hunting): picked mon.noname-c con 0x7ff3b4139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4137990 con 0x7ff3b4099090 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40a3090 con 0x7ff3b4097b60 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b409b670 con 0x7ff3b4139500 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient(hunting): authenticate will time out at 1323.642581s 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b9558700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff3b4099090 0x7ff3b409af60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49530/0 (socket says 172.21.15.76:49530) 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b9558700 1 -- 172.21.15.76:0/1056286132 learned_addr learned my addr 172.21.15.76:0/1056286132 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724833901 0 0) 0x7ff3b4137990 con 0x7ff3b4099090 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): my global_id is 184344 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00035d0 con 0x7ff3b4099090 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925860066 0 0) 0x7ff3b409b670 con 0x7ff3b4139500 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): my global_id is 194114 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4137990 con 0x7ff3b4139500 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747387623 0 0) 0x7ff3a00035d0 con 0x7ff3b4099090 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b409b670 con 0x7ff3b4099090 2024-09-16T06:46:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3a8001a10 con 0x7ff3b4099090 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): dump: 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661117335 0 0) 0x7ff3b40a3090 con 0x7ff3b4097b60 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): my global_id is 174667 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00035d0 con 0x7ff3b4097b60 2024-09-16T06:46:16.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2173760271 0 0) 0x7ff3b4137990 con 0x7ff3b4139500 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b40a3090 con 0x7ff3b4139500 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3ac001a00 con 0x7ff3b4139500 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient(hunting): dump: 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 314452063 0 0) 0x7ff3b409b670 con 0x7ff3b4099090 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6790/0 conn(0x7ff3b4139500 legacy=0x7ff3b41d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.189:6789/0 conn(0x7ff3b4097b60 legacy=0x7ff3b409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 monclient: found mon.a 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b409ba60 con 0x7ff3b4099090 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 5 monclient: authenticate success, global_id 184344 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _renew_subs 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b41376b0 con 0x7ff3b4099090 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _renew_subs 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.504+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b409c5f0 con 0x7ff3b4099090 2024-09-16T06:46:16.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3a8001d60 con 0x7ff3b4099090 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3a8004ce0 con 0x7ff3b4099090 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b3fff700 4 set_mon_vals no callback set 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: dump: 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3bad5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff394006590 con 0x7ff394002830 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff3a800f750 con 0x7ff3b4099090 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7ff3a8045850 con 0x7ff3b4099090 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3ba55a700 10 monclient: get_auth_request con 0x7ff3a0036450 auth_method 0 2024-09-16T06:46:16.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37350108 0 0) 0x7ff394006590 con 0x7ff394002830 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient(hunting): my global_id is 194117 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b409b670 con 0x7ff394002830 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2566116541 0 0) 0x7ff3b409b670 con 0x7ff394002830 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff394006590 con 0x7ff394002830 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3a40019d0 con 0x7ff394002830 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: dump: 2024-09-16T06:46:16.531 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2543292186 0 0) 0x7ff394006590 con 0x7ff394002830 2024-09-16T06:46:16.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff39400a190 con 0x7ff394002830 2024-09-16T06:46:16.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff3a4001d80 con 0x7ff394002830 2024-09-16T06:46:16.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:16.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6790/0 conn(0x7ff394002830 legacy=0x7ff394004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.508+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6790/0 conn(0x7ff394002830 legacy=0x7ff394004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:16.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff394009e50 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047407951 0 0) 0x7ff394009e50 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient(hunting): my global_id is 194120 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394006590 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1848714713 0 0) 0x7ff394006590 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff394009e50 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3a40051a0 con 0x7ff394009170 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient: dump: 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 419413902 0 0) 0x7ff394009e50 con 0x7ff394009170 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff39400c340 con 0x7ff394009170 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 266563266) 0x7ff3a40053b0 con 0x7ff394009170 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6790/0 conn(0x7ff394009170 legacy=0x7ff394009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3b27fc700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6790/0 conn(0x7ff394009170 legacy=0x7ff394009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:16.551 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff3a0036450 legacy=0x7ff3a0038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 10 monclient: shutdown 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 >> v1:172.21.15.76:6789/0 conn(0x7ff3b4099090 legacy=0x7ff3b409af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3ba55a700 1 -- 172.21.15.76:0/1056286132 reap_dead start 2024-09-16T06:46:16.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3ba55a700 1 -- 172.21.15.76:0/1056286132 reap_dead start 2024-09-16T06:46:16.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 shutdown_connections 2024-09-16T06:46:16.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 >> 172.21.15.76:0/1056286132 conn(0x7ff3b40a0030 msgr2=0x7ff3b4131610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 shutdown_connections 2024-09-16T06:46:16.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.516+0000 7ff3bad5b700 1 -- 172.21.15.76:0/1056286132 wait complete. 2024-09-16T06:46:16.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:16.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:46:16.553 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 10 monclient: get_monmap_and_config 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 10 monclient: monmap: 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.672 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:16.675003+0000 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:16.675003+0000 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:16.673 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 1 Processor -- start 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 1 -- start start 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.668+0000 7fcbfb43e700 10 monclient: init 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-b con 0x7fcbf4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-c con 0x7fcbf4099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-a con 0x7fcbf4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf4098990 con 0x7fcbf4139500 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf4097f40 con 0x7fcbf4099090 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf409d610 con 0x7fcbf4097b60 2024-09-16T06:46:16.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient(hunting): authenticate will time out at 1323.808323s 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfa43c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcbf4097b60 0x7fcbf409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34002/0 (socket says 172.21.15.76:34002) 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfac3d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcbf4139500 0x7fcbf413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49534/0 (socket says 172.21.15.76:49534) 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfa43c700 1 -- 172.21.15.76:0/3475289455 learned_addr learned my addr 172.21.15.76:0/3475289455 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162965741 0 0) 0x7fcbf4098990 con 0x7fcbf4139500 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): my global_id is 184347 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00035d0 con 0x7fcbf4139500 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849826007 0 0) 0x7fcbf409d610 con 0x7fcbf4097b60 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): my global_id is 194123 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf4098990 con 0x7fcbf4097b60 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931305379 0 0) 0x7fcbf4097f40 con 0x7fcbf4099090 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient(hunting): my global_id is 174670 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf409d610 con 0x7fcbf4099090 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133791441 0 0) 0x7fcbe00035d0 con 0x7fcbf4139500 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4097b60 legacy=0x7fcbf409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 >> v1:172.21.15.189:6789/0 conn(0x7fcbf4099090 legacy=0x7fcbf409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 monclient: found mon.noname-a 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40986b0 con 0x7fcbf4139500 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 5 monclient: authenticate success, global_id 184347 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbec001990 con 0x7fcbf4139500 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:16.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: dump: 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbec001c70 con 0x7fcbf4139500 2024-09-16T06:46:16.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 1 -- 172.21.15.76:0/3475289455 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbec002000 con 0x7fcbf4139500 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 4 set_mon_vals no callback set 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbf943a700 10 monclient: dump: 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 10 monclient: shutdown 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3475289455 >> v1:172.21.15.76:6789/0 conn(0x7fcbf4139500 legacy=0x7fcbf413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3475289455 shutdown_connections 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3475289455 >> 172.21.15.76:0/3475289455 conn(0x7fcbf40a0030 msgr2=0x7fcbf40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3475289455 shutdown_connections 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.672+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3475289455 wait complete. 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient: monmap: 2024-09-16T06:46:16.684 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:16.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 1 Processor -- start 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 1 -- start start 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient: init 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-a con 0x7fcbf4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-c con 0x7fcbf4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient(hunting): picked mon.noname-b con 0x7fcbf4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf406d140 con 0x7fcbf4097b60 2024-09-16T06:46:16.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf41392f0 con 0x7fcbf4139500 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf41387e0 con 0x7fcbf4099090 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfa43c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcbf4097b60 0x7fcbf4136df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49536/0 (socket says 172.21.15.76:49536) 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfa43c700 1 -- 172.21.15.76:0/3887424744 learned_addr learned my addr 172.21.15.76:0/3887424744 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbf9c3b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 0x7fcbf4137500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34014/0 (socket says 172.21.15.76:34014) 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbfb43e700 10 monclient(hunting): authenticate will time out at 1323.814942s 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbf9c3b700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 legacy=0x7fcbf4137500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbf9c3b700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 legacy=0x7fcbf4137500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbf9c3b700 1 --1- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 0x7fcbf4137500 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbf9c3b700 1 --1- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 0x7fcbf4137500 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:46:16.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297912562 0 0) 0x7fcbf406d140 con 0x7fcbf4097b60 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): my global_id is 184350 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd80035d0 con 0x7fcbf4097b60 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990232942 0 0) 0x7fcbf41392f0 con 0x7fcbf4139500 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): my global_id is 174673 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf406d140 con 0x7fcbf4139500 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254415469 0 0) 0x7fcbd80035d0 con 0x7fcbf4097b60 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf41392f0 con 0x7fcbf4097b60 2024-09-16T06:46:16.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbf00018f0 con 0x7fcbf4097b60 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.694 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.676+0000 7fcbe6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 765166751 0 0) 0x7fcbf41392f0 con 0x7fcbf4097b60 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6790/0 conn(0x7fcbf4099090 legacy=0x7fcbf4137500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6789/0 conn(0x7fcbf4139500 legacy=0x7fcbf4137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 monclient: found mon.a 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf413c270 con 0x7fcbf4097b60 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 5 monclient: authenticate success, global_id 184350 2024-09-16T06:46:16.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 10 monclient: _renew_subs 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf40a2bc0 con 0x7fcbf4097b60 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 10 monclient: _renew_subs 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf41d0f00 con 0x7fcbf4097b60 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbf0004890 con 0x7fcbf4097b60 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbf0004ba0 con 0x7fcbf4097b60 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: dump: 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.696 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbf8c39700 4 set_mon_vals no callback set 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcbf40008d0 con 0x7fcbf413b050 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcbf000f610 con 0x7fcbf4097b60 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7fcbf0045850 con 0x7fcbf4097b60 2024-09-16T06:46:16.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbfac3d700 10 monclient: get_auth_request con 0x7fcbd8036220 auth_method 0 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583522859 0 0) 0x7fcbf40008d0 con 0x7fcbf413b050 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient(hunting): my global_id is 174676 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf41392f0 con 0x7fcbf413b050 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3150095474 0 0) 0x7fcbf41392f0 con 0x7fcbf413b050 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf40008d0 con 0x7fcbf413b050 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbe80016c0 con 0x7fcbf413b050 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 10 monclient: dump: 2024-09-16T06:46:16.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1855587054 0 0) 0x7fcbf40008d0 con 0x7fcbf413b050 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.680+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbf406cef0 con 0x7fcbf413b050 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.684+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fcbe8001b10 con 0x7fcbf413b050 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.684+0000 7fcbe6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.684+0000 7fcbe6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:16.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.684+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6789/0 conn(0x7fcbf413b050 legacy=0x7fcbf413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.684+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6789/0 conn(0x7fcbf413b050 legacy=0x7fcbf413b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcbf413bf50 con 0x7fcbf413bb40 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340449222 0 0) 0x7fcbf413bf50 con 0x7fcbf413bb40 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient(hunting): my global_id is 174679 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf40008d0 con 0x7fcbf413bb40 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2707290559 0 0) 0x7fcbf40008d0 con 0x7fcbf413bb40 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf413bf50 con 0x7fcbf413bb40 2024-09-16T06:46:16.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcbe8003860 con 0x7fcbf413bb40 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient: dump: 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3571617057 0 0) 0x7fcbf413bf50 con 0x7fcbf413bb40 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcbf409fb80 con 0x7fcbf413bb40 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 1644659614) 0x7fcbe8005870 con 0x7fcbf413bb40 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6789/0 conn(0x7fcbf413bb40 legacy=0x7fcbf402c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbe6ffd700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6789/0 conn(0x7fcbf413bb40 legacy=0x7fcbf402c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:16.703 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:16.704 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:16.705 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:16.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:16.708 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.711 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:16.712 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.713 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:16.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:16.715 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:16.716 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:16.716 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:16.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcbd8036220 legacy=0x7fcbd80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 10 monclient: shutdown 2024-09-16T06:46:16.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 >> v1:172.21.15.76:6789/0 conn(0x7fcbf4097b60 legacy=0x7fcbf4136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfac3d700 1 -- 172.21.15.76:0/3887424744 reap_dead start 2024-09-16T06:46:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfac3d700 1 -- 172.21.15.76:0/3887424744 reap_dead start 2024-09-16T06:46:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 shutdown_connections 2024-09-16T06:46:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 >> 172.21.15.76:0/3887424744 conn(0x7fcbf40a0030 msgr2=0x7fcbf41326b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 shutdown_connections 2024-09-16T06:46:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.696+0000 7fcbfb43e700 1 -- 172.21.15.76:0/3887424744 wait complete. 2024-09-16T06:46:16.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:16.718 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:46:16.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:46:16.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: get_monmap_and_config 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: monmap: 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:16.841634+0000 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:16.841634+0000 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.839 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:16.840 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:16.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 1 Processor -- start 2024-09-16T06:46:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 1 -- start start 2024-09-16T06:46:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: init 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient(hunting): picked mon.noname-a con 0x7f24000973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient(hunting): picked mon.noname-c con 0x7f2400097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient(hunting): picked mon.noname-b con 0x7f24001400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400099350 con 0x7f24000973c0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240009f690 con 0x7f2400097eb0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400099bf0 con 0x7f24001400d0 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240691e700 10 monclient(hunting): authenticate will time out at 1323.975000s 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240611d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f24001400d0 0x7f24001424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34022/0 (socket says 172.21.15.76:34022) 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240591c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f24000973c0 0x7f24000977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49540/0 (socket says 172.21.15.76:49540) 2024-09-16T06:46:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240611d700 1 -- 172.21.15.76:0/54743067 learned_addr learned my addr 172.21.15.76:0/54743067 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509153656 0 0) 0x7f2400099350 con 0x7f24000973c0 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): my global_id is 184353 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e80035d0 con 0x7f24000973c0 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363255207 0 0) 0x7f2400099bf0 con 0x7f24001400d0 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): my global_id is 194126 2024-09-16T06:46:16.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2400099350 con 0x7f24001400d0 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949545132 0 0) 0x7f240009f690 con 0x7f2400097eb0 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.836+0000 7f240491a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient(hunting): my global_id is 174682 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2400099bf0 con 0x7f2400097eb0 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525157259 0 0) 0x7f23e80035d0 con 0x7f24000973c0 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 >> v1:172.21.15.76:6790/0 conn(0x7f24001400d0 legacy=0x7f24001424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 >> v1:172.21.15.189:6789/0 conn(0x7f2400097eb0 legacy=0x7f240013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 monclient: found mon.noname-a 2024-09-16T06:46:16.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2400098360 con 0x7f24000973c0 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 5 monclient: authenticate success, global_id 184353 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f4001610 con 0x7f24000973c0 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:16.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: dump: 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f4001830 con 0x7f24000973c0 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 1 -- 172.21.15.76:0/54743067 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f4001bc0 con 0x7f24000973c0 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240491a700 10 monclient: dump: 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.850 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 4 set_mon_vals no callback set 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 10 monclient: shutdown 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 1 -- 172.21.15.76:0/54743067 >> v1:172.21.15.76:6789/0 conn(0x7f24000973c0 legacy=0x7f24000977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 1 -- 172.21.15.76:0/54743067 shutdown_connections 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 1 -- 172.21.15.76:0/54743067 >> 172.21.15.76:0/54743067 conn(0x7f24000a0030 msgr2=0x7f24000a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:16.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 1 -- 172.21.15.76:0/54743067 shutdown_connections 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 1 -- 172.21.15.76:0/54743067 wait complete. 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 10 monclient: build_initial_monmap 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.840+0000 7f240691e700 10 monclient: monmap: 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:16.852 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 Processor -- start 2024-09-16T06:46:16.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- start start 2024-09-16T06:46:16.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: init 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient(hunting): picked mon.noname-b con 0x7f24000973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient(hunting): picked mon.noname-c con 0x7f2400097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient(hunting): picked mon.noname-a con 0x7f24001400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2400072660 con 0x7f24001400d0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240013e390 con 0x7f24000973c0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240006c7a0 con 0x7f2400097eb0 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient(hunting): authenticate will time out at 1323.980784s 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240511b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2400097eb0 0x7f24001d3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34026/0 (socket says 172.21.15.76:34026) 2024-09-16T06:46:16.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240511b700 1 -- 172.21.15.76:0/573955050 learned_addr learned my addr 172.21.15.76:0/573955050 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466745805 0 0) 0x7f2400072660 con 0x7f24001400d0 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): my global_id is 184356 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec0035d0 con 0x7f24001400d0 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790645231 0 0) 0x7f240006c7a0 con 0x7f2400097eb0 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): my global_id is 194129 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2400072660 con 0x7f2400097eb0 2024-09-16T06:46:16.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083834114 0 0) 0x7f240013e390 con 0x7f24000973c0 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): my global_id is 174685 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240006c7a0 con 0x7f24000973c0 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844587982 0 0) 0x7f2400072660 con 0x7f2400097eb0 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f240013e390 con 0x7f2400097eb0 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f80018f0 con 0x7f2400097eb0 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:16.860 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801339335 0 0) 0x7f23ec0035d0 con 0x7f24001400d0 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2400072660 con 0x7f24001400d0 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f0001bb0 con 0x7f24001400d0 2024-09-16T06:46:16.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70523574 0 0) 0x7f240006c7a0 con 0x7f24000973c0 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23ec0035d0 con 0x7f24000973c0 2024-09-16T06:46:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f4001370 con 0x7f24000973c0 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3166248815 0 0) 0x7f240013e390 con 0x7f2400097eb0 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 >> v1:172.21.15.189:6789/0 conn(0x7f24000973c0 legacy=0x7f24001ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 >> v1:172.21.15.76:6789/0 conn(0x7f24001400d0 legacy=0x7f24001d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 monclient: found mon.c 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240013ef90 con 0x7f2400097eb0 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f8001c50 con 0x7f2400097eb0 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f23f8003b10 con 0x7f2400097eb0 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 10 monclient: dump: 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:16.865 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 5 monclient: authenticate success, global_id 194129 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _renew_subs 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fffff700 4 set_mon_vals no callback set 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240006c150 con 0x7f2400097eb0 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _renew_subs 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2400138580 con 0x7f2400097eb0 2024-09-16T06:46:16.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f23f8003d40 con 0x7f2400097eb0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.844+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2400094fc0 con 0x7f2400097eb0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.848+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(217..217 src has 151..217) v4 ==== 13232+0+0 (unknown 3137642465 0 0) 0x7f23f8045810 con 0x7f2400097eb0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.848+0000 7f240591c700 10 monclient: get_auth_request con 0x7f23ec036650 auth_method 0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.852+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f23f8014810 con 0x7f2400097eb0 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.852+0000 7f23fe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:16.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:16.852+0000 7f23fe7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:17.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.070+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v218) v1 ==== 118+0+0 (unknown 3900422495 0 0) 0x7f00c4011d70 con 0x7f00d406fbd0 2024-09-16T06:46:17.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.070+0000 7f00cb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:17.074 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.070+0000 7f00cb7fe700 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-16T06:46:17.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.088+0000 7f240691e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:46:17.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.088+0000 7f240691e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:46:17.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.088+0000 7f240691e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:17.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.088+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f240009a5a0 con 0x7f2400097eb0 2024-09-16T06:46:17.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f23fe7fc700 1 -- 172.21.15.76:0/573955050 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f23f8014400 con 0x7f2400097eb0 2024-09-16T06:46:17.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f23fe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:46:17.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f23fe7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:17.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=3731604983,config=1097430493,health=2760582032,kv=1001081567,logm=918615538}) 2024-09-16T06:46:17.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=12,logm=50} crc {auth=156334898,config=1097430493,health=3056118199,kv=1001081567,logm=1505354859}) 2024-09-16T06:46:17.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=3731604983,config=1097430493,health=2760582032,kv=1001081567,logm=918615538}) 2024-09-16T06:46:17.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.092+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=13,logm=49} crc {auth=4023375398,config=1097430493,health=4105781556,kv=1001081567,logm=959280822}) 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525959619}) 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750818068}) 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525959619}) 2024-09-16T06:46:17.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314332575}) 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 >> v1:172.21.15.189:6812/3098810674 conn(0x7f23ec036650 legacy=0x7f23ec038b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 10 monclient: shutdown 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 >> v1:172.21.15.76:6790/0 conn(0x7f2400097eb0 legacy=0x7f24001d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 shutdown_connections 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 >> 172.21.15.76:0/573955050 conn(0x7f24000a0030 msgr2=0x7f240009c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 shutdown_connections 2024-09-16T06:46:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f240691e700 1 -- 172.21.15.76:0/573955050 wait complete. 2024-09-16T06:46:17.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302292630}) 2024-09-16T06:46:17.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564695313}) 2024-09-16T06:46:17.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302292630}) 2024-09-16T06:46:17.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.096+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268163674}) 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123402799}) 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942334351}) 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123402799}) 2024-09-16T06:46:17.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3193955617}) 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629413000}) 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221854529}) 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629413000}) 2024-09-16T06:46:17.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.100+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182329910}) 2024-09-16T06:46:17.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411919903}) 2024-09-16T06:46:17.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499881032}) 2024-09-16T06:46:17.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411919903}) 2024-09-16T06:46:17.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2634209174}) 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024177911}) 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921769732}) 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024177911}) 2024-09-16T06:46:17.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.104+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778171926}) 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3365779300}) 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784352057}) 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3365779300}) 2024-09-16T06:46:17.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333903304}) 2024-09-16T06:46:17.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153939932}) 2024-09-16T06:46:17.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265616037}) 2024-09-16T06:46:17.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153939932}) 2024-09-16T06:46:17.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111043087}) 2024-09-16T06:46:17.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274401747}) 2024-09-16T06:46:17.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256931383}) 2024-09-16T06:46:17.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274401747}) 2024-09-16T06:46:17.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.108+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149019931}) 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=911774784}) 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748501363}) 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=911774784}) 2024-09-16T06:46:17.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101676061}) 2024-09-16T06:46:17.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650828951}) 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428597299}) 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650828951}) 2024-09-16T06:46:17.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=522971715}) 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539417743}) 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134263395}) 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539417743}) 2024-09-16T06:46:17.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.112+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933373307}) 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719909993}) 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2507453160}) 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719909993}) 2024-09-16T06:46:17.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1344196724}) 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094025269}) 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818386985}) 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094025269}) 2024-09-16T06:46:17.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=329925562}) 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844982232}) 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464968927}) 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844982232}) 2024-09-16T06:46:17.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.116+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3782334531}) 2024-09-16T06:46:17.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830773296}) 2024-09-16T06:46:17.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530173837}) 2024-09-16T06:46:17.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830773296}) 2024-09-16T06:46:17.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689015367}) 2024-09-16T06:46:17.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3688567785}) 2024-09-16T06:46:17.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192437058}) 2024-09-16T06:46:17.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3688567785}) 2024-09-16T06:46:17.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094495461}) 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903539706}) 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223826996}) 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903539706}) 2024-09-16T06:46:17.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=182242255}) 2024-09-16T06:46:17.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671494909}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203614044}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671494909}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.120+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24740573}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640336184}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634019501}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640336184}) 2024-09-16T06:46:17.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922197498}) 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319502595}) 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2092229205}) 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319502595}) 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3562415068}) 2024-09-16T06:46:17.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642470762}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343513960}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642470762}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650176197}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385695109}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931050374}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385695109}) 2024-09-16T06:46:17.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.124+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3039550695}) 2024-09-16T06:46:17.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2431778637}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161146065}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2431778637}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224139432}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994726369}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757089440}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994726369}) 2024-09-16T06:46:17.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30041921}) 2024-09-16T06:46:17.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346798638}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8001931}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346798638}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471852864}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330263291}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335026348}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330263291}) 2024-09-16T06:46:17.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.128+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668329973}) 2024-09-16T06:46:17.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615479267}) 2024-09-16T06:46:17.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888772673}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615479267}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964096469}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317723164}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781988029}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317723164}) 2024-09-16T06:46:17.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700177322}) 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839148924}) 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205582912}) 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839148924}) 2024-09-16T06:46:17.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233901749}) 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465180690}) 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597557194}) 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465180690}) 2024-09-16T06:46:17.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.132+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657867708}) 2024-09-16T06:46:17.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538124338}) 2024-09-16T06:46:17.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864130233}) 2024-09-16T06:46:17.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538124338}) 2024-09-16T06:46:17.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390617044}) 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2843907892}) 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005425702}) 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2843907892}) 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667556627}) 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3080466611}) 2024-09-16T06:46:17.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255480495}) 2024-09-16T06:46:17.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3080466611}) 2024-09-16T06:46:17.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.136+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971484495}) 2024-09-16T06:46:17.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193109008}) 2024-09-16T06:46:17.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094508891}) 2024-09-16T06:46:17.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193109008}) 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317875152}) 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559706155}) 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337986424}) 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559706155}) 2024-09-16T06:46:17.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623509006}) 2024-09-16T06:46:17.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=543308677}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932039850}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=543308677}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1582796655}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=963099375,mdsmap=1531151339,mgr=2667678790}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1011342465,mdsmap=1531151339,mgr=3836259515}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=963099375,mdsmap=1531151339,mgr=2667678790}) 2024-09-16T06:46:17.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.140+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2109271115,mdsmap=1531151339}) 2024-09-16T06:46:17.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3346712378,mon_config_key=4279594037,monmap=1887442722,osd_metadata=501347707,osd_pg_creating=294854412,osd_snap=2033388227,osdmap=1150601871}) 2024-09-16T06:46:17.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=56} crc {mgr=208945578,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3981333586,mon_config_key=2813972085,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=294854412,osd_snap=2033388227,osdmap=220009483}) 2024-09-16T06:46:17.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3346712378,mon_config_key=4279594037,monmap=1887442722,osd_metadata=501347707,osd_pg_creating=294854412,osd_snap=2033388227,osdmap=1150601871}) 2024-09-16T06:46:17.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.144+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4281343587,mon_config_key=3176833388,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=294854412,osd_snap=2033388227,osdmap=2329503943}) 2024-09-16T06:46:17.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1698176603}) 2024-09-16T06:46:17.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=3783424465}) 2024-09-16T06:46:17.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:17.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1698176603}) 2024-09-16T06:46:17.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:17.148+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=3842070416}) 2024-09-16T06:46:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:17.188 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7f00b80023c0 con 0x7f00d406fbd0 2024-09-16T06:46:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00cb7fe700 1 -- 172.21.15.189:0/2475176034 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v218) v1 ==== 118+0+0 (unknown 3468470574 0 0) 0x7f00c40134f0 con 0x7f00d406fbd0 2024-09-16T06:46:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00cb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T06:46:17.189 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00cb7fe700 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00b4036560 legacy=0x7f00b4038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 10 monclient: shutdown 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 >> v1:172.21.15.76:6790/0 conn(0x7f00d406fbd0 legacy=0x7f00d4096550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 shutdown_connections 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 >> 172.21.15.189:0/2475176034 conn(0x7f00d409fe70 msgr2=0x7f00d40a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 shutdown_connections 2024-09-16T06:46:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.186+0000 7f00d3fff700 1 -- 172.21.15.189:0/2475176034 wait complete. 2024-09-16T06:46:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h5' does not appear in the crush map 2024-09-16T06:46:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:17.202 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h6 2024-09-16T06:46:17.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.302+0000 7f83719b0700 10 monclient: get_monmap_and_config 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.302+0000 7f83719b0700 10 monclient: build_initial_monmap 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.302+0000 7f83719b0700 10 monclient: monmap: 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:17.308212+0000 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:17.308212+0000 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:17.308 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:17.309 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:17.309 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 Processor -- start 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- start start 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient: init 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-c con 0x7f836c06a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:17.312 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-a con 0x7f836c09d5b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-b con 0x7f836c13bb20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c06b880 con 0x7f836c09d5b0 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c06be90 con 0x7f836c06a340 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c0a1a40 con 0x7f836c13bb20 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient(hunting): authenticate will time out at 1329.821295s 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f836affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f836c06a340 0x7f836c06a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51892/0 (socket says 172.21.15.189:51892) 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f836affd700 1 -- 172.21.15.189:0/2810296965 learned_addr learned my addr 172.21.15.189:0/2810296965 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663979665 0 0) 0x7f836c06be90 con 0x7f836c06a340 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:17.313 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient(hunting): my global_id is 174688 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83540035d0 con 0x7f836c06a340 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2759312427 0 0) 0x7f83540035d0 con 0x7f836c06a340 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 >> v1:172.21.15.76:6790/0 conn(0x7f836c13bb20 legacy=0x7f836c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 >> v1:172.21.15.76:6789/0 conn(0x7f836c09d5b0 legacy=0x7f836c06ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 monclient: found mon.noname-c 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c0a16f0 con 0x7f836c06a340 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:17.314 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 5 monclient: authenticate success, global_id 174688 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f835c001610 con 0x7f836c06a340 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: dump: 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:17.315 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f835c001930 con 0x7f836c06a340 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 1 -- 172.21.15.189:0/2810296965 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f835c001cc0 con 0x7f836c06a340 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 4 set_mon_vals no callback set 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f8369ffb700 10 monclient: dump: 2024-09-16T06:46:17.316 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 10 monclient: shutdown 2024-09-16T06:46:17.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- 172.21.15.189:0/2810296965 >> v1:172.21.15.189:6789/0 conn(0x7f836c06a340 legacy=0x7f836c06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- 172.21.15.189:0/2810296965 shutdown_connections 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- 172.21.15.189:0/2810296965 >> 172.21.15.189:0/2810296965 conn(0x7f836c0a21f0 msgr2=0x7f836c0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- 172.21.15.189:0/2810296965 shutdown_connections 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.306+0000 7f83719b0700 1 -- 172.21.15.189:0/2810296965 wait complete. 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: build_initial_monmap 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: monmap: 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:17.318 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:17.319 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:17.319 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:17.319 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 Processor -- start 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- start start 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: init 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-a con 0x7f836c06a340 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:17.323 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-c con 0x7f836c09d5b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient(hunting): picked mon.noname-b con 0x7f836c13bb20 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c138bf0 con 0x7f836c06a340 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c09c2c0 con 0x7f836c13bb20 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c134b50 con 0x7f836c09d5b0 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient(hunting): authenticate will time out at 1329.826192s 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f836b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f836c13bb20 0x7f836c1d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51902/0 (socket says 172.21.15.189:51902) 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f836b7fe700 1 -- 172.21.15.189:0/2760201848 learned_addr learned my addr 172.21.15.189:0/2760201848 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873585770 0 0) 0x7f836c09c2c0 con 0x7f836c13bb20 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:17.324 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): my global_id is 174691 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83580035d0 con 0x7f836c13bb20 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59527761 0 0) 0x7f836c134b50 con 0x7f836c09d5b0 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): my global_id is 194135 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f836c09c2c0 con 0x7f836c09d5b0 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288072565 0 0) 0x7f836c138bf0 con 0x7f836c06a340 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:17.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): my global_id is 184362 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f836c134b50 con 0x7f836c06a340 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850353416 0 0) 0x7f83580035d0 con 0x7f836c13bb20 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f836c138bf0 con 0x7f836c13bb20 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8364001ba0 con 0x7f836c13bb20 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): dump: 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:17.326 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 861758081 0 0) 0x7f836c138bf0 con 0x7f836c13bb20 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 >> v1:172.21.15.76:6790/0 conn(0x7f836c09d5b0 legacy=0x7f836c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 >> v1:172.21.15.76:6789/0 conn(0x7f836c06a340 legacy=0x7f836c1c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 monclient: found mon.b 2024-09-16T06:46:17.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c09c700 con 0x7f836c13bb20 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 5 monclient: authenticate success, global_id 174691 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _renew_subs 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c0721b0 con 0x7f836c13bb20 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _renew_subs 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.328 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c1d39f0 con 0x7f836c13bb20 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8364003a40 con 0x7f836c13bb20 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8364003d50 con 0x7f836c13bb20 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f83697fa700 4 set_mon_vals no callback set 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.310+0000 7f8353fff700 10 monclient: dump: 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:17.329 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8364003f30 con 0x7f836c13bb20 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f83719b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f83719b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f83719b0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c005100 con 0x7f836c13bb20 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(218..218 src has 151..218) v4 ==== 13022+0+0 (unknown 3596867018 0 0) 0x7f8364045a10 con 0x7f836c13bb20 2024-09-16T06:46:17.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.314+0000 7f836affd700 10 monclient: get_auth_request con 0x7f8358036360 auth_method 0 2024-09-16T06:46:17.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.322+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8364014a60 con 0x7f836c13bb20 2024-09-16T06:46:17.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.322+0000 7f8353fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:17.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.322+0000 7f8353fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:17.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.498+0000 7f83719b0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:17.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.498+0000 7f83719b0700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:17.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.498+0000 7f83719b0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:17.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:17.498+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f834c0053b0 con 0x7f836c13bb20 2024-09-16T06:46:18.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.098+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v219) v1 ==== 118+0+0 (unknown 1841789691 0 0) 0x7f8364012ed0 con 0x7f836c13bb20 2024-09-16T06:46:18.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.098+0000 7f8353fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:18.102 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.098+0000 7f8353fff700 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-16T06:46:18.116 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:46:18.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.206+0000 7f83719b0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.206+0000 7f83719b0700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.206+0000 7f83719b0700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.206+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f834c005650 con 0x7f836c13bb20 2024-09-16T06:46:18.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.210+0000 7f8353fff700 1 -- 172.21.15.189:0/2760201848 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v219) v1 ==== 118+0+0 (unknown 186786463 0 0) 0x7f8364014650 con 0x7f836c13bb20 2024-09-16T06:46:18.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.210+0000 7f8353fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T06:46:18.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.210+0000 7f8353fff700 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8358036360 legacy=0x7f8358038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 10 monclient: shutdown 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 >> v1:172.21.15.189:6789/0 conn(0x7f836c13bb20 legacy=0x7f836c1d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 shutdown_connections 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 >> 172.21.15.189:0/2760201848 conn(0x7f836c0a21f0 msgr2=0x7f836c09eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 shutdown_connections 2024-09-16T06:46:18.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.214+0000 7f83719b0700 1 -- 172.21.15.189:0/2760201848 wait complete. 2024-09-16T06:46:18.217 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h6' does not appear in the crush map 2024-09-16T06:46:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:18.226 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h7 2024-09-16T06:46:18.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: get_monmap_and_config 2024-09-16T06:46:18.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: monmap: 2024-09-16T06:46:18.229 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.229 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:18.232498+0000 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:18.232498+0000 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:18.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 1 Processor -- start 2024-09-16T06:46:18.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 1 -- start start 2024-09-16T06:46:18.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: init 2024-09-16T06:46:18.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-c con 0x7f2e88135500 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-a con 0x7f2e8809a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-b con 0x7f2e8813b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880a67b0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880a7320 con 0x7f2e88135500 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8809ade0 con 0x7f2e8813b980 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 10 monclient(hunting): authenticate will time out at 1325.366682s 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8dba3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2e8809a9d0 0x7f2e8813a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49554/0 (socket says 172.21.15.76:49554) 2024-09-16T06:46:18.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8eba5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2e8813b980 0x7f2e8813dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34042/0 (socket says 172.21.15.76:34042) 2024-09-16T06:46:18.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8dba3700 1 -- 172.21.15.76:0/1137978104 learned_addr learned my addr 172.21.15.76:0/1137978104 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788977256 0 0) 0x7f2e8809ade0 con 0x7f2e8813b980 2024-09-16T06:46:18.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): my global_id is 194138 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e740035d0 con 0x7f2e8813b980 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442092107 0 0) 0x7f2e880a67b0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): my global_id is 184365 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8809ade0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774919360 0 0) 0x7f2e880a7320 con 0x7f2e88135500 2024-09-16T06:46:18.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient(hunting): my global_id is 174694 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e880a67b0 con 0x7f2e88135500 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244529016 0 0) 0x7f2e740035d0 con 0x7f2e8813b980 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 >> v1:172.21.15.189:6789/0 conn(0x7f2e88135500 legacy=0x7f2e881358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 >> v1:172.21.15.76:6789/0 conn(0x7f2e8809a9d0 legacy=0x7f2e8813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 monclient: found mon.noname-b 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88099d10 con 0x7f2e8813b980 2024-09-16T06:46:18.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 5 monclient: authenticate success, global_id 194138 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e800019b0 con 0x7f2e8813b980 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e8d3a2700 10 monclient: dump: 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.238 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.228+0000 7f2e90608700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e80001c90 con 0x7f2e8813b980 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 4 set_mon_vals no callback set 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 1 -- 172.21.15.76:0/1137978104 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e80002020 con 0x7f2e8813b980 2024-09-16T06:46:18.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8d3a2700 10 monclient: dump: 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: shutdown 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/1137978104 >> v1:172.21.15.76:6790/0 conn(0x7f2e8813b980 legacy=0x7f2e8813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/1137978104 shutdown_connections 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/1137978104 >> 172.21.15.76:0/1137978104 conn(0x7f2e880a42f0 msgr2=0x7f2e880a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/1137978104 shutdown_connections 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/1137978104 wait complete. 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: monmap: 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 Processor -- start 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- start start 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: init 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-a con 0x7f2e8809a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-c con 0x7f2e88135500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient(hunting): picked mon.noname-b con 0x7f2e8813b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e881380d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8806d430 con 0x7f2e8813b980 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e88135d00 con 0x7f2e88135500 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient(hunting): authenticate will time out at 1325.370205s 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8dba3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2e88135500 0x7f2e881cf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34058/0 (socket says 172.21.15.76:34058) 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e8dba3700 1 -- 172.21.15.76:0/3236289582 learned_addr learned my addr 172.21.15.76:0/3236289582 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110089346 0 0) 0x7f2e8806d430 con 0x7f2e8813b980 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): my global_id is 174697 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c0035d0 con 0x7f2e8813b980 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079543083 0 0) 0x7f2e88135d00 con 0x7f2e88135500 2024-09-16T06:46:18.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): my global_id is 194141 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8806d430 con 0x7f2e88135500 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422285549 0 0) 0x7f2e881380d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): my global_id is 184368 2024-09-16T06:46:18.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e88135d00 con 0x7f2e8809a9d0 2024-09-16T06:46:18.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088770264 0 0) 0x7f2e88135d00 con 0x7f2e8809a9d0 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e881380d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1401940559 0 0) 0x7f2e8806d430 con 0x7f2e88135500 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e88135d00 con 0x7f2e88135500 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e840018b0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): dump: 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e780017f0 con 0x7f2e88135500 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): dump: 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.248 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681453802 0 0) 0x7f2e6c0035d0 con 0x7f2e8813b980 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e8806d430 con 0x7f2e8813b980 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e80001e00 con 0x7f2e8813b980 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:18.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient(hunting): dump: 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401200686 0 0) 0x7f2e881380d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 >> v1:172.21.15.76:6790/0 conn(0x7f2e88135500 legacy=0x7f2e881cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 >> v1:172.21.15.189:6789/0 conn(0x7f2e8813b980 legacy=0x7f2e881d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 monclient: found mon.a 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e881362f0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 5 monclient: authenticate success, global_id 184368 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: _renew_subs 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.232+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e88137830 con 0x7f2e8809a9d0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 10 monclient: _renew_subs 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e88137500 con 0x7f2e8809a9d0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e840036c0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e840039d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e8cba1700 4 set_mon_vals no callback set 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 10 monclient: dump: 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2e8400e440 con 0x7f2e8809a9d0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e70005100 con 0x7f2e8809a9d0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f2e840443e0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.236+0000 7f2e8dba3700 10 monclient: get_auth_request con 0x7f2e6c0364f0 auth_method 0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.244+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2e84013330 con 0x7f2e8809a9d0 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.244+0000 7f2e7effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.244+0000 7f2e7effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.326+0000 7fc189fbb700 10 monclient: get_monmap_and_config 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.326+0000 7fc189fbb700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: monmap: 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:18.332536+0000 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:18.332536+0000 2024-09-16T06:46:18.332 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:18.333 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:18.333 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 Processor -- start 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- start start 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: init 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:18.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-a con 0x7fc1840a27a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-b con 0x7fc1840a3d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-c con 0x7fc1840a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184099a40 con 0x7fc1840a27a0 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184099ef0 con 0x7fc1840a2350 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18409a1d0 con 0x7fc1840a3d10 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient(hunting): authenticate will time out at 1330.845762s 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc183fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc1840a2350 0x7fc18413dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51910/0 (socket says 172.21.15.189:51910) 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc183fff700 1 -- 172.21.15.189:0/2759022603 learned_addr learned my addr 172.21.15.189:0/2759022603 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820485486 0 0) 0x7fc184099ef0 con 0x7fc1840a2350 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): my global_id is 174700 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16c0035d0 con 0x7fc1840a2350 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020522988 0 0) 0x7fc18409a1d0 con 0x7fc1840a3d10 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): my global_id is 194144 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184099ef0 con 0x7fc1840a3d10 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700307395 0 0) 0x7fc184099a40 con 0x7fc1840a27a0 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient(hunting): my global_id is 184371 2024-09-16T06:46:18.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18409a1d0 con 0x7fc1840a27a0 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 769821780 0 0) 0x7fc16c0035d0 con 0x7fc1840a2350 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 >> v1:172.21.15.76:6790/0 conn(0x7fc1840a3d10 legacy=0x7fc18413a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 >> v1:172.21.15.76:6789/0 conn(0x7fc1840a27a0 legacy=0x7fc1840a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 monclient: found mon.noname-c 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840a3290 con 0x7fc1840a2350 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 5 monclient: authenticate success, global_id 174700 2024-09-16T06:46:18.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc17c001990 con 0x7fc1840a2350 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: dump: 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.341 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc17c001c70 con 0x7fc1840a2350 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 1 -- 172.21.15.189:0/2759022603 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc17c002000 con 0x7fc1840a2350 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 4 set_mon_vals no callback set 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc1827fc700 10 monclient: dump: 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.342 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 10 monclient: shutdown 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- 172.21.15.189:0/2759022603 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a2350 legacy=0x7fc18413dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- 172.21.15.189:0/2759022603 shutdown_connections 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- 172.21.15.189:0/2759022603 >> 172.21.15.189:0/2759022603 conn(0x7fc1840a7d70 msgr2=0x7fc1840a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- 172.21.15.189:0/2759022603 shutdown_connections 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.330+0000 7fc189fbb700 1 -- 172.21.15.189:0/2759022603 wait complete. 2024-09-16T06:46:18.343 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: monmap: 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 Processor -- start 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- start start 2024-09-16T06:46:18.344 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: init 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-c con 0x7fc1840a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-b con 0x7fc1840a27a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient(hunting): picked mon.noname-a con 0x7fc1841234a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.349 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184124c30 con 0x7fc1841234a0 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18409b670 con 0x7fc1840a27a0 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18406bea0 con 0x7fc1840a2350 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient(hunting): authenticate will time out at 1330.849957s 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc182ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 0x7fc184122d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51914/0 (socket says 172.21.15.189:51914) 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc182ffd700 1 -- 172.21.15.189:0/1179214241 learned_addr learned my addr 172.21.15.189:0/1179214241 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569531334 0 0) 0x7fc18409b670 con 0x7fc1840a27a0 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): my global_id is 174703 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1700035d0 con 0x7fc1840a27a0 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272123827 0 0) 0x7fc1700035d0 con 0x7fc1840a27a0 2024-09-16T06:46:18.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc18409b670 con 0x7fc1840a27a0 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc174001910 con 0x7fc1840a27a0 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): dump: 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.351 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977420709 0 0) 0x7fc184124c30 con 0x7fc1841234a0 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): my global_id is 184374 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1700035d0 con 0x7fc1841234a0 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720189705 0 0) 0x7fc18406bea0 con 0x7fc1840a2350 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.352 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient(hunting): my global_id is 194147 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184124c30 con 0x7fc1840a2350 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093960953 0 0) 0x7fc18409b670 con 0x7fc1840a27a0 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.76:6790/0 conn(0x7fc1840a2350 legacy=0x7fc184122680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.76:6789/0 conn(0x7fc1841234a0 legacy=0x7fc1841d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 monclient: found mon.b 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840a0180 con 0x7fc1840a27a0 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 5 monclient: authenticate success, global_id 174703 2024-09-16T06:46:18.353 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _renew_subs 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc184126600 con 0x7fc1840a27a0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _renew_subs 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841238a0 con 0x7fc1840a27a0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc174003740 con 0x7fc1840a27a0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc181ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc174003ad0 con 0x7fc1840a27a0 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.334+0000 7fc16bfff700 10 monclient: dump: 2024-09-16T06:46:18.354 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc189fbb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc189fbb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc189fbb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc164005100 con 0x7fc1840a27a0 2024-09-16T06:46:18.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc174003cb0 con 0x7fc1840a27a0 2024-09-16T06:46:18.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7fc1740456d0 con 0x7fc1840a27a0 2024-09-16T06:46:18.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.338+0000 7fc1837fe700 10 monclient: get_auth_request con 0x7fc1700363e0 auth_method 0 2024-09-16T06:46:18.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.346+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc174014850 con 0x7fc1840a27a0 2024-09-16T06:46:18.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.346+0000 7fc16bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.346+0000 7fc16bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:18.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e90608700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:18.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e90608700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:18.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e90608700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2e70005930 con 0x7f2e8809a9d0 2024-09-16T06:46:18.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e7effd700 1 -- 172.21.15.76:0/3236289582 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2548023689) 0x7f2e840119d0 con 0x7f2e8809a9d0 2024-09-16T06:46:18.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e7effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:18.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.480+0000 7f2e7effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:18.483 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:18.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2e6c0364f0 legacy=0x7f2e6c0389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 10 monclient: shutdown 2024-09-16T06:46:18.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 >> v1:172.21.15.76:6789/0 conn(0x7f2e8809a9d0 legacy=0x7f2e881ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 shutdown_connections 2024-09-16T06:46:18.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 >> 172.21.15.76:0/3236289582 conn(0x7f2e880a42f0 msgr2=0x7f2e8813e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 shutdown_connections 2024-09-16T06:46:18.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.484+0000 7f2e90608700 1 -- 172.21.15.76:0/3236289582 wait complete. 2024-09-16T06:46:18.500 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:46:18.500 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:46:18.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:46:18.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:46:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.518+0000 7fc189fbb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.518+0000 7fc189fbb700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.518+0000 7fc189fbb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:18.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:18.518+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7fc1640053b0 con 0x7fc1840a27a0 2024-09-16T06:46:18.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: get_monmap_and_config 2024-09-16T06:46:18.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: monmap: 2024-09-16T06:46:18.615 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:18.617893+0000 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:18.617893+0000 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.616 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.617 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:18.617 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:18.617 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 1 Processor -- start 2024-09-16T06:46:18.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 1 -- start start 2024-09-16T06:46:18.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: init 2024-09-16T06:46:18.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc0a48f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-b con 0x7f5bdc09e4a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-a con 0x7f5bdc0a53e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc06bd80 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc09ff70 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc06da30 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.612+0000 7f5be2d83700 10 monclient(hunting): authenticate will time out at 1325.751974s 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5bdc09e4a0 0x7f5bdc13a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34072/0 (socket says 172.21.15.76:34072) 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdbfff700 1 -- 172.21.15.76:0/111876314 learned_addr learned my addr 172.21.15.76:0/111876314 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965037622 0 0) 0x7f5bdc06bd80 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): my global_id is 184377 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc40035d0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392604217 0 0) 0x7f5bdc06da30 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): my global_id is 194150 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc06bd80 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215574077 0 0) 0x7f5bdc09ff70 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient(hunting): my global_id is 174706 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc06da30 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397468981 0 0) 0x7f5bc40035d0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 >> v1:172.21.15.76:6790/0 conn(0x7f5bdc09e4a0 legacy=0x7f5bdc13a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 >> v1:172.21.15.189:6789/0 conn(0x7f5bdc0a48f0 legacy=0x7f5bdc0a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc06cf90 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 5 monclient: authenticate success, global_id 184377 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4001990 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: dump: 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd4001c70 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 4 set_mon_vals no callback set 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 1 -- 172.21.15.76:0/111876314 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4002040 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5bdb7fe700 10 monclient: dump: 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 10 monclient: shutdown 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- 172.21.15.76:0/111876314 >> v1:172.21.15.76:6789/0 conn(0x7f5bdc0a53e0 legacy=0x7f5bdc13dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- 172.21.15.76:0/111876314 shutdown_connections 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- 172.21.15.76:0/111876314 >> 172.21.15.76:0/111876314 conn(0x7f5bdc0a24e0 msgr2=0x7f5bdc0a28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- 172.21.15.76:0/111876314 shutdown_connections 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- 172.21.15.76:0/111876314 wait complete. 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 10 monclient: monmap: 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:18.647 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 Processor -- start 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 1 -- start start 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.616+0000 7f5be2d83700 10 monclient: init 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:18.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-a con 0x7f5bdc09e4a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-b con 0x7f5bdc0a48f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc0a53e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc11d510 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0a1710 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc072650 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient(hunting): authenticate will time out at 1325.756609s 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be0b1f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5bdc09e4a0 0x7f5bdc13bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49580/0 (socket says 172.21.15.76:49580) 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be1320700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5bdc0a53e0 0x7f5bdc13c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34076/0 (socket says 172.21.15.76:34076) 2024-09-16T06:46:18.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be0b1f700 1 -- 172.21.15.76:0/994485766 learned_addr learned my addr 172.21.15.76:0/994485766 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797562806 0 0) 0x7f5bdc072650 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): my global_id is 194153 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc80035d0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356678102 0 0) 0x7f5bdc11d510 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): my global_id is 184380 2024-09-16T06:46:18.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc072650 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401641856 0 0) 0x7f5bdc0a1710 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): my global_id is 174709 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc11d510 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338962777 0 0) 0x7f5bc80035d0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bdc0a1710 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4001950 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:18.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): dump: 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:18.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581375626 0 0) 0x7f5bdc072650 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc80035d0 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd00018b0 con 0x7f5bdc09e4a0 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient(hunting): dump: 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1263951870 0 0) 0x7f5bdc11d510 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bdc072650 con 0x7f5bdc0a48f0 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3424946934 0 0) 0x7f5bdc0a1710 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.189:6789/0 conn(0x7f5bdc0a48f0 legacy=0x7f5bdc13c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6789/0 conn(0x7f5bdc09e4a0 legacy=0x7f5bdc13bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 monclient: found mon.c 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc13e0e0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 5 monclient: authenticate success, global_id 194153 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _renew_subs 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc11cc70 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _renew_subs 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc1d32c0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd4003e60 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bd4004170 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bdaffd700 4 set_mon_vals no callback set 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 10 monclient: dump: 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5bd40042d0 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bbc006590 con 0x7f5bbc002830 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.620+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f5bd4045a50 con 0x7f5bdc0a53e0 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5be0b1f700 10 monclient: get_auth_request con 0x7f5bc8036550 auth_method 0 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679435014 0 0) 0x7f5bbc006590 con 0x7f5bbc002830 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient(hunting): my global_id is 184383 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc0a1710 con 0x7f5bbc002830 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1469105210 0 0) 0x7f5bdc0a1710 con 0x7f5bbc002830 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bbc006590 con 0x7f5bbc002830 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bcc004890 con 0x7f5bbc002830 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient: dump: 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2548978275 0 0) 0x7f5bbc006590 con 0x7f5bbc002830 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bbc00a190 con 0x7f5bbc002830 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5bcc006ad0 con 0x7f5bbc002830 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6789/0 conn(0x7f5bbc002830 legacy=0x7f5bbc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.624+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6789/0 conn(0x7f5bbc002830 legacy=0x7f5bbc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5be2d83700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:18.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bbc009e10 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927532801 0 0) 0x7f5bbc009e10 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient(hunting): my global_id is 184386 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc006590 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1227519722 0 0) 0x7f5bbc006590 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bbc009e10 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bcc004f00 con 0x7f5bbc009150 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient: dump: 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 856552553 0 0) 0x7f5bbc009e10 con 0x7f5bbc009150 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bbc00c340 con 0x7f5bbc009150 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 488909054) 0x7f5bcc006380 con 0x7f5bbc009150 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6789/0 conn(0x7f5bbc009150 legacy=0x7f5bbc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.632+0000 7f5bd97fa700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6789/0 conn(0x7f5bbc009150 legacy=0x7f5bbc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:18.662 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.664 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:18.665 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.666 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:18.667 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.668 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.670 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:18.671 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.672 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:18.673 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5bc8036550 legacy=0x7f5bc8038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 10 monclient: shutdown 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 >> v1:172.21.15.76:6790/0 conn(0x7f5bdc0a53e0 legacy=0x7f5bdc13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be1320700 1 -- 172.21.15.76:0/994485766 reap_dead start 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be1320700 1 -- 172.21.15.76:0/994485766 reap_dead start 2024-09-16T06:46:18.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 shutdown_connections 2024-09-16T06:46:18.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 >> 172.21.15.76:0/994485766 conn(0x7f5bdc0a24e0 msgr2=0x7f5bdc13eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 shutdown_connections 2024-09-16T06:46:18.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.636+0000 7f5be2d83700 1 -- 172.21.15.76:0/994485766 wait complete. 2024-09-16T06:46:18.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:18.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:46:18.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.792+0000 7ff9325a9700 10 monclient: get_monmap_and_config 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.792+0000 7ff9325a9700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.792+0000 7ff9325a9700 10 monclient: monmap: 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:18.799484+0000 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:18.799484+0000 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 Processor -- start 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- start start 2024-09-16T06:46:18.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient: init 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-a con 0x7ff92c0964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-c con 0x7ff92c09b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-b con 0x7ff92c137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c098560 con 0x7ff92c0964f0 2024-09-16T06:46:18.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c099ec0 con 0x7ff92c09b1c0 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c09b5f0 con 0x7ff92c137e70 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient(hunting): authenticate will time out at 1325.933016s 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff931da8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff92c137e70 0x7ff92c13a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34092/0 (socket says 172.21.15.76:34092) 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff931da8700 1 -- 172.21.15.76:0/2295798048 learned_addr learned my addr 172.21.15.76:0/2295798048 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633978843 0 0) 0x7ff92c098560 con 0x7ff92c0964f0 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): my global_id is 184389 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9180057e0 con 0x7ff92c0964f0 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879375809 0 0) 0x7ff92c09b5f0 con 0x7ff92c137e70 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient(hunting): my global_id is 194156 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c098560 con 0x7ff92c137e70 2024-09-16T06:46:18.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416153826 0 0) 0x7ff9180057e0 con 0x7ff92c0964f0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 >> v1:172.21.15.76:6790/0 conn(0x7ff92c137e70 legacy=0x7ff92c13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 >> v1:172.21.15.189:6789/0 conn(0x7ff92c09b1c0 legacy=0x7ff92c136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 monclient: found mon.noname-a 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c099be0 con 0x7ff92c0964f0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff928001990 con 0x7ff92c0964f0 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: dump: 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 5 monclient: authenticate success, global_id 184389 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff928001c30 con 0x7ff92c0964f0 2024-09-16T06:46:18.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 1 -- 172.21.15.76:0/2295798048 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff928001fc0 con 0x7ff92c0964f0 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 4 set_mon_vals no callback set 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff923fff700 10 monclient: dump: 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.807 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 10 monclient: shutdown 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- 172.21.15.76:0/2295798048 >> v1:172.21.15.76:6789/0 conn(0x7ff92c0964f0 legacy=0x7ff92c0968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- 172.21.15.76:0/2295798048 shutdown_connections 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- 172.21.15.76:0/2295798048 >> 172.21.15.76:0/2295798048 conn(0x7ff92c0a0030 msgr2=0x7ff92c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- 172.21.15.76:0/2295798048 shutdown_connections 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.796+0000 7ff9325a9700 1 -- 172.21.15.76:0/2295798048 wait complete. 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: monmap: 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.808 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 Processor -- start 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- start start 2024-09-16T06:46:18.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: init 2024-09-16T06:46:18.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-b con 0x7ff92c0964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-c con 0x7ff92c09b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient(hunting): picked mon.noname-a con 0x7ff92c137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c090210 con 0x7ff92c137e70 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c08dfc0 con 0x7ff92c0964f0 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0a2e20 con 0x7ff92c09b1c0 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient(hunting): authenticate will time out at 1325.937873s 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff931da8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff92c137e70 0x7ff92c1ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49618/0 (socket says 172.21.15.76:49618) 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff931da8700 1 -- 172.21.15.76:0/2668372378 learned_addr learned my addr 172.21.15.76:0/2668372378 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158221335 0 0) 0x7ff92c090210 con 0x7ff92c137e70 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): my global_id is 184392 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9100035d0 con 0x7ff92c137e70 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751630873 0 0) 0x7ff92c0a2e20 con 0x7ff92c09b1c0 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): my global_id is 194159 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c090210 con 0x7ff92c09b1c0 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198665438 0 0) 0x7ff92c08dfc0 con 0x7ff92c0964f0 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): my global_id is 174715 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c0a2e20 con 0x7ff92c0964f0 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211141028 0 0) 0x7ff9100035d0 con 0x7ff92c137e70 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff92c08dfc0 con 0x7ff92c137e70 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff9240018d0 con 0x7ff92c137e70 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.819 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954345942 0 0) 0x7ff92c090210 con 0x7ff92c09b1c0 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9100035d0 con 0x7ff92c09b1c0 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff91c001870 con 0x7ff92c09b1c0 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 866871639 0 0) 0x7ff92c08dfc0 con 0x7ff92c137e70 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6790/0 conn(0x7ff92c09b1c0 legacy=0x7ff92c08c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.189:6789/0 conn(0x7ff92c0964f0 legacy=0x7ff92c08c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 monclient: found mon.a 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c097d90 con 0x7ff92c137e70 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 5 monclient: authenticate success, global_id 184392 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _renew_subs 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c09a540 con 0x7ff92c137e70 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _renew_subs 2024-09-16T06:46:18.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c08fdf0 con 0x7ff92c137e70 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff924004890 con 0x7ff92c137e70 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff924004c20 con 0x7ff92c137e70 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff9237fe700 4 set_mon_vals no callback set 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.800+0000 7ff921ffb700 10 monclient: dump: 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.823 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff924004d80 con 0x7ff92c137e70 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff9325a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff90c006590 con 0x7ff90c002830 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7ff9240454b0 con 0x7ff92c137e70 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff930da6700 10 monclient: get_auth_request con 0x7ff910036210 auth_method 0 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714571601 0 0) 0x7ff90c006590 con 0x7ff90c002830 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient(hunting): my global_id is 194162 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c08dfc0 con 0x7ff90c002830 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1352103244 0 0) 0x7ff92c08dfc0 con 0x7ff90c002830 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff90c006590 con 0x7ff90c002830 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff928003cc0 con 0x7ff90c002830 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient: dump: 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2295522487 0 0) 0x7ff90c006590 con 0x7ff90c002830 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff90c00a190 con 0x7ff90c002830 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff928006020 con 0x7ff90c002830 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6790/0 conn(0x7ff90c002830 legacy=0x7ff90c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.804+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6790/0 conn(0x7ff90c002830 legacy=0x7ff90c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff9325a9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff90c009e50 con 0x7ff90c009170 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748847403 0 0) 0x7ff90c009e50 con 0x7ff90c009170 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient(hunting): my global_id is 194165 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff90c006590 con 0x7ff90c009170 2024-09-16T06:46:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1512106919 0 0) 0x7ff90c006590 con 0x7ff90c009170 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff90c009e50 con 0x7ff90c009170 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff928004490 con 0x7ff90c009170 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient: dump: 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 564188098 0 0) 0x7ff90c009e50 con 0x7ff90c009170 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff90c00c340 con 0x7ff90c009170 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3881 (unknown 0 0 3362884958) 0x7ff928005800 con 0x7ff90c009170 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6790/0 conn(0x7ff90c009170 legacy=0x7ff90c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.816+0000 7ff921ffb700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6790/0 conn(0x7ff90c009170 legacy=0x7ff90c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:18.830 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:18.831 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.832 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.833 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:18.834 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.835 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:18.838 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:18.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:18.840 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:18.841 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:18.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff910036210 legacy=0x7ff9100386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 10 monclient: shutdown 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 >> v1:172.21.15.76:6789/0 conn(0x7ff92c137e70 legacy=0x7ff92c1ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff931da8700 1 -- 172.21.15.76:0/2668372378 reap_dead start 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff931da8700 1 -- 172.21.15.76:0/2668372378 reap_dead start 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 shutdown_connections 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 >> 172.21.15.76:0/2668372378 conn(0x7ff92c0a0030 msgr2=0x7ff92c0a18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 shutdown_connections 2024-09-16T06:46:18.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.820+0000 7ff9325a9700 1 -- 172.21.15.76:0/2668372378 wait complete. 2024-09-16T06:46:18.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:18.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:46:18.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:46:18.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.956+0000 7fa697fff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:18.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.956+0000 7fa697fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.956+0000 7fa697fff700 10 monclient: monmap: 2024-09-16T06:46:18.961 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.961 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.962 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:18.963534+0000 2024-09-16T06:46:18.962 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:18.963534+0000 2024-09-16T06:46:18.962 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.962 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.962 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.963 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:18.963 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:18.963 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 Processor -- start 2024-09-16T06:46:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- start start 2024-09-16T06:46:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient: init 2024-09-16T06:46:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa6980a04e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa6980a0fd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa6980a1420 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806c400 con 0x7fa6980a1420 2024-09-16T06:46:18.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806c9a0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69806c1c0 con 0x7fa6980a04e0 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient(hunting): authenticate will time out at 1326.097358s 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa696ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa6980a04e0 0x7fa6980a08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34116/0 (socket says 172.21.15.76:34116) 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa696ffd700 1 -- 172.21.15.76:0/1761028907 learned_addr learned my addr 172.21.15.76:0/1761028907 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6977fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa6980a1420 0x7fa69813dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49624/0 (socket says 172.21.15.76:49624) 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914277634 0 0) 0x7fa69806c400 con 0x7fa6980a1420 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): my global_id is 184395 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6840035d0 con 0x7fa6980a1420 2024-09-16T06:46:18.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315948366 0 0) 0x7fa69806c1c0 con 0x7fa6980a04e0 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient(hunting): my global_id is 194168 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69806c400 con 0x7fa6980a04e0 2024-09-16T06:46:18.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248383390 0 0) 0x7fa6840035d0 con 0x7fa6980a1420 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 >> v1:172.21.15.76:6790/0 conn(0x7fa6980a04e0 legacy=0x7fa6980a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 >> v1:172.21.15.189:6789/0 conn(0x7fa6980a0fd0 legacy=0x7fa69813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69809f8f0 con 0x7fa6980a1420 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 5 monclient: authenticate success, global_id 184395 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa690001990 con 0x7fa6980a1420 2024-09-16T06:46:18.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: dump: 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa690001c70 con 0x7fa6980a1420 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 1 -- 172.21.15.76:0/1761028907 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa690002040 con 0x7fa6980a1420 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 4 set_mon_vals no callback set 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa6967fc700 10 monclient: dump: 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.988 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 10 monclient: shutdown 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- 172.21.15.76:0/1761028907 >> v1:172.21.15.76:6789/0 conn(0x7fa6980a1420 legacy=0x7fa69813dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- 172.21.15.76:0/1761028907 shutdown_connections 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- 172.21.15.76:0/1761028907 >> 172.21.15.76:0/1761028907 conn(0x7fa6980a5870 msgr2=0x7fa6980a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- 172.21.15.76:0/1761028907 shutdown_connections 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.960+0000 7fa697fff700 1 -- 172.21.15.76:0/1761028907 wait complete. 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: monmap: 2024-09-16T06:46:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 Processor -- start 2024-09-16T06:46:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- start start 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: init 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa6980a04e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa6980a0fd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa6980a1420 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa698098f00 con 0x7fa6980a0fd0 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69813d000 con 0x7fa6980a04e0 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69813d8a0 con 0x7fa6980a1420 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient(hunting): authenticate will time out at 1326.101301s 2024-09-16T06:46:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa6977fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa6980a1420 0x7fa6981d2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34126/0 (socket says 172.21.15.76:34126) 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa6977fe700 1 -- 172.21.15.76:0/337781060 learned_addr learned my addr 172.21.15.76:0/337781060 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230450490 0 0) 0x7fa698098f00 con 0x7fa6980a0fd0 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): my global_id is 184398 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6780035d0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948153872 0 0) 0x7fa69813d8a0 con 0x7fa6980a1420 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): my global_id is 194171 2024-09-16T06:46:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698098f00 con 0x7fa6980a1420 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226224209 0 0) 0x7fa6780035d0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69813d8a0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289794642 0 0) 0x7fa69813d000 con 0x7fa6980a04e0 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): my global_id is 174721 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6780035d0 con 0x7fa6980a04e0 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa680001850 con 0x7fa6980a0fd0 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987935428 0 0) 0x7fa698098f00 con 0x7fa6980a1420 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69813d000 con 0x7fa6980a1420 2024-09-16T06:46:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa690001f80 con 0x7fa6980a1420 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3064059583 0 0) 0x7fa69813d8a0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.76:6790/0 conn(0x7fa6980a1420 legacy=0x7fa6981d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6789/0 conn(0x7fa6980a04e0 legacy=0x7fa698139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 monclient: found mon.a 2024-09-16T06:46:18.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69813a850 con 0x7fa6980a0fd0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 5 monclient: authenticate success, global_id 184398 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _renew_subs 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69813c260 con 0x7fa6980a0fd0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _renew_subs 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:18.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69809d590 con 0x7fa6980a0fd0 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6800036c0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6800039d0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa695ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 10 monclient: dump: 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:18.998 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa68000e440 con 0x7fa6980a0fd0 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.964+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa67c006590 con 0x7fa67c002830 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7fa6800443d0 con 0x7fa6980a0fd0 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa6977fe700 10 monclient: get_auth_request con 0x7fa678036470 auth_method 0 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429549275 0 0) 0x7fa67c006590 con 0x7fa67c002830 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:18.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient(hunting): my global_id is 174724 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69813d8a0 con 0x7fa67c002830 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1940021550 0 0) 0x7fa69813d8a0 con 0x7fa67c002830 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa67c006590 con 0x7fa67c002830 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa688001740 con 0x7fa67c002830 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient: dump: 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.000 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 848483978 0 0) 0x7fa67c006590 con 0x7fa67c002830 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa67c00a190 con 0x7fa67c002830 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa688001b50 con 0x7fa67c002830 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6789/0 conn(0x7fa67c002830 legacy=0x7fa67c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.968+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6789/0 conn(0x7fa67c002830 legacy=0x7fa67c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa697fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa67c009e50 con 0x7fa67c009170 2024-09-16T06:46:19.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405493480 0 0) 0x7fa67c009e50 con 0x7fa67c009170 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient(hunting): my global_id is 174727 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa67c006590 con 0x7fa67c009170 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 78546102 0 0) 0x7fa67c006590 con 0x7fa67c009170 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa67c009e50 con 0x7fa67c009170 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa688001d80 con 0x7fa67c009170 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient: dump: 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.003 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2827600070 0 0) 0x7fa67c009e50 con 0x7fa67c009170 2024-09-16T06:46:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa67c00c340 con 0x7fa67c009170 2024-09-16T06:46:19.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2916280046) 0x7fa688005980 con 0x7fa67c009170 2024-09-16T06:46:19.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:19.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:19.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6789/0 conn(0x7fa67c009170 legacy=0x7fa67c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.976+0000 7fa694ff9700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6789/0 conn(0x7fa67c009170 legacy=0x7fa67c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:19.006 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:19.006 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:19.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:19.006 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:19.006 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 74, 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:19.007 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:19.008 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:19.009 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:19.010 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:19.011 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:19.012 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:19.013 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:19.014 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:19.016 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:19.017 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:19.018 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:19.019 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:19.022 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:19.023 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.024 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:19.025 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:19.026 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:19.027 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:19.028 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa678036470 legacy=0x7fa678038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 10 monclient: shutdown 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 >> v1:172.21.15.76:6789/0 conn(0x7fa6980a0fd0 legacy=0x7fa69813a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa6977fe700 1 -- 172.21.15.76:0/337781060 reap_dead start 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa6977fe700 1 -- 172.21.15.76:0/337781060 reap_dead start 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 shutdown_connections 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 >> 172.21.15.76:0/337781060 conn(0x7fa6980a5870 msgr2=0x7fa69809b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:19.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 shutdown_connections 2024-09-16T06:46:19.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:18.980+0000 7fa697fff700 1 -- 172.21.15.76:0/337781060 wait complete. 2024-09-16T06:46:19.030 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:19.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:46:19.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:46:19.030 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:46:19.030 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:46:19.031 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:46:19.031 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.034+0000 7fc182ffd700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 legacy=0x7fc184122d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.034+0000 7fc182ffd700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 legacy=0x7fc184122d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.034+0000 7fc182ffd700 1 --1- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 0x7fc184122d90 :-1 s=OPENED pgs=737 cs=1 l=1).handle_message read tag failed 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.034+0000 7fc182ffd700 1 --1- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 0x7fc184122d90 :-1 s=OPENED pgs=737 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: ms_handle_reset current mon v1:172.21.15.189:6789/0 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc1840a27a0 legacy=0x7fc184122d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:19.040 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): picked mon.b con 0x7fc170049d30 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc182ffd700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc170049d30 legacy=0x7fc17004c140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): picked mon.a con 0x7fc1841234a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): picked mon.c con 0x7fc1840a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc18409b670 con 0x7fc1841234a0 2024-09-16T06:46:19.041 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc18406bea0 con 0x7fc170049d30 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc1700571f0 con 0x7fc1840a2350 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565801253 0 0) 0x7fc18409b670 con 0x7fc1841234a0 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:46:19.042 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc170057380 con 0x7fc1841234a0 2024-09-16T06:46:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308034023 0 0) 0x7fc1700571f0 con 0x7fc1840a2350 2024-09-16T06:46:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:46:19.043 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc18409b670 con 0x7fc1840a2350 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1993941152 0 0) 0x7fc170057380 con 0x7fc1841234a0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.76:6790/0 conn(0x7fc1840a2350 legacy=0x7fc170053630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6789/0 conn(0x7fc170049d30 legacy=0x7fc17004c140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 monclient: found mon.a 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fc170058410 con 0x7fc1841234a0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7fc18406bea0 con 0x7fc1841234a0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _finish_auth 0 2024-09-16T06:46:19.044 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc17c001f80 con 0x7fc1841234a0 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.038+0000 7fc16bfff700 10 monclient: dump: 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.045 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.046 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.046 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:19.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.042+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc17c001de0 con 0x7fc1841234a0 2024-09-16T06:46:19.046 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.042+0000 7fc16bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:19.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.046+0000 7fc181ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:19.048 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:19.046+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc17c0017b0 con 0x7fc1841234a0 2024-09-16T06:46:19.079 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:46:19.079 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:46:19.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:19.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:19.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: build_initial_monmap 2024-09-16T06:46:19.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: monmap: 2024-09-16T06:46:19.203 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:19.194386+0000 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:19.194386+0000 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 1 Processor -- start 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 1 -- start start 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: init 2024-09-16T06:46:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c098990 con 0x7f5e5c099090 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c097f40 con 0x7f5e5c097b60 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09d610 con 0x7f5e5c139500 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5bfff700 10 monclient(hunting): authenticate will time out at 1326.327772s 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5affd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5e5c097b60 legacy=0x7f5e5c09ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e5c139500 0x7f5e5c13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34130/0 (socket says 172.21.15.76:34130) 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.188+0000 7f5e5a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5e5c099090 0x7f5e5c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49632/0 (socket says 172.21.15.76:49632) 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5b7fe700 1 -- 172.21.15.76:0/2221810178 learned_addr learned my addr 172.21.15.76:0/2221810178 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326167592 0 0) 0x7f5e5c09d610 con 0x7f5e5c139500 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient(hunting): my global_id is 194174 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e440035d0 con 0x7f5e5c139500 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116954216 0 0) 0x7f5e440035d0 con 0x7f5e5c139500 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 >> v1:172.21.15.189:6789/0 conn(0x7f5e5c097b60 legacy=0x7f5e5c09ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 >> v1:172.21.15.76:6789/0 conn(0x7f5e5c099090 legacy=0x7f5e5c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 monclient: found mon.noname-b 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0986b0 con 0x7f5e5c139500 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e54001990 con 0x7f5e5c139500 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 5 monclient: authenticate success, global_id 194174 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: dump: 2024-09-16T06:46:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e54001cb0 con 0x7f5e5c139500 2024-09-16T06:46:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 1 -- 172.21.15.76:0/2221810178 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e54002040 con 0x7f5e5c139500 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 4 set_mon_vals no callback set 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e59ffb700 10 monclient: dump: 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 10 monclient: shutdown 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 1 -- 172.21.15.76:0/2221810178 >> v1:172.21.15.76:6790/0 conn(0x7f5e5c139500 legacy=0x7f5e5c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 1 -- 172.21.15.76:0/2221810178 shutdown_connections 2024-09-16T06:46:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 1 -- 172.21.15.76:0/2221810178 >> 172.21.15.76:0/2221810178 conn(0x7f5e5c0a0030 msgr2=0x7f5e5c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 1 -- 172.21.15.76:0/2221810178 shutdown_connections 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 1 -- 172.21.15.76:0/2221810178 wait complete. 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 10 monclient: build_initial_monmap 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.192+0000 7f5e5bfff700 10 monclient: monmap: 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:19.211 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 Processor -- start 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- start start 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: init 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09bc90 con 0x7f5e5c097b60 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09bf70 con 0x7f5e5c099090 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5e5c097b60 0x7f5e5c070510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49634/0 (socket says 172.21.15.76:49634) 2024-09-16T06:46:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e5c139500 0x7f5e5c071330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34146/0 (socket says 172.21.15.76:34146) 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09c250 con 0x7f5e5c139500 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5affd700 1 -- 172.21.15.76:0/1820724485 learned_addr learned my addr 172.21.15.76:0/1820724485 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5a7fc700 1 -- 172.21.15.76:0/1820724485 >> v1:172.21.15.189:6789/0 conn(0x7f5e5c099090 legacy=0x7f5e5c070c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient(hunting): authenticate will time out at 1326.334256s 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122603160 0 0) 0x7f5e5c09c250 con 0x7f5e5c139500 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): my global_id is 194177 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e480035d0 con 0x7f5e5c139500 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056178424 0 0) 0x7f5e480035d0 con 0x7f5e5c139500 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e5c09c250 con 0x7f5e5c139500 2024-09-16T06:46:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e54001950 con 0x7f5e5c139500 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): dump: 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.214 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 926974396 0 0) 0x7f5e5c09c250 con 0x7f5e5c139500 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 >> v1:172.21.15.189:6789/0 conn(0x7f5e5c099090 legacy=0x7f5e5c070c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 >> v1:172.21.15.76:6789/0 conn(0x7f5e5c097b60 legacy=0x7f5e5c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 monclient: found mon.c 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c09b9b0 con 0x7f5e5c139500 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 5 monclient: authenticate success, global_id 194177 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _renew_subs 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c09b660 con 0x7f5e5c139500 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _renew_subs 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c1d2fb0 con 0x7f5e5c139500 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e54005070 con 0x7f5e5c139500 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e54005380 con 0x7f5e5c139500 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e58ff9700 4 set_mon_vals no callback set 2024-09-16T06:46:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.196+0000 7f5e43fff700 10 monclient: dump: 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5e5400fdf0 con 0x7f5e5c139500 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f5e54045b70 con 0x7f5e5c139500 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e5bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e5bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c005100 con 0x7f5e5c139500 2024-09-16T06:46:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.200+0000 7f5e5affd700 10 monclient: get_auth_request con 0x7f5e480360a0 auth_method 0 2024-09-16T06:46:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.208+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5e54014e30 con 0x7f5e5c139500 2024-09-16T06:46:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.208+0000 7f5e43fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.208+0000 7f5e43fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e5bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e5bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e3c005930 con 0x7f5e5c139500 2024-09-16T06:46:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e43fff700 1 -- 172.21.15.76:0/1820724485 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 791584438) 0x7f5e540132a0 con 0x7f5e5c139500 2024-09-16T06:46:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e43fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.436+0000 7f5e43fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:19.439 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5e480360a0 legacy=0x7f5e48038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 10 monclient: shutdown 2024-09-16T06:46:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 >> v1:172.21.15.76:6790/0 conn(0x7f5e5c139500 legacy=0x7f5e5c071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 shutdown_connections 2024-09-16T06:46:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 >> 172.21.15.76:0/1820724485 conn(0x7f5e5c0a0030 msgr2=0x7f5e5c1325b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 shutdown_connections 2024-09-16T06:46:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:19.440+0000 7f5e5bfff700 1 -- 172.21.15.76:0/1820724485 wait complete. 2024-09-16T06:46:21.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:21.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:21.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:21.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:21.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:21.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:21.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:21.334+0000 7fc16affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:22.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:22.338+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:22.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:22.338+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:22.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:22.338+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:22.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:22.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.571+0000 7f30c8b76700 10 monclient: get_monmap_and_config 2024-09-16T06:46:22.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.571+0000 7f30c8b76700 10 monclient: build_initial_monmap 2024-09-16T06:46:22.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.571+0000 7f30c8b76700 10 monclient: monmap: 2024-09-16T06:46:22.576 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:22.576 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:22.579337+0000 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:22.579337+0000 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:22.577 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 Processor -- start 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- start start 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient: init 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-b con 0x7f30c4132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-c con 0x7f30c406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-a con 0x7f30c406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c406c280 con 0x7f30c406cf50 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c406bfa0 con 0x7f30c406d480 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c409ace0 con 0x7f30c4132330 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient(hunting): authenticate will time out at 1329.713189s 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c2ffd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f30c406d480 legacy=0x7f30c413a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f30c406cf50 0x7f30c413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:49648/0 (socket says 172.21.15.76:49648) 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f30c4132330 0x7f30c4132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34158/0 (socket says 172.21.15.76:34158) 2024-09-16T06:46:22.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c3fff700 1 -- 172.21.15.76:0/329630426 learned_addr learned my addr 172.21.15.76:0/329630426 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127834021 0 0) 0x7f30c409ace0 con 0x7f30c4132330 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient(hunting): my global_id is 194180 2024-09-16T06:46:22.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac0035d0 con 0x7f30c4132330 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721922494 0 0) 0x7f30ac0035d0 con 0x7f30c4132330 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 >> v1:172.21.15.189:6789/0 conn(0x7f30c406d480 legacy=0x7f30c413a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 >> v1:172.21.15.76:6789/0 conn(0x7f30c406cf50 legacy=0x7f30c413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 monclient: found mon.noname-b 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c409ae70 con 0x7f30c4132330 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f30b4001610 con 0x7f30c4132330 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 5 monclient: authenticate success, global_id 194180 2024-09-16T06:46:22.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: dump: 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:22.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b4001930 con 0x7f30c4132330 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 1 -- 172.21.15.76:0/329630426 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f30b4001b20 con 0x7f30c4132330 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c27fc700 10 monclient: dump: 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:22.585 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 4 set_mon_vals no callback set 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 10 monclient: shutdown 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- 172.21.15.76:0/329630426 >> v1:172.21.15.76:6790/0 conn(0x7f30c4132330 legacy=0x7f30c4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:22.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- 172.21.15.76:0/329630426 shutdown_connections 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.575+0000 7f30c8b76700 1 -- 172.21.15.76:0/329630426 >> 172.21.15.76:0/329630426 conn(0x7f30c40a41c0 msgr2=0x7f30c40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- 172.21.15.76:0/329630426 shutdown_connections 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- 172.21.15.76:0/329630426 wait complete. 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient: build_initial_monmap 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient: monmap: 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:22.587 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:22.588 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 Processor -- start 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- start start 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient: init 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:22.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-b con 0x7f30c406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-c con 0x7f30c406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient(hunting): picked mon.noname-a con 0x7f30c4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c412df20 con 0x7f30c4132330 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c412ffe0 con 0x7f30c406cf50 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c41ca670 con 0x7f30c406d480 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c37fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f30c406cf50 legacy=0x7f30c41c9850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c8b76700 10 monclient(hunting): authenticate will time out at 1329.719814s 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f30c406d480 0x7f30c41c9f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34172/0 (socket says 172.21.15.76:34172) 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.579+0000 7f30c2ffd700 1 -- 172.21.15.76:0/3194190243 learned_addr learned my addr 172.21.15.76:0/3194190243 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624243350 0 0) 0x7f30c41ca670 con 0x7f30c406d480 2024-09-16T06:46:22.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): my global_id is 194183 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00035d0 con 0x7f30c406d480 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872923032 0 0) 0x7f30b00035d0 con 0x7f30c406d480 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30c41ca670 con 0x7f30c406d480 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f30b8004230 con 0x7f30c406d480 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): dump: 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:22.593 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1884905232 0 0) 0x7f30c41ca670 con 0x7f30c406d480 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 >> v1:172.21.15.189:6789/0 conn(0x7f30c406cf50 legacy=0x7f30c41c9850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 >> v1:172.21.15.76:6789/0 conn(0x7f30c4132330 legacy=0x7f30c41d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:22.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 monclient: found mon.c 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c409e8b0 con 0x7f30c406d480 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: _finish_auth 0 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 5 monclient: authenticate success, global_id 194183 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _renew_subs 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c412fc70 con 0x7f30c406d480 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _renew_subs 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c412e540 con 0x7f30c406d480 2024-09-16T06:46:22.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b8004450 con 0x7f30c406d480 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f30b80047e0 con 0x7f30c406d480 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 10 monclient: dump: 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:22.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c17fa700 4 set_mon_vals no callback set 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f30b8004940 con 0x7f30c406d480 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4005100 con 0x7f30c406d480 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f30b80455a0 con 0x7f30c406d480 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.583+0000 7f30c37fe700 10 monclient: get_auth_request con 0x7f30b0036180 auth_method 0 2024-09-16T06:46:22.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.591+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f30b8005110 con 0x7f30c406d480 2024-09-16T06:46:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.591+0000 7f30abfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:22.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.591+0000 7f30abfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f30a4005930 con 0x7f30c406d480 2024-09-16T06:46:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30abfff700 1 -- 172.21.15.76:0/3194190243 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2143441047) 0x7f30b8012b90 con 0x7f30c406d480 2024-09-16T06:46:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30abfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30abfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:22.830 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 >> v1:172.21.15.189:6812/3098810674 conn(0x7f30b0036180 legacy=0x7f30b0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 10 monclient: shutdown 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 >> v1:172.21.15.76:6790/0 conn(0x7f30c406d480 legacy=0x7f30c41c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 shutdown_connections 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 >> 172.21.15.76:0/3194190243 conn(0x7f30c40a41c0 msgr2=0x7f30c409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 shutdown_connections 2024-09-16T06:46:22.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:22.827+0000 7f30c8b76700 1 -- 172.21.15.76:0/3194190243 wait complete. 2024-09-16T06:46:23.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:23.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:23.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:23.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:23.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:23.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:24.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:24.338+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:24.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:24.338+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:24.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:24.338+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:25.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:25.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:25.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:25.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:25.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:25.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:25.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: get_monmap_and_config 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: build_initial_monmap 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: monmap: 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:25.964824+0000 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:25.964824+0000 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:25.962 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:25.963 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 1 Processor -- start 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 1 -- start start 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: init 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-c con 0x7f7d18132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-b con 0x7f7d1806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:25.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-a con 0x7f7d1806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1806c280 con 0x7f7d1806cf50 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1806bfa0 con 0x7f7d18132330 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1809ace0 con 0x7f7d1806d480 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1e021700 10 monclient(hunting): authenticate will time out at 1333.098976s 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1c81e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7d1806d480 0x7f7d1813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59646/0 (socket says 172.21.15.76:59646) 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1c81e700 1 -- 172.21.15.76:0/2050582134 learned_addr learned my addr 172.21.15.76:0/2050582134 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:25.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:25.963+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d18132330 legacy=0x7f7d18132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:26.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:26.167+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d18132330 legacy=0x7f7d18132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:26.338 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:26.338+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:26.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:26.338+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:26.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:26.338+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:26.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:26.571+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d18132330 legacy=0x7f7d18132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:27.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:27.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:27.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:27.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:27.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:27.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:27.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:27.375+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d18132330 legacy=0x7f7d18132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:28.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:28.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:28.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:28.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): tick 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6790/0 conn(0x7f7d1806d480 legacy=0x7f7d1813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:28.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d18132330 legacy=0x7f7d18132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6789/0 conn(0x7f7d1806cf50 legacy=0x7f7d1813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-c con 0x7f7d04003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f7d1806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f7d0400ab10 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1806bfa0 con 0x7f7d0400ab10 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0400f860 con 0x7f7d04003740 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d04010a70 con 0x7f7d1806d480 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d16ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:28.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:28.963+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:29.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:29.167+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:29.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:29.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:29.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:29.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:29.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:29.571+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:30.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:30.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:30.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:30.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:30.375+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:31.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:31.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:31.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:31.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:31.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:31.334+0000 7fc16affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): tick 2024-09-16T06:46:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:46:31.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6790/0 conn(0x7f7d1806d480 legacy=0x7f7d040093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6789/0 conn(0x7f7d0400ab10 legacy=0x7f7d0400cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-c con 0x7f7d04003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f7d04010db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f7d1806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d04011c80 con 0x7f7d04010db0 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0400f860 con 0x7f7d04003740 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d04011a40 con 0x7f7d1806d480 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d16ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:31.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:31.963+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:32.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:32.167+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:32.339 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:32.338+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:32.338+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:32.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:32.338+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:32.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:32.571+0000 7f7d1d01f700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022858819 0 0) 0x7f7d04011c80 con 0x7f7d04010db0 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient(hunting): my global_id is 184401 2024-09-16T06:46:33.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d000035d0 con 0x7f7d04010db0 2024-09-16T06:46:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691816023 0 0) 0x7f7d000035d0 con 0x7f7d04010db0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6790/0 conn(0x7f7d1806d480 legacy=0x7f7d040093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.189:6789/0 conn(0x7f7d04003740 legacy=0x7f7d04005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 monclient: found mon.noname-a 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0400bff0 con 0x7f7d04010db0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d0c001510 con 0x7f7d04010db0 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: dump: 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 5 monclient: authenticate success, global_id 184401 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d0c001780 con 0x7f7d04010db0 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 1 -- 172.21.15.76:0/2050582134 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d0c001b10 con 0x7f7d04010db0 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 4 set_mon_vals no callback set 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d17fff700 10 monclient: dump: 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.131 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 10 monclient: shutdown 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 1 -- 172.21.15.76:0/2050582134 >> v1:172.21.15.76:6789/0 conn(0x7f7d04010db0 legacy=0x7f7d04011190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 1 -- 172.21.15.76:0/2050582134 shutdown_connections 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 1 -- 172.21.15.76:0/2050582134 >> 172.21.15.76:0/2050582134 conn(0x7f7d180a41c0 msgr2=0x7f7d180a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 1 -- 172.21.15.76:0/2050582134 shutdown_connections 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.127+0000 7f7d1e021700 1 -- 172.21.15.76:0/2050582134 wait complete. 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: build_initial_monmap 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: monmap: 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 Processor -- start 2024-09-16T06:46:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- start start 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: init 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-a con 0x7f7d04003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-b con 0x7f7d04010db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient(hunting): picked mon.noname-c con 0x7f7d1806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d180a7340 con 0x7f7d04003740 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d18131c70 con 0x7f7d04010db0 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1809e8b0 con 0x7f7d1806d480 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient(hunting): authenticate will time out at 1340.266027s 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1d01f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7d04003740 0x7f7d18130030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50652/0 (socket says 172.21.15.76:50652) 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1c81e700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7d04010db0 legacy=0x7f7d181cf0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1d01f700 1 -- 172.21.15.76:0/1626699513 learned_addr learned my addr 172.21.15.76:0/1626699513 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558143518 0 0) 0x7f7d180a7340 con 0x7f7d04003740 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): my global_id is 184404 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0400d380 con 0x7f7d04003740 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2823860362 0 0) 0x7f7d0400d380 con 0x7f7d04003740 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d180a7340 con 0x7f7d04003740 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d080047b0 con 0x7f7d04003740 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1608641267 0 0) 0x7f7d180a7340 con 0x7f7d04003740 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 >> v1:172.21.15.76:6790/0 conn(0x7f7d1806d480 legacy=0x7f7d181d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 >> v1:172.21.15.189:6789/0 conn(0x7f7d04010db0 legacy=0x7f7d181cf0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 monclient: found mon.a 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d18124780 con 0x7f7d04003740 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 5 monclient: authenticate success, global_id 184404 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _renew_subs 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d18124a60 con 0x7f7d04003740 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _renew_subs 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d18122700 con 0x7f7d04003740 2024-09-16T06:46:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d08004c40 con 0x7f7d04003740 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d08004fd0 con 0x7f7d04003740 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d177fe700 4 set_mon_vals no callback set 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.131+0000 7f7d15ffb700 10 monclient: dump: 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.140 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d1e021700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7d080051b0 con 0x7f7d04003740 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d1e021700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d1e021700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf8005100 con 0x7f7d04003740 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f7d08045ac0 con 0x7f7d04003740 2024-09-16T06:46:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.135+0000 7f7d1c81e700 10 monclient: get_auth_request con 0x7f7d040377f0 auth_method 0 2024-09-16T06:46:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.139+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7d08006700 con 0x7f7d04003740 2024-09-16T06:46:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.139+0000 7f7d15ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.139+0000 7f7d15ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:33.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:33.334+0000 7fc16affd700 10 monclient: tick 2024-09-16T06:46:33.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:33.334+0000 7fc16affd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:33.334+0000 7fc16affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d1e021700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:33.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d1e021700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:33.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d1e021700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7cf8005930 con 0x7f7d04003740 2024-09-16T06:46:33.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d15ffb700 1 -- 172.21.15.76:0/1626699513 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 4056141227) 0x7f7d080130b0 con 0x7f7d04003740 2024-09-16T06:46:33.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d15ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:33.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.375+0000 7f7d15ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:33.374 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7d040377f0 legacy=0x7f7d04039cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 10 monclient: shutdown 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 >> v1:172.21.15.76:6789/0 conn(0x7f7d04003740 legacy=0x7f7d18130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 shutdown_connections 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 >> 172.21.15.76:0/1626699513 conn(0x7f7d180a41c0 msgr2=0x7f7d1809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 shutdown_connections 2024-09-16T06:46:33.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.379+0000 7f7d1e021700 1 -- 172.21.15.76:0/1626699513 wait complete. 2024-09-16T06:46:33.392 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:46:33.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:46:33.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: get_monmap_and_config 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: build_initial_monmap 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: monmap: 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:33.508962+0000 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:33.508962+0000 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:33.507 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 Processor -- start 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 -- start start 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: init 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-c con 0x7f835c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-a con 0x7f835c098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-b con 0x7f835c099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:33.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c09d560 con 0x7f835c098be0 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c09f140 con 0x7f835c09ed60 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c09d790 con 0x7f835c099030 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient(hunting): authenticate will time out at 1340.642173s 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83619e4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f835c098be0 0x7f835c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50656/0 (socket says 172.21.15.76:50656) 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83619e4700 1 -- 172.21.15.76:0/1086033844 learned_addr learned my addr 172.21.15.76:0/1086033844 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83621e5700 1 -- 172.21.15.76:0/1086033844 >> v1:172.21.15.189:6789/0 conn(0x7f835c09ed60 legacy=0x7f835c0984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372076543 0 0) 0x7f835c09d560 con 0x7f835c098be0 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): my global_id is 184407 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83480035d0 con 0x7f835c098be0 2024-09-16T06:46:33.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650815262 0 0) 0x7f835c09d790 con 0x7f835c099030 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient(hunting): my global_id is 194186 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c09d560 con 0x7f835c099030 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054984441 0 0) 0x7f83480035d0 con 0x7f835c098be0 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 >> v1:172.21.15.76:6790/0 conn(0x7f835c099030 legacy=0x7f835c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 >> v1:172.21.15.189:6789/0 conn(0x7f835c09ed60 legacy=0x7f835c0984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 monclient: found mon.noname-a 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c09f620 con 0x7f835c098be0 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 5 monclient: authenticate success, global_id 184407 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f834c001610 con 0x7f835c098be0 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: dump: 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.518 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f834c0018f0 con 0x7f835c098be0 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 4 set_mon_vals no callback set 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 1 -- 172.21.15.76:0/1086033844 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f834c001c80 con 0x7f835c098be0 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83611e3700 10 monclient: dump: 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 10 monclient: shutdown 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.507+0000 7f83631e7700 1 -- 172.21.15.76:0/1086033844 >> v1:172.21.15.76:6789/0 conn(0x7f835c098be0 legacy=0x7f835c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- 172.21.15.76:0/1086033844 shutdown_connections 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- 172.21.15.76:0/1086033844 >> 172.21.15.76:0/1086033844 conn(0x7f835c0a0030 msgr2=0x7f835c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- 172.21.15.76:0/1086033844 shutdown_connections 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- 172.21.15.76:0/1086033844 wait complete. 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient: build_initial_monmap 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient: monmap: 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:33.521 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 Processor -- start 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- start start 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient: init 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-c con 0x7f835c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-a con 0x7f835c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:33.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient(hunting): picked mon.noname-b con 0x7f835c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c09a5b0 con 0x7f835c099030 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0927d0 con 0x7f835c09ed60 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c092d90 con 0x7f835c098be0 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83631e7700 10 monclient(hunting): authenticate will time out at 1340.646815s 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83621e5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f835c098be0 0x7f835c0965c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59678/0 (socket says 172.21.15.76:59678) 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83621e5700 1 -- 172.21.15.76:0/164708819 learned_addr learned my addr 172.21.15.76:0/164708819 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f83629e6700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.189:6789/0 conn(0x7f835c09ed60 legacy=0x7f835c1d0090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107562409 0 0) 0x7f835c092d90 con 0x7f835c098be0 2024-09-16T06:46:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): my global_id is 194189 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83400035d0 con 0x7f835c098be0 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734819812 0 0) 0x7f835c09f620 con 0x7f835c099030 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): my global_id is 184410 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c092d90 con 0x7f835c099030 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129479952 0 0) 0x7f835c092d90 con 0x7f835c099030 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c09f620 con 0x7f835c099030 2024-09-16T06:46:33.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f834c001a60 con 0x7f835c099030 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121742447 0 0) 0x7f83400035d0 con 0x7f835c098be0 2024-09-16T06:46:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c092d90 con 0x7f835c098be0 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8358004230 con 0x7f835c098be0 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.511+0000 7f8352ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4105317497 0 0) 0x7f835c09f620 con 0x7f835c099030 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6790/0 conn(0x7f835c098be0 legacy=0x7f835c0965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.189:6789/0 conn(0x7f835c09ed60 legacy=0x7f835c1d0090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 monclient: found mon.a 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c092ab0 con 0x7f835c099030 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 5 monclient: authenticate success, global_id 184410 2024-09-16T06:46:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 10 monclient: _renew_subs 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c0918d0 con 0x7f835c099030 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 10 monclient: _renew_subs 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c090b80 con 0x7f835c099030 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f834c001d30 con 0x7f835c099030 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f834c0010a0 con 0x7f835c099030 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83609e2700 4 set_mon_vals no callback set 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: dump: 2024-09-16T06:46:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f835c0970b0 con 0x7f835c096cd0 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f834c004ce0 con 0x7f835c099030 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f834c045460 con 0x7f835c099030 2024-09-16T06:46:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183438767 0 0) 0x7f835c0970b0 con 0x7f835c096cd0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient(hunting): my global_id is 184413 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c09f620 con 0x7f835c096cd0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f83629e6700 10 monclient: get_auth_request con 0x7f83400362c0 auth_method 0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3620018866 0 0) 0x7f835c09f620 con 0x7f835c096cd0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c0970b0 con 0x7f835c096cd0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83580048c0 con 0x7f835c096cd0 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: dump: 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1091886681 0 0) 0x7f835c0970b0 con 0x7f835c096cd0 2024-09-16T06:46:33.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f835c09aaa0 con 0x7f835c096cd0 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8358004cd0 con 0x7f835c096cd0 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6789/0 conn(0x7f835c096cd0 legacy=0x7f835c061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.515+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6789/0 conn(0x7f835c096cd0 legacy=0x7f835c061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f835c097810 con 0x7f835c02c820 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950887687 0 0) 0x7f835c092d90 con 0x7f835c02c820 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient(hunting): my global_id is 184416 2024-09-16T06:46:33.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c0970b0 con 0x7f835c02c820 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2856891206 0 0) 0x7f835c0970b0 con 0x7f835c02c820 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c092d90 con 0x7f835c02c820 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8358005140 con 0x7f835c02c820 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient: dump: 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.534 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 252793137 0 0) 0x7f835c092d90 con 0x7f835c02c820 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f835c09b0f0 con 0x7f835c02c820 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4200 (unknown 0 0 1746552565) 0x7f8358006710 con 0x7f835c02c820 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6789/0 conn(0x7f835c02c820 legacy=0x7f835c070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f8352ffd700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6789/0 conn(0x7f835c02c820 legacy=0x7f835c070f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:33.535 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 78, 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:33.536 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:33.537 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:33.538 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:33.539 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:33.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.541 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:33.542 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.543 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.545 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.546 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:33.547 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.189:6812/3098810674 conn(0x7f83400362c0 legacy=0x7f8340038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 10 monclient: shutdown 2024-09-16T06:46:33.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 >> v1:172.21.15.76:6789/0 conn(0x7f835c099030 legacy=0x7f835c1cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83629e6700 1 -- 172.21.15.76:0/164708819 reap_dead start 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83629e6700 1 -- 172.21.15.76:0/164708819 reap_dead start 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 shutdown_connections 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 >> 172.21.15.76:0/164708819 conn(0x7f835c0a0030 msgr2=0x7f835c06e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 shutdown_connections 2024-09-16T06:46:33.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.523+0000 7f83631e7700 1 -- 172.21.15.76:0/164708819 wait complete. 2024-09-16T06:46:33.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:46:33.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:46:33.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: build_initial_monmap 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: monmap: 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:33.670846+0000 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:33.670846+0000 2024-09-16T06:46:33.668 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 1 Processor -- start 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 1 -- start start 2024-09-16T06:46:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: init 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.667+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f75a0097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f75a0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f75a0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0098990 con 0x7f75a0099090 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0097f40 con 0x7f75a0139500 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a009d610 con 0x7f75a0097b60 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): authenticate will time out at 1340.804084s 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f75a0139500 legacy=0x7f75a013b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f75a0099090 0x7f75a009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50694/0 (socket says 172.21.15.76:50694) 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759e7fc700 1 -- 172.21.15.76:0/73383723 learned_addr learned my addr 172.21.15.76:0/73383723 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759effd700 1 --1- 172.21.15.76:0/73383723 >> v1:172.21.15.76:6790/0 conn(0x7f75a0097b60 0x7f75a009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59686/0 (socket says 172.21.15.76:59686) 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268693626 0 0) 0x7f75a0098990 con 0x7f75a0099090 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): my global_id is 184419 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75880035d0 con 0x7f75a0099090 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132073302 0 0) 0x7f75a009d610 con 0x7f75a0097b60 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient(hunting): my global_id is 194192 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0098990 con 0x7f75a0097b60 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215242270 0 0) 0x7f75880035d0 con 0x7f75a0099090 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 >> v1:172.21.15.76:6790/0 conn(0x7f75a0097b60 legacy=0x7f75a009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 >> v1:172.21.15.189:6789/0 conn(0x7f75a0139500 legacy=0x7f75a013b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 monclient: found mon.noname-a 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00986b0 con 0x7f75a0099090 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 5 monclient: authenticate success, global_id 184419 2024-09-16T06:46:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7590001610 con 0x7f75a0099090 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: dump: 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.680 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7590001830 con 0x7f75a0099090 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 1 -- 172.21.15.76:0/73383723 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7590001bc0 con 0x7f75a0099090 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759dffb700 10 monclient: dump: 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.681 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 4 set_mon_vals no callback set 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: shutdown 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- 172.21.15.76:0/73383723 >> v1:172.21.15.76:6789/0 conn(0x7f75a0099090 legacy=0x7f75a009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- 172.21.15.76:0/73383723 shutdown_connections 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- 172.21.15.76:0/73383723 >> 172.21.15.76:0/73383723 conn(0x7f75a00a0030 msgr2=0x7f75a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- 172.21.15.76:0/73383723 shutdown_connections 2024-09-16T06:46:33.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- 172.21.15.76:0/73383723 wait complete. 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: build_initial_monmap 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: monmap: 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 Processor -- start 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- start start 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: init 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f75a0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f75a0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f75a0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a00a12c0 con 0x7f75a0099090 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0138af0 con 0x7f75a0097b60 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a01369d0 con 0x7f75a0139500 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759ffff700 10 monclient(hunting): authenticate will time out at 1340.807582s 2024-09-16T06:46:33.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f75a0099090 0x7f75a01362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50698/0 (socket says 172.21.15.76:50698) 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759e7fc700 1 -- 172.21.15.76:0/3947970194 learned_addr learned my addr 172.21.15.76:0/3947970194 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.671+0000 7f759effd700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.189:6789/0 conn(0x7f75a0097b60 legacy=0x7f75a0135bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145001025 0 0) 0x7f75a00986b0 con 0x7f75a0099090 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): my global_id is 184422 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f758c0035d0 con 0x7f75a0099090 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813026350 0 0) 0x7f75a01369d0 con 0x7f75a0139500 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): my global_id is 194195 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a00986b0 con 0x7f75a0139500 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2874315175 0 0) 0x7f75a00986b0 con 0x7f75a0139500 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a01369d0 con 0x7f75a0139500 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7598004390 con 0x7f75a0139500 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): dump: 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.686 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014598530 0 0) 0x7f758c0035d0 con 0x7f75a0099090 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a00986b0 con 0x7f75a0099090 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7590001370 con 0x7f75a0099090 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): dump: 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.687 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3310294499 0 0) 0x7f75a01369d0 con 0x7f75a0139500 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.189:6789/0 conn(0x7f75a0097b60 legacy=0x7f75a0135bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6789/0 conn(0x7f75a0099090 legacy=0x7f75a01362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:33.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 monclient: found mon.c 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0071a60 con 0x7f75a0139500 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 5 monclient: authenticate success, global_id 194195 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 10 monclient: _renew_subs 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a009a540 con 0x7f75a0139500 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 10 monclient: _renew_subs 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:33.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a0132270 con 0x7f75a0139500 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7598004600 con 0x7f75a0139500 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7598004990 con 0x7f75a0139500 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: dump: 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.690 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759d7fa700 4 set_mon_vals no callback set 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7598004af0 con 0x7f75a0139500 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7580006590 con 0x7f7580002830 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(219..219 src has 151..219) v4 ==== 12812+0+0 (unknown 1622609905 0 0) 0x7f7598045750 con 0x7f75a0139500 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f759e7fc700 10 monclient: get_auth_request con 0x7f758c0362d0 auth_method 0 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381388445 0 0) 0x7f7580006590 con 0x7f7580002830 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient(hunting): my global_id is 194198 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a01369d0 con 0x7f7580002830 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2126279091 0 0) 0x7f75a01369d0 con 0x7f7580002830 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7580006590 con 0x7f7580002830 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7594001740 con 0x7f7580002830 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: dump: 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.692 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3653496035 0 0) 0x7f7580006590 con 0x7f7580002830 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f758000a190 con 0x7f7580002830 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7594001b00 con 0x7f7580002830 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6790/0 conn(0x7f7580002830 legacy=0x7f7580004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.675+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6790/0 conn(0x7f7580002830 legacy=0x7f7580004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f759ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7580009e30 con 0x7f7580009150 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052394091 0 0) 0x7f7580009e30 con 0x7f7580009150 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient(hunting): my global_id is 194201 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7580006590 con 0x7f7580009150 2024-09-16T06:46:33.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3672808261 0 0) 0x7f7580006590 con 0x7f7580009150 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7580009e30 con 0x7f7580009150 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7594004a30 con 0x7f7580009150 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 10 monclient: dump: 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:33.696 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3945128542 0 0) 0x7f7580009e30 con 0x7f7580009150 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.683+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f758000c340 con 0x7f7580009150 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4036 (unknown 0 0 728014484) 0x7f7594005fd0 con 0x7f7580009150 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f7587fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f7587fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6790/0 conn(0x7f7580009150 legacy=0x7f75800095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f7587fff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6790/0 conn(0x7f7580009150 legacy=0x7f75800095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:33.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 78, 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.698 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:33.699 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:33.700 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:33.701 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.702 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.703 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:33.704 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:33.705 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.706 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.707 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:33.708 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.189:6812/3098810674 conn(0x7f758c0362d0 legacy=0x7f758c038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 10 monclient: shutdown 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 >> v1:172.21.15.76:6790/0 conn(0x7f75a0139500 legacy=0x7f75a01d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759f7fe700 1 -- 172.21.15.76:0/3947970194 reap_dead start 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759f7fe700 1 -- 172.21.15.76:0/3947970194 reap_dead start 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 shutdown_connections 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 >> 172.21.15.76:0/3947970194 conn(0x7f75a00a0030 msgr2=0x7f75a00702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:33.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 shutdown_connections 2024-09-16T06:46:33.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:33.687+0000 7f759ffff700 1 -- 172.21.15.76:0/3947970194 wait complete. 2024-09-16T06:46:33.711 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:46:33.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:46:34.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.126+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v220) v1 ==== 118+0+0 (unknown 3863250310 0 0) 0x7fc17c014340 con 0x7fc1841234a0 2024-09-16T06:46:34.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.126+0000 7fc16bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:34.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.126+0000 7fc16bfff700 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-16T06:46:34.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.238+0000 7fc189fbb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:34.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.238+0000 7fc189fbb700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.238+0000 7fc189fbb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.238+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7fc164005650 con 0x7fc1841234a0 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc16bfff700 1 -- 172.21.15.189:0/1179214241 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v220) v1 ==== 118+0+0 (unknown 1139029950 0 0) 0x7fc17c014340 con 0x7fc1841234a0 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc16bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc16bfff700 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc1700363e0 legacy=0x7fc1700388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 10 monclient: shutdown 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 >> v1:172.21.15.76:6789/0 conn(0x7fc1841234a0 legacy=0x7fc17004faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 shutdown_connections 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 >> 172.21.15.189:0/1179214241 conn(0x7fc1840a7d70 msgr2=0x7fc18409ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:34.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 shutdown_connections 2024-09-16T06:46:34.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.242+0000 7fc189fbb700 1 -- 172.21.15.189:0/1179214241 wait complete. 2024-09-16T06:46:34.249 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h7' does not appear in the crush map 2024-09-16T06:46:34.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:34.257 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h8 2024-09-16T06:46:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: get_monmap_and_config 2024-09-16T06:46:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: build_initial_monmap 2024-09-16T06:46:34.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: monmap: 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:34.362013+0000 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:34.362013+0000 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:34.362 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 1 Processor -- start 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 1 -- start start 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: init 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-c con 0x7f1f6009f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-a con 0x7f1f60099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-b con 0x7f1f601396a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6009f520 con 0x7f1f60099700 2024-09-16T06:46:34.364 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6009bc10 con 0x7f1f6009f140 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6009b810 con 0x7f1f601396a0 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f654fc700 10 monclient(hunting): authenticate will time out at 1346.875069s 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f5effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1f6009f140 legacy=0x7f1f6009a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f5e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1f60099700 0x7f1f6009b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54816/0 (socket says 172.21.15.189:54816) 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.358+0000 7f1f5e7fc700 1 -- 172.21.15.189:0/1613546310 learned_addr learned my addr 172.21.15.189:0/1613546310 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143888853 0 0) 0x7f1f6009f520 con 0x7f1f60099700 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): my global_id is 184425 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c0035d0 con 0x7f1f60099700 2024-09-16T06:46:34.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414609486 0 0) 0x7f1f6009b810 con 0x7f1f601396a0 2024-09-16T06:46:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient(hunting): my global_id is 194204 2024-09-16T06:46:34.366 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6009f520 con 0x7f1f601396a0 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726629557 0 0) 0x7f1f4c0035d0 con 0x7f1f60099700 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 >> v1:172.21.15.76:6790/0 conn(0x7f1f601396a0 legacy=0x7f1f6013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 >> v1:172.21.15.189:6789/0 conn(0x7f1f6009f140 legacy=0x7f1f6009a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 monclient: found mon.noname-a 2024-09-16T06:46:34.368 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6009ceb0 con 0x7f1f60099700 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 5 monclient: authenticate success, global_id 184425 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f50001610 con 0x7f1f60099700 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: dump: 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.369 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f50001830 con 0x7f1f60099700 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 1 -- 172.21.15.189:0/1613546310 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f50001c00 con 0x7f1f60099700 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:34.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 4 set_mon_vals no callback set 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f5dffb700 10 monclient: dump: 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: shutdown 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- 172.21.15.189:0/1613546310 >> v1:172.21.15.76:6789/0 conn(0x7f1f60099700 legacy=0x7f1f6009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- 172.21.15.189:0/1613546310 shutdown_connections 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- 172.21.15.189:0/1613546310 >> 172.21.15.189:0/1613546310 conn(0x7f1f600a03e0 msgr2=0x7f1f600a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- 172.21.15.189:0/1613546310 shutdown_connections 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- 172.21.15.189:0/1613546310 wait complete. 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: build_initial_monmap 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: monmap: 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:34.372 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 Processor -- start 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- start start 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: init 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-a con 0x7f1f60099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-c con 0x7f1f6009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:34.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient(hunting): picked mon.noname-b con 0x7f1f601396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6013c410 con 0x7f1f60099700 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6006c8a0 con 0x7f1f601396a0 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f600a2d40 con 0x7f1f6009f140 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.362+0000 7f1f654fc700 10 monclient(hunting): authenticate will time out at 1346.879966s 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f5f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f1f601396a0 legacy=0x7f1f601d0000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f5e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1f6009f140 0x7f1f6006c190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35214/0 (socket says 172.21.15.189:35214) 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f5e7fc700 1 -- 172.21.15.189:0/2763103328 learned_addr learned my addr 172.21.15.189:0/2763103328 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:34.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f5effd700 1 --1- 172.21.15.189:0/2763103328 >> v1:172.21.15.76:6789/0 conn(0x7f1f60099700 0x7f1f6006ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54830/0 (socket says 172.21.15.189:54830) 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445067380 0 0) 0x7f1f600a2d40 con 0x7f1f6009f140 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): my global_id is 194207 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f400035d0 con 0x7f1f6009f140 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426496219 0 0) 0x7f1f48003af0 con 0x7f1f60099700 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:34.378 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): my global_id is 184428 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f600a2d40 con 0x7f1f60099700 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1322737548 0 0) 0x7f1f400035d0 con 0x7f1f6009f140 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f48003af0 con 0x7f1f6009f140 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f50001b80 con 0x7f1f6009f140 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): dump: 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:34.379 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158946405 0 0) 0x7f1f600a2d40 con 0x7f1f60099700 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f400035d0 con 0x7f1f60099700 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f48004160 con 0x7f1f60099700 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient(hunting): dump: 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:34.380 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1415541136 0 0) 0x7f1f48003af0 con 0x7f1f6009f140 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 >> v1:172.21.15.189:6789/0 conn(0x7f1f601396a0 legacy=0x7f1f601d0000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 >> v1:172.21.15.76:6789/0 conn(0x7f1f60099700 legacy=0x7f1f6006ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 monclient: found mon.c 2024-09-16T06:46:34.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6009a6e0 con 0x7f1f6009f140 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 5 monclient: authenticate success, global_id 194207 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _renew_subs 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6006e750 con 0x7f1f6009f140 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _renew_subs 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6006cd90 con 0x7f1f6009f140 2024-09-16T06:46:34.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f50004b90 con 0x7f1f6009f140 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1f50004f20 con 0x7f1f6009f140 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f5d7fa700 4 set_mon_vals no callback set 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 10 monclient: dump: 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:34.383 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1f50005100 con 0x7f1f6009f140 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.366+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f44005100 con 0x7f1f6009f140 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.370+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(220..220 src has 151..220) v4 ==== 12602+0+0 (unknown 1749261712 0 0) 0x7f1f50045820 con 0x7f1f6009f140 2024-09-16T06:46:34.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.370+0000 7f1f5effd700 10 monclient: get_auth_request con 0x7f1f400362a0 auth_method 0 2024-09-16T06:46:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.378+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1f50014b30 con 0x7f1f6009f140 2024-09-16T06:46:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.378+0000 7f1f577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:34.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.378+0000 7f1f577fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:34.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.554+0000 7f1f654fc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:34.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.554+0000 7f1f654fc700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:34.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.554+0000 7f1f654fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:34.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:34.554+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f1f440053b0 con 0x7f1f6009f140 2024-09-16T06:46:35.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.134+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v221) v1 ==== 118+0+0 (unknown 3811790355 0 0) 0x7f1f50012fa0 con 0x7f1f6009f140 2024-09-16T06:46:35.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.134+0000 7f1f577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:35.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.134+0000 7f1f577fe700 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-16T06:46:35.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.250+0000 7f1f654fc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:35.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.250+0000 7f1f654fc700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:35.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.250+0000 7f1f654fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.250+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f1f440023c0 con 0x7f1f6009f140 2024-09-16T06:46:35.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f577fe700 1 -- 172.21.15.189:0/2763103328 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v221) v1 ==== 118+0+0 (unknown 2422100913 0 0) 0x7f1f50014720 con 0x7f1f6009f140 2024-09-16T06:46:35.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f577fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T06:46:35.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f577fe700 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1f400362a0 legacy=0x7f1f40038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 10 monclient: shutdown 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 >> v1:172.21.15.76:6790/0 conn(0x7f1f6009f140 legacy=0x7f1f6006c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 shutdown_connections 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 >> 172.21.15.189:0/2763103328 conn(0x7f1f600a03e0 msgr2=0x7f1f6013b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 shutdown_connections 2024-09-16T06:46:35.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.254+0000 7f1f654fc700 1 -- 172.21.15.189:0/2763103328 wait complete. 2024-09-16T06:46:35.260 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h8' does not appear in the crush map 2024-09-16T06:46:35.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:35.270 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h9 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 10 monclient: get_monmap_and_config 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 10 monclient: build_initial_monmap 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 10 monclient: monmap: 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:35.374283+0000 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:35.374283+0000 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:35.374 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:35.375 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 1 Processor -- start 2024-09-16T06:46:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 1 -- start start 2024-09-16T06:46:35.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.370+0000 7f29338e4700 10 monclient: init 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-b con 0x7f292c0a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-a con 0x7f292c0a3d10 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-c con 0x7f292c0a2350 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c099a40 con 0x7f292c0a3d10 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c099ef0 con 0x7f292c0a2350 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c09a1d0 con 0x7f292c0a27a0 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2931e81700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f292c0a2350 legacy=0x7f292c13dcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:35.380 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient(hunting): authenticate will time out at 1347.888435s 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2931680700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f292c0a27a0 0x7f292c0a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35222/0 (socket says 172.21.15.189:35222) 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2931680700 1 -- 172.21.15.189:0/1974594805 learned_addr learned my addr 172.21.15.189:0/1974594805 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447354606 0 0) 0x7f292c09a1d0 con 0x7f292c0a27a0 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:35.381 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): my global_id is 194210 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29180035d0 con 0x7f292c0a27a0 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946600704 0 0) 0x7f292c099a40 con 0x7f292c0a3d10 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient(hunting): my global_id is 184431 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f292c09a1d0 con 0x7f292c0a3d10 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877769247 0 0) 0x7f29180035d0 con 0x7f292c0a27a0 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 >> v1:172.21.15.189:6789/0 conn(0x7f292c0a2350 legacy=0x7f292c13dcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 >> v1:172.21.15.76:6789/0 conn(0x7f292c0a3d10 legacy=0x7f292c13a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:35.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 monclient: found mon.noname-b 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0a3290 con 0x7f292c0a27a0 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 5 monclient: authenticate success, global_id 194210 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2928001610 con 0x7f292c0a27a0 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:35.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: dump: 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:35.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29280018f0 con 0x7f292c0a27a0 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 1 -- 172.21.15.189:0/1974594805 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2928001c80 con 0x7f292c0a27a0 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 4 set_mon_vals no callback set 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f2923fff700 10 monclient: dump: 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:35.385 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 10 monclient: shutdown 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- 172.21.15.189:0/1974594805 >> v1:172.21.15.76:6790/0 conn(0x7f292c0a27a0 legacy=0x7f292c0a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- 172.21.15.189:0/1974594805 shutdown_connections 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- 172.21.15.189:0/1974594805 >> 172.21.15.189:0/1974594805 conn(0x7f292c0a7d70 msgr2=0x7f292c0a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- 172.21.15.189:0/1974594805 shutdown_connections 2024-09-16T06:46:35.386 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.374+0000 7f29338e4700 1 -- 172.21.15.189:0/1974594805 wait complete. 2024-09-16T06:46:35.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient: build_initial_monmap 2024-09-16T06:46:35.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient: monmap: 2024-09-16T06:46:35.391 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:35.391 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 1 Processor -- start 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 1 -- start start 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient: init 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:35.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-a con 0x7f292c0a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-b con 0x7f292c0a27a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient(hunting): picked mon.noname-c con 0x7f292c13bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c13e9f0 con 0x7f292c0a2350 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c13c620 con 0x7f292c0a27a0 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c06bea0 con 0x7f292c13bb00 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f29338e4700 10 monclient(hunting): authenticate will time out at 1347.895019s 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2930e7f700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f292c0a27a0 legacy=0x7f292c13b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2931680700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f292c0a2350 0x7f292c13ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54852/0 (socket says 172.21.15.189:54852) 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2931680700 1 -- 172.21.15.189:0/1175589577 learned_addr learned my addr 172.21.15.189:0/1175589577 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474322747 0 0) 0x7f292c13e9f0 con 0x7f292c0a2350 2024-09-16T06:46:35.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): my global_id is 184434 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29100035d0 con 0x7f292c0a2350 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760244518 0 0) 0x7f292c06bea0 con 0x7f292c13bb00 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 10 monclient(hunting): my global_id is 194213 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.378+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f292c13e9f0 con 0x7f292c13bb00 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861864652 0 0) 0x7f292c13e9f0 con 0x7f292c13bb00 2024-09-16T06:46:35.394 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f292c06bea0 con 0x7f292c13bb00 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221104481 0 0) 0x7f29100035d0 con 0x7f292c0a2350 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f292c13e9f0 con 0x7f292c0a2350 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2924004360 con 0x7f292c13bb00 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.395 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:35.396 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:35.396 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:35.396 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29280015d0 con 0x7f292c0a2350 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:35.397 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1856141340 0 0) 0x7f292c06bea0 con 0x7f292c13bb00 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 >> v1:172.21.15.189:6789/0 conn(0x7f292c0a27a0 legacy=0x7f292c13b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 >> v1:172.21.15.76:6789/0 conn(0x7f292c0a2350 legacy=0x7f292c13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 monclient: found mon.c 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c09ff00 con 0x7f292c13bb00 2024-09-16T06:46:35.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 5 monclient: authenticate success, global_id 194213 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _renew_subs 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292c09b3c0 con 0x7f292c13bb00 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _renew_subs 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f292c13e150 con 0x7f292c13bb00 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2924004550 con 0x7f292c13bb00 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2924004860 con 0x7f292c13bb00 2024-09-16T06:46:35.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29237fe700 4 set_mon_vals no callback set 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 10 monclient: dump: 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:35.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290c005100 con 0x7f292c13bb00 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2924004a80 con 0x7f292c13bb00 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(221..221 src has 151..221) v4 ==== 12392+0+0 (unknown 3642558029 0 0) 0x7f2924045600 con 0x7f292c13bb00 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.382+0000 7f2931680700 10 monclient: get_auth_request con 0x7f29100362a0 auth_method 0 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.394+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f29240052a0 con 0x7f292c13bb00 2024-09-16T06:46:35.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.394+0000 7f2921ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:35.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.394+0000 7f2921ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.570+0000 7f29338e4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.570+0000 7f29338e4700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:35.574 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.570+0000 7f29338e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:35.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:35.570+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f290c0053b0 con 0x7f292c13bb00 2024-09-16T06:46:36.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.146+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v222) v1 ==== 118+0+0 (unknown 1725623047 0 0) 0x7f2924012d40 con 0x7f292c13bb00 2024-09-16T06:46:36.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.146+0000 7f2921ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:36.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.146+0000 7f2921ffb700 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-16T06:46:36.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f29338e4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:36.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f29338e4700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:36.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f29338e4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f290c005650 con 0x7f292c13bb00 2024-09-16T06:46:36.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f2921ffb700 1 -- 172.21.15.189:0/1175589577 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v222) v1 ==== 118+0+0 (unknown 3606191353 0 0) 0x7f29240144c0 con 0x7f292c13bb00 2024-09-16T06:46:36.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f2921ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T06:46:36.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.258+0000 7f2921ffb700 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 >> v1:172.21.15.189:6812/3098810674 conn(0x7f29100362a0 legacy=0x7f2910038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 10 monclient: shutdown 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 >> v1:172.21.15.76:6790/0 conn(0x7f292c13bb00 legacy=0x7f292c1d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 shutdown_connections 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 >> 172.21.15.189:0/1175589577 conn(0x7f292c0a7d70 msgr2=0x7f292c0a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 shutdown_connections 2024-09-16T06:46:36.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.262+0000 7f29338e4700 1 -- 172.21.15.189:0/1175589577 wait complete. 2024-09-16T06:46:36.267 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h9' does not appear in the crush map 2024-09-16T06:46:36.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:36.277 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h10 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: get_monmap_and_config 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: build_initial_monmap 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: monmap: 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:36.382488+0000 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:36.382488+0000 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.382 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:36.383 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:36.383 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:36.383 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 1 Processor -- start 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 1 -- start start 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: init 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.378+0000 7f53388db700 10 monclient(hunting): picked mon.noname-b con 0x7f53340a5c70 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient(hunting): picked mon.noname-a con 0x7f53340a7100 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:36.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient(hunting): picked mon.noname-c con 0x7f533413bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340a5a90 con 0x7f53340a7100 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340a6a90 con 0x7f533413bab0 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5334099e40 con 0x7f53340a5c70 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient(hunting): authenticate will time out at 1348.896084s 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f5332d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f533413bab0 legacy=0x7f533413dea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f5331d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f53340a7100 0x7f533413a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54860/0 (socket says 172.21.15.189:54860) 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f5331d9b700 1 -- 172.21.15.189:0/1933511751 learned_addr learned my addr 172.21.15.189:0/1933511751 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:36.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533259c700 1 --1- 172.21.15.189:0/1933511751 >> v1:172.21.15.76:6790/0 conn(0x7f53340a5c70 0x7f53340a6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35230/0 (socket says 172.21.15.189:35230) 2024-09-16T06:46:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069032779 0 0) 0x7f53340a5a90 con 0x7f53340a7100 2024-09-16T06:46:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:36.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): my global_id is 184437 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531c0035d0 con 0x7f53340a7100 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205067004 0 0) 0x7f5334099e40 con 0x7f53340a5c70 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient(hunting): my global_id is 194216 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53340a5a90 con 0x7f53340a5c70 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930990003 0 0) 0x7f53340a5a90 con 0x7f53340a5c70 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 >> v1:172.21.15.189:6789/0 conn(0x7f533413bab0 legacy=0x7f533413dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 >> v1:172.21.15.76:6789/0 conn(0x7f53340a7100 legacy=0x7f533413a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 monclient: found mon.noname-b 2024-09-16T06:46:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533409a120 con 0x7f53340a5c70 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: _finish_auth 0 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 5 monclient: authenticate success, global_id 194216 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5328001570 con 0x7f53340a5c70 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: dump: 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.389 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5328001790 con 0x7f53340a5c70 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 1 -- 172.21.15.189:0/1933511751 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5328001b20 con 0x7f53340a5c70 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 4 set_mon_vals no callback set 2024-09-16T06:46:36.390 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f533159a700 10 monclient: dump: 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient: shutdown 2024-09-16T06:46:36.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:36.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- 172.21.15.189:0/1933511751 >> v1:172.21.15.76:6790/0 conn(0x7f53340a5c70 legacy=0x7f53340a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:36.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- 172.21.15.189:0/1933511751 shutdown_connections 2024-09-16T06:46:36.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- 172.21.15.189:0/1933511751 >> 172.21.15.189:0/1933511751 conn(0x7f53340a79e0 msgr2=0x7f53340a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:36.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- 172.21.15.189:0/1933511751 shutdown_connections 2024-09-16T06:46:36.392 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 1 -- 172.21.15.189:0/1933511751 wait complete. 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.382+0000 7f53388db700 10 monclient: build_initial_monmap 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: monmap: 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:36.396 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 Processor -- start 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- start start 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: init 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient(hunting): picked mon.noname-a con 0x7f53340a5c70 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient(hunting): picked mon.noname-c con 0x7f53340a7100 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:36.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient(hunting): picked mon.noname-b con 0x7f533413bab0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340a6f50 con 0x7f53340a5c70 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533413acc0 con 0x7f533413bab0 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53340a42c0 con 0x7f53340a7100 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient(hunting): authenticate will time out at 1348.901297s 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5332d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f533413bab0 legacy=0x7f53341d1e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5331d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f53340a7100 0x7f53341ca340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35232/0 (socket says 172.21.15.189:35232) 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5331d9b700 1 -- 172.21.15.189:0/2024393652 learned_addr learned my addr 172.21.15.189:0/2024393652 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f533259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f53340a5c70 0x7f53341c9c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54876/0 (socket says 172.21.15.189:54876) 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266257850 0 0) 0x7f53340a42c0 con 0x7f53340a7100 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:36.398 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): my global_id is 194219 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53140035d0 con 0x7f53340a7100 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398127784 0 0) 0x7f53340a6f50 con 0x7f53340a5c70 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): my global_id is 184440 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53340a42c0 con 0x7f53340a5c70 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2252640650 0 0) 0x7f53140035d0 con 0x7f53340a7100 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53340a6f50 con 0x7f53340a7100 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f53240019f0 con 0x7f53340a7100 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:36.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3128217305 0 0) 0x7f53340a42c0 con 0x7f53340a5c70 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53140035d0 con 0x7f53340a5c70 2024-09-16T06:46:36.400 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5328001470 con 0x7f53340a5c70 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.401 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3329325900 0 0) 0x7f53340a6f50 con 0x7f53340a7100 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 >> v1:172.21.15.189:6789/0 conn(0x7f533413bab0 legacy=0x7f53341d1e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 >> v1:172.21.15.76:6789/0 conn(0x7f53340a5c70 legacy=0x7f53341c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 monclient: found mon.c 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53341caa50 con 0x7f53340a7100 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 5 monclient: authenticate success, global_id 194219 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _renew_subs 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533406bd50 con 0x7f53340a7100 2024-09-16T06:46:36.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _renew_subs 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f533411f660 con 0x7f53340a7100 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5324001c90 con 0x7f53340a7100 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5324004ce0 con 0x7f53340a7100 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5330d99700 4 set_mon_vals no callback set 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.386+0000 7f5322ffd700 10 monclient: dump: 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:36.403 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f53388db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f53388db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f53388db700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318005100 con 0x7f53340a7100 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f532400f750 con 0x7f53340a7100 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(222..222 src has 151..222) v4 ==== 12182+0+0 (unknown 290790654 0 0) 0x7f5324045430 con 0x7f53340a7100 2024-09-16T06:46:36.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.390+0000 7f533259c700 10 monclient: get_auth_request con 0x7f5314036270 auth_method 0 2024-09-16T06:46:36.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.398+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5324014600 con 0x7f53340a7100 2024-09-16T06:46:36.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.398+0000 7f5322ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:36.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.398+0000 7f5322ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:36.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.570+0000 7f53388db700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:36.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.570+0000 7f53388db700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:36.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.570+0000 7f53388db700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:36.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:36.570+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f53180053b0 con 0x7f53340a7100 2024-09-16T06:46:37.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.157+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v223) v1 ==== 120+0+0 (unknown 290862865 0 0) 0x7f5324012ca0 con 0x7f53340a7100 2024-09-16T06:46:37.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.157+0000 7f5322ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:37.162 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.157+0000 7f5322ffd700 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-16T06:46:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f53388db700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f53388db700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:37.277 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f53388db700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f5318005650 con 0x7f53340a7100 2024-09-16T06:46:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f5322ffd700 1 -- 172.21.15.189:0/2024393652 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v223) v1 ==== 120+0+0 (unknown 3944623496 0 0) 0x7f5324014420 con 0x7f53340a7100 2024-09-16T06:46:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f5322ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T06:46:37.278 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.273+0000 7f5322ffd700 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-16T06:46:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5314036270 legacy=0x7f5314038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 10 monclient: shutdown 2024-09-16T06:46:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 >> v1:172.21.15.76:6790/0 conn(0x7f53340a7100 legacy=0x7f53341ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:37.281 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 shutdown_connections 2024-09-16T06:46:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 >> 172.21.15.189:0/2024393652 conn(0x7f53340a79e0 msgr2=0x7f5334138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 shutdown_connections 2024-09-16T06:46:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.277+0000 7f53388db700 1 -- 172.21.15.189:0/2024393652 wait complete. 2024-09-16T06:46:37.282 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h10' does not appear in the crush map 2024-09-16T06:46:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:37.292 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h11 2024-09-16T06:46:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: get_monmap_and_config 2024-09-16T06:46:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: build_initial_monmap 2024-09-16T06:46:37.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: monmap: 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:37.398076+0000 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:37.398076+0000 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:37.398 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 1 Processor -- start 2024-09-16T06:46:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 1 -- start start 2024-09-16T06:46:37.399 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: init 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.393+0000 7f5734815700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient(hunting): picked mon.noname-b con 0x7f57300993c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient(hunting): picked mon.noname-a con 0x7f5730099eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient(hunting): picked mon.noname-c con 0x7f5730139e00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573009b0c0 con 0x7f5730099eb0 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573009b680 con 0x7f5730139e00 2024-09-16T06:46:37.402 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5730139e00 legacy=0x7f573013c1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573009b3a0 con 0x7f57300993c0 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient(hunting): authenticate will time out at 1349.912238s 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f57300993c0 0x7f57300997a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35234/0 (socket says 172.21.15.189:35234) 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572f7fe700 1 -- 172.21.15.189:0/670386693 learned_addr learned my addr 172.21.15.189:0/670386693 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008058824 0 0) 0x7f573009b3a0 con 0x7f57300993c0 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): my global_id is 194222 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c0035d0 con 0x7f57300993c0 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984371789 0 0) 0x7f573009b0c0 con 0x7f5730099eb0 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:37.403 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient(hunting): my global_id is 184443 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573009b3a0 con 0x7f5730099eb0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146827834 0 0) 0x7f571c0035d0 con 0x7f57300993c0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 >> v1:172.21.15.189:6789/0 conn(0x7f5730139e00 legacy=0x7f573013c1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 >> v1:172.21.15.76:6789/0 conn(0x7f5730099eb0 legacy=0x7f57301386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 monclient: found mon.noname-b 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573009f7d0 con 0x7f57300993c0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: _finish_auth 0 2024-09-16T06:46:37.404 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5720001610 con 0x7f57300993c0 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: dump: 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.405 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 5 monclient: authenticate success, global_id 194222 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57200018f0 con 0x7f57300993c0 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 1 -- 172.21.15.189:0/670386693 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5720001c80 con 0x7f57300993c0 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 4 set_mon_vals no callback set 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:37.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f572effd700 10 monclient: dump: 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:37.407 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 10 monclient: shutdown 2024-09-16T06:46:37.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- 172.21.15.189:0/670386693 >> v1:172.21.15.76:6790/0 conn(0x7f57300993c0 legacy=0x7f57300997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- 172.21.15.189:0/670386693 shutdown_connections 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- 172.21.15.189:0/670386693 >> 172.21.15.189:0/670386693 conn(0x7f57300a2270 msgr2=0x7f57300a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- 172.21.15.189:0/670386693 shutdown_connections 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.397+0000 7f5734815700 1 -- 172.21.15.189:0/670386693 wait complete. 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient: build_initial_monmap 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient: monmap: 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:37.413 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 1 Processor -- start 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 1 -- start start 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient: init 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient(hunting): picked mon.noname-a con 0x7f57300993c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient(hunting): picked mon.noname-c con 0x7f5730099eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient(hunting): picked mon.noname-b con 0x7f5730139e00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:37.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300a1860 con 0x7f57300993c0 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573009c540 con 0x7f5730139e00 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5730139e00 legacy=0x7f57301cffd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5730138950 con 0x7f5730099eb0 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5734815700 10 monclient(hunting): authenticate will time out at 1349.918755s 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5726ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5730099eb0 0x7f5730138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35248/0 (socket says 172.21.15.189:35248) 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f57300993c0 0x7f5730137b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:54902/0 (socket says 172.21.15.189:54902) 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f5726ffd700 1 -- 172.21.15.189:0/2918601923 learned_addr learned my addr 172.21.15.189:0/2918601923 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251575486 0 0) 0x7f5730138950 con 0x7f5730099eb0 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:37.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): my global_id is 194225 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57100035d0 con 0x7f5730099eb0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12236096 0 0) 0x7f57300a1860 con 0x7f57300993c0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 10 monclient(hunting): my global_id is 184446 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.401+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5730138950 con 0x7f57300993c0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815776980 0 0) 0x7f57100035d0 con 0x7f5730099eb0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57300a1860 con 0x7f5730099eb0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5718001970 con 0x7f5730099eb0 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): dump: 2024-09-16T06:46:37.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 423622994 0 0) 0x7f5730138950 con 0x7f57300993c0 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57100035d0 con 0x7f57300993c0 2024-09-16T06:46:37.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57200015d0 con 0x7f57300993c0 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient(hunting): dump: 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 591177135 0 0) 0x7f57300a1860 con 0x7f5730099eb0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 >> v1:172.21.15.189:6789/0 conn(0x7f5730139e00 legacy=0x7f57301cffd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 >> v1:172.21.15.76:6789/0 conn(0x7f57300993c0 legacy=0x7f5730137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 monclient: found mon.c 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573012c780 con 0x7f5730099eb0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 5 monclient: authenticate success, global_id 194225 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _renew_subs 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57300a4360 con 0x7f5730099eb0 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _renew_subs 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f573012c4a0 con 0x7f5730099eb0 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5718001ce0 con 0x7f5730099eb0 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5718003b10 con 0x7f5730099eb0 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572e7fc700 4 set_mon_vals no callback set 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 10 monclient: dump: 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.420 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5718003c70 con 0x7f5730099eb0 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5714005100 con 0x7f5730099eb0 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(223..223 src has 151..223) v4 ==== 11969+0+0 (unknown 3035316234 0 0) 0x7f57180452d0 con 0x7f5730099eb0 2024-09-16T06:46:37.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.405+0000 7f572f7fe700 10 monclient: get_auth_request con 0x7f57100362c0 auth_method 0 2024-09-16T06:46:37.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.417+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5718014760 con 0x7f5730099eb0 2024-09-16T06:46:37.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.417+0000 7f572cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:37.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.417+0000 7f572cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:37.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.593+0000 7f5734815700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:37.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.593+0000 7f5734815700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:37.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.593+0000 7f5734815700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:37.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:37.593+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f57140053b0 con 0x7f5730099eb0 2024-09-16T06:46:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.165+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v224) v1 ==== 120+0+0 (unknown 1914542532 0 0) 0x7f5718012bd0 con 0x7f5730099eb0 2024-09-16T06:46:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.165+0000 7f572cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:38.172 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.165+0000 7f572cff9700 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-16T06:46:38.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.281+0000 7f5734815700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:38.286 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.281+0000 7f5734815700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:38.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.281+0000 7f5734815700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.281+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f5714005650 con 0x7f5730099eb0 2024-09-16T06:46:38.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.281+0000 7f572cff9700 1 -- 172.21.15.189:0/2918601923 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v224) v1 ==== 120+0+0 (unknown 1766862666 0 0) 0x7f5718014350 con 0x7f5730099eb0 2024-09-16T06:46:38.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f572cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T06:46:38.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f572cff9700 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 >> v1:172.21.15.189:6812/3098810674 conn(0x7f57100362c0 legacy=0x7f5710038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f5734815700 10 monclient: shutdown 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f5734815700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 >> v1:172.21.15.76:6790/0 conn(0x7f5730099eb0 legacy=0x7f5730138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.285+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 shutdown_connections 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.289+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 >> 172.21.15.189:0/2918601923 conn(0x7f57300a2270 msgr2=0x7f57300701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.289+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 shutdown_connections 2024-09-16T06:46:38.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.289+0000 7f5734815700 1 -- 172.21.15.189:0/2918601923 wait complete. 2024-09-16T06:46:38.293 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h11' does not appear in the crush map 2024-09-16T06:46:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:38.306 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h12 2024-09-16T06:46:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.409+0000 7f4405bb1700 10 monclient: get_monmap_and_config 2024-09-16T06:46:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.409+0000 7f4405bb1700 10 monclient: build_initial_monmap 2024-09-16T06:46:38.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.409+0000 7f4405bb1700 10 monclient: monmap: 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:38.415890+0000 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:38.415890+0000 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:38.416 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 Processor -- start 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- start start 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient: init 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-a con 0x7f440009f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-b con 0x7f4400099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-c con 0x7f44001396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440009f520 con 0x7f440009f140 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440009bc10 con 0x7f44001396a0 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440009b810 con 0x7f4400099700 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient(hunting): authenticate will time out at 1350.928893s 2024-09-16T06:46:38.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43fffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f44001396a0 legacy=0x7f440013ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43f7fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4400099700 0x7f440009b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35254/0 (socket says 172.21.15.189:35254) 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43f7fff700 1 -- 172.21.15.189:0/405449396 learned_addr learned my addr 172.21.15.189:0/405449396 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94405053 0 0) 0x7f440009f520 con 0x7f440009f140 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): my global_id is 184449 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ec0035d0 con 0x7f440009f140 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171216625 0 0) 0x7f440009b810 con 0x7f4400099700 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:38.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient(hunting): my global_id is 194228 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440009f520 con 0x7f4400099700 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2057674140 0 0) 0x7f440009f520 con 0x7f4400099700 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 >> v1:172.21.15.189:6789/0 conn(0x7f44001396a0 legacy=0x7f440013ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 >> v1:172.21.15.76:6789/0 conn(0x7f440009f140 legacy=0x7f440009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 monclient: found mon.noname-b 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440009ceb0 con 0x7f4400099700 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:38.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 5 monclient: authenticate success, global_id 194228 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e80015f0 con 0x7f4400099700 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: dump: 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:38.424 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e8001910 con 0x7f4400099700 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 1 -- 172.21.15.189:0/405449396 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e8001ca0 con 0x7f4400099700 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 4 set_mon_vals no callback set 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f43feffd700 10 monclient: dump: 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 10 monclient: shutdown 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- 172.21.15.189:0/405449396 >> v1:172.21.15.76:6790/0 conn(0x7f4400099700 legacy=0x7f440009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- 172.21.15.189:0/405449396 shutdown_connections 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- 172.21.15.189:0/405449396 >> 172.21.15.189:0/405449396 conn(0x7f44000a03e0 msgr2=0x7f44000a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- 172.21.15.189:0/405449396 shutdown_connections 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.413+0000 7f4405bb1700 1 -- 172.21.15.189:0/405449396 wait complete. 2024-09-16T06:46:38.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: build_initial_monmap 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: monmap: 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 Processor -- start 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- start start 2024-09-16T06:46:38.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: init 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-a con 0x7f4400099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-c con 0x7f440009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient(hunting): picked mon.noname-b con 0x7f44001396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440009a750 con 0x7f4400099700 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000949e0 con 0x7f44001396a0 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400093cd0 con 0x7f440009f140 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:38.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient(hunting): authenticate will time out at 1350.933479s 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f44001396a0 legacy=0x7f44001d1fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43f7fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f440009f140 0x7f44001ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:35256/0 (socket says 172.21.15.189:35256) 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43f7fff700 1 -- 172.21.15.189:0/2023591870 learned_addr learned my addr 172.21.15.189:0/2023591870 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741753536 0 0) 0x7f4400093cd0 con 0x7f440009f140 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): my global_id is 194231 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e00035d0 con 0x7f440009f140 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608638332 0 0) 0x7f43ec0035d0 con 0x7f4400099700 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:38.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): my global_id is 184452 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4400093cd0 con 0x7f4400099700 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1501101051 0 0) 0x7f43e00035d0 con 0x7f440009f140 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43ec0035d0 con 0x7f440009f140 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e80015b0 con 0x7f440009f140 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): dump: 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271911353 0 0) 0x7f4400093cd0 con 0x7f4400099700 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e00035d0 con 0x7f4400099700 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43f0001730 con 0x7f4400099700 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:38.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient(hunting): dump: 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 982805770 0 0) 0x7f43ec0035d0 con 0x7f440009f140 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 >> v1:172.21.15.189:6789/0 conn(0x7f44001396a0 legacy=0x7f44001d1fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:38.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 >> v1:172.21.15.76:6789/0 conn(0x7f4400099700 legacy=0x7f4400096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 monclient: found mon.c 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400094700 con 0x7f440009f140 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 5 monclient: authenticate success, global_id 194231 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _renew_subs 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000713c0 con 0x7f440009f140 2024-09-16T06:46:38.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _renew_subs 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4400093810 con 0x7f440009f140 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e8004ca0 con 0x7f440009f140 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f43e8005030 con 0x7f440009f140 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fe7fc700 4 set_mon_vals no callback set 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.417+0000 7f43fcff9700 10 monclient: dump: 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:38.438 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f43e8005190 con 0x7f440009f140 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f4405bb1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f4405bb1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f4405bb1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e4005100 con 0x7f440009f140 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(224..224 src has 151..224) v4 ==== 11756+0+0 (unknown 406138485 0 0) 0x7f43e8045440 con 0x7f440009f140 2024-09-16T06:46:38.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.421+0000 7f43ff7fe700 10 monclient: get_auth_request con 0x7f43e0036280 auth_method 0 2024-09-16T06:46:38.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.433+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f43e8014870 con 0x7f440009f140 2024-09-16T06:46:38.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.433+0000 7f43fcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:38.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.433+0000 7f43fcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:38.613 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.609+0000 7f4405bb1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:38.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.609+0000 7f4405bb1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:38.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.609+0000 7f4405bb1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:38.614 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:38.609+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f43e40053b0 con 0x7f440009f140 2024-09-16T06:46:39.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.169+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v225) v1 ==== 120+0+0 (unknown 1358270317 0 0) 0x7f43e8012f10 con 0x7f440009f140 2024-09-16T06:46:39.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.169+0000 7f43fcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:39.174 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.169+0000 7f43fcff9700 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-16T06:46:39.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.281+0000 7f4405bb1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:39.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.281+0000 7f4405bb1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:39.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.281+0000 7f4405bb1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.287 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.281+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f43e40023c0 con 0x7f440009f140 2024-09-16T06:46:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f43fcff9700 1 -- 172.21.15.189:0/2023591870 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v225) v1 ==== 120+0+0 (unknown 2860704110 0 0) 0x7f43e8014690 con 0x7f440009f140 2024-09-16T06:46:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f43fcff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T06:46:39.288 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f43fcff9700 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-16T06:46:39.290 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 >> v1:172.21.15.189:6812/3098810674 conn(0x7f43e0036280 legacy=0x7f43e0038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 10 monclient: shutdown 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 >> v1:172.21.15.76:6790/0 conn(0x7f440009f140 legacy=0x7f44001ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 shutdown_connections 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 >> 172.21.15.189:0/2023591870 conn(0x7f44000a03e0 msgr2=0x7f44000a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 shutdown_connections 2024-09-16T06:46:39.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.285+0000 7f4405bb1700 1 -- 172.21.15.189:0/2023591870 wait complete. 2024-09-16T06:46:39.292 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h12' does not appear in the crush map 2024-09-16T06:46:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:39.301 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h13 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.401+0000 7fe11b458700 10 monclient: get_monmap_and_config 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.401+0000 7fe11b458700 10 monclient: build_initial_monmap 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.401+0000 7fe11b458700 10 monclient: monmap: 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.407 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:39.407618+0000 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:39.407618+0000 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:39.408 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.401+0000 7fe11b458700 1 Processor -- start 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- start start 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient: init 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-b con 0x7fe11409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-c con 0x7fe11409c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-a con 0x7fe114138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11409cca0 con 0x7fe114138da0 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11409f750 con 0x7fe11409c5f0 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe114097620 con 0x7fe11409bb00 2024-09-16T06:46:39.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient(hunting): authenticate will time out at 1351.920727s 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe1189f3700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe11409c5f0 legacy=0x7fe114137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe1191f4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe11409bb00 0x7fe11409bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48160/0 (socket says 172.21.15.189:48160) 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe1199f5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe114138da0 0x7fe11413b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48422/0 (socket says 172.21.15.189:48422) 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe1191f4700 1 -- 172.21.15.189:0/3367796646 learned_addr learned my addr 172.21.15.189:0/3367796646 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382607965 0 0) 0x7fe114097620 con 0x7fe11409bb00 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): my global_id is 194234 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1000057e0 con 0x7fe11409bb00 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957400371 0 0) 0x7fe11409cca0 con 0x7fe114138da0 2024-09-16T06:46:39.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient(hunting): my global_id is 184455 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe114097620 con 0x7fe114138da0 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093398236 0 0) 0x7fe1000057e0 con 0x7fe11409bb00 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 >> v1:172.21.15.189:6789/0 conn(0x7fe11409c5f0 legacy=0x7fe114137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 >> v1:172.21.15.76:6789/0 conn(0x7fe114138da0 legacy=0x7fe11413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 monclient: found mon.noname-b 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114097900 con 0x7fe11409bb00 2024-09-16T06:46:39.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: _finish_auth 0 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 5 monclient: authenticate success, global_id 194234 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe104001610 con 0x7fe11409bb00 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: dump: 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.417 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1040018f0 con 0x7fe11409bb00 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 1 -- 172.21.15.189:0/3367796646 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe104001c80 con 0x7fe11409bb00 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 4 set_mon_vals no callback set 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:39.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe10bfff700 10 monclient: dump: 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 10 monclient: shutdown 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- 172.21.15.189:0/3367796646 >> v1:172.21.15.76:6790/0 conn(0x7fe11409bb00 legacy=0x7fe11409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:39.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- 172.21.15.189:0/3367796646 shutdown_connections 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- 172.21.15.189:0/3367796646 >> 172.21.15.189:0/3367796646 conn(0x7fe1140a03e0 msgr2=0x7fe1140a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- 172.21.15.189:0/3367796646 shutdown_connections 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.405+0000 7fe11b458700 1 -- 172.21.15.189:0/3367796646 wait complete. 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: build_initial_monmap 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: monmap: 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:39.420 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 Processor -- start 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- start start 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: init 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-c con 0x7fe11409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-a con 0x7fe114138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient(hunting): picked mon.noname-b con 0x7fe114091950 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe114099640 con 0x7fe114138da0 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe114094540 con 0x7fe114091950 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe114094820 con 0x7fe11409c5f0 2024-09-16T06:46:39.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient(hunting): authenticate will time out at 1351.925510s 2024-09-16T06:46:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe1199f5700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fe114091950 legacy=0x7fe1141d28c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe1189f3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe114138da0 0x7fe1141cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48438/0 (socket says 172.21.15.189:48438) 2024-09-16T06:46:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe1191f4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe11409c5f0 0x7fe1141ca1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48168/0 (socket says 172.21.15.189:48168) 2024-09-16T06:46:39.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe1189f3700 1 -- 172.21.15.189:0/1500011124 learned_addr learned my addr 172.21.15.189:0/1500011124 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075452071 0 0) 0x7fe114099640 con 0x7fe114138da0 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): my global_id is 184458 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f80035d0 con 0x7fe114138da0 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045361065 0 0) 0x7fe114094820 con 0x7fe11409c5f0 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:39.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): my global_id is 194237 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe114099640 con 0x7fe11409c5f0 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556000302 0 0) 0x7fe114099640 con 0x7fe11409c5f0 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe114094820 con 0x7fe11409c5f0 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1040015d0 con 0x7fe11409c5f0 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:39.427 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474449422 0 0) 0x7fe0f80035d0 con 0x7fe114138da0 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe114099640 con 0x7fe114138da0 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe110004200 con 0x7fe114138da0 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:39.428 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 30558304 0 0) 0x7fe114094820 con 0x7fe11409c5f0 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 >> v1:172.21.15.189:6789/0 conn(0x7fe114091950 legacy=0x7fe1141d28c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 >> v1:172.21.15.76:6789/0 conn(0x7fe114138da0 legacy=0x7fe1141cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 monclient: found mon.c 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114092450 con 0x7fe11409c5f0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 5 monclient: authenticate success, global_id 194237 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _renew_subs 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11409a6e0 con 0x7fe11409c5f0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _renew_subs 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe114093f80 con 0x7fe11409c5f0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe104004ca0 con 0x7fe11409c5f0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe104004fb0 con 0x7fe11409c5f0 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:39.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe10b7fe700 4 set_mon_vals no callback set 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.409+0000 7fe109ffb700 10 monclient: dump: 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe11b458700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe11b458700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe11b458700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.431 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f4005100 con 0x7fe11409c5f0 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe10400fa20 con 0x7fe11409c5f0 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(225..225 src has 151..225) v4 ==== 11543+0+0 (unknown 871425822 0 0) 0x7fe104045330 con 0x7fe11409c5f0 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.413+0000 7fe1189f3700 10 monclient: get_auth_request con 0x7fe0f80362c0 auth_method 0 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.421+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe1040147f0 con 0x7fe11409c5f0 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.421+0000 7fe109ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:39.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.421+0000 7fe109ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.593+0000 7fe11b458700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.593+0000 7fe11b458700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.593+0000 7fe11b458700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:39.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:39.593+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7fe0f40053b0 con 0x7fe11409c5f0 2024-09-16T06:46:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.185+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v226) v1 ==== 120+0+0 (unknown 2110496802 0 0) 0x7fe104012e90 con 0x7fe11409c5f0 2024-09-16T06:46:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.185+0000 7fe109ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:40.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.185+0000 7fe109ffb700 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-16T06:46:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.297+0000 7fe11b458700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.297+0000 7fe11b458700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:40.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.297+0000 7fe11b458700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.297+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7fe0f4005650 con 0x7fe11409c5f0 2024-09-16T06:46:40.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe109ffb700 1 -- 172.21.15.189:0/1500011124 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v226) v1 ==== 120+0+0 (unknown 80480485 0 0) 0x7fe104014610 con 0x7fe11409c5f0 2024-09-16T06:46:40.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe109ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T06:46:40.304 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe109ffb700 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe0f80362c0 legacy=0x7fe0f8038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 10 monclient: shutdown 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 >> v1:172.21.15.76:6790/0 conn(0x7fe11409c5f0 legacy=0x7fe1141ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 shutdown_connections 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 >> 172.21.15.189:0/1500011124 conn(0x7fe1140a03e0 msgr2=0x7fe11409b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 shutdown_connections 2024-09-16T06:46:40.307 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.301+0000 7fe11b458700 1 -- 172.21.15.189:0/1500011124 wait complete. 2024-09-16T06:46:40.308 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h13' does not appear in the crush map 2024-09-16T06:46:40.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:40.317 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h14 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: get_monmap_and_config 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: build_initial_monmap 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: monmap: 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:40.425300+0000 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:40.425300+0000 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 1 Processor -- start 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 1 -- start start 2024-09-16T06:46:40.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: init 2024-09-16T06:46:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-b con 0x7f3888073b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-c con 0x7f3888073f40 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:40.428 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-a con 0x7f3888139840 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888098910 con 0x7f3888139840 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388809e6f0 con 0x7f3888073f40 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388809db30 con 0x7f3888073b60 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f388d2f3700 10 monclient(hunting): authenticate will time out at 1352.939504s 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f38867fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3888073f40 legacy=0x7f3888074370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f38877fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3888139840 0x7f388813bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48452/0 (socket says 172.21.15.189:48452) 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.421+0000 7f38877fe700 1 -- 172.21.15.189:0/1312655203 learned_addr learned my addr 172.21.15.189:0/1312655203 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389327196 0 0) 0x7f3888098910 con 0x7f3888139840 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:40.429 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): my global_id is 184461 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38700035d0 con 0x7f3888139840 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174674411 0 0) 0x7f388809db30 con 0x7f3888073b60 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient(hunting): my global_id is 194240 2024-09-16T06:46:40.430 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888098910 con 0x7f3888073b60 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790672775 0 0) 0x7f38700035d0 con 0x7f3888139840 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 >> v1:172.21.15.76:6790/0 conn(0x7f3888073b60 legacy=0x7f388806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 >> v1:172.21.15.189:6789/0 conn(0x7f3888073f40 legacy=0x7f3888074370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 monclient: found mon.noname-a 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388809dcc0 con 0x7f3888139840 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:40.433 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 5 monclient: authenticate success, global_id 184461 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38800018c0 con 0x7f3888139840 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: dump: 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.434 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3880001ba0 con 0x7f3888139840 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 4 set_mon_vals no callback set 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 1 -- 172.21.15.189:0/1312655203 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3880001f30 con 0x7f3888139840 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:40.435 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f3885ffb700 10 monclient: dump: 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 10 monclient: shutdown 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:40.436 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 1 -- 172.21.15.189:0/1312655203 >> v1:172.21.15.76:6789/0 conn(0x7f3888139840 legacy=0x7f388813bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 1 -- 172.21.15.189:0/1312655203 shutdown_connections 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 1 -- 172.21.15.189:0/1312655203 >> 172.21.15.189:0/1312655203 conn(0x7f38880a1100 msgr2=0x7f38880a1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 1 -- 172.21.15.189:0/1312655203 shutdown_connections 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.425+0000 7f388d2f3700 1 -- 172.21.15.189:0/1312655203 wait complete. 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient: build_initial_monmap 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient: monmap: 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:40.437 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 1 Processor -- start 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 1 -- start start 2024-09-16T06:46:40.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient: init 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-a con 0x7f3888073b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-c con 0x7f3888073f40 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient(hunting): picked mon.noname-b con 0x7f3888139840 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888099610 con 0x7f3888073b60 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880713e0 con 0x7f3888139840 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3888099b60 con 0x7f3888073f40 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:40.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f388d2f3700 10 monclient(hunting): authenticate will time out at 1352.946263s 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f38877fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f3888139840 legacy=0x7f38881cfb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f3886ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3888073b60 0x7f3888071be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48462/0 (socket says 172.21.15.189:48462) 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f3886ffd700 1 -- 172.21.15.189:0/2276697339 learned_addr learned my addr 172.21.15.189:0/2276697339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159555182 0 0) 0x7f3888099610 con 0x7f3888073b60 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): my global_id is 184464 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38740035d0 con 0x7f3888073b60 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597865829 0 0) 0x7f3888099b60 con 0x7f3888073f40 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:40.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): my global_id is 194243 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888099610 con 0x7f3888073f40 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548026 0 0) 0x7f3888099610 con 0x7f3888073f40 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3888099b60 con 0x7f3888073f40 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326966990 0 0) 0x7f38740035d0 con 0x7f3888073b60 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3888099610 con 0x7f3888073b60 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f387c004170 con 0x7f3888073f40 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:40.444 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3878001970 con 0x7f3888073b60 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.429+0000 7f386f7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:40.445 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1462817062 0 0) 0x7f3888099b60 con 0x7f3888073f40 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 >> v1:172.21.15.189:6789/0 conn(0x7f3888139840 legacy=0x7f38881cfb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 >> v1:172.21.15.76:6789/0 conn(0x7f3888073b60 legacy=0x7f3888071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:40.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 monclient: found mon.c 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388809b610 con 0x7f3888073f40 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 5 monclient: authenticate success, global_id 194243 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _renew_subs 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388806fc50 con 0x7f3888073f40 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _renew_subs 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3888138660 con 0x7f3888073f40 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f387c004650 con 0x7f3888073f40 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f387c0049e0 con 0x7f3888073f40 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 10 monclient: dump: 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:40.448 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f38857fa700 4 set_mon_vals no callback set 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868005100 con 0x7f3888073f40 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f387c004b40 con 0x7f3888073f40 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(226..226 src has 151..226) v4 ==== 11330+0+0 (unknown 453996001 0 0) 0x7f387c044f30 con 0x7f3888073f40 2024-09-16T06:46:40.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.433+0000 7f3886ffd700 10 monclient: get_auth_request con 0x7f3874036340 auth_method 0 2024-09-16T06:46:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.445+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f387c006130 con 0x7f3888073f40 2024-09-16T06:46:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.445+0000 7f386f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:40.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.445+0000 7f386f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.613+0000 7f388d2f3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.613+0000 7f388d2f3700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.613+0000 7f388d2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:40.620 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.613+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f38680053b0 con 0x7f3888073f40 2024-09-16T06:46:40.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.965+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v227) v1 ==== 120+0+0 (unknown 1899353898 0 0) 0x7f387c012ab0 con 0x7f3888073f40 2024-09-16T06:46:40.969 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.965+0000 7f386f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:40.970 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:40.965+0000 7f386f7fe700 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-16T06:46:41.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f388d2f3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:41.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f388d2f3700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:41.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f388d2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:41.085 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f3868005650 con 0x7f3888073f40 2024-09-16T06:46:41.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f386f7fe700 1 -- 172.21.15.189:0/2276697339 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v227) v1 ==== 120+0+0 (unknown 684395479 0 0) 0x7f387c014410 con 0x7f3888073f40 2024-09-16T06:46:41.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f386f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T06:46:41.086 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.081+0000 7f386f7fe700 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-16T06:46:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3874036340 legacy=0x7f3874038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 10 monclient: shutdown 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 >> v1:172.21.15.76:6790/0 conn(0x7f3888073f40 legacy=0x7f38881cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 shutdown_connections 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 >> 172.21.15.189:0/2276697339 conn(0x7f38880a1100 msgr2=0x7f388813bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 shutdown_connections 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.085+0000 7f388d2f3700 1 -- 172.21.15.189:0/2276697339 wait complete. 2024-09-16T06:46:41.089 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h14' does not appear in the crush map 2024-09-16T06:46:41.098 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:41.099 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h15 2024-09-16T06:46:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: get_monmap_and_config 2024-09-16T06:46:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: build_initial_monmap 2024-09-16T06:46:41.204 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: monmap: 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:41.204875+0000 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:41.204875+0000 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:41.205 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 1 Processor -- start 2024-09-16T06:46:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 1 -- start start 2024-09-16T06:46:41.206 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: init 2024-09-16T06:46:41.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:41.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:41.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-a con 0x7f982c131670 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-c con 0x7f982c09f7e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-b con 0x7f982c1375e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c09bcb0 con 0x7f982c131670 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c09b9d0 con 0x7f982c09f7e0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c09bf90 con 0x7f982c1375e0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9831cf8700 10 monclient(hunting): authenticate will time out at 1353.719181s 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f982bfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f982c09f7e0 legacy=0x7f982c131a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9830cf6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f982c131670 0x7f982c09c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48464/0 (socket says 172.21.15.189:48464) 2024-09-16T06:46:41.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f98314f7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f982c1375e0 0x7f982c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48206/0 (socket says 172.21.15.189:48206) 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.201+0000 7f9830cf6700 1 -- 172.21.15.189:0/1234845344 learned_addr learned my addr 172.21.15.189:0/1234845344 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379412445 0 0) 0x7f982c09bcb0 con 0x7f982c131670 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): my global_id is 184467 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f981c0035d0 con 0x7f982c131670 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325767746 0 0) 0x7f982c09bf90 con 0x7f982c1375e0 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:41.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient(hunting): my global_id is 194246 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c09bcb0 con 0x7f982c1375e0 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3770775335 0 0) 0x7f981c0035d0 con 0x7f982c131670 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 >> v1:172.21.15.76:6790/0 conn(0x7f982c1375e0 legacy=0x7f982c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 >> v1:172.21.15.189:6789/0 conn(0x7f982c09f7e0 legacy=0x7f982c131a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c132160 con 0x7f982c131670 2024-09-16T06:46:41.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 5 monclient: authenticate success, global_id 184467 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9820001610 con 0x7f982c131670 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: dump: 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.214 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9820001830 con 0x7f982c131670 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 4 set_mon_vals no callback set 2024-09-16T06:46:41.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 1 -- 172.21.15.189:0/1234845344 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9820001b40 con 0x7f982c131670 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f982b7fe700 10 monclient: dump: 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:41.216 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 10 monclient: shutdown 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 1 -- 172.21.15.189:0/1234845344 >> v1:172.21.15.76:6789/0 conn(0x7f982c131670 legacy=0x7f982c09c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 1 -- 172.21.15.189:0/1234845344 shutdown_connections 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 1 -- 172.21.15.189:0/1234845344 >> 172.21.15.189:0/1234845344 conn(0x7f982c0a0660 msgr2=0x7f982c0a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 1 -- 172.21.15.189:0/1234845344 shutdown_connections 2024-09-16T06:46:41.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 1 -- 172.21.15.189:0/1234845344 wait complete. 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 10 monclient: build_initial_monmap 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.205+0000 7f9831cf8700 10 monclient: monmap: 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:41.223 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 Processor -- start 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- start start 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: init 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:41.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-a con 0x7f982c09f7e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-b con 0x7f982c131670 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient(hunting): picked mon.noname-c con 0x7f982c1375e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c13a9f0 con 0x7f982c09f7e0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f982bfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f982c131670 legacy=0x7f982c1cc830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c072210 con 0x7f982c131670 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c136fb0 con 0x7f982c1375e0 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient(hunting): authenticate will time out at 1353.725857s 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98314f7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f982c1375e0 0x7f982c1cff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48216/0 (socket says 172.21.15.189:48216) 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9830cf6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f982c09f7e0 0x7f982c130640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48476/0 (socket says 172.21.15.189:48476) 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98314f7700 1 -- 172.21.15.189:0/983769606 learned_addr learned my addr 172.21.15.189:0/983769606 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:41.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618585677 0 0) 0x7f982c13a9f0 con 0x7f982c09f7e0 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): my global_id is 184470 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100035d0 con 0x7f982c09f7e0 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182945834 0 0) 0x7f982c136fb0 con 0x7f982c1375e0 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:41.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): my global_id is 194249 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c13a9f0 con 0x7f982c1375e0 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2434148583 0 0) 0x7f98100035d0 con 0x7f982c09f7e0 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f982c136fb0 con 0x7f982c09f7e0 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f98200015d0 con 0x7f982c09f7e0 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): dump: 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:41.227 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1267750174 0 0) 0x7f982c13a9f0 con 0x7f982c1375e0 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98100035d0 con 0x7f982c1375e0 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9824001ae0 con 0x7f982c1375e0 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:41.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient(hunting): dump: 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1089950227 0 0) 0x7f982c136fb0 con 0x7f982c09f7e0 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 >> v1:172.21.15.76:6790/0 conn(0x7f982c1375e0 legacy=0x7f982c1cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 >> v1:172.21.15.189:6789/0 conn(0x7f982c131670 legacy=0x7f982c1cc830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 monclient: found mon.a 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c135810 con 0x7f982c09f7e0 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:41.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 5 monclient: authenticate success, global_id 184470 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _renew_subs 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c135cb0 con 0x7f982c09f7e0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _renew_subs 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c066500 con 0x7f982c09f7e0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9820004b40 con 0x7f982c09f7e0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9820004e50 con 0x7f982c09f7e0 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f982affd700 4 set_mon_vals no callback set 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:41.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.209+0000 7f98297fa700 10 monclient: dump: 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:41.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f9831cf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f9831cf8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f9831cf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c005100 con 0x7f982c09f7e0 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9820004fb0 con 0x7f982c09f7e0 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(227..227 src has 151..227) v4 ==== 11117+0+0 (unknown 3068146711 0 0) 0x7f98200450f0 con 0x7f982c09f7e0 2024-09-16T06:46:41.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.213+0000 7f982bfff700 10 monclient: get_auth_request con 0x7f98100362f0 auth_method 0 2024-09-16T06:46:41.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.225+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9820014970 con 0x7f982c09f7e0 2024-09-16T06:46:41.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.225+0000 7f98297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:41.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.225+0000 7f98297fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:41.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.401+0000 7f9831cf8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:41.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.401+0000 7f9831cf8700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:41.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.401+0000 7f9831cf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:41.406 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:41.401+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f980c0053b0 con 0x7f982c09f7e0 2024-09-16T06:46:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.197+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v228) v1 ==== 120+0+0 (unknown 2841329582 0 0) 0x7f9820012de0 con 0x7f982c09f7e0 2024-09-16T06:46:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.197+0000 7f98297fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:42.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.197+0000 7f98297fa700 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-16T06:46:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f9831cf8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f9831cf8700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f9831cf8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.317 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f980c005650 con 0x7f982c09f7e0 2024-09-16T06:46:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f98297fa700 1 -- 172.21.15.189:0/983769606 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v228) v1 ==== 120+0+0 (unknown 2993184788 0 0) 0x7f9820014740 con 0x7f982c09f7e0 2024-09-16T06:46:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f98297fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T06:46:42.319 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.313+0000 7f98297fa700 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-16T06:46:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 >> v1:172.21.15.189:6812/3098810674 conn(0x7f98100362f0 legacy=0x7f98100387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 10 monclient: shutdown 2024-09-16T06:46:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:42.321 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 >> v1:172.21.15.76:6789/0 conn(0x7f982c09f7e0 legacy=0x7f982c130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 shutdown_connections 2024-09-16T06:46:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 >> 172.21.15.189:0/983769606 conn(0x7f982c0a0660 msgr2=0x7f982c13a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 shutdown_connections 2024-09-16T06:46:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.317+0000 7f9831cf8700 1 -- 172.21.15.189:0/983769606 wait complete. 2024-09-16T06:46:42.322 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h15' does not appear in the crush map 2024-09-16T06:46:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:42.332 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h16 2024-09-16T06:46:42.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: get_monmap_and_config 2024-09-16T06:46:42.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: build_initial_monmap 2024-09-16T06:46:42.432 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: monmap: 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:42.432802+0000 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:42.432802+0000 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:42.433 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 Processor -- start 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 -- start start 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: init 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-b con 0x7f876006a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-c con 0x7f876006ae30 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-a con 0x7f876006b240 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600a3c30 con 0x7f876006b240 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876006c170 con 0x7f876006ae30 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876009ec40 con 0x7f876006a340 2024-09-16T06:46:42.442 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient(hunting): authenticate will time out at 1354.945732s 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875effd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f876006ae30 legacy=0x7f876013a820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f876006a340 0x7f876006a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48222/0 (socket says 172.21.15.189:48222) 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f876006b240 0x7f876013df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48486/0 (socket says 172.21.15.189:48486) 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875f7fe700 1 -- 172.21.15.189:0/2724409945 learned_addr learned my addr 172.21.15.189:0/2724409945 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700687041 0 0) 0x7f87600a3c30 con 0x7f876006b240 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): my global_id is 184473 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87480035d0 con 0x7f876006b240 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820936921 0 0) 0x7f876009ec40 con 0x7f876006a340 2024-09-16T06:46:42.443 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient(hunting): my global_id is 194252 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87600a3c30 con 0x7f876006a340 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2076985951 0 0) 0x7f87480035d0 con 0x7f876006b240 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 >> v1:172.21.15.76:6790/0 conn(0x7f876006a340 legacy=0x7f876006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 >> v1:172.21.15.189:6789/0 conn(0x7f876006ae30 legacy=0x7f876013a820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600a3eb0 con 0x7f876006b240 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:42.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758001990 con 0x7f876006b240 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 5 monclient: authenticate success, global_id 184473 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: dump: 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.445 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8758001c70 con 0x7f876006b240 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 1 -- 172.21.15.189:0/2724409945 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758002000 con 0x7f876006b240 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f875e7fc700 10 monclient: dump: 2024-09-16T06:46:42.446 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 4 set_mon_vals no callback set 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 10 monclient: shutdown 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.429+0000 7f8764a7f700 1 -- 172.21.15.189:0/2724409945 >> v1:172.21.15.76:6789/0 conn(0x7f876006b240 legacy=0x7f876013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/2724409945 shutdown_connections 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/2724409945 >> 172.21.15.189:0/2724409945 conn(0x7f87600a45c0 msgr2=0x7f87600a49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/2724409945 shutdown_connections 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/2724409945 wait complete. 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: build_initial_monmap 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: monmap: 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:42.448 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 Processor -- start 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- start start 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: init 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:42.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-b con 0x7f876006a340 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-a con 0x7f876006ae30 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient(hunting): picked mon.noname-c con 0x7f876006b240 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600938e0 con 0x7f876006ae30 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8760094d90 con 0x7f876006a340 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600a65b0 con 0x7f876006b240 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient(hunting): authenticate will time out at 1354.949552s 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f875f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f876006a340 legacy=0x7f876009a9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f875effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f876006ae30 0x7f87601ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48502/0 (socket says 172.21.15.189:48502) 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f875effd700 1 -- 172.21.15.189:0/34921935 learned_addr learned my addr 172.21.15.189:0/34921935 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285901616 0 0) 0x7f87600938e0 con 0x7f876006ae30 2024-09-16T06:46:42.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): my global_id is 184476 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c0035d0 con 0x7f876006ae30 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199244648 0 0) 0x7f87600a65b0 con 0x7f876006b240 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): my global_id is 194255 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87600938e0 con 0x7f876006b240 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822856680 0 0) 0x7f874c0035d0 con 0x7f876006ae30 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87600a65b0 con 0x7f876006ae30 2024-09-16T06:46:42.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8750004200 con 0x7f876006ae30 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): dump: 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:42.452 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1041846678 0 0) 0x7f87600938e0 con 0x7f876006b240 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f874c0035d0 con 0x7f876006b240 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758001f80 con 0x7f876006b240 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient(hunting): dump: 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.453 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398435328 0 0) 0x7f87600a65b0 con 0x7f876006ae30 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 >> v1:172.21.15.76:6790/0 conn(0x7f876006b240 legacy=0x7f87601d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 >> v1:172.21.15.189:6789/0 conn(0x7f876006a340 legacy=0x7f876009a9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 monclient: found mon.a 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600a6060 con 0x7f876006ae30 2024-09-16T06:46:42.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 5 monclient: authenticate success, global_id 184476 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _renew_subs 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8760092ca0 con 0x7f876006ae30 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _renew_subs 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8760095070 con 0x7f876006ae30 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8750004470 con 0x7f876006ae30 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:42.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f875dffb700 4 set_mon_vals no callback set 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8750004800 con 0x7f876006ae30 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.433+0000 7f8747fff700 10 monclient: dump: 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:42.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8750004960 con 0x7f876006ae30 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8764a7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8764a7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8764a7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(228..228 src has 151..228) v4 ==== 10904+0+0 (unknown 2147401745 0 0) 0x7f8750044e50 con 0x7f876006ae30 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8740005100 con 0x7f876006ae30 2024-09-16T06:46:42.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.437+0000 7f875f7fe700 10 monclient: get_auth_request con 0x7f874c0362a0 auth_method 0 2024-09-16T06:46:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.445+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8750005160 con 0x7f876006ae30 2024-09-16T06:46:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.445+0000 7f8747fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:42.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.445+0000 7f8747fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:42.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.621+0000 7f8764a7f700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:42.624 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.621+0000 7f8764a7f700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:42.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.621+0000 7f8764a7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:42.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:42.621+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f87400053b0 con 0x7f876006ae30 2024-09-16T06:46:43.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.209+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v229) v1 ==== 120+0+0 (unknown 994912501 0 0) 0x7f8750012bc0 con 0x7f876006ae30 2024-09-16T06:46:43.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.209+0000 7f8747fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:43.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.209+0000 7f8747fff700 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-16T06:46:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8764a7f700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:43.325 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8764a7f700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:43.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8764a7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.326 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f87400023c0 con 0x7f876006ae30 2024-09-16T06:46:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8747fff700 1 -- 172.21.15.189:0/34921935 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v229) v1 ==== 120+0+0 (unknown 1908059696 0 0) 0x7f8750014340 con 0x7f876006ae30 2024-09-16T06:46:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8747fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T06:46:43.327 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.321+0000 7f8747fff700 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 >> v1:172.21.15.189:6812/3098810674 conn(0x7f874c0362a0 legacy=0x7f874c038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 10 monclient: shutdown 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 >> v1:172.21.15.76:6789/0 conn(0x7f876006ae30 legacy=0x7f87601ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 shutdown_connections 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 >> 172.21.15.189:0/34921935 conn(0x7f87600a45c0 msgr2=0x7f876009f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:43.330 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 shutdown_connections 2024-09-16T06:46:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.325+0000 7f8764a7f700 1 -- 172.21.15.189:0/34921935 wait complete. 2024-09-16T06:46:43.331 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h16' does not appear in the crush map 2024-09-16T06:46:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:43.341 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h17 2024-09-16T06:46:43.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.441+0000 7fbf6ad45700 10 monclient: get_monmap_and_config 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.441+0000 7fbf6ad45700 10 monclient: build_initial_monmap 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.441+0000 7fbf6ad45700 10 monclient: monmap: 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:43.447225+0000 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:43.447225+0000 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:43.447 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.441+0000 7fbf6ad45700 1 Processor -- start 2024-09-16T06:46:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- start start 2024-09-16T06:46:43.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient: init 2024-09-16T06:46:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-a con 0x7fbf6409bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-b con 0x7fbf6409c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:43.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-c con 0x7fbf64138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6409cca0 con 0x7fbf6409bb00 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6409f750 con 0x7fbf64138da0 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf64097620 con 0x7fbf6409c5f0 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient(hunting): authenticate will time out at 1355.960940s 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf692e2700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbf64138da0 legacy=0x7fbf6413b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf63fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fbf6409c5f0 0x7fbf64137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48234/0 (socket says 172.21.15.189:48234) 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf63fff700 1 -- 172.21.15.189:0/2318879458 learned_addr learned my addr 172.21.15.189:0/2318879458 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:43.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf68ae1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbf6409bb00 0x7fbf6409bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48512/0 (socket says 172.21.15.189:48512) 2024-09-16T06:46:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452309261 0 0) 0x7fbf64097620 con 0x7fbf6409c5f0 2024-09-16T06:46:43.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): my global_id is 194258 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf4c0035d0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275836958 0 0) 0x7fbf6409cca0 con 0x7fbf6409bb00 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient(hunting): my global_id is 184479 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf64097620 con 0x7fbf6409bb00 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277037313 0 0) 0x7fbf4c0035d0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 >> v1:172.21.15.189:6789/0 conn(0x7fbf64138da0 legacy=0x7fbf6413b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:43.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 >> v1:172.21.15.76:6789/0 conn(0x7fbf6409bb00 legacy=0x7fbf6409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 monclient: found mon.noname-b 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf64097900 con 0x7fbf6409c5f0 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbf58001610 con 0x7fbf6409c5f0 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 5 monclient: authenticate success, global_id 194258 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:43.454 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: dump: 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf580018f0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:43.455 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 4 set_mon_vals no callback set 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 1 -- 172.21.15.189:0/2318879458 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbf58001c80 con 0x7fbf6409c5f0 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf637fe700 10 monclient: dump: 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:43.456 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 10 monclient: shutdown 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- 172.21.15.189:0/2318879458 >> v1:172.21.15.76:6790/0 conn(0x7fbf6409c5f0 legacy=0x7fbf64137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- 172.21.15.189:0/2318879458 shutdown_connections 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- 172.21.15.189:0/2318879458 >> 172.21.15.189:0/2318879458 conn(0x7fbf640a03e0 msgr2=0x7fbf640a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- 172.21.15.189:0/2318879458 shutdown_connections 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.445+0000 7fbf6ad45700 1 -- 172.21.15.189:0/2318879458 wait complete. 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: build_initial_monmap 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: monmap: 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:43.457 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 Processor -- start 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- start start 2024-09-16T06:46:43.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: init 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-c con 0x7fbf6409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-a con 0x7fbf6409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient(hunting): picked mon.noname-b con 0x7fbf64138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf64090800 con 0x7fbf6409c5f0 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6408d9d0 con 0x7fbf64138da0 2024-09-16T06:46:43.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf640a12f0 con 0x7fbf6409bb00 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient(hunting): authenticate will time out at 1355.965568s 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf692e2700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fbf64138da0 legacy=0x7fbf641cdf60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf63fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbf6409c5f0 0x7fbf6408d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48514/0 (socket says 172.21.15.189:48514) 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf63fff700 1 -- 172.21.15.189:0/3183265213 learned_addr learned my addr 172.21.15.189:0/3183265213 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341824468 0 0) 0x7fbf64090800 con 0x7fbf6409c5f0 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): my global_id is 184482 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf500035d0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401362662 0 0) 0x7fbf640a12f0 con 0x7fbf6409bb00 2024-09-16T06:46:43.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): my global_id is 194261 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf64090800 con 0x7fbf6409bb00 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956910643 0 0) 0x7fbf500035d0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf640a12f0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbf580015d0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): dump: 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.463 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185213453 0 0) 0x7fbf64090800 con 0x7fbf6409bb00 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf500035d0 con 0x7fbf6409bb00 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbf540018b0 con 0x7fbf6409bb00 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:43.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient(hunting): dump: 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650412000 0 0) 0x7fbf640a12f0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 >> v1:172.21.15.76:6790/0 conn(0x7fbf6409bb00 legacy=0x7fbf6408cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:43.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 >> v1:172.21.15.189:6789/0 conn(0x7fbf64138da0 legacy=0x7fbf641cdf60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 monclient: found mon.a 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6409a6e0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 5 monclient: authenticate success, global_id 184482 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _renew_subs 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf640a2e50 con 0x7fbf6409c5f0 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _renew_subs 2024-09-16T06:46:43.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6408f010 con 0x7fbf6409c5f0 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf58004ca0 con 0x7fbf6409c5f0 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbf58005030 con 0x7fbf6409c5f0 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf617fa700 10 monclient: dump: 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:43.467 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.449+0000 7fbf62ffd700 4 set_mon_vals no callback set 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbf58005210 con 0x7fbf6409c5f0 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf6ad45700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf6ad45700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf6ad45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf44005100 con 0x7fbf6409c5f0 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(229..229 src has 151..229) v4 ==== 10691+0+0 (unknown 298651910 0 0) 0x7fbf58045060 con 0x7fbf6409c5f0 2024-09-16T06:46:43.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.453+0000 7fbf68ae1700 10 monclient: get_auth_request con 0x7fbf500362c0 auth_method 0 2024-09-16T06:46:43.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.461+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbf58014b50 con 0x7fbf6409c5f0 2024-09-16T06:46:43.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.461+0000 7fbf617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:43.469 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.461+0000 7fbf617fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:43.641 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.637+0000 7fbf6ad45700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:43.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.637+0000 7fbf6ad45700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:43.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.637+0000 7fbf6ad45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:43.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:43.637+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7fbf440053b0 con 0x7fbf6409c5f0 2024-09-16T06:46:44.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.217+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v230) v1 ==== 120+0+0 (unknown 2780793259 0 0) 0x7fbf58012fc0 con 0x7fbf6409c5f0 2024-09-16T06:46:44.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.217+0000 7fbf617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:44.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.217+0000 7fbf617fa700 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-16T06:46:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf6ad45700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf6ad45700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:44.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf6ad45700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.333 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7fbf44005650 con 0x7fbf6409c5f0 2024-09-16T06:46:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf617fa700 1 -- 172.21.15.189:0/3183265213 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v230) v1 ==== 120+0+0 (unknown 3757328315 0 0) 0x7fbf58014740 con 0x7fbf6409c5f0 2024-09-16T06:46:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf617fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T06:46:44.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.329+0000 7fbf617fa700 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbf500362c0 legacy=0x7fbf50038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 10 monclient: shutdown 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 >> v1:172.21.15.76:6789/0 conn(0x7fbf6409c5f0 legacy=0x7fbf6408d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 shutdown_connections 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 >> 172.21.15.189:0/3183265213 conn(0x7fbf640a03e0 msgr2=0x7fbf6409a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 shutdown_connections 2024-09-16T06:46:44.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.333+0000 7fbf6ad45700 1 -- 172.21.15.189:0/3183265213 wait complete. 2024-09-16T06:46:44.338 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h17' does not appear in the crush map 2024-09-16T06:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:44.348 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h18 2024-09-16T06:46:44.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: get_monmap_and_config 2024-09-16T06:46:44.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: build_initial_monmap 2024-09-16T06:46:44.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: monmap: 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:44.441738+0000 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:44.441738+0000 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:44.442 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 1 Processor -- start 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 1 -- start start 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: init 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-b con 0x7f9f9c09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-c con 0x7f9f9c09beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:44.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-a con 0x7f9f9c09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c099ed0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c09d2e0 con 0x7f9f9c09beb0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f93fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9f9c09beb0 legacy=0x7f9f9c137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c09ef80 con 0x7f9f9c09eba0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa13bb700 10 monclient(hunting): authenticate will time out at 1356.955107s 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa0bba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9f9c09c2e0 0x7f9f9c13ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48528/0 (socket says 172.21.15.189:48528) 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9fa0bba700 1 -- 172.21.15.189:0/259985526 learned_addr learned my addr 172.21.15.189:0/259985526 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402950976 0 0) 0x7f9f9c099ed0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:44.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.437+0000 7f9f9b7fe700 10 monclient(hunting): my global_id is 184485 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f880057e0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668338613 0 0) 0x7f9f9c09ef80 con 0x7f9f9c09eba0 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient(hunting): my global_id is 194264 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9c099ed0 con 0x7f9f9c09eba0 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603274576 0 0) 0x7f9f880057e0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 >> v1:172.21.15.76:6790/0 conn(0x7f9f9c09eba0 legacy=0x7f9f9c09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 >> v1:172.21.15.189:6789/0 conn(0x7f9f9c09beb0 legacy=0x7f9f9c137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:44.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c09cc70 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f940018c0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 5 monclient: authenticate success, global_id 184485 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: dump: 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.449 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f94001ae0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 1 -- 172.21.15.189:0/259985526 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f94001df0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 4 set_mon_vals no callback set 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:44.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f9b7fe700 10 monclient: dump: 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: shutdown 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- 172.21.15.189:0/259985526 >> v1:172.21.15.76:6789/0 conn(0x7f9f9c09c2e0 legacy=0x7f9f9c13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:44.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- 172.21.15.189:0/259985526 shutdown_connections 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- 172.21.15.189:0/259985526 >> 172.21.15.189:0/259985526 conn(0x7f9f9c09fe70 msgr2=0x7f9f9c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- 172.21.15.189:0/259985526 shutdown_connections 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- 172.21.15.189:0/259985526 wait complete. 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: build_initial_monmap 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: monmap: 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:44.452 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:44.453 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:44.453 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 Processor -- start 2024-09-16T06:46:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- start start 2024-09-16T06:46:44.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: init 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-c con 0x7f9f9c09beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-a con 0x7f9f9c09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient(hunting): picked mon.noname-b con 0x7f9f9c09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c12acc0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c12b1c0 con 0x7f9f9c09eba0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c09a380 con 0x7f9f9c09beb0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa0bba700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9f9c09eba0 legacy=0x7f9f9c1cdd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9fa13bb700 10 monclient(hunting): authenticate will time out at 1356.959637s 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f93fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9f9c09c2e0 0x7f9f9c1ca610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48542/0 (socket says 172.21.15.189:48542) 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.441+0000 7f9f93fff700 1 -- 172.21.15.189:0/3351793947 learned_addr learned my addr 172.21.15.189:0/3351793947 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:44.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871354729 0 0) 0x7f9f9c12acc0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): my global_id is 184488 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c0035d0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671701378 0 0) 0x7f9f9c09a380 con 0x7f9f9c09beb0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): my global_id is 194267 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9c12acc0 con 0x7f9f9c09beb0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 454267652 0 0) 0x7f9f7c0035d0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f9c09a380 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.459 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f84004200 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): dump: 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:44.460 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1440858016 0 0) 0x7f9f9c12acc0 con 0x7f9f9c09beb0 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f7c0035d0 con 0x7f9f9c09beb0 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f8c001870 con 0x7f9f9c09beb0 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient(hunting): dump: 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.461 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149839742 0 0) 0x7f9f9c09a380 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 >> v1:172.21.15.76:6790/0 conn(0x7f9f9c09beb0 legacy=0x7f9f9c132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 >> v1:172.21.15.189:6789/0 conn(0x7f9f9c09eba0 legacy=0x7f9f9c1cdd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 monclient: found mon.a 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:44.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c12c760 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 5 monclient: authenticate success, global_id 184488 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _renew_subs 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c098bc0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _renew_subs 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c132e40 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f840044e0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:44.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9f84004870 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f9affd700 4 set_mon_vals no callback set 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 10 monclient: dump: 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:44.464 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f80005100 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9f840049d0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(230..230 src has 151..230) v4 ==== 10478+0+0 (unknown 2841933832 0 0) 0x7f9f84044d00 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.445+0000 7f9f9bfff700 10 monclient: get_auth_request con 0x7f9f7c036280 auth_method 0 2024-09-16T06:46:44.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.453+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9f840051d0 con 0x7f9f9c09c2e0 2024-09-16T06:46:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.453+0000 7f9f997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:44.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.453+0000 7f9f997fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:44.625 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.621+0000 7f9fa13bb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:44.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.621+0000 7f9fa13bb700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:44.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.621+0000 7f9fa13bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:44.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:44.621+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f9f800053b0 con 0x7f9f9c09c2e0 2024-09-16T06:46:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.217+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v231) v1 ==== 120+0+0 (unknown 1453658801 0 0) 0x7f9f84012c10 con 0x7f9f9c09c2e0 2024-09-16T06:46:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.217+0000 7f9f997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:45.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.217+0000 7f9f997fa700 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-16T06:46:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.325+0000 7f9fa13bb700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.325+0000 7f9fa13bb700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.325+0000 7f9fa13bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.332 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.325+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f9f80005650 con 0x7f9f9c09c2e0 2024-09-16T06:46:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.329+0000 7f9f997fa700 1 -- 172.21.15.189:0/3351793947 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v231) v1 ==== 120+0+0 (unknown 699705428 0 0) 0x7f9f84014390 con 0x7f9f9c09c2e0 2024-09-16T06:46:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.329+0000 7f9f997fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T06:46:45.334 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.329+0000 7f9f997fa700 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-16T06:46:45.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9f7c036280 legacy=0x7f9f7c038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:45.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 10 monclient: shutdown 2024-09-16T06:46:45.336 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:45.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 >> v1:172.21.15.76:6789/0 conn(0x7f9f9c09c2e0 legacy=0x7f9f9c1ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:45.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 shutdown_connections 2024-09-16T06:46:45.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 >> 172.21.15.189:0/3351793947 conn(0x7f9f9c09fe70 msgr2=0x7f9f9c09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:45.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 shutdown_connections 2024-09-16T06:46:45.337 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.333+0000 7f9fa13bb700 1 -- 172.21.15.189:0/3351793947 wait complete. 2024-09-16T06:46:45.338 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h18' does not appear in the crush map 2024-09-16T06:46:45.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:45.347 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h19 2024-09-16T06:46:45.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: get_monmap_and_config 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: build_initial_monmap 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: monmap: 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:45.457216+0000 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:45.457216+0000 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:45.457 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:45.458 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 1 Processor -- start 2024-09-16T06:46:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 1 -- start start 2024-09-16T06:46:45.458 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: init 2024-09-16T06:46:45.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-b con 0x7fd81009a9f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-c con 0x7fd81009a230 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-a con 0x7fd81009b4e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81009bca0 con 0x7fd81009b4e0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81009bef0 con 0x7fd81009a230 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81009f770 con 0x7fd81009a9f0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd8182db700 10 monclient(hunting): authenticate will time out at 1357.971131s 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd815876700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd81009a230 legacy=0x7fd8101388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd816077700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd81009a9f0 0x7fd81009add0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48276/0 (socket says 172.21.15.189:48276) 2024-09-16T06:46:45.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd816878700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd81009b4e0 0x7fd81013bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48544/0 (socket says 172.21.15.189:48544) 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.453+0000 7fd816077700 1 -- 172.21.15.189:0/1109437456 learned_addr learned my addr 172.21.15.189:0/1109437456 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592818384 0 0) 0x7fd81009bca0 con 0x7fd81009b4e0 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _negotiate 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): my global_id is 184491 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc0035d0 con 0x7fd81009b4e0 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168606960 0 0) 0x7fd81009f770 con 0x7fd81009a9f0 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _negotiate 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient(hunting): my global_id is 194270 2024-09-16T06:46:45.464 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd81009bca0 con 0x7fd81009a9f0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304655198 0 0) 0x7fd7fc0035d0 con 0x7fd81009b4e0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 >> v1:172.21.15.76:6790/0 conn(0x7fd81009a9f0 legacy=0x7fd81009add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 >> v1:172.21.15.189:6789/0 conn(0x7fd81009a230 legacy=0x7fd8101388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: _finish_hunting 0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 monclient: found mon.noname-a 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81009c1d0 con 0x7fd81009b4e0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: _finish_auth 0 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: _check_auth_tickets 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:45.465 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 5 monclient: authenticate success, global_id 184491 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd808001990 con 0x7fd81009b4e0 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: dump: 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:45.466 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd808001c70 con 0x7fd81009b4e0 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 1 -- 172.21.15.189:0/1109437456 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd808002000 con 0x7fd81009b4e0 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 4 set_mon_vals no callback set 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd815075700 10 monclient: dump: 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.467 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 10 monclient: shutdown 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- 172.21.15.189:0/1109437456 >> v1:172.21.15.76:6789/0 conn(0x7fd81009b4e0 legacy=0x7fd81013bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- 172.21.15.189:0/1109437456 shutdown_connections 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- 172.21.15.189:0/1109437456 >> 172.21.15.189:0/1109437456 conn(0x7fd8100a03e0 msgr2=0x7fd8100a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- 172.21.15.189:0/1109437456 shutdown_connections 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- 172.21.15.189:0/1109437456 wait complete. 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 10 monclient: build_initial_monmap 2024-09-16T06:46:45.468 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 10 monclient: monmap: 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:45.469 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 Processor -- start 2024-09-16T06:46:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 1 -- start start 2024-09-16T06:46:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.457+0000 7fd8182db700 10 monclient: init 2024-09-16T06:46:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:45.474 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-b con 0x7fd81009a230 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-a con 0x7fd81009a9f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient(hunting): picked mon.noname-c con 0x7fd81009b4e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81009a6e0 con 0x7fd81009a9f0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd810130670 con 0x7fd81009a230 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd810071600 con 0x7fd81009b4e0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd816077700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fd81009a230 legacy=0x7fd810133ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient(hunting): authenticate will time out at 1357.976392s 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd815876700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd81009a9f0 0x7fd8101ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48558/0 (socket says 172.21.15.189:48558) 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd815876700 1 -- 172.21.15.189:0/3522974162 learned_addr learned my addr 172.21.15.189:0/3522974162 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089351686 0 0) 0x7fd81009a6e0 con 0x7fd81009a9f0 2024-09-16T06:46:45.475 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): my global_id is 184494 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f40035d0 con 0x7fd81009a9f0 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541252948 0 0) 0x7fd810071600 con 0x7fd81009b4e0 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): my global_id is 194273 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd81009a6e0 con 0x7fd81009b4e0 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1472884443 0 0) 0x7fd7f40035d0 con 0x7fd81009a9f0 2024-09-16T06:46:45.476 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd810071600 con 0x7fd81009a9f0 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd800004200 con 0x7fd81009a9f0 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:45.477 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2484087901 0 0) 0x7fd81009a6e0 con 0x7fd81009b4e0 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7f40035d0 con 0x7fd81009b4e0 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8080017b0 con 0x7fd81009b4e0 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:45.478 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 430259539 0 0) 0x7fd810071600 con 0x7fd81009a9f0 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 >> v1:172.21.15.76:6790/0 conn(0x7fd81009b4e0 legacy=0x7fd8101d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 >> v1:172.21.15.189:6789/0 conn(0x7fd81009a230 legacy=0x7fd810133ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 monclient: found mon.a 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81009ea20 con 0x7fd81009a9f0 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:45.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 5 monclient: authenticate success, global_id 184494 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _renew_subs 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81012ee70 con 0x7fd81009a9f0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _renew_subs 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101342b0 con 0x7fd81009a9f0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd800004420 con 0x7fd81009a9f0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8000047b0 con 0x7fd81009a9f0 2024-09-16T06:46:45.480 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd814874700 4 set_mon_vals no callback set 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 10 monclient: dump: 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:45.481 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.461+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd800004910 con 0x7fd81009a9f0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd8182db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd8182db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd8182db700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f8005100 con 0x7fd81009a9f0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(231..231 src has 151..231) v4 ==== 10265+0+0 (unknown 78508237 0 0) 0x7fd800044b90 con 0x7fd81009a9f0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.465+0000 7fd816077700 10 monclient: get_auth_request con 0x7fd7f4036400 auth_method 0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.473+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd800014700 con 0x7fd81009a9f0 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.473+0000 7fd806ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:45.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.473+0000 7fd806ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:45.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.649+0000 7fd8182db700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:45.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.649+0000 7fd8182db700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:45.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.649+0000 7fd8182db700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:45.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:45.649+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7fd7f80053b0 con 0x7fd81009a9f0 2024-09-16T06:46:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.237+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v232) v1 ==== 120+0+0 (unknown 2506175799 0 0) 0x7fd800012b70 con 0x7fd81009a9f0 2024-09-16T06:46:46.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.237+0000 7fd806ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:46.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.237+0000 7fd806ffd700 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-16T06:46:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.349+0000 7fd8182db700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:46.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.349+0000 7fd8182db700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.349+0000 7fd8182db700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.349+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7fd7f80023c0 con 0x7fd81009a9f0 2024-09-16T06:46:46.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.353+0000 7fd806ffd700 1 -- 172.21.15.189:0/3522974162 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v232) v1 ==== 120+0+0 (unknown 3670853383 0 0) 0x7fd8000142f0 con 0x7fd81009a9f0 2024-09-16T06:46:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.353+0000 7fd806ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T06:46:46.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.353+0000 7fd806ffd700 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-16T06:46:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd7f4036400 legacy=0x7fd7f40388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 10 monclient: shutdown 2024-09-16T06:46:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:46.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 >> v1:172.21.15.76:6789/0 conn(0x7fd81009a9f0 legacy=0x7fd8101ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 shutdown_connections 2024-09-16T06:46:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 >> 172.21.15.189:0/3522974162 conn(0x7fd8100a03e0 msgr2=0x7fd81006f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 shutdown_connections 2024-09-16T06:46:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.357+0000 7fd8182db700 1 -- 172.21.15.189:0/3522974162 wait complete. 2024-09-16T06:46:46.361 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h19' does not appear in the crush map 2024-09-16T06:46:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:46.371 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h20 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.473+0000 7fa1d0328700 10 monclient: get_monmap_and_config 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.473+0000 7fa1d0328700 10 monclient: build_initial_monmap 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.473+0000 7fa1d0328700 10 monclient: monmap: 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:46.479487+0000 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:46.479487+0000 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:46.479 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.480 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:46.480 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:46.480 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:46.480 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.473+0000 7fa1d0328700 1 Processor -- start 2024-09-16T06:46:46.481 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- start start 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient: init 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-c con 0x7fa1c809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-b con 0x7fa1c809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-a con 0x7fa1c8138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c809cca0 con 0x7fa1c8138da0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c809f750 con 0x7fa1c809bb00 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c8097620 con 0x7fa1c809c5f0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient(hunting): authenticate will time out at 1358.992748s 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1ce0c4700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa1c809bb00 legacy=0x7fa1c809bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:46.482 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1ce8c5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa1c8138da0 0x7fa1c813b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48574/0 (socket says 172.21.15.189:48574) 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1ce8c5700 1 -- 172.21.15.189:0/56344558 learned_addr learned my addr 172.21.15.189:0/56344558 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095041064 0 0) 0x7fa1c809cca0 con 0x7fa1c8138da0 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): my global_id is 184497 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b40035d0 con 0x7fa1c8138da0 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902054087 0 0) 0x7fa1c8097620 con 0x7fa1c809c5f0 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient(hunting): my global_id is 194276 2024-09-16T06:46:46.483 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c809cca0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782285984 0 0) 0x7fa1b40035d0 con 0x7fa1c8138da0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 >> v1:172.21.15.76:6790/0 conn(0x7fa1c809c5f0 legacy=0x7fa1c8137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 >> v1:172.21.15.189:6789/0 conn(0x7fa1c809bb00 legacy=0x7fa1c809bee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: _finish_hunting 0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 monclient: found mon.noname-a 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8097900 con 0x7fa1c8138da0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: _finish_auth 0 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: _check_auth_tickets 2024-09-16T06:46:46.489 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 5 monclient: authenticate success, global_id 184497 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1c0001990 con 0x7fa1c8138da0 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: dump: 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.490 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c0001bf0 con 0x7fa1c8138da0 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 1 -- 172.21.15.189:0/56344558 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1c0001f80 con 0x7fa1c8138da0 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 4 set_mon_vals no callback set 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1cd0c2700 10 monclient: dump: 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:46.491 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 10 monclient: shutdown 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- 172.21.15.189:0/56344558 >> v1:172.21.15.76:6789/0 conn(0x7fa1c8138da0 legacy=0x7fa1c813b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:46.492 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- 172.21.15.189:0/56344558 shutdown_connections 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- 172.21.15.189:0/56344558 >> 172.21.15.189:0/56344558 conn(0x7fa1c80a03e0 msgr2=0x7fa1c80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- 172.21.15.189:0/56344558 shutdown_connections 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.477+0000 7fa1d0328700 1 -- 172.21.15.189:0/56344558 wait complete. 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: build_initial_monmap 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: monmap: 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:46.493 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 Processor -- start 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- start start 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: init 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-c con 0x7fa1c809bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-a con 0x7fa1c809c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient(hunting): picked mon.noname-b con 0x7fa1c8138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c8132670 con 0x7fa1c809c5f0 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c809a6e0 con 0x7fa1c8138da0 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c8133220 con 0x7fa1c809bb00 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:46.494 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient(hunting): authenticate will time out at 1358.997379s 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1ce8c5700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa1c8138da0 legacy=0x7fa1c81d1e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1ce0c4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa1c809bb00 0x7fa1c81c9af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48296/0 (socket says 172.21.15.189:48296) 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1ce0c4700 1 -- 172.21.15.189:0/2255133606 learned_addr learned my addr 172.21.15.189:0/2255133606 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1cd8c3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa1c809c5f0 0x7fa1c8138020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48590/0 (socket says 172.21.15.189:48590) 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891881422 0 0) 0x7fa1c8132670 con 0x7fa1c809c5f0 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): my global_id is 184500 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ac0035d0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142686122 0 0) 0x7fa1c8133220 con 0x7fa1c809bb00 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:46.495 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): my global_id is 194279 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c8132670 con 0x7fa1c809bb00 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521730135 0 0) 0x7fa1ac0035d0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1c8133220 con 0x7fa1c809c5f0 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1c40011d0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): dump: 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.496 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4162125248 0 0) 0x7fa1c8132670 con 0x7fa1c809bb00 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ac0035d0 con 0x7fa1c809bb00 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1b8004200 con 0x7fa1c809bb00 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient(hunting): dump: 2024-09-16T06:46:46.497 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 713200847 0 0) 0x7fa1c8133220 con 0x7fa1c809c5f0 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 >> v1:172.21.15.76:6790/0 conn(0x7fa1c809bb00 legacy=0x7fa1c81c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 >> v1:172.21.15.189:6789/0 conn(0x7fa1c8138da0 legacy=0x7fa1c81d1e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:46.498 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 monclient: found mon.a 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8133500 con 0x7fa1c809c5f0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 5 monclient: authenticate success, global_id 184500 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _renew_subs 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c813c160 con 0x7fa1c809c5f0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _renew_subs 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.499 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8138730 con 0x7fa1c809c5f0 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c40036a0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa1c40039b0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1cc8c1700 4 set_mon_vals no callback set 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.481+0000 7fa1beffd700 10 monclient: dump: 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:46.500 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1d0328700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1d0328700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1d0328700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b0005100 con 0x7fa1c809c5f0 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa1c4003bd0 con 0x7fa1c809c5f0 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(232..232 src has 151..232) v4 ==== 10052+0+0 (unknown 1747650315 0 0) 0x7fa1c4044b10 con 0x7fa1c809c5f0 2024-09-16T06:46:46.501 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.485+0000 7fa1ce0c4700 10 monclient: get_auth_request con 0x7fa1ac0362c0 auth_method 0 2024-09-16T06:46:46.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.493+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa1c4014700 con 0x7fa1c809c5f0 2024-09-16T06:46:46.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.493+0000 7fa1beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:46.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.493+0000 7fa1beffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:46.662 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.657+0000 7fa1d0328700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:46.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.657+0000 7fa1d0328700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:46.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.657+0000 7fa1d0328700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:46.663 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:46.657+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fa1b00053b0 con 0x7fa1c809c5f0 2024-09-16T06:46:47.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.245+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v233) v1 ==== 120+0+0 (unknown 1721632000 0 0) 0x7fa1c4012b70 con 0x7fa1c809c5f0 2024-09-16T06:46:47.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.245+0000 7fa1beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:47.249 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.245+0000 7fa1beffd700 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-16T06:46:47.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1d0328700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:47.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1d0328700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:47.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1d0328700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.365 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fa1b0005650 con 0x7fa1c809c5f0 2024-09-16T06:46:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1beffd700 1 -- 172.21.15.189:0/2255133606 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v233) v1 ==== 120+0+0 (unknown 550076410 0 0) 0x7fa1c40142f0 con 0x7fa1c809c5f0 2024-09-16T06:46:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1beffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T06:46:47.367 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.361+0000 7fa1beffd700 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa1ac0362c0 legacy=0x7fa1ac038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 10 monclient: shutdown 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 >> v1:172.21.15.76:6789/0 conn(0x7fa1c809c5f0 legacy=0x7fa1c8138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 shutdown_connections 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 >> 172.21.15.189:0/2255133606 conn(0x7fa1c80a03e0 msgr2=0x7fa1c813b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 shutdown_connections 2024-09-16T06:46:47.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.365+0000 7fa1d0328700 1 -- 172.21.15.189:0/2255133606 wait complete. 2024-09-16T06:46:47.371 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h20' does not appear in the crush map 2024-09-16T06:46:47.380 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:47.380 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h21 2024-09-16T06:46:47.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: get_monmap_and_config 2024-09-16T06:46:47.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: build_initial_monmap 2024-09-16T06:46:47.490 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: monmap: 2024-09-16T06:46:47.490 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:47.490 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:47.490723+0000 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:47.490723+0000 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:47.491 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 1 Processor -- start 2024-09-16T06:46:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 1 -- start start 2024-09-16T06:46:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: init 2024-09-16T06:46:47.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-b con 0x7fb7900a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-c con 0x7fb79009aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-a con 0x7fb7900a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79009a040 con 0x7fb7900a3ca0 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7900a33d0 con 0x7fb79009aa20 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb790099d60 con 0x7fb7900a2350 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.485+0000 7fb794e24700 10 monclient(hunting): authenticate will time out at 1360.003937s 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78dd9b700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb79009aa20 legacy=0x7fb79013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb7900a2350 0x7fb7900a2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:48308/0 (socket says 172.21.15.189:48308) 2024-09-16T06:46:47.503 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78e59c700 1 -- 172.21.15.189:0/1956235499 learned_addr learned my addr 172.21.15.189:0/1956235499 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810973739 0 0) 0x7fb790099d60 con 0x7fb7900a2350 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): my global_id is 194282 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7780035d0 con 0x7fb7900a2350 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412274785 0 0) 0x7fb79009a040 con 0x7fb7900a3ca0 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:47.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient(hunting): my global_id is 184503 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb790099d60 con 0x7fb7900a3ca0 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812458329 0 0) 0x7fb7780035d0 con 0x7fb7900a2350 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 >> v1:172.21.15.189:6789/0 conn(0x7fb79009aa20 legacy=0x7fb79013a6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 >> v1:172.21.15.76:6789/0 conn(0x7fb7900a3ca0 legacy=0x7fb79013dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 monclient: found mon.noname-b 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:47.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7900a3920 con 0x7fb7900a2350 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: _finish_auth 0 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 5 monclient: authenticate success, global_id 194282 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb784001610 con 0x7fb7900a2350 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: dump: 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.506 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7840018f0 con 0x7fb7900a2350 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 1 -- 172.21.15.189:0/1956235499 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb784001c80 con 0x7fb7900a2350 2024-09-16T06:46:47.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 4 set_mon_vals no callback set 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb78d59a700 10 monclient: dump: 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:47.508 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 10 monclient: shutdown 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 -- 172.21.15.189:0/1956235499 >> v1:172.21.15.76:6790/0 conn(0x7fb7900a2350 legacy=0x7fb7900a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 -- 172.21.15.189:0/1956235499 shutdown_connections 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 -- 172.21.15.189:0/1956235499 >> 172.21.15.189:0/1956235499 conn(0x7fb7900a7d70 msgr2=0x7fb7900a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 -- 172.21.15.189:0/1956235499 shutdown_connections 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 -- 172.21.15.189:0/1956235499 wait complete. 2024-09-16T06:46:47.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 10 monclient: build_initial_monmap 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 10 monclient: monmap: 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:47.510 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.489+0000 7fb794e24700 1 Processor -- start 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- start start 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: init 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-b con 0x7fb79009aa20 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-c con 0x7fb7900a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient(hunting): picked mon.noname-a con 0x7fb7900a3ca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79006bed0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79009b640 con 0x7fb79009aa20 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb78e59c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fb79009aa20 legacy=0x7fb790135520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:47.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb790136190 con 0x7fb7900a2350 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient(hunting): authenticate will time out at 1360.008811s 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb78ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb7900a3ca0 0x7fb7901d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:48612/0 (socket says 172.21.15.189:48612) 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb78ed9d700 1 -- 172.21.15.189:0/4198767286 learned_addr learned my addr 172.21.15.189:0/4198767286 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238495694 0 0) 0x7fb79006bed0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): my global_id is 184506 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7700035d0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393619062 0 0) 0x7fb790136190 con 0x7fb7900a2350 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): my global_id is 194285 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb79006bed0 con 0x7fb7900a2350 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2241718107 0 0) 0x7fb7700035d0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb790136190 con 0x7fb7900a3ca0 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb788001c50 con 0x7fb7900a3ca0 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): dump: 2024-09-16T06:46:47.513 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587503900 0 0) 0x7fb79006bed0 con 0x7fb7900a2350 2024-09-16T06:46:47.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7700035d0 con 0x7fb7900a2350 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb780004170 con 0x7fb7900a2350 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient(hunting): dump: 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:47.515 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3496840496 0 0) 0x7fb790136190 con 0x7fb7900a3ca0 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 >> v1:172.21.15.76:6790/0 conn(0x7fb7900a2350 legacy=0x7fb7901cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 >> v1:172.21.15.189:6789/0 conn(0x7fb79009aa20 legacy=0x7fb790135520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 monclient: found mon.a 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790122c00 con 0x7fb7900a3ca0 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: _finish_auth 0 2024-09-16T06:46:47.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 5 monclient: authenticate success, global_id 184506 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _renew_subs 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790122ee0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _renew_subs 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb790120b10 con 0x7fb7900a3ca0 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb788001fe0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:47.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb788003e90 con 0x7fb7900a3ca0 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 10 monclient: dump: 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:47.518 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb78cd99700 4 set_mon_vals no callback set 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb794e24700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.493+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb7880040b0 con 0x7fb7900a3ca0 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.497+0000 7fb794e24700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.497+0000 7fb794e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.497+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb774005100 con 0x7fb7900a3ca0 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.497+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(233..233 src has 151..233) v4 ==== 9839+0+0 (unknown 3664576898 0 0) 0x7fb788044e90 con 0x7fb7900a3ca0 2024-09-16T06:46:47.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.497+0000 7fb78e59c700 10 monclient: get_auth_request con 0x7fb7700362a0 auth_method 0 2024-09-16T06:46:47.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.509+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fb788014980 con 0x7fb7900a3ca0 2024-09-16T06:46:47.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.509+0000 7fb77effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:47.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.509+0000 7fb77effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:47.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.681+0000 7fb794e24700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:47.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.681+0000 7fb794e24700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:47.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.681+0000 7fb794e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:47.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:47.681+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fb7740053b0 con 0x7fb7900a3ca0 2024-09-16T06:46:48.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.265+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v234) v1 ==== 120+0+0 (unknown 701841564 0 0) 0x7fb788013020 con 0x7fb7900a3ca0 2024-09-16T06:46:48.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.265+0000 7fb77effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:48.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.265+0000 7fb77effd700 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-16T06:46:48.382 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.377+0000 7fb794e24700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:48.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.377+0000 7fb794e24700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:48.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.377+0000 7fb794e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.383 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.377+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fb774005650 con 0x7fb7900a3ca0 2024-09-16T06:46:48.384 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb77effd700 1 -- 172.21.15.189:0/4198767286 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v234) v1 ==== 120+0+0 (unknown 2391250545 0 0) 0x7fb7880147a0 con 0x7fb7900a3ca0 2024-09-16T06:46:48.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb77effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T06:46:48.385 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb77effd700 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-16T06:46:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb7700362a0 legacy=0x7fb770038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:48.387 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 10 monclient: shutdown 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 >> v1:172.21.15.76:6789/0 conn(0x7fb7900a3ca0 legacy=0x7fb7901d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 shutdown_connections 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 >> 172.21.15.189:0/4198767286 conn(0x7fb7900a7d70 msgr2=0x7fb79009b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 shutdown_connections 2024-09-16T06:46:48.388 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.381+0000 7fb794e24700 1 -- 172.21.15.189:0/4198767286 wait complete. 2024-09-16T06:46:48.389 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h21' does not appear in the crush map 2024-09-16T06:46:48.398 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:48.398 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h22 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: get_monmap_and_config 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: build_initial_monmap 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: monmap: 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.502 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:48.502737+0000 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:48.502737+0000 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:48.503 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.504 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 1 Processor -- start 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 1 -- start start 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: init 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-b con 0x7f9200098070 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-c con 0x7f9200099a60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-a con 0x7f9200138a80 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920009c460 con 0x7f9200138a80 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920009c1e0 con 0x7f9200099a60 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920009bd60 con 0x7f9200098070 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f9206afa700 10 monclient(hunting): authenticate will time out at 1361.015745s 2024-09-16T06:46:48.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.497+0000 7f91fffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9200099a60 legacy=0x7f9200099e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9204896700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9200098070 0x7f920009f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46106/0 (socket says 172.21.15.189:46106) 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9205097700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9200138a80 0x7f920013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57272/0 (socket says 172.21.15.189:57272) 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9204896700 1 -- 172.21.15.189:0/534157907 learned_addr learned my addr 172.21.15.189:0/534157907 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073355542 0 0) 0x7f920009c460 con 0x7f9200138a80 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): my global_id is 184509 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec0035d0 con 0x7f9200138a80 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304555619 0 0) 0x7f920009bd60 con 0x7f9200098070 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:48.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient(hunting): my global_id is 194288 2024-09-16T06:46:48.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920009c460 con 0x7f9200098070 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458568798 0 0) 0x7f91ec0035d0 con 0x7f9200138a80 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 >> v1:172.21.15.76:6790/0 conn(0x7f9200098070 legacy=0x7f920009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 >> v1:172.21.15.189:6789/0 conn(0x7f9200099a60 legacy=0x7f9200099e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:48.511 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920009d490 con 0x7f9200138a80 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 5 monclient: authenticate success, global_id 184509 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91f8001610 con 0x7f9200138a80 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: dump: 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:48.512 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91f8001830 con 0x7f9200138a80 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 1 -- 172.21.15.189:0/534157907 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91f8001b40 con 0x7f9200138a80 2024-09-16T06:46:48.513 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 4 set_mon_vals no callback set 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f91ff7fe700 10 monclient: dump: 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:48.514 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 10 monclient: shutdown 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- 172.21.15.189:0/534157907 >> v1:172.21.15.76:6789/0 conn(0x7f9200138a80 legacy=0x7f920013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- 172.21.15.189:0/534157907 shutdown_connections 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- 172.21.15.189:0/534157907 >> 172.21.15.189:0/534157907 conn(0x7f92000a03e0 msgr2=0x7f92000a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- 172.21.15.189:0/534157907 shutdown_connections 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- 172.21.15.189:0/534157907 wait complete. 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 10 monclient: build_initial_monmap 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 10 monclient: monmap: 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:48.515 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 Processor -- start 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 1 -- start start 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.501+0000 7f9206afa700 10 monclient: init 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-a con 0x7f9200098070 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-c con 0x7f9200099a60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient(hunting): picked mon.noname-b con 0x7f9200138a80 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:48.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920009a6e0 con 0x7f9200098070 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920006e2c0 con 0x7f9200138a80 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920006f4d0 con 0x7f9200099a60 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient(hunting): authenticate will time out at 1361.020157s 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9205097700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9200138a80 legacy=0x7f92001d6730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9204896700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9200098070 0x7f9200070ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57286/0 (socket says 172.21.15.189:57286) 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9204896700 1 -- 172.21.15.189:0/4102872638 learned_addr learned my addr 172.21.15.189:0/4102872638 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261121154 0 0) 0x7f920009a6e0 con 0x7f9200098070 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): my global_id is 184512 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e00035d0 con 0x7f9200098070 2024-09-16T06:46:48.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277294985 0 0) 0x7f920006f4d0 con 0x7f9200099a60 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): my global_id is 194291 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920009a6e0 con 0x7f9200099a60 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322151638 0 0) 0x7f91e00035d0 con 0x7f9200098070 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920006f4d0 con 0x7f9200098070 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e80018f0 con 0x7f9200098070 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.518 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747020191 0 0) 0x7f920009a6e0 con 0x7f9200099a60 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91e00035d0 con 0x7f9200099a60 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91f00044f0 con 0x7f9200099a60 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:48.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3293863561 0 0) 0x7f920006f4d0 con 0x7f9200098070 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 >> v1:172.21.15.76:6790/0 conn(0x7f9200099a60 legacy=0x7f92001d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 >> v1:172.21.15.189:6789/0 conn(0x7f9200138a80 legacy=0x7f92001d6730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:48.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 monclient: found mon.a 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920006e020 con 0x7f9200098070 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 5 monclient: authenticate success, global_id 184512 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _renew_subs 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920006ef70 con 0x7f9200098070 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _renew_subs 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f920006db80 con 0x7f9200098070 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91e8004890 con 0x7f9200098070 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f91e8004a00 con 0x7f9200098070 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fe7fc700 4 set_mon_vals no callback set 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 10 monclient: dump: 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:48.522 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f91e800f470 con 0x7f9200098070 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.523 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e4005100 con 0x7f9200098070 2024-09-16T06:46:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(234..234 src has 151..234) v4 ==== 9626+0+0 (unknown 1415875341 0 0) 0x7f91e80448e0 con 0x7f9200098070 2024-09-16T06:46:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.505+0000 7f91fffff700 10 monclient: get_auth_request con 0x7f91e0036100 auth_method 0 2024-09-16T06:46:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.517+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f91e8010a60 con 0x7f9200098070 2024-09-16T06:46:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.517+0000 7f91fd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:48.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.517+0000 7f91fd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.693+0000 7f9206afa700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.693+0000 7f9206afa700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.693+0000 7f9206afa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:48.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:48.693+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f91e40053b0 con 0x7f9200098070 2024-09-16T06:46:49.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.273+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v235) v1 ==== 120+0+0 (unknown 1478098323 0 0) 0x7f91e8012ae0 con 0x7f9200098070 2024-09-16T06:46:49.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.273+0000 7f91fd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:49.279 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.273+0000 7f91fd7fa700 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-16T06:46:49.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.385+0000 7f9206afa700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:49.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.385+0000 7f9206afa700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:49.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.385+0000 7f9206afa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.391 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.385+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f91e4005650 con 0x7f9200098070 2024-09-16T06:46:49.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f91fd7fa700 1 -- 172.21.15.189:0/4102872638 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v235) v1 ==== 120+0+0 (unknown 1297438805 0 0) 0x7f91e8014260 con 0x7f9200098070 2024-09-16T06:46:49.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f91fd7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T06:46:49.393 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f91fd7fa700 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 >> v1:172.21.15.189:6812/3098810674 conn(0x7f91e0036100 legacy=0x7f91e00385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 10 monclient: shutdown 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 >> v1:172.21.15.76:6789/0 conn(0x7f9200098070 legacy=0x7f9200070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 shutdown_connections 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 >> 172.21.15.189:0/4102872638 conn(0x7f92000a03e0 msgr2=0x7f920009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 shutdown_connections 2024-09-16T06:46:49.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.389+0000 7f9206afa700 1 -- 172.21.15.189:0/4102872638 wait complete. 2024-09-16T06:46:49.396 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h22' does not appear in the crush map 2024-09-16T06:46:49.405 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:49.406 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h23 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: get_monmap_and_config 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: build_initial_monmap 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: monmap: 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:49.510505+0000 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:49.510505+0000 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:49.510 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.511 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:49.511 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:49.511 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:49.511 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 1 Processor -- start 2024-09-16T06:46:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 1 -- start start 2024-09-16T06:46:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: init 2024-09-16T06:46:49.514 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-a con 0x7f9d4009f140 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-b con 0x7f9d40099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-c con 0x7f9d401396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4009f520 con 0x7f9d4009f140 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4009bc10 con 0x7f9d401396a0 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4009b810 con 0x7f9d40099700 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d469f0700 10 monclient(hunting): authenticate will time out at 1362.023417s 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d44f8d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9d401396a0 legacy=0x7f9d4013ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d377fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9d40099700 0x7f9d4009b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46122/0 (socket says 172.21.15.189:46122) 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d377fe700 1 -- 172.21.15.189:0/1653278124 learned_addr learned my addr 172.21.15.189:0/1653278124 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.505+0000 7f9d3ffff700 1 --1- 172.21.15.189:0/1653278124 >> v1:172.21.15.76:6789/0 conn(0x7f9d4009f140 0x7f9d4009a9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57294/0 (socket says 172.21.15.189:57294) 2024-09-16T06:46:49.515 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045053184 0 0) 0x7f9d4009b810 con 0x7f9d40099700 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): my global_id is 194294 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c0035d0 con 0x7f9d40099700 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879371334 0 0) 0x7f9d4009f520 con 0x7f9d4009f140 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient(hunting): my global_id is 184515 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4009b810 con 0x7f9d4009f140 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2725129530 0 0) 0x7f9d2c0035d0 con 0x7f9d40099700 2024-09-16T06:46:49.516 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 >> v1:172.21.15.189:6789/0 conn(0x7f9d401396a0 legacy=0x7f9d4013ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 >> v1:172.21.15.76:6789/0 conn(0x7f9d4009f140 legacy=0x7f9d4009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4009ceb0 con 0x7f9d40099700 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 5 monclient: authenticate success, global_id 194294 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d28001630 con 0x7f9d40099700 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:49.517 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: dump: 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d28001910 con 0x7f9d40099700 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:49.518 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 1 -- 172.21.15.189:0/1653278124 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d28001ca0 con 0x7f9d40099700 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 4 set_mon_vals no callback set 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3f7fe700 10 monclient: dump: 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:49.519 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: shutdown 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- 172.21.15.189:0/1653278124 >> v1:172.21.15.76:6790/0 conn(0x7f9d40099700 legacy=0x7f9d4009b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- 172.21.15.189:0/1653278124 shutdown_connections 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- 172.21.15.189:0/1653278124 >> 172.21.15.189:0/1653278124 conn(0x7f9d400a03e0 msgr2=0x7f9d400a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- 172.21.15.189:0/1653278124 shutdown_connections 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- 172.21.15.189:0/1653278124 wait complete. 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: build_initial_monmap 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: monmap: 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:49.520 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:49.521 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 Processor -- start 2024-09-16T06:46:49.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- start start 2024-09-16T06:46:49.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: init 2024-09-16T06:46:49.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:49.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-b con 0x7f9d40099700 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-c con 0x7f9d4009f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d3ffff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f9d40099700 legacy=0x7f9d400965d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 10 monclient(hunting): picked mon.noname-a con 0x7f9d401396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.509+0000 7f9d469f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d400925b0 con 0x7f9d401396a0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4009a6e0 con 0x7f9d40099700 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d40071580 con 0x7f9d4009f140 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient(hunting): authenticate will time out at 1362.028117s 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d44f8d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9d401396a0 0x7f9d401cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57308/0 (socket says 172.21.15.189:57308) 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d44f8d700 1 -- 172.21.15.189:0/2258743607 learned_addr learned my addr 172.21.15.189:0/2258743607 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410777958 0 0) 0x7f9d400925b0 con 0x7f9d401396a0 2024-09-16T06:46:49.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): my global_id is 184518 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d200035d0 con 0x7f9d401396a0 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745972078 0 0) 0x7f9d40071580 con 0x7f9d4009f140 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): my global_id is 194297 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d400925b0 con 0x7f9d4009f140 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2155913448 0 0) 0x7f9d200035d0 con 0x7f9d401396a0 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d40071580 con 0x7f9d401396a0 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4063380732 0 0) 0x7f9d400925b0 con 0x7f9d4009f140 2024-09-16T06:46:49.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d200035d0 con 0x7f9d4009f140 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d380043a0 con 0x7f9d401396a0 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:49.528 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d28001450 con 0x7f9d4009f140 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:49.529 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025355975 0 0) 0x7f9d40071580 con 0x7f9d401396a0 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 >> v1:172.21.15.76:6790/0 conn(0x7f9d4009f140 legacy=0x7f9d401cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 >> v1:172.21.15.189:6789/0 conn(0x7f9d40099700 legacy=0x7f9d400965d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 monclient: found mon.a 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d400a2eb0 con 0x7f9d401396a0 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:49.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 5 monclient: authenticate success, global_id 184518 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _renew_subs 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d40096ce0 con 0x7f9d401396a0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _renew_subs 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d40090a80 con 0x7f9d401396a0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d380045d0 con 0x7f9d401396a0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9d38004960 con 0x7f9d401396a0 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3effd700 4 set_mon_vals no callback set 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 10 monclient: dump: 2024-09-16T06:46:49.531 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9d38004ac0 con 0x7f9d401396a0 2024-09-16T06:46:49.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d400008d0 con 0x7f9d401396a0 2024-09-16T06:46:49.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.513+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(235..235 src has 151..235) v4 ==== 9413+0+0 (unknown 2236536427 0 0) 0x7f9d38044a60 con 0x7f9d401396a0 2024-09-16T06:46:49.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.517+0000 7f9d3ffff700 10 monclient: get_auth_request con 0x7f9d200362b0 auth_method 0 2024-09-16T06:46:49.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.525+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9d380052c0 con 0x7f9d401396a0 2024-09-16T06:46:49.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.525+0000 7f9d3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:49.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.525+0000 7f9d3d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:49.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.709+0000 7f9d469f0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:49.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.709+0000 7f9d469f0700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:49.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.709+0000 7f9d469f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:49.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:49.709+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f9d40138cc0 con 0x7f9d401396a0 2024-09-16T06:46:50.291 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.285+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v236) v1 ==== 120+0+0 (unknown 1103777940 0 0) 0x7f9d38012da0 con 0x7f9d401396a0 2024-09-16T06:46:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.285+0000 7f9d3d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:50.292 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.285+0000 7f9d3d7fa700 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-16T06:46:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d469f0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d469f0700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d469f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:50.405 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f9d40138e50 con 0x7f9d401396a0 2024-09-16T06:46:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d3d7fa700 1 -- 172.21.15.189:0/2258743607 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v236) v1 ==== 120+0+0 (unknown 3623236502 0 0) 0x7f9d40138e50 con 0x7f9d401396a0 2024-09-16T06:46:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d3d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T06:46:50.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.401+0000 7f9d3d7fa700 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-16T06:46:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9d200362b0 legacy=0x7f9d20038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:50.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 10 monclient: shutdown 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 >> v1:172.21.15.76:6789/0 conn(0x7f9d401396a0 legacy=0x7f9d401cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 shutdown_connections 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 >> 172.21.15.189:0/2258743607 conn(0x7f9d400a03e0 msgr2=0x7f9d4006f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 shutdown_connections 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.405+0000 7f9d469f0700 1 -- 172.21.15.189:0/2258743607 wait complete. 2024-09-16T06:46:50.410 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h23' does not appear in the crush map 2024-09-16T06:46:50.419 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:50.419 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h24 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: get_monmap_and_config 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: build_initial_monmap 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: monmap: 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:50.522504+0000 2024-09-16T06:46:50.522 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:50.522504+0000 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:50.523 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 1 Processor -- start 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 1 -- start start 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: init 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-c con 0x7f6fec1352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-a con 0x7f6fec135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:50.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-b con 0x7f6fec0a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec09b050 con 0x7f6fec135db0 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec06ab20 con 0x7f6fec1352c0 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec06a840 con 0x7f6fec0a2030 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff1456700 10 monclient(hunting): authenticate will time out at 1363.035563s 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6febfff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6fec1352c0 legacy=0x7f6fec1356a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff0c55700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6fec0a2030 0x7f6fec13df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46140/0 (socket says 172.21.15.189:46140) 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.517+0000 7f6ff0c55700 1 -- 172.21.15.189:0/3076196617 learned_addr learned my addr 172.21.15.189:0/3076196617 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763462227 0 0) 0x7f6fec06a840 con 0x7f6fec0a2030 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:50.526 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): my global_id is 194300 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd80035d0 con 0x7f6fec0a2030 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260460388 0 0) 0x7f6fec09b050 con 0x7f6fec135db0 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient(hunting): my global_id is 184521 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fec06a840 con 0x7f6fec135db0 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539528558 0 0) 0x7f6fd80035d0 con 0x7f6fec0a2030 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 >> v1:172.21.15.189:6789/0 conn(0x7f6fec1352c0 legacy=0x7f6fec1356a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 >> v1:172.21.15.76:6789/0 conn(0x7f6fec135db0 legacy=0x7f6fec13a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 monclient: found mon.noname-b 2024-09-16T06:46:50.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec06c1b0 con 0x7f6fec0a2030 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe4001610 con 0x7f6fec0a2030 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: dump: 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.528 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 5 monclient: authenticate success, global_id 194300 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe40018f0 con 0x7f6fec0a2030 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 1 -- 172.21.15.189:0/3076196617 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe4001cc0 con 0x7f6fec0a2030 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:50.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6feaffd700 10 monclient: dump: 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:50.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 4 set_mon_vals no callback set 2024-09-16T06:46:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 10 monclient: shutdown 2024-09-16T06:46:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:50.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- 172.21.15.189:0/3076196617 >> v1:172.21.15.76:6790/0 conn(0x7f6fec0a2030 legacy=0x7f6fec13df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- 172.21.15.189:0/3076196617 shutdown_connections 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- 172.21.15.189:0/3076196617 >> 172.21.15.189:0/3076196617 conn(0x7f6fec0a4550 msgr2=0x7f6fec0a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- 172.21.15.189:0/3076196617 shutdown_connections 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- 172.21.15.189:0/3076196617 wait complete. 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 10 monclient: build_initial_monmap 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 10 monclient: monmap: 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:50.535 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 Processor -- start 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 1 -- start start 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.521+0000 7f6ff1456700 10 monclient: init 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-a con 0x7f6fec0a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-c con 0x7f6fec1352c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient(hunting): picked mon.noname-b con 0x7f6fec135db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:50.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec134f90 con 0x7f6fec0a2030 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0a6280 con 0x7f6fec135db0 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff0c55700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f6fec135db0 legacy=0x7f6fec1d2280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec09e820 con 0x7f6fec1352c0 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient(hunting): authenticate will time out at 1363.040994s 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6feb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6fec1352c0 0x7f6fec1ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46146/0 (socket says 172.21.15.189:46146) 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6feb7fe700 1 -- 172.21.15.189:0/837356057 learned_addr learned my addr 172.21.15.189:0/837356057 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011538673 0 0) 0x7f6fec09e820 con 0x7f6fec1352c0 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:50.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): my global_id is 194303 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc0035d0 con 0x7f6fec1352c0 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703862654 0 0) 0x7f6fec134f90 con 0x7f6fec0a2030 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): my global_id is 184524 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fec09e820 con 0x7f6fec0a2030 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3149977992 0 0) 0x7f6fcc0035d0 con 0x7f6fec1352c0 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fec134f90 con 0x7f6fec1352c0 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fdc001a70 con 0x7f6fec1352c0 2024-09-16T06:46:50.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:50.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1127581307 0 0) 0x7f6fec09e820 con 0x7f6fec0a2030 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fcc0035d0 con 0x7f6fec0a2030 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fd40044f0 con 0x7f6fec0a2030 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:50.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671296764 0 0) 0x7f6fec134f90 con 0x7f6fec1352c0 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 >> v1:172.21.15.189:6789/0 conn(0x7f6fec135db0 legacy=0x7f6fec1d2280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 >> v1:172.21.15.76:6789/0 conn(0x7f6fec0a2030 legacy=0x7f6fec13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 monclient: found mon.c 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec13b210 con 0x7f6fec1352c0 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:50.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 5 monclient: authenticate success, global_id 194303 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _renew_subs 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec13b4f0 con 0x7f6fec1352c0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _renew_subs 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec13f150 con 0x7f6fec1352c0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fdc001d40 con 0x7f6fec1352c0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fdc003af0 con 0x7f6fec1352c0 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:50.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe9ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.525+0000 7f6fe8ff9700 10 monclient: dump: 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6fdc003cd0 con 0x7f6fec1352c0 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(236..236 src has 151..236) v4 ==== 9200+0+0 (unknown 2121822092 0 0) 0x7f6fdc0449d0 con 0x7f6fec1352c0 2024-09-16T06:46:50.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6ff1456700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6ff1456700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6ff1456700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0005100 con 0x7f6fec1352c0 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.529+0000 7f6febfff700 10 monclient: get_auth_request con 0x7f6fcc0362c0 auth_method 0 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.541+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6fdc014990 con 0x7f6fec1352c0 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.541+0000 7f6fe8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:50.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.541+0000 7f6fe8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:50.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.709+0000 7f6ff1456700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:50.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.709+0000 7f6ff1456700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:50.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.709+0000 7f6ff1456700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:50.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.709+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f6fd00053b0 con 0x7f6fec1352c0 2024-09-16T06:46:50.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.969+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v237) v1 ==== 120+0+0 (unknown 1295263644 0 0) 0x7f6fdc012e00 con 0x7f6fec1352c0 2024-09-16T06:46:50.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.969+0000 7f6fe8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:50.974 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:50.969+0000 7f6fe8ff9700 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-16T06:46:51.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6ff1456700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:51.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6ff1456700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:51.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6ff1456700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f6fd00023c0 con 0x7f6fec1352c0 2024-09-16T06:46:51.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6fe8ff9700 1 -- 172.21.15.189:0/837356057 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v237) v1 ==== 120+0+0 (unknown 4226887844 0 0) 0x7f6fdc014760 con 0x7f6fec1352c0 2024-09-16T06:46:51.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6fe8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T06:46:51.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.081+0000 7f6fe8ff9700 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6fcc0362c0 legacy=0x7f6fcc038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 10 monclient: shutdown 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 >> v1:172.21.15.76:6790/0 conn(0x7f6fec1352c0 legacy=0x7f6fec1ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 shutdown_connections 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 >> 172.21.15.189:0/837356057 conn(0x7f6fec0a4550 msgr2=0x7f6fec133a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 shutdown_connections 2024-09-16T06:46:51.091 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.085+0000 7f6ff1456700 1 -- 172.21.15.189:0/837356057 wait complete. 2024-09-16T06:46:51.092 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h24' does not appear in the crush map 2024-09-16T06:46:51.101 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:51.101 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h25 2024-09-16T06:46:51.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: get_monmap_and_config 2024-09-16T06:46:51.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: build_initial_monmap 2024-09-16T06:46:51.201 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: monmap: 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:51.201395+0000 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:51.201395+0000 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:51.202 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.203 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:51.203 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:51.203 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:51.203 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 1 Processor -- start 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 1 -- start start 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: init 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-b con 0x7f68a8133380 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:51.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-c con 0x7f68a809f760 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-a con 0x7f68a80a1190 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8132ec0 con 0x7f68a80a1190 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a809e4c0 con 0x7f68a809f760 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a809e7a0 con 0x7f68a8133380 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68ae951700 10 monclient(hunting): authenticate will time out at 1363.715592s 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68a77fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f68a809f760 legacy=0x7f68a8138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68a7fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f68a8133380 0x7f68a8133760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46152/0 (socket says 172.21.15.189:46152) 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.197+0000 7f68a7fff700 1 -- 172.21.15.189:0/2296841581 learned_addr learned my addr 172.21.15.189:0/2296841581 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819641316 0 0) 0x7f68a809e7a0 con 0x7f68a8133380 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:51.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): my global_id is 194306 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68900035d0 con 0x7f68a8133380 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570044129 0 0) 0x7f68a8132ec0 con 0x7f68a80a1190 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient(hunting): my global_id is 184527 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a809e7a0 con 0x7f68a80a1190 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3691176956 0 0) 0x7f68900035d0 con 0x7f68a8133380 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 >> v1:172.21.15.189:6789/0 conn(0x7f68a809f760 legacy=0x7f68a8138240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:51.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 >> v1:172.21.15.76:6789/0 conn(0x7f68a80a1190 legacy=0x7f68a813b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 monclient: found mon.noname-b 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80a1cb0 con 0x7f68a8133380 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 5 monclient: authenticate success, global_id 194306 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:51.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6898001610 con 0x7f68a8133380 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: dump: 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:51.217 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6898001830 con 0x7f68a8133380 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 4 set_mon_vals no callback set 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 1 -- 172.21.15.189:0/2296841581 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6898001bc0 con 0x7f68a8133380 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68a6ffd700 10 monclient: dump: 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.218 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 10 monclient: shutdown 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 1 -- 172.21.15.189:0/2296841581 >> v1:172.21.15.76:6790/0 conn(0x7f68a8133380 legacy=0x7f68a8133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 1 -- 172.21.15.189:0/2296841581 shutdown_connections 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 1 -- 172.21.15.189:0/2296841581 >> 172.21.15.189:0/2296841581 conn(0x7f68a80a21f0 msgr2=0x7f68a80a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 1 -- 172.21.15.189:0/2296841581 shutdown_connections 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 1 -- 172.21.15.189:0/2296841581 wait complete. 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.201+0000 7f68ae951700 10 monclient: build_initial_monmap 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient: monmap: 2024-09-16T06:46:51.219 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 1 Processor -- start 2024-09-16T06:46:51.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 1 -- start start 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient: init 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-b con 0x7f68a809f760 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-c con 0x7f68a80a1190 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient(hunting): picked mon.noname-a con 0x7f68a8133380 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a812cb40 con 0x7f68a8133380 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a806f180 con 0x7f68a809f760 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a806f460 con 0x7f68a80a1190 2024-09-16T06:46:51.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68ae951700 10 monclient(hunting): authenticate will time out at 1363.722203s 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a7fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f68a809f760 legacy=0x7f68a812a810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f68a80a1190 0x7f68a812af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46168/0 (socket says 172.21.15.189:46168) 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a77fe700 1 -- 172.21.15.189:0/2762525732 learned_addr learned my addr 172.21.15.189:0/2762525732 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458433072 0 0) 0x7f68a809e7a0 con 0x7f68a8133380 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): my global_id is 184530 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940035d0 con 0x7f68a8133380 2024-09-16T06:46:51.222 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215736167 0 0) 0x7f68a806f460 con 0x7f68a80a1190 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): my global_id is 194309 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a809e7a0 con 0x7f68a80a1190 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2363628863 0 0) 0x7f68a809e7a0 con 0x7f68a80a1190 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a806f460 con 0x7f68a80a1190 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3370622334 0 0) 0x7f68940035d0 con 0x7f68a8133380 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a809e7a0 con 0x7f68a8133380 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c0041f0 con 0x7f68a80a1190 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:51.223 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68a0001a10 con 0x7f68a8133380 2024-09-16T06:46:51.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.205+0000 7f68a4ff9700 10 monclient(hunting): dump: 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:51.225 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2861348424 0 0) 0x7f68a806f460 con 0x7f68a80a1190 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 >> v1:172.21.15.189:6789/0 conn(0x7f68a809f760 legacy=0x7f68a812a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 >> v1:172.21.15.76:6789/0 conn(0x7f68a8133380 legacy=0x7f68a81d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 monclient: found mon.c 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a812b630 con 0x7f68a80a1190 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:51.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 5 monclient: authenticate success, global_id 194309 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _renew_subs 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a809c6c0 con 0x7f68a80a1190 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _renew_subs 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80a4bb0 con 0x7f68a80a1190 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c0046d0 con 0x7f68a80a1190 2024-09-16T06:46:51.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f689c004a60 con 0x7f68a80a1190 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a67fc700 4 set_mon_vals no callback set 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 10 monclient: dump: 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.228 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:51.229 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:51.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:51.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:51.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6888005100 con 0x7f68a80a1190 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f689c00f550 con 0x7f68a80a1190 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(237..237 src has 151..237) v4 ==== 8987+0+0 (unknown 1173069655 0 0) 0x7f689c044740 con 0x7f68a80a1190 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.209+0000 7f68a7fff700 10 monclient: get_auth_request con 0x7f68940362f0 auth_method 0 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.221+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f689c0110b0 con 0x7f68a80a1190 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.221+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:51.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.221+0000 7f68a4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.389+0000 7f68ae951700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:51.395 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.389+0000 7f68ae951700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.389+0000 7f68ae951700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:51.389+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f68880053b0 con 0x7f68a80a1190 2024-09-16T06:46:52.296 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.293+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v238) v1 ==== 120+0+0 (unknown 3505654049 0 0) 0x7f689c012be0 con 0x7f68a80a1190 2024-09-16T06:46:52.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.293+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:52.297 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.293+0000 7f68a4ff9700 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-16T06:46:52.409 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68ae951700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68ae951700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68ae951700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7f6888005650 con 0x7f68a80a1190 2024-09-16T06:46:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68a4ff9700 1 -- 172.21.15.189:0/2762525732 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v238) v1 ==== 120+0+0 (unknown 1438636335 0 0) 0x7f689c014360 con 0x7f68a80a1190 2024-09-16T06:46:52.410 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68a4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T06:46:52.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.405+0000 7f68a4ff9700 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 >> v1:172.21.15.189:6812/3098810674 conn(0x7f68940362f0 legacy=0x7f68940387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 10 monclient: shutdown 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 >> v1:172.21.15.76:6790/0 conn(0x7f68a80a1190 legacy=0x7f68a812af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 shutdown_connections 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 >> 172.21.15.189:0/2762525732 conn(0x7f68a80a21f0 msgr2=0x7f68a80a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 shutdown_connections 2024-09-16T06:46:52.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.409+0000 7f68ae951700 1 -- 172.21.15.189:0/2762525732 wait complete. 2024-09-16T06:46:52.414 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h25' does not appear in the crush map 2024-09-16T06:46:52.424 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:52.425 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h26 2024-09-16T06:46:52.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.521+0000 7f8c8f4b6700 10 monclient: get_monmap_and_config 2024-09-16T06:46:52.527 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.521+0000 7f8c8f4b6700 10 monclient: build_initial_monmap 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.521+0000 7f8c8f4b6700 10 monclient: monmap: 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:52.527901+0000 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:52.527901+0000 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:52.528 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 Processor -- start 2024-09-16T06:46:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- start start 2024-09-16T06:46:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient: init 2024-09-16T06:46:52.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-c con 0x7f8c8809f140 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-b con 0x7f8c88099700 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-a con 0x7f8c881396a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8809f520 con 0x7f8c881396a0 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8809bc10 con 0x7f8c8809f140 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8809b810 con 0x7f8c88099700 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient(hunting): authenticate will time out at 1365.041136s 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8d252700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8c8809f140 legacy=0x7f8c8809a9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8da53700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8c881396a0 0x7f8c8813ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57358/0 (socket says 172.21.15.189:57358) 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8da53700 1 -- 172.21.15.189:0/3479774467 learned_addr learned my addr 172.21.15.189:0/3479774467 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:52.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8ca51700 1 --1- 172.21.15.189:0/3479774467 >> v1:172.21.15.76:6790/0 conn(0x7f8c88099700 0x7f8c8809b100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46172/0 (socket says 172.21.15.189:46172) 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877663559 0 0) 0x7f8c8809f520 con 0x7f8c881396a0 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): my global_id is 184533 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c740035d0 con 0x7f8c881396a0 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771551017 0 0) 0x7f8c8809b810 con 0x7f8c88099700 2024-09-16T06:46:52.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient(hunting): my global_id is 194312 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8809f520 con 0x7f8c88099700 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3664410716 0 0) 0x7f8c740035d0 con 0x7f8c881396a0 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 >> v1:172.21.15.76:6790/0 conn(0x7f8c88099700 legacy=0x7f8c8809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 >> v1:172.21.15.189:6789/0 conn(0x7f8c8809f140 legacy=0x7f8c8809a9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:52.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 monclient: found mon.noname-a 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8809ceb0 con 0x7f8c881396a0 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: _finish_auth 0 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 5 monclient: authenticate success, global_id 184533 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c80001990 con 0x7f8c881396a0 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:52.536 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: dump: 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:52.537 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c80001c70 con 0x7f8c881396a0 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 1 -- 172.21.15.189:0/3479774467 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c80002000 con 0x7f8c881396a0 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 4 set_mon_vals no callback set 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c7ffff700 10 monclient: dump: 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.538 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 10 monclient: shutdown 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/3479774467 >> v1:172.21.15.76:6789/0 conn(0x7f8c881396a0 legacy=0x7f8c8813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/3479774467 shutdown_connections 2024-09-16T06:46:52.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/3479774467 >> 172.21.15.189:0/3479774467 conn(0x7f8c880a03e0 msgr2=0x7f8c880a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/3479774467 shutdown_connections 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.525+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/3479774467 wait complete. 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: build_initial_monmap 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: monmap: 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:52.540 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:52.541 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 Processor -- start 2024-09-16T06:46:52.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- start start 2024-09-16T06:46:52.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: init 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-a con 0x7f8c88099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-c con 0x7f8c8809f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient(hunting): picked mon.noname-b con 0x7f8c881396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8812bef0 con 0x7f8c88099700 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8812a840 con 0x7f8c881396a0 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8812cdc0 con 0x7f8c8809f140 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient(hunting): authenticate will time out at 1365.045862s 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8da53700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8c881396a0 legacy=0x7f8c881d2360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:52.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8ca51700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8c8809f140 0x7f8c8812a130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:46186/0 (socket says 172.21.15.189:46186) 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8d252700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8c88099700 0x7f8c88129a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57374/0 (socket says 172.21.15.189:57374) 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8ca51700 1 -- 172.21.15.189:0/2081996542 learned_addr learned my addr 172.21.15.189:0/2081996542 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658417421 0 0) 0x7f8c8812cdc0 con 0x7f8c8809f140 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): my global_id is 194315 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c0035d0 con 0x7f8c8809f140 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835174865 0 0) 0x7f8c8812bef0 con 0x7f8c88099700 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): my global_id is 184536 2024-09-16T06:46:52.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8812cdc0 con 0x7f8c88099700 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533770944 0 0) 0x7f8c6c0035d0 con 0x7f8c8809f140 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c8812bef0 con 0x7f8c8809f140 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c780018d0 con 0x7f8c8809f140 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): dump: 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.548 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405647255 0 0) 0x7f8c8812cdc0 con 0x7f8c88099700 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c6c0035d0 con 0x7f8c88099700 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c840040a0 con 0x7f8c88099700 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient(hunting): dump: 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.549 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.550 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3573065794 0 0) 0x7f8c8812bef0 con 0x7f8c8809f140 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 >> v1:172.21.15.189:6789/0 conn(0x7f8c881396a0 legacy=0x7f8c881d2360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 >> v1:172.21.15.76:6789/0 conn(0x7f8c88099700 legacy=0x7f8c88129a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 monclient: found mon.c 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8812d0a0 con 0x7f8c8809f140 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:52.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c7dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 5 monclient: authenticate success, global_id 194315 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _renew_subs 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8812d380 con 0x7f8c8809f140 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _renew_subs 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.529+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8812b390 con 0x7f8c8809f140 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c78001ca0 con 0x7f8c8809f140 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c78004ce0 con 0x7f8c8809f140 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7effd700 4 set_mon_vals no callback set 2024-09-16T06:46:52.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 10 monclient: dump: 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c8f4b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:52.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c8f4b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c8f4b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c68005100 con 0x7f8c8809f140 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8c78004ec0 con 0x7f8c8809f140 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(238..238 src has 151..238) v4 ==== 8774+0+0 (unknown 2262568899 0 0) 0x7f8c78044750 con 0x7f8c8809f140 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.533+0000 7f8c8d252700 10 monclient: get_auth_request con 0x7f8c6c036220 auth_method 0 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.541+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8c780148e0 con 0x7f8c8809f140 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.541+0000 7f8c7dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:52.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.541+0000 7f8c7dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.717+0000 7f8c8f4b6700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:52.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.717+0000 7f8c8f4b6700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.717+0000 7f8c8f4b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:52.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:52.717+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7f8c680053b0 con 0x7f8c8809f140 2024-09-16T06:46:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.297+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v239) v1 ==== 120+0+0 (unknown 1122115194 0 0) 0x7f8c78012d50 con 0x7f8c8809f140 2024-09-16T06:46:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.297+0000 7f8c7dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:53.303 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.297+0000 7f8c7dffb700 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-16T06:46:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7f8c68005650 con 0x7f8c8809f140 2024-09-16T06:46:53.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c7dffb700 1 -- 172.21.15.189:0/2081996542 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v239) v1 ==== 120+0+0 (unknown 2523731723 0 0) 0x7f8c780144d0 con 0x7f8c8809f140 2024-09-16T06:46:53.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c7dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T06:46:53.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c7dffb700 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8c6c036220 legacy=0x7f8c6c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 10 monclient: shutdown 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 >> v1:172.21.15.76:6790/0 conn(0x7f8c8809f140 legacy=0x7f8c8812a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 shutdown_connections 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.413+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 >> 172.21.15.189:0/2081996542 conn(0x7f8c880a03e0 msgr2=0x7f8c880a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.417+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 shutdown_connections 2024-09-16T06:46:53.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.417+0000 7f8c8f4b6700 1 -- 172.21.15.189:0/2081996542 wait complete. 2024-09-16T06:46:53.421 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h26' does not appear in the crush map 2024-09-16T06:46:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:53.431 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h27 2024-09-16T06:46:53.539 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 10 monclient: get_monmap_and_config 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 10 monclient: build_initial_monmap 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 10 monclient: monmap: 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:53.535750+0000 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:53.535750+0000 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:53.540 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 1 Processor -- start 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 1 -- start start 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.529+0000 7f8979ae7700 10 monclient: init 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-b con 0x7f897409bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-a con 0x7f897409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-c con 0x7f8974138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:53.541 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897409cca0 con 0x7f897409c5f0 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897409f750 con 0x7f8974138da0 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8974097620 con 0x7f897409bb00 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient(hunting): authenticate will time out at 1366.048566s 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8973fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f8974138da0 legacy=0x7f897413b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8972ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f897409c5f0 0x7f8974137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57376/0 (socket says 172.21.15.189:57376) 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8972ffd700 1 -- 172.21.15.189:0/2635909086 learned_addr learned my addr 172.21.15.189:0/2635909086 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898506566 0 0) 0x7f897409cca0 con 0x7f897409c5f0 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): my global_id is 184539 2024-09-16T06:46:53.542 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c0035d0 con 0x7f897409c5f0 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983730460 0 0) 0x7f8974097620 con 0x7f897409bb00 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient(hunting): my global_id is 194318 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f897409cca0 con 0x7f897409bb00 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 370495011 0 0) 0x7f895c0035d0 con 0x7f897409c5f0 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 >> v1:172.21.15.76:6790/0 conn(0x7f897409bb00 legacy=0x7f897409bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 >> v1:172.21.15.189:6789/0 conn(0x7f8974138da0 legacy=0x7f897413b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.543 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8974097900 con 0x7f897409c5f0 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 5 monclient: authenticate success, global_id 184539 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8964001610 con 0x7f897409c5f0 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: dump: 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.544 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89640018f0 con 0x7f897409c5f0 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 1 -- 172.21.15.189:0/2635909086 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8964001c80 con 0x7f897409c5f0 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 4 set_mon_vals no callback set 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:53.545 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f89727fc700 10 monclient: dump: 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: shutdown 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- 172.21.15.189:0/2635909086 >> v1:172.21.15.76:6789/0 conn(0x7f897409c5f0 legacy=0x7f8974137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.546 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- 172.21.15.189:0/2635909086 shutdown_connections 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- 172.21.15.189:0/2635909086 >> 172.21.15.189:0/2635909086 conn(0x7f89740a03e0 msgr2=0x7f89740a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- 172.21.15.189:0/2635909086 shutdown_connections 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 1 -- 172.21.15.189:0/2635909086 wait complete. 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: build_initial_monmap 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.533+0000 7f8979ae7700 10 monclient: monmap: 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:53.547 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 Processor -- start 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- start start 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: init 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-b con 0x7f897409bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-a con 0x7f897409c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient(hunting): picked mon.noname-c con 0x7f8974138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8974091120 con 0x7f897409c5f0 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897408f490 con 0x7f897409bb00 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89740a12f0 con 0x7f8974138da0 2024-09-16T06:46:53.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient(hunting): authenticate will time out at 1366.053157s 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f89737fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f897409bb00 legacy=0x7f897408cee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8972ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f897409c5f0 0x7f897408d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:57390/0 (socket says 172.21.15.189:57390) 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8972ffd700 1 -- 172.21.15.189:0/341456291 learned_addr learned my addr 172.21.15.189:0/341456291 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171106905 0 0) 0x7f8974091120 con 0x7f897409c5f0 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): my global_id is 184542 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89600035d0 con 0x7f897409c5f0 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713098240 0 0) 0x7f89740a12f0 con 0x7f8974138da0 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): my global_id is 194321 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8974091120 con 0x7f8974138da0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2554733877 0 0) 0x7f8974091120 con 0x7f8974138da0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89740a12f0 con 0x7f8974138da0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506311277 0 0) 0x7f89600035d0 con 0x7f897409c5f0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8974091120 con 0x7f897409c5f0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f896c0043f0 con 0x7f8974138da0 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): dump: 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:53.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8964001c00 con 0x7f897409c5f0 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient(hunting): dump: 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:53.555 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 248984073 0 0) 0x7f89740a12f0 con 0x7f8974138da0 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 >> v1:172.21.15.189:6789/0 conn(0x7f897409bb00 legacy=0x7f897408cee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 >> v1:172.21.15.76:6789/0 conn(0x7f897409c5f0 legacy=0x7f897408d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:53.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 monclient: found mon.c 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8974090940 con 0x7f8974138da0 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: _finish_auth 0 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 5 monclient: authenticate success, global_id 194321 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _renew_subs 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89740a2e10 con 0x7f8974138da0 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _renew_subs 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f897408f210 con 0x7f8974138da0 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f896c004610 con 0x7f8974138da0 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f896c004920 con 0x7f8974138da0 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f8971ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.537+0000 7f895bfff700 10 monclient: dump: 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.558 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f896c004b00 con 0x7f8974138da0 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f8979ae7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f8979ae7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f8979ae7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(239..239 src has 151..239) v4 ==== 8561+0+0 (unknown 989705829 0 0) 0x7f896c0446f0 con 0x7f8974138da0 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8954005100 con 0x7f8974138da0 2024-09-16T06:46:53.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.541+0000 7f89737fe700 10 monclient: get_auth_request con 0x7f89600362f0 auth_method 0 2024-09-16T06:46:53.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.549+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f896c0148c0 con 0x7f8974138da0 2024-09-16T06:46:53.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.549+0000 7f895bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.549+0000 7f895bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:53.711 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:46:53.711 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:46:53.712 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:46:53.712 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:46:53.714 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:46:53.715 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:46:53.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:53.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.725+0000 7f8979ae7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:53.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.725+0000 7f8979ae7700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:53.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.725+0000 7f8979ae7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:53.731 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:53.725+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f89540053b0 con 0x7f8974138da0 2024-09-16T06:46:53.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: get_monmap_and_config 2024-09-16T06:46:53.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: build_initial_monmap 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: monmap: 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:53.837789+0000 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:53.837789+0000 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:53.836 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:53.836 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:53.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 1 Processor -- start 2024-09-16T06:46:53.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 1 -- start start 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: init 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-c con 0x7f731009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-b con 0x7f731009fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-a con 0x7f7310139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:53.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731009cd80 con 0x7f7310139b70 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7310099fd0 con 0x7f731009f230 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731009a410 con 0x7f731009fd20 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f7315d1f700 10 monclient(hunting): authenticate will time out at 1360.971464s 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f730ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7310139b70 0x7f731013bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:32914/0 (socket says 172.21.15.76:32914) 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f730ffff700 1 -- 172.21.15.76:0/4221927672 learned_addr learned my addr 172.21.15.76:0/4221927672 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:53.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.835+0000 7f730effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f731009fd20 0x7f7310138450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38338/0 (socket says 172.21.15.76:38338) 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891916163 0 0) 0x7f731009a410 con 0x7f731009fd20 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): my global_id is 194324 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f00035d0 con 0x7f731009fd20 2024-09-16T06:46:53.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638332322 0 0) 0x7f731009cd80 con 0x7f7310139b70 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient(hunting): my global_id is 184545 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731009a410 con 0x7f7310139b70 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4057793018 0 0) 0x7f731009a410 con 0x7f7310139b70 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 >> v1:172.21.15.76:6790/0 conn(0x7f731009fd20 legacy=0x7f7310138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 >> v1:172.21.15.189:6789/0 conn(0x7f731009f230 legacy=0x7f731009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731009de90 con 0x7f7310139b70 2024-09-16T06:46:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f73080015f0 con 0x7f7310139b70 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: dump: 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:53.857 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 5 monclient: authenticate success, global_id 184545 2024-09-16T06:46:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73080018a0 con 0x7f7310139b70 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 1 -- 172.21.15.76:0/4221927672 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7308001c70 con 0x7f7310139b70 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 4 set_mon_vals no callback set 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f730e7fc700 10 monclient: dump: 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.859 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 10 monclient: shutdown 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 1 -- 172.21.15.76:0/4221927672 >> v1:172.21.15.76:6789/0 conn(0x7f7310139b70 legacy=0x7f731013bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 1 -- 172.21.15.76:0/4221927672 shutdown_connections 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 1 -- 172.21.15.76:0/4221927672 >> 172.21.15.76:0/4221927672 conn(0x7f73100a3e50 msgr2=0x7f73100a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 1 -- 172.21.15.76:0/4221927672 shutdown_connections 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 1 -- 172.21.15.76:0/4221927672 wait complete. 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 10 monclient: build_initial_monmap 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.839+0000 7f7315d1f700 10 monclient: monmap: 2024-09-16T06:46:53.860 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 Processor -- start 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- start start 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: init 2024-09-16T06:46:53.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-b con 0x7f731009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-a con 0x7f731009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient(hunting): picked mon.noname-c con 0x7f7310139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7310135250 con 0x7f731009fd20 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7310139490 con 0x7f731009f230 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100706d0 con 0x7f7310139b70 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient(hunting): authenticate will time out at 1360.978202s 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f730effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f731009fd20 0x7f7310134b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:32916/0 (socket says 172.21.15.76:32916) 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f730ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7310139b70 0x7f73101d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:38354/0 (socket says 172.21.15.76:38354) 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f730effd700 1 -- 172.21.15.76:0/1084460553 learned_addr learned my addr 172.21.15.76:0/1084460553 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:53.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907855625 0 0) 0x7f73100706d0 con 0x7f7310139b70 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): my global_id is 194327 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f40035d0 con 0x7f7310139b70 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089673954 0 0) 0x7f7310135250 con 0x7f731009fd20 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): my global_id is 184548 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73100706d0 con 0x7f731009fd20 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2232586244 0 0) 0x7f73100706d0 con 0x7f731009fd20 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7310135250 con 0x7f731009fd20 2024-09-16T06:46:53.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72f80019f0 con 0x7f731009fd20 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): dump: 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.864 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417299570 0 0) 0x7f72f40035d0 con 0x7f7310139b70 2024-09-16T06:46:53.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73100706d0 con 0x7f7310139b70 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7308001bb0 con 0x7f7310139b70 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient(hunting): dump: 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.866 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2171123927 0 0) 0x7f7310135250 con 0x7f731009fd20 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 >> v1:172.21.15.76:6790/0 conn(0x7f7310139b70 legacy=0x7f73101d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 >> v1:172.21.15.189:6789/0 conn(0x7f731009f230 legacy=0x7f7310134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 monclient: found mon.a 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7310072450 con 0x7f731009fd20 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:53.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7307fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 5 monclient: authenticate success, global_id 184548 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _renew_subs 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731013a070 con 0x7f731009fd20 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _renew_subs 2024-09-16T06:46:53.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.843+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7310135a10 con 0x7f731009fd20 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72f8004890 con 0x7f731009fd20 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72f8004ba0 con 0x7f731009fd20 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f730dffb700 4 set_mon_vals no callback set 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 10 monclient: dump: 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:53.869 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f72f800f610 con 0x7f731009fd20 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7315d1f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7315d1f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7315d1f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(239..239 src has 151..239) v4 ==== 8561+0+0 (unknown 989705829 0 0) 0x7f72f80444d0 con 0x7f731009fd20 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e8005100 con 0x7f731009fd20 2024-09-16T06:46:53.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.847+0000 7f730f7fe700 10 monclient: get_auth_request con 0x7f72f40363b0 auth_method 0 2024-09-16T06:46:53.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.855+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f72f80146a0 con 0x7f731009fd20 2024-09-16T06:46:53.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.855+0000 7f7307fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:53.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:53.855+0000 7f7307fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f72e8005930 con 0x7f731009fd20 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7307fff700 1 -- 172.21.15.76:0/1084460553 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2155399449) 0x7f72f8014290 con 0x7f731009fd20 2024-09-16T06:46:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7307fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7307fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:54.087 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 >> v1:172.21.15.189:6812/3098810674 conn(0x7f72f40363b0 legacy=0x7f72f4038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 10 monclient: shutdown 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 >> v1:172.21.15.76:6789/0 conn(0x7f731009fd20 legacy=0x7f7310134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 shutdown_connections 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.087+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 >> 172.21.15.76:0/1084460553 conn(0x7f73100a3e50 msgr2=0x7f731012d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.091+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 shutdown_connections 2024-09-16T06:46:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.091+0000 7f7315d1f700 1 -- 172.21.15.76:0/1084460553 wait complete. 2024-09-16T06:46:54.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:46:54.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:46:54.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:46:54.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.211+0000 7f81b359e700 10 monclient: get_monmap_and_config 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.211+0000 7f81b359e700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.211+0000 7f81b359e700 10 monclient: monmap: 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:54.215016+0000 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:54.215016+0000 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:54.212 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.211+0000 7f81b359e700 1 Processor -- start 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- start start 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient: init 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-b con 0x7f81b409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-a con 0x7f81b4098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-c con 0x7f81b4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b409d560 con 0x7f81b4098be0 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b409f140 con 0x7f81b4099030 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b409d790 con 0x7f81b409ed60 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient(hunting): authenticate will time out at 1361.348870s 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f81b409ed60 0x7f81b40984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37006/0 (socket says 172.21.15.76:37006) 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b259c700 1 -- 172.21.15.76:0/1192672787 learned_addr learned my addr 172.21.15.76:0/1192672787 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874547005 0 0) 0x7f81b409d790 con 0x7f81b409ed60 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): my global_id is 194330 2024-09-16T06:46:54.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c0057e0 con 0x7f81b409ed60 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714093061 0 0) 0x7f81b409d560 con 0x7f81b4098be0 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient(hunting): my global_id is 184551 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b409d790 con 0x7f81b4098be0 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1021582168 0 0) 0x7f819c0057e0 con 0x7f81b409ed60 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 >> v1:172.21.15.189:6789/0 conn(0x7f81b4099030 legacy=0x7f81b413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 >> v1:172.21.15.76:6789/0 conn(0x7f81b4098be0 legacy=0x7f81b41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 monclient: found mon.noname-b 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b409f620 con 0x7f81b409ed60 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a4001610 con 0x7f81b409ed60 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: dump: 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 5 monclient: authenticate success, global_id 194330 2024-09-16T06:46:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a4001830 con 0x7f81b409ed60 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 4 set_mon_vals no callback set 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 1 -- 172.21.15.76:0/1192672787 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a4001bc0 con 0x7f81b409ed60 2024-09-16T06:46:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b159a700 10 monclient: dump: 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 10 monclient: shutdown 2024-09-16T06:46:54.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- 172.21.15.76:0/1192672787 >> v1:172.21.15.76:6790/0 conn(0x7f81b409ed60 legacy=0x7f81b40984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- 172.21.15.76:0/1192672787 shutdown_connections 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- 172.21.15.76:0/1192672787 >> 172.21.15.76:0/1192672787 conn(0x7f81b40a0030 msgr2=0x7f81b40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- 172.21.15.76:0/1192672787 shutdown_connections 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.215+0000 7f81b359e700 1 -- 172.21.15.76:0/1192672787 wait complete. 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient: monmap: 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:54.227 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 1 Processor -- start 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 1 -- start start 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient: init 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-c con 0x7f81b4098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-b con 0x7f81b4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient(hunting): picked mon.noname-a con 0x7f81b409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b4092c70 con 0x7f81b409ed60 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b40a16a0 con 0x7f81b4099030 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b40925c0 con 0x7f81b4098be0 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b359e700 10 monclient(hunting): authenticate will time out at 1361.353051s 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b2d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f81b409ed60 0x7f81b41d2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43086/0 (socket says 172.21.15.76:43086) 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f81b4098be0 0x7f81b40917a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37016/0 (socket says 172.21.15.76:37016) 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81b2d9d700 1 -- 172.21.15.76:0/4162532021 learned_addr learned my addr 172.21.15.76:0/4162532021 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595643229 0 0) 0x7f81b40925c0 con 0x7f81b4098be0 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): my global_id is 194333 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81940035d0 con 0x7f81b4098be0 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028474688 0 0) 0x7f81b4092c70 con 0x7f81b409ed60 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): my global_id is 184554 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b40925c0 con 0x7f81b409ed60 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1877961524 0 0) 0x7f81940035d0 con 0x7f81b4098be0 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81b4092c70 con 0x7f81b4098be0 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a4001510 con 0x7f81b4098be0 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.219+0000 7f81a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4215564431 0 0) 0x7f81b4092c70 con 0x7f81b4098be0 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.189:6789/0 conn(0x7f81b4099030 legacy=0x7f81b4091eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6789/0 conn(0x7f81b409ed60 legacy=0x7f81b41d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 monclient: found mon.c 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40a1250 con 0x7f81b4098be0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 5 monclient: authenticate success, global_id 194333 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 10 monclient: _renew_subs 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b409a540 con 0x7f81b4098be0 2024-09-16T06:46:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 10 monclient: _renew_subs 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b4093b70 con 0x7f81b4098be0 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a4004b40 con 0x7f81b4098be0 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a4004e50 con 0x7f81b4098be0 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b0d99700 4 set_mon_vals no callback set 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: dump: 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8198006590 con 0x7f8198002830 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968788026 0 0) 0x7f8198006590 con 0x7f8198002830 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient(hunting): my global_id is 184557 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4092c70 con 0x7f8198002830 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 435071677 0 0) 0x7f81b4092c70 con 0x7f8198002830 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8198006590 con 0x7f8198002830 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a8001660 con 0x7f8198002830 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: dump: 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3171981882 0 0) 0x7f8198006590 con 0x7f8198002830 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f819800a190 con 0x7f8198002830 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f81a400f8c0 con 0x7f81b4098be0 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(239..239 src has 151..239) v4 ==== 8561+0+0 (unknown 989705829 0 0) 0x7f81a4044640 con 0x7f81b4098be0 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f81a8001a80 con 0x7f8198002830 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6789/0 conn(0x7f8198002830 legacy=0x7f8198004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6789/0 conn(0x7f8198002830 legacy=0x7f8198004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.223+0000 7f81b2d9d700 10 monclient: get_auth_request con 0x7f8194036740 auth_method 0 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81b359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f819800c4e0 con 0x7f8198009170 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144422246 0 0) 0x7f819800c4e0 con 0x7f8198009170 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 10 monclient(hunting): my global_id is 184560 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8198006590 con 0x7f8198009170 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.231+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1373714684 0 0) 0x7f8198006590 con 0x7f8198009170 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f819800c4e0 con 0x7f8198009170 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81a8003a50 con 0x7f8198009170 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 10 monclient: dump: 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3138831154 0 0) 0x7f819800c4e0 con 0x7f8198009170 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8198009d10 con 0x7f8198009170 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2422128288) 0x7f81a8003d40 con 0x7f8198009170 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6789/0 conn(0x7f8198009170 legacy=0x7f8198009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81a2ffd700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6789/0 conn(0x7f8198009170 legacy=0x7f8198009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.241 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.242 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.243 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:54.244 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:54.245 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.246 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.248 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.250 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.251 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.252 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:54.253 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8194036740 legacy=0x7f8194038c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 10 monclient: shutdown 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 >> v1:172.21.15.76:6790/0 conn(0x7f81b4098be0 legacy=0x7f81b40917a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b2d9d700 1 -- 172.21.15.76:0/4162532021 reap_dead start 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b2d9d700 1 -- 172.21.15.76:0/4162532021 reap_dead start 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 shutdown_connections 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 >> 172.21.15.76:0/4162532021 conn(0x7f81b40a0030 msgr2=0x7f81b40702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 shutdown_connections 2024-09-16T06:46:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.235+0000 7f81b359e700 1 -- 172.21.15.76:0/4162532021 wait complete. 2024-09-16T06:46:54.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:54.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:46:54.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:46:54.315 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.309+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v240) v1 ==== 120+0+0 (unknown 1385181404 0 0) 0x7f896c012d30 con 0x7f8974138da0 2024-09-16T06:46:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.309+0000 7f895bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:54.316 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.309+0000 7f895bfff700 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-16T06:46:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.371+0000 7fc3c2ca6700 10 monclient: get_monmap_and_config 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.371+0000 7fc3c2ca6700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.371+0000 7fc3c2ca6700 10 monclient: monmap: 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:54.375197+0000 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:54.375197+0000 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:54.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 Processor -- start 2024-09-16T06:46:54.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- start start 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient: init 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-b con 0x7fc3bc0a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-c con 0x7fc3bc071d80 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-a con 0x7fc3bc138610 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc0a11a0 con 0x7fc3bc138610 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc0a4c20 con 0x7fc3bc071d80 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc0a06e0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient(hunting): authenticate will time out at 1361.509613s 2024-09-16T06:46:54.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c1ca4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc3bc0a4840 0x7fc3bc09b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37024/0 (socket says 172.21.15.76:37024) 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c1ca4700 1 -- 172.21.15.76:0/1706600135 learned_addr learned my addr 172.21.15.76:0/1706600135 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583678835 0 0) 0x7fc3bc0a11a0 con 0x7fc3bc138610 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): my global_id is 184563 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a80057e0 con 0x7fc3bc138610 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603161334 0 0) 0x7fc3bc0a06e0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): my global_id is 194336 2024-09-16T06:46:54.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc0a11a0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562805163 0 0) 0x7fc3bc0a4c20 con 0x7fc3bc071d80 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient(hunting): my global_id is 204100 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc0a06e0 con 0x7fc3bc071d80 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4264505323 0 0) 0x7fc3a80057e0 con 0x7fc3bc138610 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 >> v1:172.21.15.76:6790/0 conn(0x7fc3bc0a4840 legacy=0x7fc3bc09b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 >> v1:172.21.15.189:6789/0 conn(0x7fc3bc071d80 legacy=0x7fc3bc09ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 monclient: found mon.noname-a 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc09c120 con 0x7fc3bc138610 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 5 monclient: authenticate success, global_id 184563 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b4001990 con 0x7fc3bc138610 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: dump: 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b4001bb0 con 0x7fc3bc138610 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 4 set_mon_vals no callback set 2024-09-16T06:46:54.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 1 -- 172.21.15.76:0/1706600135 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b4001f40 con 0x7fc3bc138610 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c0ca2700 10 monclient: dump: 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 10 monclient: shutdown 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/1706600135 >> v1:172.21.15.76:6789/0 conn(0x7fc3bc138610 legacy=0x7fc3bc13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/1706600135 shutdown_connections 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/1706600135 >> 172.21.15.76:0/1706600135 conn(0x7fc3bc0a4430 msgr2=0x7fc3bc0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/1706600135 shutdown_connections 2024-09-16T06:46:54.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.375+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/1706600135 wait complete. 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: monmap: 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:54.388 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 Processor -- start 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- start start 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: init 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-a con 0x7fc3bc071d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-c con 0x7fc3bc0a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient(hunting): picked mon.noname-b con 0x7fc3bc138610 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc122270 con 0x7fc3bc071d80 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc09b0f0 con 0x7fc3bc138610 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc0a2570 con 0x7fc3bc0a4840 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient(hunting): authenticate will time out at 1361.514095s 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c1ca4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc3bc071d80 0x7fc3bc1378e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43124/0 (socket says 172.21.15.76:43124) 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c1ca4700 1 -- 172.21.15.76:0/997384725 learned_addr learned my addr 172.21.15.76:0/997384725 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688315096 0 0) 0x7fc3bc0a2570 con 0x7fc3bc0a4840 2024-09-16T06:46:54.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): my global_id is 194339 2024-09-16T06:46:54.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a00035d0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226102554 0 0) 0x7fc3bc09b0f0 con 0x7fc3bc138610 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): my global_id is 204103 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc0a2570 con 0x7fc3bc138610 2024-09-16T06:46:54.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487848435 0 0) 0x7fc3bc122270 con 0x7fc3bc071d80 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): my global_id is 184566 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc09b0f0 con 0x7fc3bc071d80 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1999984719 0 0) 0x7fc3a00035d0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3bc122270 con 0x7fc3bc0a4840 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3ac0018b0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.393 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545980353 0 0) 0x7fc3bc09b0f0 con 0x7fc3bc071d80 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a00035d0 con 0x7fc3bc071d80 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279125916 0 0) 0x7fc3bc0a2570 con 0x7fc3bc138610 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3bc09b0f0 con 0x7fc3bc138610 2024-09-16T06:46:54.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b4001ec0 con 0x7fc3bc138610 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.395 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b80018f0 con 0x7fc3bc071d80 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1577466837 0 0) 0x7fc3bc122270 con 0x7fc3bc0a4840 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.189:6789/0 conn(0x7fc3bc138610 legacy=0x7fc3bc1ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6789/0 conn(0x7fc3bc071d80 legacy=0x7fc3bc1378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 monclient: found mon.c 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc071580 con 0x7fc3bc0a4840 2024-09-16T06:46:54.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 5 monclient: authenticate success, global_id 194339 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _renew_subs 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc137ff0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _renew_subs 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.379+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc11fea0 con 0x7fc3bc0a4840 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3ac001c40 con 0x7fc3bc0a4840 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3ac003b10 con 0x7fc3bc0a4840 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: dump: 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b3fff700 4 set_mon_vals no callback set 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3c2ca6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc3a4006590 con 0x7fc3a4002830 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc3ac00e580 con 0x7fc3bc0a4840 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(240..240 src has 151..240) v4 ==== 8348+0+0 (unknown 3172574935 0 0) 0x7fc3ac043330 con 0x7fc3bc0a4840 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487863750 0 0) 0x7fc3a4006590 con 0x7fc3a4002830 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient(hunting): my global_id is 194342 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3bc122270 con 0x7fc3a4002830 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3c24a5700 10 monclient: get_auth_request con 0x7fc3a00364a0 auth_method 0 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 300586551 0 0) 0x7fc3bc122270 con 0x7fc3a4002830 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a4006590 con 0x7fc3a4002830 2024-09-16T06:46:54.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b8004a30 con 0x7fc3a4002830 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: dump: 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.402 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 799238264 0 0) 0x7fc3a4006590 con 0x7fc3a4002830 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3a400a190 con 0x7fc3a4002830 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc3b8004cd0 con 0x7fc3a4002830 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6790/0 conn(0x7fc3a4002830 legacy=0x7fc3a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.383+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6790/0 conn(0x7fc3a4002830 legacy=0x7fc3a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc3a4009e30 con 0x7fc3a4009150 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565224149 0 0) 0x7fc3a4009e30 con 0x7fc3a4009150 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient(hunting): my global_id is 194345 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a4006590 con 0x7fc3a4009150 2024-09-16T06:46:54.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1818967811 0 0) 0x7fc3a4006590 con 0x7fc3a4009150 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a4009e30 con 0x7fc3a4009150 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3b8005290 con 0x7fc3a4009150 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient: dump: 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.406 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1219033643 0 0) 0x7fc3a4009e30 con 0x7fc3a4009150 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc3a400c340 con 0x7fc3a4009150 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 363856947) 0x7fc3b80065b0 con 0x7fc3a4009150 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6790/0 conn(0x7fc3a4009150 legacy=0x7fc3a40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3b27fc700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6790/0 conn(0x7fc3a4009150 legacy=0x7fc3a40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:54.407 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.408 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.410 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:54.411 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.412 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:54.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.415 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.416 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.416 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.416 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.416 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.417 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:54.417 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:54.417 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.417 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.418 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.418 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:54.418 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.419 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.420 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.421 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:54.421 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:54.421 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.421 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.422 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.422 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:54.422 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.422 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.422 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.423 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.423 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.423 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.423 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.424 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.424 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:54.425 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:54.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.425 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.425 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.426 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:54.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.428 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.428 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.428 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:46:54.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.429 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.429 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:54.429 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:54.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc3a00364a0 legacy=0x7fc3a0038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 10 monclient: shutdown 2024-09-16T06:46:54.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 >> v1:172.21.15.76:6790/0 conn(0x7fc3bc0a4840 legacy=0x7fc3bc1caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c24a5700 1 -- 172.21.15.76:0/997384725 reap_dead start 2024-09-16T06:46:54.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c24a5700 1 -- 172.21.15.76:0/997384725 reap_dead start 2024-09-16T06:46:54.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 shutdown_connections 2024-09-16T06:46:54.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 >> 172.21.15.76:0/997384725 conn(0x7fc3bc0a4430 msgr2=0x7fc3bc09e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 shutdown_connections 2024-09-16T06:46:54.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.395+0000 7fc3c2ca6700 1 -- 172.21.15.76:0/997384725 wait complete. 2024-09-16T06:46:54.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:54.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:46:54.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:46:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.421+0000 7f8979ae7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.421+0000 7f8979ae7700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:54.438 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.421+0000 7f8979ae7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.421+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7f89540023c0 con 0x7f8974138da0 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f895bfff700 1 -- 172.21.15.189:0/341456291 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v240) v1 ==== 120+0+0 (unknown 3064164728 0 0) 0x7f896c0144b0 con 0x7f8974138da0 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f895bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f895bfff700 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 >> v1:172.21.15.189:6812/3098810674 conn(0x7f89600362f0 legacy=0x7f89600387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.439 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 10 monclient: shutdown 2024-09-16T06:46:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 >> v1:172.21.15.76:6790/0 conn(0x7f8974138da0 legacy=0x7f89741cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 shutdown_connections 2024-09-16T06:46:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 >> 172.21.15.189:0/341456291 conn(0x7f89740a03e0 msgr2=0x7f897409a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.440 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 shutdown_connections 2024-09-16T06:46:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.425+0000 7f8979ae7700 1 -- 172.21.15.189:0/341456291 wait complete. 2024-09-16T06:46:54.441 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h27' does not appear in the crush map 2024-09-16T06:46:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:54.442 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h28 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.545+0000 7fedb3c6c700 10 monclient: get_monmap_and_config 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.545+0000 7fedb3c6c700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.545+0000 7fedb3c6c700 10 monclient: monmap: 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:54.552318+0000 2024-09-16T06:46:54.552 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:54.552318+0000 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:54.553 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 Processor -- start 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- start start 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient: init 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-b con 0x7fedac09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-a con 0x7fedac099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-c con 0x7fedac1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.557 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac09f520 con 0x7fedac099700 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac09bc10 con 0x7fedac1396a0 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac09b810 con 0x7fedac09f140 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient(hunting): authenticate will time out at 1367.065375s 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb2209700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fedac1396a0 0x7fedac13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52428/0 (socket says 172.21.15.189:52428) 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb2209700 1 -- 172.21.15.189:0/3413710548 learned_addr learned my addr 172.21.15.189:0/3413710548 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465207418 0 0) 0x7fedac09bc10 con 0x7fedac1396a0 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): my global_id is 204106 2024-09-16T06:46:54.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed980035d0 con 0x7fedac1396a0 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895573834 0 0) 0x7fedac09b810 con 0x7fedac09f140 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): my global_id is 194348 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac09bc10 con 0x7fedac09f140 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632006866 0 0) 0x7fedac09f520 con 0x7fedac099700 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient(hunting): my global_id is 184569 2024-09-16T06:46:54.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac09b810 con 0x7fedac099700 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445519449 0 0) 0x7fed980035d0 con 0x7fedac1396a0 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 >> v1:172.21.15.76:6790/0 conn(0x7fedac09f140 legacy=0x7fedac09a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 >> v1:172.21.15.76:6789/0 conn(0x7fedac099700 legacy=0x7fedac09b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 monclient: found mon.noname-c 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac09ceb0 con 0x7fedac1396a0 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 5 monclient: authenticate success, global_id 204106 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda4001610 con 0x7fedac1396a0 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: dump: 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda40018f0 con 0x7fedac1396a0 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 4 set_mon_vals no callback set 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 1 -- 172.21.15.189:0/3413710548 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda4001c80 con 0x7fedac1396a0 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb0a06700 10 monclient: dump: 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.563 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 10 monclient: shutdown 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3413710548 >> v1:172.21.15.189:6789/0 conn(0x7fedac1396a0 legacy=0x7fedac13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3413710548 shutdown_connections 2024-09-16T06:46:54.564 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3413710548 >> 172.21.15.189:0/3413710548 conn(0x7fedac0a03e0 msgr2=0x7fedac0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3413710548 shutdown_connections 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.549+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3413710548 wait complete. 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient: monmap: 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.565 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:54.566 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.566 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:54.566 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:54.566 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 1 Processor -- start 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 1 -- start start 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient: init 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:54.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-a con 0x7fedac099700 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-c con 0x7fedac09f140 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient(hunting): picked mon.noname-b con 0x7fedac1396a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac092c60 con 0x7fedac099700 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac071610 con 0x7fedac1396a0 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac06ba80 con 0x7fedac09f140 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb3c6c700 10 monclient(hunting): authenticate will time out at 1367.071036s 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb2209700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fedac1396a0 0x7fedac1cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52436/0 (socket says 172.21.15.189:52436) 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fedb2209700 1 -- 172.21.15.189:0/3033188128 learned_addr learned my addr 172.21.15.189:0/3033188128 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:54.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215588409 0 0) 0x7fedac071610 con 0x7fedac1396a0 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): my global_id is 204109 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed900035d0 con 0x7fedac1396a0 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966215566 0 0) 0x7fedac092c60 con 0x7fedac099700 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): my global_id is 184572 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac071610 con 0x7fedac099700 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393971140 0 0) 0x7fedac06ba80 con 0x7fedac09f140 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 10 monclient(hunting): my global_id is 194351 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.553+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac092c60 con 0x7fedac09f140 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3100284370 0 0) 0x7fed900035d0 con 0x7fedac1396a0 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedac06ba80 con 0x7fedac1396a0 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda40015d0 con 0x7fedac1396a0 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.579 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292373835 0 0) 0x7fedac071610 con 0x7fedac099700 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed900035d0 con 0x7fedac099700 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda00018b0 con 0x7fedac099700 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202419829 0 0) 0x7fedac092c60 con 0x7fedac09f140 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedac071610 con 0x7fedac09f140 2024-09-16T06:46:54.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda8001ae0 con 0x7fedac09f140 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient(hunting): dump: 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3486242112 0 0) 0x7fedac06ba80 con 0x7fedac1396a0 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 >> v1:172.21.15.76:6790/0 conn(0x7fedac09f140 legacy=0x7fedac1cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 >> v1:172.21.15.76:6789/0 conn(0x7fedac099700 legacy=0x7fedac0963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 monclient: found mon.b 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac090900 con 0x7fedac1396a0 2024-09-16T06:46:54.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 5 monclient: authenticate success, global_id 204109 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _renew_subs 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac0914d0 con 0x7fedac1396a0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _renew_subs 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac096b50 con 0x7fedac1396a0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4001060 con 0x7fedac1396a0 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7feda4004e40 con 0x7fedac1396a0 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 10 monclient: dump: 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.585 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9f7fe700 4 set_mon_vals no callback set 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7feda400f8b0 con 0x7fedac1396a0 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(240..240 src has 151..240) v4 ==== 8348+0+0 (unknown 3172574935 0 0) 0x7feda4044680 con 0x7fedac1396a0 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb1a08700 10 monclient: get_auth_request con 0x7fed90036560 auth_method 0 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.557+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed94005100 con 0x7fedac1396a0 2024-09-16T06:46:54.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.565+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7feda4014a60 con 0x7fedac1396a0 2024-09-16T06:46:54.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.565+0000 7fed9e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.565+0000 7fed9e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:54.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:54.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 10 monclient: monmap: 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:54.567079+0000 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:54.567079+0000 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:54.593 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 1 Processor -- start 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 1 -- start start 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.563+0000 7f8cb3fff700 10 monclient: init 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8cb40a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8cb40a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8cb413b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb409b3e0 con 0x7f8cb40a20c0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40a3450 con 0x7f8cb413b6b0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb409b1a0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient(hunting): authenticate will time out at 1361.700460s 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cabfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8cb40a24a0 0x7f8cb40a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37060/0 (socket says 172.21.15.76:37060) 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cabfff700 1 -- 172.21.15.76:0/1392177283 learned_addr learned my addr 172.21.15.76:0/1392177283 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210181560 0 0) 0x7f8cb409b1a0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): my global_id is 194354 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca00035d0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314198813 0 0) 0x7f8cb409b3e0 con 0x7f8cb40a20c0 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient(hunting): my global_id is 184575 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb409b1a0 con 0x7f8cb40a20c0 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281622085 0 0) 0x7f8ca00035d0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 >> v1:172.21.15.189:6789/0 conn(0x7f8cb413b6b0 legacy=0x7f8cb413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 >> v1:172.21.15.76:6789/0 conn(0x7f8cb40a20c0 legacy=0x7f8cb406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 monclient: found mon.noname-b 2024-09-16T06:46:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb406bd30 con 0x7f8cb40a24a0 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 5 monclient: authenticate success, global_id 194354 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c9c001610 con 0x7f8cb40a24a0 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: dump: 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c9c001830 con 0x7f8cb40a24a0 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 1 -- 172.21.15.76:0/1392177283 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c9c001b40 con 0x7f8cb40a24a0 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 4 set_mon_vals no callback set 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb27fc700 10 monclient: dump: 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.599 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: shutdown 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1392177283 >> v1:172.21.15.76:6790/0 conn(0x7f8cb40a24a0 legacy=0x7f8cb40a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1392177283 shutdown_connections 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1392177283 >> 172.21.15.76:0/1392177283 conn(0x7f8cb40a41c0 msgr2=0x7f8cb40a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1392177283 shutdown_connections 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1392177283 wait complete. 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: monmap: 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:54.601 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 Processor -- start 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 1 -- start start 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.567+0000 7f8cb3fff700 10 monclient: init 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:54.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8cb40a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8cb40a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8cb413b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb41397e0 con 0x7f8cb413b6b0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb4138670 con 0x7f8cb40a24a0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb4137760 con 0x7f8cb40a20c0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient(hunting): authenticate will time out at 1361.704662s 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8cb40a20c0 0x7f8cb4135570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37064/0 (socket says 172.21.15.76:37064) 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb2ffd700 1 -- 172.21.15.76:0/1542356340 learned_addr learned my addr 172.21.15.76:0/1542356340 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317932695 0 0) 0x7f8cb41397e0 con 0x7f8cb413b6b0 2024-09-16T06:46:54.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): my global_id is 184578 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c940035d0 con 0x7f8cb413b6b0 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238226306 0 0) 0x7f8cb4138670 con 0x7f8cb40a24a0 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): my global_id is 204112 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb41397e0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577085292 0 0) 0x7f8cb4137760 con 0x7f8cb40a20c0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): my global_id is 194357 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4138670 con 0x7f8cb40a20c0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500345381 0 0) 0x7f8cb41397e0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb4137760 con 0x7f8cb40a24a0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c9c001510 con 0x7f8cb40a24a0 2024-09-16T06:46:54.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.606 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114712048 0 0) 0x7f8c940035d0 con 0x7f8cb413b6b0 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb41397e0 con 0x7f8cb413b6b0 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cac001a20 con 0x7f8cb413b6b0 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.607 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1835602634 0 0) 0x7f8cb4138670 con 0x7f8cb40a20c0 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c940035d0 con 0x7f8cb40a20c0 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ca4001980 con 0x7f8cb40a20c0 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.609 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 441912463 0 0) 0x7f8cb4137760 con 0x7f8cb40a24a0 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.76:6790/0 conn(0x7f8cb40a20c0 legacy=0x7f8cb4135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.76:6789/0 conn(0x7f8cb413b6b0 legacy=0x7f8cb41d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 monclient: found mon.b 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4139220 con 0x7f8cb40a24a0 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 5 monclient: authenticate success, global_id 204112 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _renew_subs 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb4139500 con 0x7f8cb40a24a0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _renew_subs 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb4138330 con 0x7f8cb40a24a0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c9c000f90 con 0x7f8cb40a24a0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c9c004d60 con 0x7f8cb40a24a0 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb1ffb700 4 set_mon_vals no callback set 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 10 monclient: dump: 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.612 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8c9c004ec0 con 0x7f8cb40a24a0 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(240..240 src has 151..240) v4 ==== 8348+0+0 (unknown 3172574935 0 0) 0x7f8c9c044630 con 0x7f8cb40a24a0 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb2ffd700 10 monclient: get_auth_request con 0x7f8c94036580 auth_method 0 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.571+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c98006590 con 0x7f8c98002830 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591975074 0 0) 0x7f8c98006590 con 0x7f8c98002830 2024-09-16T06:46:54.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient(hunting): my global_id is 204115 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4137760 con 0x7f8c98002830 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3071987795 0 0) 0x7f8cb4137760 con 0x7f8c98002830 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c98006590 con 0x7f8c98002830 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cac003a20 con 0x7f8c98002830 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient: dump: 2024-09-16T06:46:54.614 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2235189536 0 0) 0x7f8c98006590 con 0x7f8c98002830 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c9800a190 con 0x7f8c98002830 2024-09-16T06:46:54.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8cac004a70 con 0x7f8c98002830 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6789/0 conn(0x7f8c98002830 legacy=0x7f8c98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.575+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6789/0 conn(0x7f8c98002830 legacy=0x7f8c98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.579+0000 7f8cb3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.579+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c98009e50 con 0x7f8c98009170 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949806265 0 0) 0x7f8c98009e50 con 0x7f8c98009170 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient(hunting): my global_id is 204118 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c98006590 con 0x7f8c98009170 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 754591536 0 0) 0x7f8c98006590 con 0x7f8c98009170 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c98009e50 con 0x7f8c98009170 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8cac004020 con 0x7f8c98009170 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient: dump: 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.617 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 294992694 0 0) 0x7f8c98009e50 con 0x7f8c98009170 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8c9800c340 con 0x7f8c98009170 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 1685485242) 0x7f8cac004210 con 0x7f8c98009170 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:54.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6789/0 conn(0x7f8c98009170 legacy=0x7f8c98009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cab7fe700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6789/0 conn(0x7f8c98009170 legacy=0x7f8c98009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:54.632 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8c94036580 legacy=0x7f8c94038a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 10 monclient: shutdown 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 >> v1:172.21.15.189:6789/0 conn(0x7f8cb40a24a0 legacy=0x7f8cb41ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb37fe700 1 -- 172.21.15.76:0/1542356340 reap_dead start 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb37fe700 1 -- 172.21.15.76:0/1542356340 reap_dead start 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 shutdown_connections 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 >> 172.21.15.76:0/1542356340 conn(0x7f8cb40a41c0 msgr2=0x7f8cb413aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 shutdown_connections 2024-09-16T06:46:54.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.583+0000 7f8cb3fff700 1 -- 172.21.15.76:0/1542356340 wait complete. 2024-09-16T06:46:54.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:54.635 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:46:54.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:46:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.737+0000 7fedb3c6c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.737+0000 7fedb3c6c700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.737+0000 7fedb3c6c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:54.743 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:54.737+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7fed940053b0 con 0x7fedac1396a0 2024-09-16T06:46:54.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: monmap: 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:54.760043+0000 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:54.760043+0000 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:54.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 1 Processor -- start 2024-09-16T06:46:54.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 1 -- start start 2024-09-16T06:46:54.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: init 2024-09-16T06:46:54.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:54.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fc3700a4840 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fc370071d80 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fc370138610 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700a11a0 con 0x7fc3700a4840 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700a4c20 con 0x7fc370138610 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36dd9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc370071d80 0x7fc37009ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37076/0 (socket says 172.21.15.76:37076) 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36dd9b700 1 -- 172.21.15.76:0/3821537115 learned_addr learned my addr 172.21.15.76:0/3821537115 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700a06e0 con 0x7fc370071d80 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36f59e700 10 monclient(hunting): authenticate will time out at 1361.895259s 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555690811 0 0) 0x7fc3700a11a0 con 0x7fc3700a4840 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): my global_id is 184581 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3580035d0 con 0x7fc3700a4840 2024-09-16T06:46:54.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413598981 0 0) 0x7fc3700a06e0 con 0x7fc370071d80 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): my global_id is 194360 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3700a11a0 con 0x7fc370071d80 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486710591 0 0) 0x7fc3700a4c20 con 0x7fc370138610 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 10 monclient(hunting): my global_id is 204121 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.759+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3700a06e0 con 0x7fc370138610 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244838595 0 0) 0x7fc3700a11a0 con 0x7fc370071d80 2024-09-16T06:46:54.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 >> v1:172.21.15.189:6789/0 conn(0x7fc370138610 legacy=0x7fc37013aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 >> v1:172.21.15.76:6789/0 conn(0x7fc3700a4840 legacy=0x7fc37009b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 monclient: found mon.noname-b 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37009c120 con 0x7fc370071d80 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3640015f0 con 0x7fc370071d80 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 5 monclient: authenticate success, global_id 194360 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: dump: 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc364001890 con 0x7fc370071d80 2024-09-16T06:46:54.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 1 -- 172.21.15.76:0/3821537115 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc364001c60 con 0x7fc370071d80 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 4 set_mon_vals no callback set 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36d59a700 10 monclient: dump: 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: shutdown 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- 172.21.15.76:0/3821537115 >> v1:172.21.15.76:6790/0 conn(0x7fc370071d80 legacy=0x7fc37009ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- 172.21.15.76:0/3821537115 shutdown_connections 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- 172.21.15.76:0/3821537115 >> 172.21.15.76:0/3821537115 conn(0x7fc3700a4430 msgr2=0x7fc3700a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- 172.21.15.76:0/3821537115 shutdown_connections 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- 172.21.15.76:0/3821537115 wait complete. 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: build_initial_monmap 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: monmap: 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:54.775 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 Processor -- start 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- start start 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: init 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:54.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fc370071d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fc3700a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fc370138610 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37009b0f0 con 0x7fc370071d80 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3700a36a0 con 0x7fc3700a4840 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37009bc50 con 0x7fc370138610 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36f59e700 10 monclient(hunting): authenticate will time out at 1361.899070s 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc370071d80 0x7fc370131e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43158/0 (socket says 172.21.15.76:43158) 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc36e59c700 1 -- 172.21.15.76:0/71756840 learned_addr learned my addr 172.21.15.76:0/71756840 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680992601 0 0) 0x7fc37009bc50 con 0x7fc370138610 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): my global_id is 194363 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3500035d0 con 0x7fc370138610 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158037808 0 0) 0x7fc37009b0f0 con 0x7fc370071d80 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): my global_id is 184584 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37009bc50 con 0x7fc370071d80 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760739702 0 0) 0x7fc3700a36a0 con 0x7fc3700a4840 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 10 monclient(hunting): my global_id is 204124 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37009b0f0 con 0x7fc3700a4840 2024-09-16T06:46:54.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.763+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283122358 0 0) 0x7fc3500035d0 con 0x7fc370138610 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3700a36a0 con 0x7fc370138610 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc368001bb0 con 0x7fc370138610 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): dump: 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.779 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415009999 0 0) 0x7fc37009bc50 con 0x7fc370071d80 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3500035d0 con 0x7fc370071d80 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc360001650 con 0x7fc370071d80 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): dump: 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835863473 0 0) 0x7fc37009b0f0 con 0x7fc3700a4840 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc37009bc50 con 0x7fc3700a4840 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3640014f0 con 0x7fc3700a4840 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient(hunting): dump: 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272803288 0 0) 0x7fc3500035d0 con 0x7fc370071d80 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 >> v1:172.21.15.76:6790/0 conn(0x7fc370138610 legacy=0x7fc3701d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 >> v1:172.21.15.189:6789/0 conn(0x7fc3700a4840 legacy=0x7fc3701cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 monclient: found mon.a 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:54.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370120530 con 0x7fc370071d80 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: _finish_auth 0 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 5 monclient: authenticate success, global_id 184584 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _renew_subs 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc370120810 con 0x7fc370071d80 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _renew_subs 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37011e440 con 0x7fc370071d80 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3600036a0 con 0x7fc370071d80 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:54.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3600039b0 con 0x7fc370071d80 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36cd99700 4 set_mon_vals no callback set 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 10 monclient: dump: 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:54.784 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc354005100 con 0x7fc370071d80 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc360003b10 con 0x7fc370071d80 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(240..240 src has 151..240) v4 ==== 8348+0+0 (unknown 3172574935 0 0) 0x7fc3600444d0 con 0x7fc370071d80 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.767+0000 7fc36dd9b700 10 monclient: get_auth_request con 0x7fc350036530 auth_method 0 2024-09-16T06:46:54.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.775+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc360014780 con 0x7fc370071d80 2024-09-16T06:46:54.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.775+0000 7fc35effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:54.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:54.775+0000 7fc35effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.003+0000 7fc36f59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:46:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.003+0000 7fc36f59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:46:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.003+0000 7fc36f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.003+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc354005bc0 con 0x7fc370071d80 2024-09-16T06:46:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.007+0000 7fc35effd700 1 -- 172.21.15.76:0/71756840 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fc360012bf0 con 0x7fc370071d80 2024-09-16T06:46:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.007+0000 7fc35effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:46:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.007+0000 7fc35effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:55.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=2695187554,config=1097430493,health=3798748726,kv=2335881212,logm=575902826}) 2024-09-16T06:46:55.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=13,logm=46} crc {auth=3089719528,config=1097430493,health=3798748726,kv=1465867468,logm=2233062021}) 2024-09-16T06:46:55.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=2695187554,config=1097430493,health=3798748726,kv=2335881212,logm=575902826}) 2024-09-16T06:46:55.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.007+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=12,logm=48} crc {auth=1994705789,config=1097430493,health=1585885715,kv=71370927,logm=2472812573}) 2024-09-16T06:46:55.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097697046}) 2024-09-16T06:46:55.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279599356}) 2024-09-16T06:46:55.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097697046}) 2024-09-16T06:46:55.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208856123}) 2024-09-16T06:46:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc350036530 legacy=0x7fc3500389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 10 monclient: shutdown 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 >> v1:172.21.15.76:6789/0 conn(0x7fc370071d80 legacy=0x7fc370131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 shutdown_connections 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 >> 172.21.15.76:0/71756840 conn(0x7fc3700a4430 msgr2=0x7fc3700a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:55.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 shutdown_connections 2024-09-16T06:46:55.013 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256841739}) 2024-09-16T06:46:55.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:55.011+0000 7fc36f59e700 1 -- 172.21.15.76:0/71756840 wait complete. 2024-09-16T06:46:55.013 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161462827}) 2024-09-16T06:46:55.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256841739}) 2024-09-16T06:46:55.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.011+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3677934173}) 2024-09-16T06:46:55.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189064811}) 2024-09-16T06:46:55.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2391808325}) 2024-09-16T06:46:55.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189064811}) 2024-09-16T06:46:55.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.015+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1321929489}) 2024-09-16T06:46:55.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796720533}) 2024-09-16T06:46:55.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534320103}) 2024-09-16T06:46:55.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796720533}) 2024-09-16T06:46:55.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099307319}) 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535032217}) 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605237460}) 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535032217}) 2024-09-16T06:46:55.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780471431}) 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021872901}) 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848796218}) 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021872901}) 2024-09-16T06:46:55.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.019+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576119671}) 2024-09-16T06:46:55.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889097936}) 2024-09-16T06:46:55.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446956988}) 2024-09-16T06:46:55.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889097936}) 2024-09-16T06:46:55.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175370834}) 2024-09-16T06:46:55.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630276785}) 2024-09-16T06:46:55.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901800723}) 2024-09-16T06:46:55.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630276785}) 2024-09-16T06:46:55.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.023+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095228514}) 2024-09-16T06:46:55.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292428356}) 2024-09-16T06:46:55.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920196548}) 2024-09-16T06:46:55.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292428356}) 2024-09-16T06:46:55.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940957559}) 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656149746}) 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337185353}) 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656149746}) 2024-09-16T06:46:55.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.027+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635185980}) 2024-09-16T06:46:55.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:46:55.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099946089}) 2024-09-16T06:46:55.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746595892}) 2024-09-16T06:46:55.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099946089}) 2024-09-16T06:46:55.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524188080}) 2024-09-16T06:46:55.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127424017}) 2024-09-16T06:46:55.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960167840}) 2024-09-16T06:46:55.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127424017}) 2024-09-16T06:46:55.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259939137}) 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916342484}) 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309895165}) 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916342484}) 2024-09-16T06:46:55.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.031+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2351903948}) 2024-09-16T06:46:55.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206190005}) 2024-09-16T06:46:55.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138514723}) 2024-09-16T06:46:55.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206190005}) 2024-09-16T06:46:55.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660163144}) 2024-09-16T06:46:55.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728597652}) 2024-09-16T06:46:55.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682358763}) 2024-09-16T06:46:55.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728597652}) 2024-09-16T06:46:55.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176295250}) 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2607567292}) 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982322481}) 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2607567292}) 2024-09-16T06:46:55.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.035+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2696672601}) 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383848239}) 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155350891}) 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383848239}) 2024-09-16T06:46:55.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692936808}) 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797003744}) 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172055216}) 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797003744}) 2024-09-16T06:46:55.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266005370}) 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500754317}) 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259192011}) 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500754317}) 2024-09-16T06:46:55.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=117654084}) 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269881761}) 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098223163}) 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269881761}) 2024-09-16T06:46:55.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.039+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323694617}) 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575644629}) 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689869047}) 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575644629}) 2024-09-16T06:46:55.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3617332825}) 2024-09-16T06:46:55.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007458317}) 2024-09-16T06:46:55.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933398585}) 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007458317}) 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.043+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116455041}) 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886166862}) 2024-09-16T06:46:55.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634021268}) 2024-09-16T06:46:55.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886166862}) 2024-09-16T06:46:55.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283881166}) 2024-09-16T06:46:55.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925025092}) 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123960993}) 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925025092}) 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872204286}) 2024-09-16T06:46:55.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913486439}) 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834011060}) 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913486439}) 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583005395}) 2024-09-16T06:46:55.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136214074}) 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=504367210}) 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136214074}) 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145804690}) 2024-09-16T06:46:55.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011752154}) 2024-09-16T06:46:55.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682782097}) 2024-09-16T06:46:55.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011752154}) 2024-09-16T06:46:55.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.047+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=279422008}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114512329}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249644961}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114512329}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53600303}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862134160}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845420356}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862134160}) 2024-09-16T06:46:55.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443380152}) 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648660675}) 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088612711}) 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648660675}) 2024-09-16T06:46:55.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959455825}) 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4001842537}) 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90744333}) 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4001842537}) 2024-09-16T06:46:55.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.051+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4290237970}) 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330370027}) 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832950573}) 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330370027}) 2024-09-16T06:46:55.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777224290}) 2024-09-16T06:46:55.054 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.054 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=747500970}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077048830}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=747500970}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562315915}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909082124}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743455405}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909082124}) 2024-09-16T06:46:55.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983451390}) 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=782600065}) 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488400284}) 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=782600065}) 2024-09-16T06:46:55.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.055+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356897664}) 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969947624}) 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287369837}) 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969947624}) 2024-09-16T06:46:55.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2530645575}) 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113063392}) 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=574503710}) 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113063392}) 2024-09-16T06:46:55.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270885205}) 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228542223}) 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857726001}) 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228542223}) 2024-09-16T06:46:55.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811946919}) 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955351735}) 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794229685}) 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955351735}) 2024-09-16T06:46:55.060 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260471377}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372330602}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2509004119}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372330602}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.059+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401050423}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266006933}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309352209}) 2024-09-16T06:46:55.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266006933}) 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372816255}) 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985689379}) 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090517666}) 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985689379}) 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1709492948}) 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=2,mgr=4} crc {logm=428921020,mdsmap=3585761405,mgr=401727391}) 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1500334645,mdsmap=1531151339}) 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=2,mgr=4} crc {logm=428921020,mdsmap=3585761405,mgr=401727391}) 2024-09-16T06:46:55.064 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.063+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=2722749154,mdsmap=1531151339,mgr=1223607153}) 2024-09-16T06:46:55.064 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.064 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=866058972,mon_config_key=3974076015,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=353356854,osd_snap=2033388227,osdmap=3168260057}) 2024-09-16T06:46:55.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=866058972,mon_config_key=3219578969,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=353356854,osd_snap=2033388227,osdmap=3933073345}) 2024-09-16T06:46:55.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=866058972,mon_config_key=3974076015,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=353356854,osd_snap=2033388227,osdmap=3168260057}) 2024-09-16T06:46:55.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.067+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mgr=2314766964,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3247640354,mon_config_key=362158945,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=353356854,osd_snap=2033388227,osdmap=309679158}) 2024-09-16T06:46:55.072 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.072 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=96} crc {osdmap=2370793963}) 2024-09-16T06:46:55.073 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=96} crc {osdmap=4102764581}) 2024-09-16T06:46:55.073 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:46:55.073 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=96} crc {osdmap=2370793963}) 2024-09-16T06:46:55.073 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:46:55.075+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=91} crc {osdmap=1398140417}) 2024-09-16T06:46:55.340 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.337+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v241) v1 ==== 121+0+0 (unknown 2659064173 0 0) 0x7feda401cf80 con 0x7fedac1396a0 2024-09-16T06:46:55.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.337+0000 7fed9e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:55.341 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.337+0000 7fed9e7fc700 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-16T06:46:55.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.445+0000 7fedb3c6c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.445+0000 7fedb3c6c700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.445+0000 7fedb3c6c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.452 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.445+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7fed940023c0 con 0x7fedac1396a0 2024-09-16T06:46:55.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fed9e7fc700 1 -- 172.21.15.189:0/3033188128 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v241) v1 ==== 120+0+0 (unknown 1088659095 0 0) 0x7feda4012ed0 con 0x7fedac1396a0 2024-09-16T06:46:55.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fed9e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T06:46:55.453 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fed9e7fc700 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 >> v1:172.21.15.189:6812/3098810674 conn(0x7fed90036560 legacy=0x7fed90038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 10 monclient: shutdown 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 >> v1:172.21.15.189:6789/0 conn(0x7fedac1396a0 legacy=0x7fedac1cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 shutdown_connections 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.449+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 >> 172.21.15.189:0/3033188128 conn(0x7fedac0a03e0 msgr2=0x7fedac06f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.453+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 shutdown_connections 2024-09-16T06:46:55.456 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.453+0000 7fedb3c6c700 1 -- 172.21.15.189:0/3033188128 wait complete. 2024-09-16T06:46:55.457 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h28' does not appear in the crush map 2024-09-16T06:46:55.466 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:55.466 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h29 2024-09-16T06:46:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: get_monmap_and_config 2024-09-16T06:46:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: build_initial_monmap 2024-09-16T06:46:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: monmap: 2024-09-16T06:46:55.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:55.572738+0000 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:55.572738+0000 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:55.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 1 Processor -- start 2024-09-16T06:46:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 1 -- start start 2024-09-16T06:46:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: init 2024-09-16T06:46:55.575 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient(hunting): picked mon.noname-c con 0x7f719009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient(hunting): picked mon.noname-b con 0x7f719009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient(hunting): picked mon.noname-a con 0x7f7190138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719009cca0 con 0x7f7190138da0 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719009f750 con 0x7f719009bb00 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7190097620 con 0x7f719009c5f0 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient(hunting): authenticate will time out at 1368.085670s 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71963f5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f719009bb00 0x7f719009bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52438/0 (socket says 172.21.15.189:52438) 2024-09-16T06:46:55.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71963f5700 1 -- 172.21.15.189:0/9386852 learned_addr learned my addr 172.21.15.189:0/9386852 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851357551 0 0) 0x7f719009f750 con 0x7f719009bb00 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): my global_id is 204127 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f717c0057e0 con 0x7f719009bb00 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689267374 0 0) 0x7f719009cca0 con 0x7f7190138da0 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): my global_id is 184587 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719009f750 con 0x7f7190138da0 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952413762 0 0) 0x7f7190097620 con 0x7f719009c5f0 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient(hunting): my global_id is 194366 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719009cca0 con 0x7f719009c5f0 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2610069276 0 0) 0x7f717c0057e0 con 0x7f719009bb00 2024-09-16T06:46:55.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 >> v1:172.21.15.76:6790/0 conn(0x7f719009c5f0 legacy=0x7f7190137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 >> v1:172.21.15.76:6789/0 conn(0x7f7190138da0 legacy=0x7f719013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: _finish_hunting 0 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 monclient: found mon.noname-c 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7190097900 con 0x7f719009bb00 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: _finish_auth 0 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: _check_auth_tickets 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f718c001610 con 0x7f719009bb00 2024-09-16T06:46:55.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: dump: 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 5 monclient: authenticate success, global_id 204127 2024-09-16T06:46:55.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f718c0018f0 con 0x7f719009bb00 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f7198659700 4 set_mon_vals no callback set 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 1 -- 172.21.15.189:0/9386852 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f718c001c80 con 0x7f719009bb00 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.569+0000 7f71953f3700 10 monclient: dump: 2024-09-16T06:46:55.581 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:55.582 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: shutdown 2024-09-16T06:46:55.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/9386852 >> v1:172.21.15.189:6789/0 conn(0x7f719009bb00 legacy=0x7f719009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/9386852 shutdown_connections 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/9386852 >> 172.21.15.189:0/9386852 conn(0x7f71900a03e0 msgr2=0x7f71900a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/9386852 shutdown_connections 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/9386852 wait complete. 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: build_initial_monmap 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: monmap: 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:55.587 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 Processor -- start 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- start start 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: init 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient(hunting): picked mon.noname-b con 0x7f719009bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient(hunting): picked mon.noname-c con 0x7f719009c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:55.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient(hunting): picked mon.noname-a con 0x7f7190138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71901cb020 con 0x7f7190138da0 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900934a0 con 0x7f719009bb00 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900700e0 con 0x7f719009c5f0 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient(hunting): authenticate will time out at 1368.089980s 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f71963f5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f719009bb00 0x7f7190096330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52442/0 (socket says 172.21.15.189:52442) 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f71963f5700 1 -- 172.21.15.189:0/3332530589 learned_addr learned my addr 172.21.15.189:0/3332530589 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988108370 0 0) 0x7f71900934a0 con 0x7f719009bb00 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): my global_id is 204130 2024-09-16T06:46:55.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71740035d0 con 0x7f719009bb00 2024-09-16T06:46:55.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328592330 0 0) 0x7f71901cb020 con 0x7f7190138da0 2024-09-16T06:46:55.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): my global_id is 184590 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71900934a0 con 0x7f7190138da0 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071151853 0 0) 0x7f71900700e0 con 0x7f719009c5f0 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:55.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): my global_id is 194369 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71901cb020 con 0x7f719009c5f0 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1170215080 0 0) 0x7f71740035d0 con 0x7f719009bb00 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71900700e0 con 0x7f719009bb00 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f718c0015d0 con 0x7f719009bb00 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): dump: 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:55.592 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568474823 0 0) 0x7f71900700e0 con 0x7f719009bb00 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 >> v1:172.21.15.76:6790/0 conn(0x7f719009c5f0 legacy=0x7f71901cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 >> v1:172.21.15.76:6789/0 conn(0x7f7190138da0 legacy=0x7f71901d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 monclient: found mon.b 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.593 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719009a6e0 con 0x7f719009bb00 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient: _finish_auth 0 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7182ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 5 monclient: authenticate success, global_id 204130 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _renew_subs 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71900997d0 con 0x7f719009bb00 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _renew_subs 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.573+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7190094e00 con 0x7f719009bb00 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f718c004ca0 con 0x7f719009bb00 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:55.594 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f718c005030 con 0x7f719009bb00 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7194bf2700 4 set_mon_vals no callback set 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 10 monclient: dump: 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:55.595 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7198659700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7198659700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7198659700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178005100 con 0x7f719009bb00 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f718c005210 con 0x7f719009bb00 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(241..241 src has 151..241) v4 ==== 8135+0+0 (unknown 67560669 0 0) 0x7f718c044500 con 0x7f719009bb00 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.577+0000 7f7195bf4700 10 monclient: get_auth_request con 0x7f71740362c0 auth_method 0 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.581+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f718c014ae0 con 0x7f719009bb00 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.581+0000 7f7182ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:55.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.581+0000 7f7182ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.757+0000 7f7198659700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.757+0000 7f7198659700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.757+0000 7f7198659700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:55.762 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:55.757+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f71780053b0 con 0x7f719009bb00 2024-09-16T06:46:56.030 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:46:56.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.147+0000 7ff1e9fa8700 10 monclient: get_monmap_and_config 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.147+0000 7ff1e9fa8700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.147+0000 7ff1e9fa8700 10 monclient: monmap: 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.148 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:56.151449+0000 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:56.151449+0000 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:56.149 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 Processor -- start 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- start start 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient: init 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-b con 0x7ff1e4097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-c con 0x7ff1e4099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-a con 0x7ff1e4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e4098990 con 0x7ff1e4139500 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e4097f40 con 0x7ff1e4099090 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e409d610 con 0x7ff1e4097b60 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient(hunting): authenticate will time out at 1363.285064s 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e97a7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff1e4139500 0x7ff1e413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43178/0 (socket says 172.21.15.76:43178) 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e97a7700 1 -- 172.21.15.76:0/3361699391 learned_addr learned my addr 172.21.15.76:0/3361699391 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926125662 0 0) 0x7ff1e4098990 con 0x7ff1e4139500 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): my global_id is 184593 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d40035d0 con 0x7ff1e4139500 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663899444 0 0) 0x7ff1e409d610 con 0x7ff1e4097b60 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): my global_id is 194372 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4098990 con 0x7ff1e4097b60 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749387740 0 0) 0x7ff1e4097f40 con 0x7ff1e4099090 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient(hunting): my global_id is 204133 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e409d610 con 0x7ff1e4099090 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25091137 0 0) 0x7ff1d40035d0 con 0x7ff1e4139500 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 >> v1:172.21.15.76:6790/0 conn(0x7ff1e4097b60 legacy=0x7ff1e409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 >> v1:172.21.15.189:6789/0 conn(0x7ff1e4099090 legacy=0x7ff1e409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40986b0 con 0x7ff1e4139500 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 5 monclient: authenticate success, global_id 184593 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1dc001610 con 0x7ff1e4139500 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: dump: 2024-09-16T06:46:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1dc001830 con 0x7ff1e4139500 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 4 set_mon_vals no callback set 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 1 -- 172.21.15.76:0/3361699391 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1dc001bc0 con 0x7ff1e4139500 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e37fe700 10 monclient: dump: 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 10 monclient: shutdown 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/3361699391 >> v1:172.21.15.76:6789/0 conn(0x7ff1e4139500 legacy=0x7ff1e413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/3361699391 shutdown_connections 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/3361699391 >> 172.21.15.76:0/3361699391 conn(0x7ff1e40a0030 msgr2=0x7ff1e40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/3361699391 shutdown_connections 2024-09-16T06:46:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.151+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/3361699391 wait complete. 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient: monmap: 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 1 Processor -- start 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 1 -- start start 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient: init 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-a con 0x7ff1e4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-c con 0x7ff1e4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient(hunting): picked mon.noname-b con 0x7ff1e4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e40a2b50 con 0x7ff1e4097b60 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e406d0e0 con 0x7ff1e4139500 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e41386d0 con 0x7ff1e4099090 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e9fa8700 10 monclient(hunting): authenticate will time out at 1363.290788s 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff1e4099090 0x7ff1e4137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37104/0 (socket says 172.21.15.76:37104) 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e8fa6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff1e4097b60 0x7ff1e4136f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43188/0 (socket says 172.21.15.76:43188) 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e3fff700 1 -- 172.21.15.76:0/4183709048 learned_addr learned my addr 172.21.15.76:0/4183709048 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756203030 0 0) 0x7ff1e41386d0 con 0x7ff1e4099090 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): my global_id is 194375 2024-09-16T06:46:56.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c80035d0 con 0x7ff1e4099090 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343973559 0 0) 0x7ff1e40a2b50 con 0x7ff1e4097b60 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): my global_id is 184596 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e41386d0 con 0x7ff1e4097b60 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957691944 0 0) 0x7ff1e406d0e0 con 0x7ff1e4139500 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): my global_id is 204136 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e40a2b50 con 0x7ff1e4139500 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609137650 0 0) 0x7ff1c80035d0 con 0x7ff1e4099090 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1e406d0e0 con 0x7ff1e4099090 2024-09-16T06:46:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1d0001990 con 0x7ff1e4099090 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): dump: 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945491419 0 0) 0x7ff1e41386d0 con 0x7ff1e4097b60 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c80035d0 con 0x7ff1e4097b60 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1d8001c70 con 0x7ff1e4097b60 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.155+0000 7ff1e17fa700 10 monclient(hunting): dump: 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.168 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1140924187 0 0) 0x7ff1e406d0e0 con 0x7ff1e4099090 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 >> v1:172.21.15.189:6789/0 conn(0x7ff1e4139500 legacy=0x7ff1e41cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 >> v1:172.21.15.76:6789/0 conn(0x7ff1e4097b60 legacy=0x7ff1e4136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 monclient: found mon.c 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e413c2d0 con 0x7ff1e4099090 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 5 monclient: authenticate success, global_id 194375 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _renew_subs 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e409a540 con 0x7ff1e4099090 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _renew_subs 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e406cb20 con 0x7ff1e4099090 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1d0001d40 con 0x7ff1e4099090 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff1d0003b10 con 0x7ff1e4099090 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e2ffd700 4 set_mon_vals no callback set 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 10 monclient: dump: 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff1d0003c70 con 0x7ff1e4099090 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4005100 con 0x7ff1e4099090 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(241..241 src has 151..241) v4 ==== 8135+0+0 (unknown 67560669 0 0) 0x7ff1d00443e0 con 0x7ff1e4099090 2024-09-16T06:46:56.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.159+0000 7ff1e8fa6700 10 monclient: get_auth_request con 0x7ff1c8036210 auth_method 0 2024-09-16T06:46:56.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.167+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff1d0014580 con 0x7ff1e4099090 2024-09-16T06:46:56.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.167+0000 7ff1e17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.167+0000 7ff1e17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.345+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v242) v1 ==== 121+0+0 (unknown 2204287237 0 0) 0x7f718c01d000 con 0x7f719009bb00 2024-09-16T06:46:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.345+0000 7f7182ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:56.348 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.345+0000 7f7182ffd700 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-16T06:46:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e9fa8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e9fa8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e9fa8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff1c4005930 con 0x7ff1e4099090 2024-09-16T06:46:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e17fa700 1 -- 172.21.15.76:0/4183709048 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 4101374358) 0x7ff1d00143a0 con 0x7ff1e4099090 2024-09-16T06:46:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.403+0000 7ff1e17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:56.403 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff1c8036210 legacy=0x7ff1c80386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 10 monclient: shutdown 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 >> v1:172.21.15.76:6790/0 conn(0x7ff1e4099090 legacy=0x7ff1e4137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 shutdown_connections 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 >> 172.21.15.76:0/4183709048 conn(0x7ff1e40a0030 msgr2=0x7ff1e40a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 shutdown_connections 2024-09-16T06:46:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.407+0000 7ff1e9fa8700 1 -- 172.21.15.76:0/4183709048 wait complete. 2024-09-16T06:46:56.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:46:56.417 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:46:56.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:46:56.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:46:56.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7198659700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:56.461 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7198659700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:56.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7198659700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f7178005650 con 0x7f719009bb00 2024-09-16T06:46:56.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7182ffd700 1 -- 172.21.15.189:0/3332530589 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v242) v1 ==== 120+0+0 (unknown 4004345628 0 0) 0x7f718c012f50 con 0x7f719009bb00 2024-09-16T06:46:56.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7182ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T06:46:56.463 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.457+0000 7f7182ffd700 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 >> v1:172.21.15.189:6812/3098810674 conn(0x7f71740362c0 legacy=0x7f7174038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 10 monclient: shutdown 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 >> v1:172.21.15.189:6789/0 conn(0x7f719009bb00 legacy=0x7f7190096330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 shutdown_connections 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 >> 172.21.15.189:0/3332530589 conn(0x7f71900a03e0 msgr2=0x7f719008afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 shutdown_connections 2024-09-16T06:46:56.466 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.461+0000 7f7198659700 1 -- 172.21.15.189:0/3332530589 wait complete. 2024-09-16T06:46:56.467 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h29' does not appear in the crush map 2024-09-16T06:46:56.476 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:46:56.476 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h30 2024-09-16T06:46:56.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 10 monclient: get_monmap_and_config 2024-09-16T06:46:56.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 10 monclient: monmap: 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:56.534845+0000 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:56.534845+0000 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:56.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 1 Processor -- start 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 1 -- start start 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 10 monclient: init 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.531+0000 7fc7ba40f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:56.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-b con 0x7fc7b409f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-a con 0x7fc7b409c460 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-c con 0x7fc7b409fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b409a040 con 0x7fc7b409c460 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b409ac80 con 0x7fc7b409fd20 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b409def0 con 0x7fc7b409f230 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient(hunting): authenticate will time out at 1363.668122s 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7abfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc7b409c460 0x7fc7b413ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43200/0 (socket says 172.21.15.76:43200) 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7abfff700 1 -- 172.21.15.76:0/3869959923 learned_addr learned my addr 172.21.15.76:0/3869959923 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457771400 0 0) 0x7fc7b409a040 con 0x7fc7b409c460 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): my global_id is 184599 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a00035d0 con 0x7fc7b409c460 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665088889 0 0) 0x7fc7b409def0 con 0x7fc7b409f230 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): my global_id is 194378 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b409a040 con 0x7fc7b409f230 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065968490 0 0) 0x7fc7ac001490 con 0x7fc7b409fd20 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient(hunting): my global_id is 204139 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b409def0 con 0x7fc7b409fd20 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1077410330 0 0) 0x7fc7a00035d0 con 0x7fc7b409c460 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 >> v1:172.21.15.76:6790/0 conn(0x7fc7b409f230 legacy=0x7fc7b409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 >> v1:172.21.15.189:6789/0 conn(0x7fc7b409fd20 legacy=0x7fc7b4142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 monclient: found mon.noname-a 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b409a410 con 0x7fc7b409c460 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7a4001610 con 0x7fc7b409c460 2024-09-16T06:46:56.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: dump: 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 5 monclient: authenticate success, global_id 184599 2024-09-16T06:46:56.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7a40018f0 con 0x7fc7b409c460 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 1 -- 172.21.15.76:0/3869959923 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7a4001c80 con 0x7fc7b409c460 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 4 set_mon_vals no callback set 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7b37fe700 10 monclient: dump: 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.541 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: shutdown 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- 172.21.15.76:0/3869959923 >> v1:172.21.15.76:6789/0 conn(0x7fc7b409c460 legacy=0x7fc7b413ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- 172.21.15.76:0/3869959923 shutdown_connections 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- 172.21.15.76:0/3869959923 >> 172.21.15.76:0/3869959923 conn(0x7fc7b40a3e50 msgr2=0x7fc7b40a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- 172.21.15.76:0/3869959923 shutdown_connections 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- 172.21.15.76:0/3869959923 wait complete. 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: monmap: 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 Processor -- start 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 1 -- start start 2024-09-16T06:46:56.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.535+0000 7fc7ba40f700 10 monclient: init 2024-09-16T06:46:56.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:46:56.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-c con 0x7fc7b409c460 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-b con 0x7fc7b409f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient(hunting): picked mon.noname-a con 0x7fc7b409fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b4141da0 con 0x7fc7b409fd20 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b413eea0 con 0x7fc7b409f230 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b89ac700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc7b409fd20 0x7fc7b41d6970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43216/0 (socket says 172.21.15.76:43216) 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b89ac700 1 -- 172.21.15.76:0/4175197325 learned_addr learned my addr 172.21.15.76:0/4175197325 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b41408f0 con 0x7fc7b409c460 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient(hunting): authenticate will time out at 1363.672733s 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242401771 0 0) 0x7fc7b41408f0 con 0x7fc7b409c460 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): my global_id is 194381 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7940035d0 con 0x7fc7b409c460 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176924308 0 0) 0x7fc7b4141da0 con 0x7fc7b409fd20 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): my global_id is 184602 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b41408f0 con 0x7fc7b409fd20 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007503566 0 0) 0x7fc7940035d0 con 0x7fc7b409c460 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b4141da0 con 0x7fc7b409c460 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc79c0018d0 con 0x7fc7b409c460 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:56.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): dump: 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915615492 0 0) 0x7fc7b413eea0 con 0x7fc7b409f230 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): my global_id is 204142 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7940035d0 con 0x7fc7b409f230 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723088609 0 0) 0x7fc7b41408f0 con 0x7fc7b409fd20 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b413eea0 con 0x7fc7b409fd20 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7ac001be0 con 0x7fc7b409fd20 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): dump: 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590916317 0 0) 0x7fc7b413eea0 con 0x7fc7b409fd20 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6790/0 conn(0x7fc7b409c460 legacy=0x7fc7b413e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.189:6789/0 conn(0x7fc7b409f230 legacy=0x7fc7b413e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 monclient: found mon.a 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b413fd20 con 0x7fc7b409fd20 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 5 monclient: authenticate success, global_id 184602 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _renew_subs 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4141ac0 con 0x7fc7b409fd20 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _renew_subs 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b41401f0 con 0x7fc7b409fd20 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7ac0020b0 con 0x7fc7b409fd20 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7ac003be0 con 0x7fc7b409fd20 2024-09-16T06:46:56.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b2ffd700 4 set_mon_vals no callback set 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient: dump: 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc7ac003d40 con 0x7fc7b409fd20 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7b40678b0 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7fc7ac044440 con 0x7fc7b409fd20 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7abfff700 10 monclient: get_auth_request con 0x7fc794036460 auth_method 0 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310035124 0 0) 0x7fc7b40678b0 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 10 monclient(hunting): my global_id is 184605 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.539+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b413eea0 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2848504252 0 0) 0x7fc7b413eea0 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b40678b0 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc79c003b10 con 0x7fc7b413aa10 2024-09-16T06:46:56.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 10 monclient: dump: 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3090189270 0 0) 0x7fc7b40678b0 con 0x7fc7b413aa10 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7b40a2be0 con 0x7fc7b413aa10 2024-09-16T06:46:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc79c003f10 con 0x7fc7b413aa10 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6789/0 conn(0x7fc7b413aa10 legacy=0x7fc7b413adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.543+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6789/0 conn(0x7fc7b413aa10 legacy=0x7fc7b413adf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.547+0000 7fc7ba40f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.547+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc7b4141920 con 0x7fc7b413b500 2024-09-16T06:46:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142815067 0 0) 0x7fc7b409a040 con 0x7fc7b413b500 2024-09-16T06:46:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient(hunting): my global_id is 184608 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b40678b0 con 0x7fc7b413b500 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 920210435 0 0) 0x7fc7b40678b0 con 0x7fc7b413b500 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7b409a040 con 0x7fc7b413b500 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc79c0042f0 con 0x7fc7b413b500 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient: dump: 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3951060428 0 0) 0x7fc7b409a040 con 0x7fc7b413b500 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc7b409b150 con 0x7fc7b413b500 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4211 (unknown 0 0 2584361135) 0x7fc79c005770 con 0x7fc7b413b500 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6789/0 conn(0x7fc7b413b500 legacy=0x7fc7b402c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b17fa700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6789/0 conn(0x7fc7b413b500 legacy=0x7fc7b402c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.562 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.563 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.564 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:56.565 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.566 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:56.567 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:56.568 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.571 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:46:56.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:56.573 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc794036460 legacy=0x7fc794038920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 10 monclient: shutdown 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 >> v1:172.21.15.76:6789/0 conn(0x7fc7b409fd20 legacy=0x7fc7b41d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b89ac700 1 -- 172.21.15.76:0/4175197325 reap_dead start 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7b89ac700 1 -- 172.21.15.76:0/4175197325 reap_dead start 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 shutdown_connections 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 >> 172.21.15.76:0/4175197325 conn(0x7fc7b40a3e50 msgr2=0x7fc7b4070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 shutdown_connections 2024-09-16T06:46:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.551+0000 7fc7ba40f700 1 -- 172.21.15.76:0/4175197325 wait complete. 2024-09-16T06:46:56.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:56.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:46:56.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 10 monclient: get_monmap_and_config 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 10 monclient: monmap: 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:46:56.579697+0000 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:46:56.579697+0000 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:56.583 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 1 Processor -- start 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 1 -- start start 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.573+0000 7fc06a7c5700 10 monclient: init 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-c con 0x7fc064071e00 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-a con 0x7fc06406b7c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-b con 0x7fc064138850 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06409a1f0 con 0x7fc06406b7c0 2024-09-16T06:46:56.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06409b6e0 con 0x7fc064071e00 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06409b4b0 con 0x7fc064138850 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): authenticate will time out at 1369.092328s 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc0697c3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc064071e00 0x7fc06409a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52446/0 (socket says 172.21.15.189:52446) 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc0697c3700 1 -- 172.21.15.189:0/1271788549 learned_addr learned my addr 172.21.15.189:0/1271788549 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635009974 0 0) 0x7fc06409b6e0 con 0x7fc064071e00 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): my global_id is 204145 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0500057e0 con 0x7fc064071e00 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181166005 0 0) 0x7fc06409a1f0 con 0x7fc06406b7c0 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): my global_id is 184611 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06409b6e0 con 0x7fc06406b7c0 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108719249 0 0) 0x7fc06409b4b0 con 0x7fc064138850 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient(hunting): my global_id is 194384 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06409a1f0 con 0x7fc064138850 2024-09-16T06:46:56.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2648855330 0 0) 0x7fc0500057e0 con 0x7fc064071e00 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 >> v1:172.21.15.76:6790/0 conn(0x7fc064138850 legacy=0x7fc06413ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 >> v1:172.21.15.76:6789/0 conn(0x7fc06406b7c0 legacy=0x7fc06409ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 monclient: found mon.noname-c 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064072250 con 0x7fc064071e00 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 5 monclient: authenticate success, global_id 204145 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc058001610 con 0x7fc064071e00 2024-09-16T06:46:56.587 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: dump: 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.588 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc058001880 con 0x7fc064071e00 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 1 -- 172.21.15.189:0/1271788549 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc058001c10 con 0x7fc064071e00 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 4 set_mon_vals no callback set 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc057fff700 10 monclient: dump: 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:56.589 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: shutdown 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1271788549 >> v1:172.21.15.189:6789/0 conn(0x7fc064071e00 legacy=0x7fc06409a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1271788549 shutdown_connections 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1271788549 >> 172.21.15.189:0/1271788549 conn(0x7fc06409fe70 msgr2=0x7fc0640a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1271788549 shutdown_connections 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1271788549 wait complete. 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: monmap: 2024-09-16T06:46:56.590 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 Processor -- start 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- start start 2024-09-16T06:46:56.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: init 2024-09-16T06:46:56.595 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-b con 0x7fc06406b7c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-a con 0x7fc064071e00 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): picked mon.noname-c con 0x7fc064138850 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06412c9b0 con 0x7fc064071e00 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06412bff0 con 0x7fc06406b7c0 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06412aa10 con 0x7fc064138850 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc06a7c5700 10 monclient(hunting): authenticate will time out at 1369.095822s 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc0697c3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc06406b7c0 0x7fc0641c9c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52452/0 (socket says 172.21.15.189:52452) 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.577+0000 7fc0697c3700 1 -- 172.21.15.189:0/1616248373 learned_addr learned my addr 172.21.15.189:0/1616248373 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626964595 0 0) 0x7fc06412bff0 con 0x7fc06406b7c0 2024-09-16T06:46:56.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): my global_id is 204148 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0480035d0 con 0x7fc06406b7c0 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865648425 0 0) 0x7fc06412c9b0 con 0x7fc064071e00 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): my global_id is 184614 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06412bff0 con 0x7fc064071e00 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3785914142 0 0) 0x7fc0480035d0 con 0x7fc06406b7c0 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc06412c9b0 con 0x7fc06406b7c0 2024-09-16T06:46:56.597 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc058001610 con 0x7fc06406b7c0 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): dump: 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:56.598 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806588279 0 0) 0x7fc06412aa10 con 0x7fc064138850 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient(hunting): my global_id is 194387 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0480035d0 con 0x7fc064138850 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2516161745 0 0) 0x7fc06412c9b0 con 0x7fc06406b7c0 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 >> v1:172.21.15.76:6790/0 conn(0x7fc064138850 legacy=0x7fc0641d2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 >> v1:172.21.15.76:6789/0 conn(0x7fc064071e00 legacy=0x7fc0641cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 monclient: found mon.b 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064129cb0 con 0x7fc06406b7c0 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.599 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 5 monclient: authenticate success, global_id 204148 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _renew_subs 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0640a2940 con 0x7fc06406b7c0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _renew_subs 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0640a2e80 con 0x7fc06406b7c0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0580011b0 con 0x7fc06406b7c0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc056ffd700 4 set_mon_vals no callback set 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc058004e80 con 0x7fc06406b7c0 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 10 monclient: dump: 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc058004fe0 con 0x7fc06406b7c0 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc044005100 con 0x7fc06406b7c0 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7fc058044610 con 0x7fc06406b7c0 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.581+0000 7fc068fc2700 10 monclient: get_auth_request con 0x7fc0480361a0 auth_method 0 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.589+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc058006600 con 0x7fc06406b7c0 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.589+0000 7fc055ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.589+0000 7fc055ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:56.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.691+0000 7f74cc8cb700 10 monclient: get_monmap_and_config 2024-09-16T06:46:56.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.691+0000 7f74cc8cb700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.691+0000 7f74cc8cb700 10 monclient: monmap: 2024-09-16T06:46:56.692 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:56.695601+0000 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:56.695601+0000 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:56.693 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 Processor -- start 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- start start 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient: init 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-a con 0x7f74c8097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-c con 0x7f74c8099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-b con 0x7f74c8139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c8098990 con 0x7f74c8097b60 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c8097f40 con 0x7f74c8099090 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809d610 con 0x7f74c8139500 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient(hunting): authenticate will time out at 1363.829371s 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f74c8097b60 0x7f74c809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43228/0 (socket says 172.21.15.76:43228) 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c77fe700 1 -- 172.21.15.76:0/2075795474 learned_addr learned my addr 172.21.15.76:0/2075795474 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683534037 0 0) 0x7f74c8098990 con 0x7f74c8097b60 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): my global_id is 184617 2024-09-16T06:46:56.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b40035d0 con 0x7f74c8097b60 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576909477 0 0) 0x7f74c8097f40 con 0x7f74c8099090 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): my global_id is 204151 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8098990 con 0x7f74c8099090 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558447687 0 0) 0x7f74c809d610 con 0x7f74c8139500 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient(hunting): my global_id is 194390 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8097f40 con 0x7f74c8139500 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 880040620 0 0) 0x7f74b40035d0 con 0x7f74c8097b60 2024-09-16T06:46:56.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 >> v1:172.21.15.76:6790/0 conn(0x7f74c8139500 legacy=0x7f74c813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 >> v1:172.21.15.189:6789/0 conn(0x7f74c8099090 legacy=0x7f74c809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c80986b0 con 0x7f74c8097b60 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 5 monclient: authenticate success, global_id 184617 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b0001610 con 0x7f74c8097b60 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: dump: 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b0001870 con 0x7f74c8097b60 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 1 -- 172.21.15.76:0/2075795474 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b0001c40 con 0x7f74c8097b60 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 4 set_mon_vals no callback set 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74c67fc700 10 monclient: dump: 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.702 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 10 monclient: shutdown 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2075795474 >> v1:172.21.15.76:6789/0 conn(0x7f74c8097b60 legacy=0x7f74c809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.695+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2075795474 shutdown_connections 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2075795474 >> 172.21.15.76:0/2075795474 conn(0x7f74c80a0030 msgr2=0x7f74c80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2075795474 shutdown_connections 2024-09-16T06:46:56.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2075795474 wait complete. 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient: monmap: 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:56.709 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 Processor -- start 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- start start 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient: init 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-b con 0x7f74c8097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-c con 0x7f74c8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient(hunting): picked mon.noname-a con 0x7f74c8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c808e480 con 0x7f74c8139500 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c809c210 con 0x7f74c8097b60 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80a2bf0 con 0x7f74c8099090 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74cc8cb700 10 monclient(hunting): authenticate will time out at 1363.835292s 2024-09-16T06:46:56.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74c6ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f74c8099090 0x7f74c81ca690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37144/0 (socket says 172.21.15.76:37144) 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74c7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f74c8139500 0x7f74c81cddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43230/0 (socket says 172.21.15.76:43230) 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74c6ffd700 1 -- 172.21.15.76:0/2531613150 learned_addr learned my addr 172.21.15.76:0/2531613150 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331060099 0 0) 0x7f74c80a2bf0 con 0x7f74c8099090 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): my global_id is 194393 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a80035d0 con 0x7f74c8099090 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6594218 0 0) 0x7f74c808e480 con 0x7f74c8139500 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): my global_id is 184620 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c80a2bf0 con 0x7f74c8139500 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114715378 0 0) 0x7f74c80986b0 con 0x7f74c8097b60 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 10 monclient(hunting): my global_id is 204154 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.699+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c808e480 con 0x7f74c8097b60 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030022818 0 0) 0x7f74a80035d0 con 0x7f74c8099090 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c80986b0 con 0x7f74c8099090 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b8001370 con 0x7f74c8099090 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): dump: 2024-09-16T06:46:56.712 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3153229461 0 0) 0x7f74c80a2bf0 con 0x7f74c8139500 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74a80035d0 con 0x7f74c8139500 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74c0001bb0 con 0x7f74c8139500 2024-09-16T06:46:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): dump: 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 493596547 0 0) 0x7f74c808e480 con 0x7f74c8097b60 2024-09-16T06:46:56.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c80a2bf0 con 0x7f74c8097b60 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b0001040 con 0x7f74c8097b60 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): dump: 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.715 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1169983990 0 0) 0x7f74c80986b0 con 0x7f74c8099090 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.189:6789/0 conn(0x7f74c8097b60 legacy=0x7f74c81389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6789/0 conn(0x7f74c8139500 legacy=0x7f74c81cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 monclient: found mon.c 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c809a540 con 0x7f74c8099090 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 5 monclient: authenticate success, global_id 194393 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 10 monclient: _renew_subs 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c8090290 con 0x7f74c8099090 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 10 monclient: _renew_subs 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8090850 con 0x7f74c8099090 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74c80994e0 con 0x7f74c81390c0 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74b8001590 con 0x7f74c8099090 2024-09-16T06:46:56.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b8003af0 con 0x7f74c8099090 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: dump: 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.718 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062023060 0 0) 0x7f74c80a2bf0 con 0x7f74c81390c0 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient(hunting): my global_id is 194396 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c80986b0 con 0x7f74c81390c0 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74c57fa700 4 set_mon_vals no callback set 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1125893475 0 0) 0x7f74c80986b0 con 0x7f74c81390c0 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c80a2bf0 con 0x7f74c81390c0 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b0001c50 con 0x7f74c81390c0 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: dump: 2024-09-16T06:46:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2690561846 0 0) 0x7f74c80a2bf0 con 0x7f74c81390c0 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74c808fcf0 con 0x7f74c81390c0 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f74b8003d50 con 0x7f74c8099090 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f74b8044220 con 0x7f74c8099090 2024-09-16T06:46:56.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f74b0006ab0 con 0x7f74c81390c0 2024-09-16T06:46:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6790/0 conn(0x7f74c81390c0 legacy=0x7f74c8061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6790/0 conn(0x7f74c81390c0 legacy=0x7f74c8061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.703+0000 7f74c7fff700 10 monclient: get_auth_request con 0x7f74a8036610 auth_method 0 2024-09-16T06:46:56.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74cc8cb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f74c808dd30 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693605339 0 0) 0x7f74c808dd30 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 10 monclient(hunting): my global_id is 194399 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.711+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c80a2bf0 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 823274133 0 0) 0x7f74c80a2bf0 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c808dd30 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74b0005170 con 0x7f74c808d920 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:56.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 10 monclient: dump: 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2534613548 0 0) 0x7f74c808dd30 con 0x7f74c808d920 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f74c809fb80 con 0x7f74c808d920 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4047 (unknown 0 0 4049505050) 0x7f74b00065e0 con 0x7f74c808d920 2024-09-16T06:46:56.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6790/0 conn(0x7f74c808d920 legacy=0x7f74c806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74bffff700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6790/0 conn(0x7f74c808d920 legacy=0x7f74c806eeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:56.725 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.726 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.727 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:56.728 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.729 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:56.730 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.731 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.734 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.735 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:56.736 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:56.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.189:6812/3098810674 conn(0x7f74a8036610 legacy=0x7f74a8038ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 10 monclient: shutdown 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 >> v1:172.21.15.76:6790/0 conn(0x7f74c8099090 legacy=0x7f74c81ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74c7fff700 1 -- 172.21.15.76:0/2531613150 reap_dead start 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74c7fff700 1 -- 172.21.15.76:0/2531613150 reap_dead start 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 shutdown_connections 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 >> 172.21.15.76:0/2531613150 conn(0x7f74c80a0030 msgr2=0x7f74c8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 shutdown_connections 2024-09-16T06:46:56.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.715+0000 7f74cc8cb700 1 -- 172.21.15.76:0/2531613150 wait complete. 2024-09-16T06:46:56.739 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:56.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:46:56.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:46:56.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.769+0000 7fc06a7c5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:46:56.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.769+0000 7fc06a7c5700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:46:56.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.769+0000 7fc06a7c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:46:56.772 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:56.769+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fc0440053b0 con 0x7fc06406b7c0 2024-09-16T06:46:56.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: monmap: 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:56.857067+0000 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:56.857067+0000 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:56.854 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 1 Processor -- start 2024-09-16T06:46:56.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 1 -- start start 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: init 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f52a0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f52a0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f52a0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0098990 con 0x7f52a0099090 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0097f40 con 0x7f52a0097b60 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a009d610 con 0x7f52a0139500 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient(hunting): authenticate will time out at 1363.990322s 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f52a0139500 0x7f52a013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37162/0 (socket says 172.21.15.76:37162) 2024-09-16T06:46:56.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f52967fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f52a0099090 0x7f52a009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43236/0 (socket says 172.21.15.76:43236) 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529f7fe700 1 -- 172.21.15.76:0/278154917 learned_addr learned my addr 172.21.15.76:0/278154917 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498660494 0 0) 0x7f52a009d610 con 0x7f52a0139500 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): my global_id is 194402 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c0035d0 con 0x7f52a0139500 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767691384 0 0) 0x7f52a0098990 con 0x7f52a0099090 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): my global_id is 184623 2024-09-16T06:46:56.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a009d610 con 0x7f52a0099090 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18127501 0 0) 0x7f52a0097f40 con 0x7f52a0097b60 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient(hunting): my global_id is 204157 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0098990 con 0x7f52a0097b60 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4174706060 0 0) 0x7f52a009d610 con 0x7f52a0099090 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 >> v1:172.21.15.76:6790/0 conn(0x7f52a0139500 legacy=0x7f52a013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 >> v1:172.21.15.189:6789/0 conn(0x7f52a0097b60 legacy=0x7f52a009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a00986b0 con 0x7f52a0099090 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f52880015f0 con 0x7f52a0099090 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: dump: 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.862 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 5 monclient: authenticate success, global_id 184623 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5288001910 con 0x7f52a0099090 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 1 -- 172.21.15.76:0/278154917 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5288001ca0 con 0x7f52a0099090 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529ffff700 4 set_mon_vals no callback set 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.855+0000 7f529e7fc700 10 monclient: dump: 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: shutdown 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- 172.21.15.76:0/278154917 >> v1:172.21.15.76:6789/0 conn(0x7f52a0099090 legacy=0x7f52a009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- 172.21.15.76:0/278154917 shutdown_connections 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- 172.21.15.76:0/278154917 >> 172.21.15.76:0/278154917 conn(0x7f52a00a0030 msgr2=0x7f52a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- 172.21.15.76:0/278154917 shutdown_connections 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- 172.21.15.76:0/278154917 wait complete. 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: build_initial_monmap 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: monmap: 2024-09-16T06:46:56.865 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 Processor -- start 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- start start 2024-09-16T06:46:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: init 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f52a0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f52a0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f52a0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0090d20 con 0x7f52a0139500 2024-09-16T06:46:56.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a013c150 con 0x7f52a0097b60 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a009a540 con 0x7f52a0099090 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529ffff700 10 monclient(hunting): authenticate will time out at 1363.995083s 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f52a0139500 0x7f52a01d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43240/0 (socket says 172.21.15.76:43240) 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f529f7fe700 1 -- 172.21.15.76:0/3987981367 learned_addr learned my addr 172.21.15.76:0/3987981367 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921590762 0 0) 0x7f52a0090d20 con 0x7f52a0139500 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): my global_id is 184626 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52800035d0 con 0x7f52a0139500 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589457944 0 0) 0x7f52a009a540 con 0x7f52a0099090 2024-09-16T06:46:56.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 10 monclient(hunting): my global_id is 194405 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.859+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0090d20 con 0x7f52a0099090 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374933122 0 0) 0x7f52a0098990 con 0x7f52a0097b60 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): my global_id is 204160 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a009a540 con 0x7f52a0097b60 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398645714 0 0) 0x7f52800035d0 con 0x7f52a0139500 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52a0098990 con 0x7f52a0139500 2024-09-16T06:46:56.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5298001d70 con 0x7f52a0139500 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): dump: 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127822631 0 0) 0x7f52a0090d20 con 0x7f52a0099090 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52800035d0 con 0x7f52a0099090 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f52880015f0 con 0x7f52a0099090 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): dump: 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.874 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3631763008 0 0) 0x7f52a0098990 con 0x7f52a0139500 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.76:6790/0 conn(0x7f52a0099090 legacy=0x7f52a0138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6789/0 conn(0x7f52a0097b60 legacy=0x7f52a0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: _finish_hunting 0 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 monclient: found mon.a 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0139330 con 0x7f52a0139500 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: _finish_auth 0 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: _check_auth_tickets 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:56.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 5 monclient: authenticate success, global_id 184626 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 10 monclient: _renew_subs 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a00a2c80 con 0x7f52a0139500 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 10 monclient: _renew_subs 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a008f1f0 con 0x7f52a0139500 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5298004c10 con 0x7f52a0139500 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5298004f20 con 0x7f52a0139500 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:46:56.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529dffb700 4 set_mon_vals no callback set 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: dump: 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5284006590 con 0x7f5284002830 2024-09-16T06:46:56.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f529800f990 con 0x7f52a0139500 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f52980443f0 con 0x7f52a0139500 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121126614 0 0) 0x7f5284006590 con 0x7f5284002830 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient(hunting): my global_id is 204163 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0098990 con 0x7f5284002830 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f52967fc700 10 monclient: get_auth_request con 0x7f52800363f0 auth_method 0 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3855735709 0 0) 0x7f52a0098990 con 0x7f5284002830 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5284006590 con 0x7f5284002830 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5290001620 con 0x7f5284002830 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 10 monclient: dump: 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3050879211 0 0) 0x7f5284006590 con 0x7f5284002830 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.863+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f528400a190 con 0x7f5284002830 2024-09-16T06:46:56.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.867+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5290001ba0 con 0x7f5284002830 2024-09-16T06:46:56.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.867+0000 7f5297fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:56.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.867+0000 7f5297fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:56.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.867+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6789/0 conn(0x7f5284002830 legacy=0x7f5284004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.867+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6789/0 conn(0x7f5284002830 legacy=0x7f5284004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.875+0000 7f529ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:46:56.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.875+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5284009e50 con 0x7f5284009170 2024-09-16T06:46:56.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577495694 0 0) 0x7f5284009e50 con 0x7f5284009170 2024-09-16T06:46:56.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient(hunting): _negotiate 2024-09-16T06:46:56.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient(hunting): my global_id is 204166 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5284006590 con 0x7f5284009170 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2914406129 0 0) 0x7f5284006590 con 0x7f5284009170 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5284009e50 con 0x7f5284009170 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5290001d50 con 0x7f5284009170 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient: dump: 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:56.882 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 467723072 0 0) 0x7f5284009e50 con 0x7f5284009170 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f528400c340 con 0x7f5284009170 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 4173801963) 0x7f5290004f30 con 0x7f5284009170 2024-09-16T06:46:56.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6789/0 conn(0x7f5284009170 legacy=0x7f5284009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f5297fff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6789/0 conn(0x7f5284009170 legacy=0x7f5284009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 80, 2024-09-16T06:46:56.884 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.885 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:46:56.887 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:46:56.888 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.889 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.890 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:46:56.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:46:56.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:46:56.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:46:56.895 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:46:56.895 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:46:56.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.189:6812/3098810674 conn(0x7f52800363f0 legacy=0x7f52800388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 10 monclient: shutdown 2024-09-16T06:46:56.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:56.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 >> v1:172.21.15.76:6789/0 conn(0x7f52a0139500 legacy=0x7f52a01d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:56.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529f7fe700 1 -- 172.21.15.76:0/3987981367 reap_dead start 2024-09-16T06:46:56.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529f7fe700 1 -- 172.21.15.76:0/3987981367 reap_dead start 2024-09-16T06:46:56.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 shutdown_connections 2024-09-16T06:46:56.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 >> 172.21.15.76:0/3987981367 conn(0x7f52a00a0030 msgr2=0x7f52a00714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:56.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 shutdown_connections 2024-09-16T06:46:56.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:56.879+0000 7f529ffff700 1 -- 172.21.15.76:0/3987981367 wait complete. 2024-09-16T06:46:56.897 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:46:56.897 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:46:56.897 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:46:56.898 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:46:56.898 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:46:56.898 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:46:56.898 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:46:56.924 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:46:56.925 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:46:56.925 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: monmap: 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:46:57.044189+0000 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:46:57.044189+0000 2024-09-16T06:46:57.041 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:57.042 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:57.042 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:57.042 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:46:57.042 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:46:57.042 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 Processor -- start 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- start start 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: init 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f73cc024d00 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f73cc0c2790 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f73cc0c62a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:57.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc01fe20 con 0x7f73cc024d00 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc0250e0 con 0x7f73cc0c2790 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc01f860 con 0x7f73cc0c62a0 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient(hunting): authenticate will time out at 1364.177664s 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d2ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f73cc024d00 legacy=0x7f73cc0227d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f73cc0c62a0 0x7f73cc0c8690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37206/0 (socket says 172.21.15.76:37206) 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d37fe700 1 -- 172.21.15.76:0/741701993 learned_addr learned my addr 172.21.15.76:0/741701993 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451820089 0 0) 0x7f73cc01f860 con 0x7f73cc0c62a0 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): my global_id is 194408 2024-09-16T06:46:57.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73bc0035d0 con 0x7f73cc0c62a0 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388893226 0 0) 0x7f73cc0250e0 con 0x7f73cc0c2790 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient(hunting): my global_id is 204169 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc01f860 con 0x7f73cc0c2790 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183611660 0 0) 0x7f73bc0035d0 con 0x7f73cc0c62a0 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 >> v1:172.21.15.189:6789/0 conn(0x7f73cc0c2790 legacy=0x7f73cc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 >> v1:172.21.15.76:6789/0 conn(0x7f73cc024d00 legacy=0x7f73cc0227d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 monclient: found mon.noname-b 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:57.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0232d0 con 0x7f73cc0c62a0 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 5 monclient: authenticate success, global_id 194408 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f73d4069400 con 0x7f73cc0c62a0 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: dump: 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:57.048 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d404ec20 con 0x7f73cc0c62a0 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:57.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 4 set_mon_vals no callback set 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 1 -- 172.21.15.76:0/741701993 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f73d40008d0 con 0x7f73cc0c62a0 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d1ffb700 10 monclient: dump: 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:57.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 10 monclient: shutdown 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- 172.21.15.76:0/741701993 >> v1:172.21.15.76:6790/0 conn(0x7f73cc0c62a0 legacy=0x7f73cc0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- 172.21.15.76:0/741701993 shutdown_connections 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.043+0000 7f73d3fff700 1 -- 172.21.15.76:0/741701993 >> 172.21.15.76:0/741701993 conn(0x7f73cc005b40 msgr2=0x7f73cc005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- 172.21.15.76:0/741701993 shutdown_connections 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- 172.21.15.76:0/741701993 wait complete. 2024-09-16T06:46:57.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: build_initial_monmap 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: monmap: 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:46:57.052 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 Processor -- start 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- start start 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: init 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f73cc024d00 addr v1:172.21.15.76:6790/0 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f73cc0c2790 addr v1:172.21.15.76:6789/0 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f73cc0c62a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc028a00 con 0x7f73cc0c2790 2024-09-16T06:46:57.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc0c5c30 con 0x7f73cc0c62a0 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc0ac920 con 0x7f73cc024d00 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient(hunting): authenticate will time out at 1364.181638s 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d27fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f73cc0c2790 legacy=0x7f73cc159660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f73cc024d00 0x7f73cc0c0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37216/0 (socket says 172.21.15.76:37216) 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d2ffd700 1 -- 172.21.15.76:0/1894623121 learned_addr learned my addr 172.21.15.76:0/1894623121 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531804289 0 0) 0x7f73cc0ac920 con 0x7f73cc024d00 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): my global_id is 194411 2024-09-16T06:46:57.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c00035d0 con 0x7f73cc024d00 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622418009 0 0) 0x7f73cc0c5c30 con 0x7f73cc0c62a0 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): my global_id is 204172 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc0ac920 con 0x7f73cc0c62a0 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3833624933 0 0) 0x7f73c00035d0 con 0x7f73cc024d00 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73cc0c5c30 con 0x7f73cc024d00 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f73c8004200 con 0x7f73cc024d00 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): dump: 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:57.058 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3278383737 0 0) 0x7f73cc0c5c30 con 0x7f73cc024d00 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 >> v1:172.21.15.189:6789/0 conn(0x7f73cc0c62a0 legacy=0x7f73cc15cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 >> v1:172.21.15.76:6789/0 conn(0x7f73cc0c2790 legacy=0x7f73cc159660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:46:57.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 monclient: found mon.c 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0acee0 con 0x7f73cc024d00 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 5 monclient: authenticate success, global_id 194411 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _renew_subs 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc0ac640 con 0x7f73cc024d00 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _renew_subs 2024-09-16T06:46:57.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc0a28f0 con 0x7f73cc024d00 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73c80044e0 con 0x7f73cc024d00 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f73c8004870 con 0x7f73cc024d00 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73d0ff9700 4 set_mon_vals no callback set 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.047+0000 7f73bb7fe700 10 monclient: dump: 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:46:57.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f73c8004a50 con 0x7f73cc024d00 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73d3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73d3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b4005100 con 0x7f73cc024d00 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f73c80442c0 con 0x7f73cc024d00 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.051+0000 7f73d27fc700 10 monclient: get_auth_request con 0x7f73c0036180 auth_method 0 2024-09-16T06:46:57.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.055+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f73c80148a0 con 0x7f73cc024d00 2024-09-16T06:46:57.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.055+0000 7f73bb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:46:57.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.055+0000 7f73bb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:46:57.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73d3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:46:57.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73d3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:57.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73d3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:46:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f73b4005930 con 0x7f73cc024d00 2024-09-16T06:46:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73bb7fe700 1 -- 172.21.15.76:0/1894623121 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1283940009) 0x7f73c801ce90 con 0x7f73cc024d00 2024-09-16T06:46:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73bb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:46:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.287+0000 7f73bb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:46:57.288 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:46:57.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 >> v1:172.21.15.189:6812/3098810674 conn(0x7f73c0036180 legacy=0x7f73c0038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 10 monclient: shutdown 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 >> v1:172.21.15.76:6790/0 conn(0x7f73cc024d00 legacy=0x7f73cc0c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 shutdown_connections 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 >> 172.21.15.76:0/1894623121 conn(0x7f73cc005b40 msgr2=0x7f73cc021c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 shutdown_connections 2024-09-16T06:46:57.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:46:57.291+0000 7f73d3fff700 1 -- 172.21.15.76:0/1894623121 wait complete. 2024-09-16T06:46:59.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:59.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:46:59.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:59.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:46:59.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:59.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:46:59.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:46:59.581+0000 7fc054ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:00.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:00.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 10 monclient: get_monmap_and_config 2024-09-16T06:47:00.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 10 monclient: build_initial_monmap 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 10 monclient: monmap: 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:00.426901+0000 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:00.426901+0000 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:00.424 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:00.425 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 1 Processor -- start 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 1 -- start start 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.423+0000 7f6933391700 10 monclient: init 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-c con 0x7f692c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-a con 0x7f692c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:00.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-b con 0x7f692c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c098990 con 0x7f692c099090 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c097f40 con 0x7f692c097b60 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c09d610 con 0x7f692c139500 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): authenticate will time out at 1367.560192s 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6931b8e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f692c099090 legacy=0x7f692c09f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6932b90700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f692c139500 0x7f692c13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37228/0 (socket says 172.21.15.76:37228) 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6932b90700 1 -- 172.21.15.76:0/1258247034 learned_addr learned my addr 172.21.15.76:0/1258247034 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404329251 0 0) 0x7f692c09d610 con 0x7f692c139500 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient(hunting): _negotiate 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:00.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient(hunting): my global_id is 194414 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69180035d0 con 0x7f692c139500 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2159665092 0 0) 0x7f69180035d0 con 0x7f692c139500 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 >> v1:172.21.15.189:6789/0 conn(0x7f692c097b60 legacy=0x7f692c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 >> v1:172.21.15.76:6789/0 conn(0x7f692c099090 legacy=0x7f692c09f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: _finish_hunting 0 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 monclient: found mon.noname-b 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0986b0 con 0x7f692c139500 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: _finish_auth 0 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: _check_auth_tickets 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:00.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6924001990 con 0x7f692c139500 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 5 monclient: authenticate success, global_id 194414 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: dump: 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6924001cb0 con 0x7f692c139500 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 1 -- 172.21.15.76:0/1258247034 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6924002040 con 0x7f692c139500 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 4 set_mon_vals no callback set 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693138d700 10 monclient: dump: 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:00.435 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: shutdown 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- 172.21.15.76:0/1258247034 >> v1:172.21.15.76:6790/0 conn(0x7f692c139500 legacy=0x7f692c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- 172.21.15.76:0/1258247034 shutdown_connections 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- 172.21.15.76:0/1258247034 >> 172.21.15.76:0/1258247034 conn(0x7f692c0a0030 msgr2=0x7f692c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:00.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- 172.21.15.76:0/1258247034 shutdown_connections 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- 172.21.15.76:0/1258247034 wait complete. 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: build_initial_monmap 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: monmap: 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:00.437 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 Processor -- start 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- start start 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: init 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-a con 0x7f692c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-b con 0x7f692c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f693238f700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f692c097b60 legacy=0x7f692c1379d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): picked mon.noname-c con 0x7f692c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0a2bd0 con 0x7f692c097b60 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c09a540 con 0x7f692c099090 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0711a0 con 0x7f692c139500 2024-09-16T06:47:00.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.427+0000 7f6933391700 10 monclient(hunting): authenticate will time out at 1367.563876s 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6932b90700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f692c139500 0x7f692c1d6590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37232/0 (socket says 172.21.15.76:37232) 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6932b90700 1 -- 172.21.15.76:0/2339967632 learned_addr learned my addr 172.21.15.76:0/2339967632 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995703321 0 0) 0x7f692c0711a0 con 0x7f692c139500 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): my global_id is 194417 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69100035d0 con 0x7f692c139500 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749447007 0 0) 0x7f69100035d0 con 0x7f692c139500 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f692c0711a0 con 0x7f692c139500 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f69240017b0 con 0x7f692c139500 2024-09-16T06:47:00.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): dump: 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1306760330 0 0) 0x7f692c0711a0 con 0x7f692c139500 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 >> v1:172.21.15.189:6789/0 conn(0x7f692c099090 legacy=0x7f692c1d2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 >> v1:172.21.15.76:6789/0 conn(0x7f692c097b60 legacy=0x7f692c1379d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 monclient: found mon.c 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c1293d0 con 0x7f692c139500 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 5 monclient: authenticate success, global_id 194417 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _renew_subs 2024-09-16T06:47:00.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c09c2d0 con 0x7f692c139500 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _renew_subs 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c070c70 con 0x7f692c139500 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6924005070 con 0x7f692c139500 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6924005380 con 0x7f692c139500 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6923fff700 4 set_mon_vals no callback set 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 10 monclient: dump: 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:00.442 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:00.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f692400fdf0 con 0x7f692c139500 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6914005100 con 0x7f692c139500 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f6924044930 con 0x7f692c139500 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.431+0000 7f693238f700 10 monclient: get_auth_request con 0x7f6910036270 auth_method 0 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.439+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6924014c60 con 0x7f692c139500 2024-09-16T06:47:00.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.439+0000 7f6922ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:00.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.439+0000 7f6922ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:00.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:00.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:00.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:00.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:00.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6933391700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:00.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6933391700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:00.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6933391700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:00.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6914005930 con 0x7f692c139500 2024-09-16T06:47:00.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6922ffd700 1 -- 172.21.15.76:0/2339967632 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 476811400) 0x7f692403c240 con 0x7f692c139500 2024-09-16T06:47:00.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6922ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:00.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.671+0000 7f6922ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:00.671 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6910036270 legacy=0x7f6910038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 10 monclient: shutdown 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 >> v1:172.21.15.76:6790/0 conn(0x7f692c139500 legacy=0x7f692c1d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 shutdown_connections 2024-09-16T06:47:00.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 >> 172.21.15.76:0/2339967632 conn(0x7f692c0a0030 msgr2=0x7f692c138720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:00.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 shutdown_connections 2024-09-16T06:47:00.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:00.675+0000 7f6933391700 1 -- 172.21.15.76:0/2339967632 wait complete. 2024-09-16T06:47:01.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:01.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:01.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:01.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:01.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:01.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:02.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:02.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:02.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:02.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:02.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:02.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:03.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:03.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:03.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:03.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:03.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:03.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:03.685 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.806+0000 7ff42204a700 10 monclient: get_monmap_and_config 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.806+0000 7ff42204a700 10 monclient: build_initial_monmap 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.806+0000 7ff42204a700 10 monclient: monmap: 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:03.811347+0000 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:03.811347+0000 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:03.808 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:03.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:03.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:03.809 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:03.809 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 1 Processor -- start 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 1 -- start start 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient: init 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-a con 0x7ff41c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-c con 0x7ff41c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:03.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-b con 0x7ff41c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c098990 con 0x7ff41c097b60 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c097f40 con 0x7ff41c099090 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c09d610 con 0x7ff41c139500 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff42204a700 10 monclient(hunting): authenticate will time out at 1370.944648s 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff421048700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff41c09ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff421849700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff41c139500 0x7ff41c13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:37248/0 (socket says 172.21.15.76:37248) 2024-09-16T06:47:03.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:03.810+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 learned_addr learned my addr 172.21.15.76:0/1944475706 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:04.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:04.014+0000 7ff421048700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff41c09ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:04.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:04.418+0000 7ff421048700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff41c09ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:04.584 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:04.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:04.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:04.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:04.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:04.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:05.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:05.222+0000 7ff421048700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff41c09ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:05.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:05.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:05.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:05.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:05.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:05.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:06.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:06.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:06.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:06.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:06.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:06.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:06.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): tick 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6790/0 conn(0x7ff41c139500 legacy=0x7ff41c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.189:6789/0 conn(0x7ff41c099090 legacy=0x7ff41c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff41c09ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-c con 0x7ff400003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-b con 0x7ff41c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-a con 0x7ff41c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:06.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c0986b0 con 0x7ff41c097b60 2024-09-16T06:47:06.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c098990 con 0x7ff400003740 2024-09-16T06:47:06.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff400010420 con 0x7ff41c139500 2024-09-16T06:47:06.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff412ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:06.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:06.810+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:07.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:07.014+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:07.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:07.418+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:07.585+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:07.585+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:07.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:07.585+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:08.222+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:08.585+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:08.585+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:08.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:08.585+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:09.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:09.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:09.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:09.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:09.581+0000 7fc054ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): tick 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6790/0 conn(0x7ff41c139500 legacy=0x7ff400009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.189:6789/0 conn(0x7ff400003740 legacy=0x7ff400005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:09.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-c con 0x7ff4000106b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-b con 0x7ff400006340 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): picked mon.noname-a con 0x7ff41c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff400011630 con 0x7ff41c097b60 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c0986b0 con 0x7ff4000106b0 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff400006720 con 0x7ff400006340 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff412ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:09.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:09.810+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:10.014+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:10.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:10.418+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:10.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:10.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:10.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:10.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:10.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:10.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.222+0000 7ff421849700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164473562 0 0) 0x7ff41c0986b0 con 0x7ff4000106b0 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient(hunting): my global_id is 204175 2024-09-16T06:47:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080035d0 con 0x7ff4000106b0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419168762 0 0) 0x7ff4080035d0 con 0x7ff4000106b0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6790/0 conn(0x7ff400006340 legacy=0x7ff400005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.76:6789/0 conn(0x7ff41c097b60 legacy=0x7ff40000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 monclient: found mon.noname-c 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000093d0 con 0x7ff4000106b0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff42204a700 5 monclient: authenticate success, global_id 204175 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c004250 con 0x7ff4000106b0 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: dump: 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff42204a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff40c004470 con 0x7ff4000106b0 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 1 -- 172.21.15.76:0/1944475706 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c004800 con 0x7ff4000106b0 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff42204a700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff42204a700 4 set_mon_vals no callback set 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.386+0000 7ff413fff700 10 monclient: dump: 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.392 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 10 monclient: shutdown 2024-09-16T06:47:11.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- 172.21.15.76:0/1944475706 >> v1:172.21.15.189:6789/0 conn(0x7ff4000106b0 legacy=0x7ff400010a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- 172.21.15.76:0/1944475706 shutdown_connections 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- 172.21.15.76:0/1944475706 >> 172.21.15.76:0/1944475706 conn(0x7ff41c0a0030 msgr2=0x7ff41c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- 172.21.15.76:0/1944475706 shutdown_connections 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- 172.21.15.76:0/1944475706 wait complete. 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 10 monclient: build_initial_monmap 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 10 monclient: monmap: 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:11.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 Processor -- start 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 1 -- start start 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.390+0000 7ff42204a700 10 monclient: init 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-b con 0x7ff400006340 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-a con 0x7ff4000106b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient(hunting): picked mon.noname-c con 0x7ff41c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c094a10 con 0x7ff4000106b0 2024-09-16T06:47:11.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c071920 con 0x7ff400006340 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c094fd0 con 0x7ff41c097b60 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient(hunting): authenticate will time out at 1378.528764s 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff420847700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff4000106b0 legacy=0x7ff41c09af60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff421849700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff41c097b60 0x7ff41c1d2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34794/0 (socket says 172.21.15.76:34794) 2024-09-16T06:47:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff421849700 1 -- 172.21.15.76:0/93002051 learned_addr learned my addr 172.21.15.76:0/93002051 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974925389 0 0) 0x7ff41c071920 con 0x7ff400006340 2024-09-16T06:47:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): my global_id is 204178 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff400011630 con 0x7ff400006340 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047369348 0 0) 0x7ff400011630 con 0x7ff400006340 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff41c071920 con 0x7ff400006340 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c0045e0 con 0x7ff400006340 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 850822428 0 0) 0x7ff41c071920 con 0x7ff400006340 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 >> v1:172.21.15.76:6790/0 conn(0x7ff41c097b60 legacy=0x7ff41c1d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 >> v1:172.21.15.76:6789/0 conn(0x7ff4000106b0 legacy=0x7ff41c09af60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 monclient: found mon.b 2024-09-16T06:47:11.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c0952b0 con 0x7ff400006340 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 5 monclient: authenticate success, global_id 204178 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _renew_subs 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c09b670 con 0x7ff400006340 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _renew_subs 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c0a2b90 con 0x7ff400006340 2024-09-16T06:47:11.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff40c004a70 con 0x7ff400006340 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff40c004e00 con 0x7ff400006340 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff4137fe700 4 set_mon_vals no callback set 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.394+0000 7ff411ffb700 10 monclient: dump: 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.404 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.405 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.405 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff40c004f60 con 0x7ff400006340 2024-09-16T06:47:11.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff42204a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff42204a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff42204a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc005100 con 0x7ff400006340 2024-09-16T06:47:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7ff40c0442c0 con 0x7ff400006340 2024-09-16T06:47:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.398+0000 7ff420847700 10 monclient: get_auth_request con 0x7ff400037740 auth_method 0 2024-09-16T06:47:11.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.406+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff40c014820 con 0x7ff400006340 2024-09-16T06:47:11.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.406+0000 7ff411ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.406+0000 7ff411ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:11.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:11.581+0000 7fc054ff9700 10 monclient: tick 2024-09-16T06:47:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:11.581+0000 7fc054ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:11.581+0000 7fc054ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff42204a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff42204a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff42204a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff3fc005930 con 0x7ff400006340 2024-09-16T06:47:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff411ffb700 1 -- 172.21.15.76:0/93002051 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 2521241401) 0x7ff40c014410 con 0x7ff400006340 2024-09-16T06:47:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff411ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff411ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:11.650 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:11.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.650+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff400037740 legacy=0x7ff400039c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 10 monclient: shutdown 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 >> v1:172.21.15.189:6789/0 conn(0x7ff400006340 legacy=0x7ff41c09a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 shutdown_connections 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 >> 172.21.15.76:0/93002051 conn(0x7ff41c0a0030 msgr2=0x7ff41c06f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 shutdown_connections 2024-09-16T06:47:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.654+0000 7ff42204a700 1 -- 172.21.15.76:0/93002051 wait complete. 2024-09-16T06:47:11.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:47:11.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:47:11.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:47:11.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.770+0000 7f6229ea5700 10 monclient: get_monmap_and_config 2024-09-16T06:47:11.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.770+0000 7f6229ea5700 10 monclient: build_initial_monmap 2024-09-16T06:47:11.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.770+0000 7f6229ea5700 10 monclient: monmap: 2024-09-16T06:47:11.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:11.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:11.774762+0000 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:11.774762+0000 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:11.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 Processor -- start 2024-09-16T06:47:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- start start 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient: init 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-b con 0x7f6224132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-c con 0x7f622406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-a con 0x7f622406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622406c280 con 0x7f622406cf50 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622406bfa0 con 0x7f622406d480 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622409ace0 con 0x7f6224132330 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient(hunting): authenticate will time out at 1378.908963s 2024-09-16T06:47:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62296a4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f622406cf50 legacy=0x7f622413da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6228ea3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6224132330 0x7f6224132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34800/0 (socket says 172.21.15.76:34800) 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6228ea3700 1 -- 172.21.15.76:0/3443087017 learned_addr learned my addr 172.21.15.76:0/3443087017 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679028670 0 0) 0x7f622409ace0 con 0x7f6224132330 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): my global_id is 194420 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62100035d0 con 0x7f6224132330 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724117778 0 0) 0x7f622406bfa0 con 0x7f622406d480 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient(hunting): my global_id is 204181 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622409ace0 con 0x7f622406d480 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323106421 0 0) 0x7f62100035d0 con 0x7f6224132330 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 >> v1:172.21.15.189:6789/0 conn(0x7f622406d480 legacy=0x7f622413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 >> v1:172.21.15.76:6789/0 conn(0x7f622406cf50 legacy=0x7f622413da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 monclient: found mon.noname-b 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622409ae70 con 0x7f6224132330 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 5 monclient: authenticate success, global_id 194420 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f620c001610 con 0x7f6224132330 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: dump: 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f620c0018f0 con 0x7f6224132330 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 1 -- 172.21.15.76:0/3443087017 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f620c001ae0 con 0x7f6224132330 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 4 set_mon_vals no callback set 2024-09-16T06:47:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f62237fe700 10 monclient: dump: 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 10 monclient: shutdown 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- 172.21.15.76:0/3443087017 >> v1:172.21.15.76:6790/0 conn(0x7f6224132330 legacy=0x7f6224132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- 172.21.15.76:0/3443087017 shutdown_connections 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- 172.21.15.76:0/3443087017 >> 172.21.15.76:0/3443087017 conn(0x7f62240a41c0 msgr2=0x7f62240a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- 172.21.15.76:0/3443087017 shutdown_connections 2024-09-16T06:47:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.774+0000 7f6229ea5700 1 -- 172.21.15.76:0/3443087017 wait complete. 2024-09-16T06:47:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: build_initial_monmap 2024-09-16T06:47:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: monmap: 2024-09-16T06:47:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 Processor -- start 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- start start 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: init 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-c con 0x7f622406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-b con 0x7f622406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient(hunting): picked mon.noname-a con 0x7f6224132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224137cf0 con 0x7f6224132330 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224139260 con 0x7f622406d480 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224136120 con 0x7f622406cf50 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient(hunting): authenticate will time out at 1378.913374s 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62296a4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6224132330 legacy=0x7f62241d2450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6228ea3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f622406cf50 0x7f6224135300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34810/0 (socket says 172.21.15.76:34810) 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6228ea3700 1 -- 172.21.15.76:0/3824621715 learned_addr learned my addr 172.21.15.76:0/3824621715 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633403179 0 0) 0x7f6224136120 con 0x7f622406cf50 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): my global_id is 194423 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62040035d0 con 0x7f622406cf50 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183535195 0 0) 0x7f62040035d0 con 0x7f622406cf50 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6224136120 con 0x7f622406cf50 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f620c004890 con 0x7f622406cf50 2024-09-16T06:47:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): dump: 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499870820 0 0) 0x7f6224139260 con 0x7f622406d480 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): my global_id is 204184 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62040035d0 con 0x7f622406d480 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578676812 0 0) 0x7f6224136120 con 0x7f622406cf50 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.189:6789/0 conn(0x7f622406d480 legacy=0x7f6224135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6789/0 conn(0x7f6224132330 legacy=0x7f62241d2450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 monclient: found mon.c 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62241316e0 con 0x7f622406cf50 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 5 monclient: authenticate success, global_id 194423 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _renew_subs 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622409e8b0 con 0x7f622406cf50 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _renew_subs 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6224136f00 con 0x7f622406cf50 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f620c004d90 con 0x7f622406cf50 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f620c005120 con 0x7f622406cf50 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6222ffd700 4 set_mon_vals no callback set 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient: dump: 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6208006590 con 0x7f6208002830 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428587825 0 0) 0x7f6208006590 con 0x7f6208002830 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 10 monclient(hunting): my global_id is 194426 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6224136120 con 0x7f6208002830 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.778+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f620c005300 con 0x7f622406cf50 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f620c044630 con 0x7f622406cf50 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 101985863 0 0) 0x7f6224136120 con 0x7f6208002830 2024-09-16T06:47:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6208006590 con 0x7f6208002830 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62140016c0 con 0x7f6208002830 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 10 monclient: dump: 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62296a4700 10 monclient: get_auth_request con 0x7f6204036210 auth_method 0 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 576518070 0 0) 0x7f6208006590 con 0x7f6208002830 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f620800a190 con 0x7f6208002830 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6214001a10 con 0x7f6208002830 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6790/0 conn(0x7f6208002830 legacy=0x7f6208004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.782+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6790/0 conn(0x7f6208002830 legacy=0x7f6208004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6208009e50 con 0x7f6208009170 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470421672 0 0) 0x7f6208009e50 con 0x7f6208009170 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient(hunting): my global_id is 194429 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6208006590 con 0x7f6208009170 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3037927264 0 0) 0x7f6208006590 con 0x7f6208009170 2024-09-16T06:47:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6208009e50 con 0x7f6208009170 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6214003840 con 0x7f6208009170 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient: dump: 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 286678425 0 0) 0x7f6208009e50 con 0x7f6208009170 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f620800c340 con 0x7f6208009170 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 3471166406) 0x7f6214005ac0 con 0x7f6208009170 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6790/0 conn(0x7f6208009170 legacy=0x7f6208009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62217fa700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6790/0 conn(0x7f6208009170 legacy=0x7f6208009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 84, 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:47:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:11.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.812 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:11.812 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6204036210 legacy=0x7f62040386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 10 monclient: shutdown 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 >> v1:172.21.15.76:6790/0 conn(0x7f622406cf50 legacy=0x7f6224135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f62296a4700 1 -- 172.21.15.76:0/3824621715 reap_dead start 2024-09-16T06:47:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 shutdown_connections 2024-09-16T06:47:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 >> 172.21.15.76:0/3824621715 conn(0x7f62240a41c0 msgr2=0x7f62240a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 shutdown_connections 2024-09-16T06:47:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.794+0000 7f6229ea5700 1 -- 172.21.15.76:0/3824621715 wait complete. 2024-09-16T06:47:11.814 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:47:11.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:47:11.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 10 monclient: get_monmap_and_config 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 10 monclient: build_initial_monmap 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 10 monclient: monmap: 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:11.931087+0000 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:11.931087+0000 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 1 Processor -- start 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 1 -- start start 2024-09-16T06:47:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.926+0000 7f518cd59700 10 monclient: init 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-a con 0x7f51880964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-b con 0x7f518809b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-c con 0x7f5188137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188098560 con 0x7f51880964f0 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188099ec0 con 0x7f5188137e70 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518809b5f0 con 0x7f518809b1c0 2024-09-16T06:47:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient(hunting): authenticate will time out at 1379.064342s 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f51877fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f51880964f0 legacy=0x7f51880968d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f517ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f518809b1c0 0x7f5188136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34826/0 (socket says 172.21.15.76:34826) 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f517ffff700 1 -- 172.21.15.76:0/2770730124 learned_addr learned my addr 172.21.15.76:0/2770730124 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299244400 0 0) 0x7f518809b5f0 con 0x7f518809b1c0 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): my global_id is 194432 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51740035d0 con 0x7f518809b1c0 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307727473 0 0) 0x7f5188099ec0 con 0x7f5188137e70 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient(hunting): my global_id is 204187 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518809b5f0 con 0x7f5188137e70 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549401227 0 0) 0x7f518809b5f0 con 0x7f5188137e70 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 >> v1:172.21.15.76:6790/0 conn(0x7f518809b1c0 legacy=0x7f5188136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 >> v1:172.21.15.76:6789/0 conn(0x7f51880964f0 legacy=0x7f51880968d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 monclient: found mon.noname-c 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188099be0 con 0x7f5188137e70 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 5 monclient: authenticate success, global_id 204187 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5180001970 con 0x7f5188137e70 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: dump: 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.938 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5180001c90 con 0x7f5188137e70 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 4 set_mon_vals no callback set 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 1 -- 172.21.15.76:0/2770730124 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5180002060 con 0x7f5188137e70 2024-09-16T06:47:11.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f5186ffd700 10 monclient: dump: 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.940 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: shutdown 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- 172.21.15.76:0/2770730124 >> v1:172.21.15.189:6789/0 conn(0x7f5188137e70 legacy=0x7f518813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- 172.21.15.76:0/2770730124 shutdown_connections 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- 172.21.15.76:0/2770730124 >> 172.21.15.76:0/2770730124 conn(0x7f51880a0030 msgr2=0x7f51880a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- 172.21.15.76:0/2770730124 shutdown_connections 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- 172.21.15.76:0/2770730124 wait complete. 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: build_initial_monmap 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: monmap: 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:11.941 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 Processor -- start 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 1 -- start start 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.930+0000 7f518cd59700 10 monclient: init 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-c con 0x7f51880964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:11.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-a con 0x7f518809b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient(hunting): picked mon.noname-b con 0x7f5188137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188092d30 con 0x7f518809b1c0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188090390 con 0x7f5188137e70 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188092260 con 0x7f51880964f0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient(hunting): authenticate will time out at 1379.068470s 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f517ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f518809b1c0 legacy=0x7f518808fc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f51877fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f51880964f0 0x7f518808f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:34836/0 (socket says 172.21.15.76:34836) 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f51877fe700 1 -- 172.21.15.76:0/1559984819 learned_addr learned my addr 172.21.15.76:0/1559984819 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12382827 0 0) 0x7f5188092260 con 0x7f51880964f0 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): my global_id is 194435 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51680035d0 con 0x7f51880964f0 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945055639 0 0) 0x7f51680035d0 con 0x7f51880964f0 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5188092260 con 0x7f51880964f0 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5178004200 con 0x7f51880964f0 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): dump: 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202757918 0 0) 0x7f5188090390 con 0x7f5188137e70 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient(hunting): my global_id is 204190 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51680035d0 con 0x7f5188137e70 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3261398247 0 0) 0x7f5188092260 con 0x7f51880964f0 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6789/0 conn(0x7f5188137e70 legacy=0x7f51881d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.76:6789/0 conn(0x7f518809b1c0 legacy=0x7f518808fc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 monclient: found mon.c 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188092a50 con 0x7f51880964f0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: _finish_auth 0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 5 monclient: authenticate success, global_id 194435 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _renew_subs 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5188090f40 con 0x7f51880964f0 2024-09-16T06:47:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _renew_subs 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5188091cd0 con 0x7f51880964f0 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5178004520 con 0x7f51880964f0 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f51780048b0 con 0x7f51880964f0 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f51867fc700 4 set_mon_vals no callback set 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 10 monclient: dump: 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5178004a90 con 0x7f51880964f0 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f516c006590 con 0x7f516c002830 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(242..242 src has 151..242) v4 ==== 7922+0+0 (unknown 2175047066 0 0) 0x7f5178044360 con 0x7f51880964f0 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.934+0000 7f5187fff700 10 monclient: get_auth_request con 0x7f5168036220 auth_method 0 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894928644 0 0) 0x7f516c006590 con 0x7f516c002830 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient(hunting): my global_id is 204193 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5188092260 con 0x7f516c002830 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4000135113 0 0) 0x7f5188092260 con 0x7f516c002830 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f516c006590 con 0x7f516c002830 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5170001850 con 0x7f516c002830 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient: dump: 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1370933785 0 0) 0x7f516c006590 con 0x7f516c002830 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f516c00a190 con 0x7f516c002830 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5170001a80 con 0x7f516c002830 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6789/0 conn(0x7f516c002830 legacy=0x7f516c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.938+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6789/0 conn(0x7f516c002830 legacy=0x7f516c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f516c009e30 con 0x7f516c009150 2024-09-16T06:47:11.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838523421 0 0) 0x7f516c009e30 con 0x7f516c009150 2024-09-16T06:47:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient(hunting): my global_id is 204196 2024-09-16T06:47:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f516c006590 con 0x7f516c009150 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2585866489 0 0) 0x7f516c006590 con 0x7f516c009150 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f516c009e30 con 0x7f516c009150 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5170004af0 con 0x7f516c009150 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient: dump: 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:11.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3346188809 0 0) 0x7f516c009e30 con 0x7f516c009150 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f516c00c340 con 0x7f516c009150 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 907640639) 0x7f5170004d60 con 0x7f516c009150 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6789/0 conn(0x7f516c009150 legacy=0x7f516c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5184ff9700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6789/0 conn(0x7f516c009150 legacy=0x7f516c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 84, 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:11.956 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.957 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.958 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:11.959 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.962 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:11.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.965 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:11.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:11.967 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:11.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5168036220 legacy=0x7f51680386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 10 monclient: shutdown 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 >> v1:172.21.15.76:6790/0 conn(0x7f51880964f0 legacy=0x7f518808f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5187fff700 1 -- 172.21.15.76:0/1559984819 reap_dead start 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f5187fff700 1 -- 172.21.15.76:0/1559984819 reap_dead start 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 shutdown_connections 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 >> 172.21.15.76:0/1559984819 conn(0x7f51880a0030 msgr2=0x7f518809e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 shutdown_connections 2024-09-16T06:47:11.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:11.950+0000 7f518cd59700 1 -- 172.21.15.76:0/1559984819 wait complete. 2024-09-16T06:47:11.969 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:47:11.970 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:47:12.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.393+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v243) v1 ==== 121+0+0 (unknown 1329427737 0 0) 0x7fc058012f80 con 0x7fc06406b7c0 2024-09-16T06:47:12.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.393+0000 7fc055ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:47:12.397 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.393+0000 7fc055ffb700 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-16T06:47:12.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc06a7c5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:47:12.505 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc06a7c5700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:47:12.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc06a7c5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.506 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fc044005650 con 0x7fc06406b7c0 2024-09-16T06:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc055ffb700 1 -- 172.21.15.189:0/1616248373 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v243) v1 ==== 120+0+0 (unknown 262190404 0 0) 0x7fc058014700 con 0x7fc06406b7c0 2024-09-16T06:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc055ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T06:47:12.507 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.501+0000 7fc055ffb700 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-16T06:47:12.509 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc0480361a0 legacy=0x7fc048038660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 10 monclient: shutdown 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 >> v1:172.21.15.189:6789/0 conn(0x7fc06406b7c0 legacy=0x7fc0641c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 shutdown_connections 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 >> 172.21.15.189:0/1616248373 conn(0x7fc06409fe70 msgr2=0x7fc06406dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 shutdown_connections 2024-09-16T06:47:12.510 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.505+0000 7fc06a7c5700 1 -- 172.21.15.189:0/1616248373 wait complete. 2024-09-16T06:47:12.511 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h30' does not appear in the crush map 2024-09-16T06:47:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:47:12.521 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h31 2024-09-16T06:47:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: get_monmap_and_config 2024-09-16T06:47:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: build_initial_monmap 2024-09-16T06:47:12.626 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: monmap: 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:12.626736+0000 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:12.626736+0000 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:12.627 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 1 Processor -- start 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 1 -- start start 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: init 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-b con 0x7fca0809eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-a con 0x7fca0809beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-c con 0x7fca0809c2e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:12.631 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca08099ed0 con 0x7fca0809beb0 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0809d2e0 con 0x7fca0809c2e0 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0809ef80 con 0x7fca0809eba0 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.621+0000 7fca0d971700 10 monclient(hunting): authenticate will time out at 1385.139904s 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca07fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fca0809beb0 legacy=0x7fca08137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d170700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fca0809c2e0 0x7fca0813ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49504/0 (socket says 172.21.15.189:49504) 2024-09-16T06:47:12.632 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d170700 1 -- 172.21.15.189:0/2393714196 learned_addr learned my addr 172.21.15.189:0/2393714196 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048489083 0 0) 0x7fca0809d2e0 con 0x7fca0809c2e0 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): my global_id is 204199 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f40057e0 con 0x7fca0809c2e0 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077592934 0 0) 0x7fca0809ef80 con 0x7fca0809eba0 2024-09-16T06:47:12.633 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient(hunting): my global_id is 194438 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0809d2e0 con 0x7fca0809eba0 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 195787299 0 0) 0x7fc9f40057e0 con 0x7fca0809c2e0 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 >> v1:172.21.15.76:6790/0 conn(0x7fca0809eba0 legacy=0x7fca0809b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 >> v1:172.21.15.76:6789/0 conn(0x7fca0809beb0 legacy=0x7fca08137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 monclient: found mon.noname-c 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.634 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0809cc70 con 0x7fca0809c2e0 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 5 monclient: authenticate success, global_id 204199 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc9f8001990 con 0x7fca0809c2e0 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:12.635 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: dump: 2024-09-16T06:47:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.636 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9f8001bb0 con 0x7fca0809c2e0 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:12.637 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 1 -- 172.21.15.189:0/2393714196 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc9f8001ec0 con 0x7fca0809c2e0 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 4 set_mon_vals no callback set 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca077fe700 10 monclient: dump: 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 10 monclient: shutdown 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- 172.21.15.189:0/2393714196 >> v1:172.21.15.189:6789/0 conn(0x7fca0809c2e0 legacy=0x7fca0813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- 172.21.15.189:0/2393714196 shutdown_connections 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- 172.21.15.189:0/2393714196 >> 172.21.15.189:0/2393714196 conn(0x7fca0809fe70 msgr2=0x7fca080a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- 172.21.15.189:0/2393714196 shutdown_connections 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- 172.21.15.189:0/2393714196 wait complete. 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 10 monclient: build_initial_monmap 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 10 monclient: monmap: 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:12.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:12.640 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:12.640 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:12.640 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 Processor -- start 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 1 -- start start 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.625+0000 7fca0d971700 10 monclient: init 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-b con 0x7fca0809beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-c con 0x7fca0809c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient(hunting): picked mon.noname-a con 0x7fca0809eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca080905d0 con 0x7fca0809eba0 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0806f8c0 con 0x7fca0809beb0 2024-09-16T06:47:12.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca080a1470 con 0x7fca0809c2e0 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d170700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fca0809eba0 legacy=0x7fca081d22b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0c96f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fca0809beb0 0x7fca080967c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49510/0 (socket says 172.21.15.189:49510) 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0c96f700 1 -- 172.21.15.189:0/3817738422 learned_addr learned my addr 172.21.15.189:0/3817738422 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient(hunting): authenticate will time out at 1385.144759s 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154954308 0 0) 0x7fca0806f8c0 con 0x7fca0809beb0 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): my global_id is 204202 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e80035d0 con 0x7fca0809beb0 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043119460 0 0) 0x7fca080a1470 con 0x7fca0809c2e0 2024-09-16T06:47:12.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): my global_id is 194441 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0806f8c0 con 0x7fca0809c2e0 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3514823482 0 0) 0x7fc9e80035d0 con 0x7fca0809beb0 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca080a1470 con 0x7fca0809beb0 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca000019b0 con 0x7fca0809beb0 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): dump: 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:12.646 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3898676437 0 0) 0x7fca080a1470 con 0x7fca0809beb0 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 >> v1:172.21.15.76:6790/0 conn(0x7fca0809c2e0 legacy=0x7fca081ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 >> v1:172.21.15.76:6789/0 conn(0x7fca0809eba0 legacy=0x7fca081d22b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:12.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 monclient: found mon.b 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca080a1090 con 0x7fca0809beb0 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 5 monclient: authenticate success, global_id 204202 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _renew_subs 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca08071150 con 0x7fca0809beb0 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _renew_subs 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0806fa90 con 0x7fca0809beb0 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca00001c50 con 0x7fca0809beb0 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca00003b10 con 0x7fca0809beb0 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca06ffd700 4 set_mon_vals no callback set 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 10 monclient: dump: 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:12.649 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fca00003cf0 con 0x7fca0809beb0 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(243..243 src has 151..243) v4 ==== 7709+0+0 (unknown 2165717199 0 0) 0x7fca00044290 con 0x7fca0809beb0 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca07fff700 10 monclient: get_auth_request con 0x7fc9e8036220 auth_method 0 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:12.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.629+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca080677c0 con 0x7fca0809beb0 2024-09-16T06:47:12.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.637+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fca00014810 con 0x7fca0809beb0 2024-09-16T06:47:12.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.637+0000 7fca057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:12.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.637+0000 7fca057fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.813+0000 7fca0d971700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.813+0000 7fca0d971700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.813+0000 7fca0d971700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:12.818 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:12.813+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fca0806f340 con 0x7fca0809beb0 2024-09-16T06:47:13.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.401+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v244) v1 ==== 121+0+0 (unknown 1208934722 0 0) 0x7fca00012c80 con 0x7fca0809beb0 2024-09-16T06:47:13.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.401+0000 7fca057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:13.407 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.401+0000 7fca057fa700 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-16T06:47:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.513+0000 7fca0d971700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.513+0000 7fca0d971700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.513+0000 7fca0d971700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.520 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.513+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fca08096ed0 con 0x7fca0809beb0 2024-09-16T06:47:13.521 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.517+0000 7fca057fa700 1 -- 172.21.15.189:0/3817738422 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v244) v1 ==== 120+0+0 (unknown 2500058759 0 0) 0x7fca00014400 con 0x7fca0809beb0 2024-09-16T06:47:13.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.517+0000 7fca057fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T06:47:13.522 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.517+0000 7fca057fa700 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.517+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc9e8036220 legacy=0x7fc9e80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 10 monclient: shutdown 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 >> v1:172.21.15.189:6789/0 conn(0x7fca0809beb0 legacy=0x7fca080967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 shutdown_connections 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 >> 172.21.15.189:0/3817738422 conn(0x7fca0809fe70 msgr2=0x7fca0808f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:13.524 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 shutdown_connections 2024-09-16T06:47:13.525 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.521+0000 7fca0d971700 1 -- 172.21.15.189:0/3817738422 wait complete. 2024-09-16T06:47:13.525 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h31' does not appear in the crush map 2024-09-16T06:47:13.534 INFO:tasks.workunit.client.0.smithi189.stderr:+ for f in `seq 1 32` 2024-09-16T06:47:13.534 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm h32 2024-09-16T06:47:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: get_monmap_and_config 2024-09-16T06:47:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: build_initial_monmap 2024-09-16T06:47:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: monmap: 2024-09-16T06:47:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:13.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:13.638587+0000 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:13.638587+0000 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:13.639 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 1 Processor -- start 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 1 -- start start 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: init 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-a con 0x7ff3cc1321f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff3cc06bac0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff3cc13bc00 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc06a340 con 0x7ff3cc1321f0 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc06bf00 con 0x7ff3cc06bac0 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0a3930 con 0x7ff3cc13bc00 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3cbfff700 10 monclient(hunting): authenticate will time out at 1386.151759s 2024-09-16T06:47:13.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3ca7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3cc06bac0 0x7ff3cc13a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49524/0 (socket says 172.21.15.189:49524) 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3ca7fc700 1 -- 172.21.15.189:0/3975876396 learned_addr learned my addr 172.21.15.189:0/3975876396 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.633+0000 7ff3caffd700 1 -- 172.21.15.189:0/3975876396 >> v1:172.21.15.76:6789/0 conn(0x7ff3cc1321f0 legacy=0x7ff3cc1325d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175317045 0 0) 0x7ff3cc06bf00 con 0x7ff3cc06bac0 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): my global_id is 204205 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b40035d0 con 0x7ff3cc06bac0 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171592289 0 0) 0x7ff3cc0a3930 con 0x7ff3cc13bc00 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:13.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient(hunting): my global_id is 194444 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3cc06bf00 con 0x7ff3cc13bc00 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659897040 0 0) 0x7ff3b40035d0 con 0x7ff3cc06bac0 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 >> v1:172.21.15.76:6790/0 conn(0x7ff3cc13bc00 legacy=0x7ff3cc13dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 >> v1:172.21.15.76:6789/0 conn(0x7ff3cc1321f0 legacy=0x7ff3cc1325d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 monclient: found mon.noname-c 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc06a820 con 0x7ff3cc06bac0 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:13.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3bc001610 con 0x7ff3cc06bac0 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: dump: 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:13.650 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 5 monclient: authenticate success, global_id 204205 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3bc0018f0 con 0x7ff3cc06bac0 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 4 set_mon_vals no callback set 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 1 -- 172.21.15.189:0/3975876396 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3bc001c80 con 0x7ff3cc06bac0 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3c9ffb700 10 monclient: dump: 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.651 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: shutdown 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- 172.21.15.189:0/3975876396 >> v1:172.21.15.189:6789/0 conn(0x7ff3cc06bac0 legacy=0x7ff3cc13a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- 172.21.15.189:0/3975876396 shutdown_connections 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- 172.21.15.189:0/3975876396 >> 172.21.15.189:0/3975876396 conn(0x7ff3cc0a4280 msgr2=0x7ff3cc0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- 172.21.15.189:0/3975876396 shutdown_connections 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- 172.21.15.189:0/3975876396 wait complete. 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: build_initial_monmap 2024-09-16T06:47:13.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: monmap: 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 Processor -- start 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- start start 2024-09-16T06:47:13.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: init 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-a con 0x7ff3cc06bac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff3cc1321f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff3cc13bc00 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc1389e0 con 0x7ff3cc06bac0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc131850 con 0x7ff3cc13bc00 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc131320 con 0x7ff3cc1321f0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cbfff700 10 monclient(hunting): authenticate will time out at 1386.155414s 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3caffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff3cc06bac0 legacy=0x7ff3cc12ff80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cb7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff3cc13bc00 0x7ff3cc1d29e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49530/0 (socket says 172.21.15.189:49530) 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3cb7fe700 1 -- 172.21.15.189:0/887641761 learned_addr learned my addr 172.21.15.189:0/887641761 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:13.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528831113 0 0) 0x7ff3cc131850 con 0x7ff3cc13bc00 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 10 monclient(hunting): my global_id is 204208 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.637+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b80035d0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686589436 0 0) 0x7ff3cc131320 con 0x7ff3cc1321f0 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): my global_id is 194447 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3cc131850 con 0x7ff3cc1321f0 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404694222 0 0) 0x7ff3b80035d0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3cc131320 con 0x7ff3cc13bc00 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3c4001c30 con 0x7ff3cc13bc00 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): dump: 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:13.656 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 800880197 0 0) 0x7ff3cc131320 con 0x7ff3cc13bc00 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 >> v1:172.21.15.76:6790/0 conn(0x7ff3cc1321f0 legacy=0x7ff3cc1cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 >> v1:172.21.15.76:6789/0 conn(0x7ff3cc06bac0 legacy=0x7ff3cc12ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 monclient: found mon.b 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc138700 con 0x7ff3cc13bc00 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: _finish_auth 0 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 5 monclient: authenticate success, global_id 204208 2024-09-16T06:47:13.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _renew_subs 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc1368f0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _renew_subs 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc135d50 con 0x7ff3cc13bc00 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3c4003a40 con 0x7ff3cc13bc00 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff3c4003dd0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3c97fa700 4 set_mon_vals no callback set 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 10 monclient: dump: 2024-09-16T06:47:13.658 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff3c4003fb0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac005100 con 0x7ff3cc13bc00 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(244..244 src has 151..244) v4 ==== 7496+0+0 (unknown 42527312 0 0) 0x7ff3c4044580 con 0x7ff3cc13bc00 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.641+0000 7ff3caffd700 10 monclient: get_auth_request con 0x7ff3b8036180 auth_method 0 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.649+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff3c4014bf0 con 0x7ff3cc13bc00 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.649+0000 7ff3b3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:13.660 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.649+0000 7ff3b3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.825+0000 7ff3cbfff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.825+0000 7ff3cbfff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.825+0000 7ff3cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:13.829 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:13.825+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7ff3ac0053b0 con 0x7ff3cc13bc00 2024-09-16T06:47:14.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.457+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v245) v1 ==== 121+0+0 (unknown 339943673 0 0) 0x7ff3c401d1e0 con 0x7ff3cc13bc00 2024-09-16T06:47:14.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.457+0000 7ff3b3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:14.462 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.457+0000 7ff3b3fff700 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-16T06:47:14.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.573+0000 7ff3cbfff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:14.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.573+0000 7ff3cbfff700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:14.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.573+0000 7ff3cbfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.573+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7ff3ac005650 con 0x7ff3cc13bc00 2024-09-16T06:47:14.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3b3fff700 1 -- 172.21.15.189:0/887641761 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v245) v1 ==== 120+0+0 (unknown 1456541859 0 0) 0x7ff3c4013060 con 0x7ff3cc13bc00 2024-09-16T06:47:14.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3b3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T06:47:14.580 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3b3fff700 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-16T06:47:14.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff3b8036180 legacy=0x7ff3b8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:14.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 10 monclient: shutdown 2024-09-16T06:47:14.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:14.582 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 >> v1:172.21.15.189:6789/0 conn(0x7ff3cc13bc00 legacy=0x7ff3cc1d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:14.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 shutdown_connections 2024-09-16T06:47:14.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 >> 172.21.15.189:0/887641761 conn(0x7ff3cc0a4280 msgr2=0x7ff3cc0a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:14.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 shutdown_connections 2024-09-16T06:47:14.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.577+0000 7ff3cbfff700 1 -- 172.21.15.189:0/887641761 wait complete. 2024-09-16T06:47:14.584 INFO:tasks.workunit.client.0.smithi189.stderr:device 'h32' does not appear in the crush map 2024-09-16T06:47:14.593 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush rm r1 2024-09-16T06:47:14.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: get_monmap_and_config 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: build_initial_monmap 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: monmap: 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:14.701131+0000 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:14.701131+0000 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:14.701 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 1 Processor -- start 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 1 -- start start 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: init 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-a con 0x7fc5dc09bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-b con 0x7fc5dc09c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-c con 0x7fc5dc138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:14.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc09cca0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc09f750 con 0x7fc5dc138da0 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc097620 con 0x7fc5dc09c5f0 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient(hunting): authenticate will time out at 1387.214543s 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5db7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5dc138da0 0x7fc5dc13b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49532/0 (socket says 172.21.15.189:49532) 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5db7fe700 1 -- 172.21.15.189:0/2414130611 learned_addr learned my addr 172.21.15.189:0/2414130611 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820863849 0 0) 0x7fc5dc09f750 con 0x7fc5dc138da0 2024-09-16T06:47:14.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient(hunting): my global_id is 204211 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c40035d0 con 0x7fc5dc138da0 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3440732058 0 0) 0x7fc5c40035d0 con 0x7fc5dc138da0 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 >> v1:172.21.15.76:6790/0 conn(0x7fc5dc09c5f0 legacy=0x7fc5dc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 >> v1:172.21.15.76:6789/0 conn(0x7fc5dc09bb00 legacy=0x7fc5dc09bee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 monclient: found mon.noname-c 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.712 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc097900 con 0x7fc5dc138da0 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5d4001990 con 0x7fc5dc138da0 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 5 monclient: authenticate success, global_id 204211 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: dump: 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.713 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5d4001c80 con 0x7fc5dc138da0 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 4 set_mon_vals no callback set 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 1 -- 172.21.15.189:0/2414130611 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5d4002090 con 0x7fc5dc138da0 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:14.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5d9ffb700 10 monclient: dump: 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.697+0000 7fc5e13ac700 10 monclient: shutdown 2024-09-16T06:47:14.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/2414130611 >> v1:172.21.15.189:6789/0 conn(0x7fc5dc138da0 legacy=0x7fc5dc13b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/2414130611 shutdown_connections 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/2414130611 >> 172.21.15.189:0/2414130611 conn(0x7fc5dc0a03e0 msgr2=0x7fc5dc0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/2414130611 shutdown_connections 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/2414130611 wait complete. 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: build_initial_monmap 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: monmap: 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:14.716 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 Processor -- start 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- start start 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: init 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-b con 0x7fc5dc09bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:14.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-a con 0x7fc5dc09c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient(hunting): picked mon.noname-c con 0x7fc5dc138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc0a2d10 con 0x7fc5dc09c5f0 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc096e30 con 0x7fc5dc09bb00 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc094f90 con 0x7fc5dc138da0 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient(hunting): authenticate will time out at 1387.217588s 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5da7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc5dc09c5f0 legacy=0x7fc5dc1cf270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5daffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5dc09bb00 0x7fc5dc1ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49542/0 (socket says 172.21.15.189:49542) 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5daffd700 1 -- 172.21.15.189:0/523026502 learned_addr learned my addr 172.21.15.189:0/523026502 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721603507 0 0) 0x7fc5dc096e30 con 0x7fc5dc09bb00 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:14.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): my global_id is 204214 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c80035d0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86808424 0 0) 0x7fc5dc094f90 con 0x7fc5dc138da0 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): my global_id is 194453 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5dc096e30 con 0x7fc5dc138da0 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556330612 0 0) 0x7fc5c80035d0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5dc094f90 con 0x7fc5dc09bb00 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc0017f0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:14.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): dump: 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409173882 0 0) 0x7fc5dc094f90 con 0x7fc5dc09bb00 2024-09-16T06:47:14.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 >> v1:172.21.15.76:6790/0 conn(0x7fc5dc138da0 legacy=0x7fc5dc1d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 >> v1:172.21.15.76:6789/0 conn(0x7fc5dc09c5f0 legacy=0x7fc5dc1cf270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 monclient: found mon.b 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc09a6e0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 5 monclient: authenticate success, global_id 204214 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _renew_subs 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc099490 con 0x7fc5dc09bb00 2024-09-16T06:47:14.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _renew_subs 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc06d700 con 0x7fc5dc09bb00 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5cc001d80 con 0x7fc5dc09bb00 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc5cc004b50 con 0x7fc5dc09bb00 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5d97fa700 4 set_mon_vals no callback set 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5c37fe700 10 monclient: dump: 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:14.722 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.701+0000 7fc5e13ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5e13ac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5e13ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc005100 con 0x7fc5dc09bb00 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc5cc00f5c0 con 0x7fc5dc09bb00 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(245..245 src has 151..245) v4 ==== 7283+0+0 (unknown 2278850787 0 0) 0x7fc5cc043f20 con 0x7fc5dc09bb00 2024-09-16T06:47:14.723 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.705+0000 7fc5da7fc700 10 monclient: get_auth_request con 0x7fc5c8035ff0 auth_method 0 2024-09-16T06:47:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.713+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc5cc010f90 con 0x7fc5dc09bb00 2024-09-16T06:47:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.713+0000 7fc5c37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:14.724 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.713+0000 7fc5c37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:14.887 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.881+0000 7fc5e13ac700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:14.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.881+0000 7fc5e13ac700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:14.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.881+0000 7fc5e13ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:14.888 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:14.881+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7fc5bc0053b0 con 0x7fc5dc09bb00 2024-09-16T06:47:15.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.465+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v246) v1 ==== 118+0+0 (unknown 1523667376 0 0) 0x7fc5cc012aa0 con 0x7fc5dc09bb00 2024-09-16T06:47:15.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.465+0000 7fc5c37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:15.470 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.465+0000 7fc5c37fe700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-16T06:47:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5e13ac700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5e13ac700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5e13ac700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.581 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7fc5bc005650 con 0x7fc5dc09bb00 2024-09-16T06:47:15.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5c37fe700 1 -- 172.21.15.189:0/523026502 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v246) v1 ==== 118+0+0 (unknown 430339250 0 0) 0x7fc5cc014220 con 0x7fc5dc09bb00 2024-09-16T06:47:15.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5c37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T06:47:15.583 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.577+0000 7fc5c37fe700 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-16T06:47:15.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc5c8035ff0 legacy=0x7fc5c80384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:15.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 10 monclient: shutdown 2024-09-16T06:47:15.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:15.585 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 >> v1:172.21.15.189:6789/0 conn(0x7fc5dc09bb00 legacy=0x7fc5dc1ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:15.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 shutdown_connections 2024-09-16T06:47:15.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 >> 172.21.15.189:0/523026502 conn(0x7fc5dc0a03e0 msgr2=0x7fc5dc09a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:15.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 shutdown_connections 2024-09-16T06:47:15.586 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.581+0000 7fc5e13ac700 1 -- 172.21.15.189:0/523026502 wait complete. 2024-09-16T06:47:15.587 INFO:tasks.workunit.client.0.smithi189.stderr:device 'r1' does not appear in the crush map 2024-09-16T06:47:15.596 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T06:47:15.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: get_monmap_and_config 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: build_initial_monmap 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: monmap: 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:15.701116+0000 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:15.701116+0000 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:15.701 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 Processor -- start 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 -- start start 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: init 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:15.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-b con 0x7fedd009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-a con 0x7fedd009c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-c con 0x7fedd0138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd009cca0 con 0x7fedd009c5f0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd009f750 con 0x7fedd0138da0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd0097620 con 0x7fedd009bb00 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient(hunting): authenticate will time out at 1388.214412s 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fedd0138da0 0x7fedd013b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49558/0 (socket says 172.21.15.189:49558) 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcf7fe700 1 -- 172.21.15.189:0/1447211877 learned_addr learned my addr 172.21.15.189:0/1447211877 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedce7fc700 1 -- 172.21.15.189:0/1447211877 >> v1:172.21.15.76:6789/0 conn(0x7fedd009c5f0 legacy=0x7fedd0137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:15.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027084432 0 0) 0x7fedd009f750 con 0x7fedd0138da0 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient(hunting): my global_id is 204217 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb80057e0 con 0x7fedd0138da0 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708902068 0 0) 0x7fedb80057e0 con 0x7fedd0138da0 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 >> v1:172.21.15.76:6790/0 conn(0x7fedd009bb00 legacy=0x7fedd009bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 >> v1:172.21.15.76:6789/0 conn(0x7fedd009c5f0 legacy=0x7fedd0137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:15.706 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 monclient: found mon.noname-c 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd0097900 con 0x7fedd0138da0 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fedc8001990 con 0x7fedd0138da0 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 5 monclient: authenticate success, global_id 204217 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:15.707 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: dump: 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:15.708 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc8001cb0 con 0x7fedd0138da0 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 1 -- 172.21.15.189:0/1447211877 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fedc8002080 con 0x7fedd0138da0 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 4 set_mon_vals no callback set 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:15.709 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedcdffb700 10 monclient: dump: 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 10 monclient: shutdown 2024-09-16T06:47:15.710 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.697+0000 7fedd529a700 1 -- 172.21.15.189:0/1447211877 >> v1:172.21.15.189:6789/0 conn(0x7fedd0138da0 legacy=0x7fedd013b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- 172.21.15.189:0/1447211877 shutdown_connections 2024-09-16T06:47:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- 172.21.15.189:0/1447211877 >> 172.21.15.189:0/1447211877 conn(0x7fedd00a03e0 msgr2=0x7fedd00a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- 172.21.15.189:0/1447211877 shutdown_connections 2024-09-16T06:47:15.711 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- 172.21.15.189:0/1447211877 wait complete. 2024-09-16T06:47:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: build_initial_monmap 2024-09-16T06:47:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: monmap: 2024-09-16T06:47:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:15.714 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 Processor -- start 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- start start 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: init 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:15.715 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-c con 0x7fedd009bb00 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-a con 0x7fedd009c5f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient(hunting): picked mon.noname-b con 0x7fedd0138da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd012d3c0 con 0x7fedd009c5f0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd012d630 con 0x7fedd0138da0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd00993f0 con 0x7fedd009bb00 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient(hunting): authenticate will time out at 1388.218574s 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedce7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fedd009c5f0 legacy=0x7fedd01cf020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedcf7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fedd0138da0 0x7fedd01d2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49574/0 (socket says 172.21.15.189:49574) 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedcf7fe700 1 -- 172.21.15.189:0/3971681833 learned_addr learned my addr 172.21.15.189:0/3971681833 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279909982 0 0) 0x7fedd012d630 con 0x7fedd0138da0 2024-09-16T06:47:15.716 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): my global_id is 204220 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedbc0035d0 con 0x7fedd0138da0 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183900392 0 0) 0x7fedd00993f0 con 0x7fedd009bb00 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): my global_id is 194459 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd012d630 con 0x7fedd009bb00 2024-09-16T06:47:15.717 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123752778 0 0) 0x7fedbc0035d0 con 0x7fedd0138da0 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedd00993f0 con 0x7fedd0138da0 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fedc8001950 con 0x7fedd0138da0 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): dump: 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:15.718 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3213152958 0 0) 0x7fedd00993f0 con 0x7fedd0138da0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 >> v1:172.21.15.76:6790/0 conn(0x7fedd009bb00 legacy=0x7fedd0096440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 >> v1:172.21.15.76:6789/0 conn(0x7fedd009c5f0 legacy=0x7fedd01cf020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 monclient: found mon.b 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd012b3e0 con 0x7fedd0138da0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:15.719 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedb77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 5 monclient: authenticate success, global_id 204220 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: _renew_subs 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.701+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd0129a20 con 0x7fedd0138da0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 10 monclient: _renew_subs 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd012e140 con 0x7fedd0138da0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc8005020 con 0x7fedd0138da0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedcd7fa700 4 set_mon_vals no callback set 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fedc80053b0 con 0x7fedd0138da0 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:15.720 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 10 monclient: dump: 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fedc8005510 con 0x7fedd0138da0 2024-09-16T06:47:15.721 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb0005100 con 0x7fedd0138da0 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(246..246 src has 151..246) v4 ==== 7073+0+0 (unknown 2628532321 0 0) 0x7fedc8044540 con 0x7fedd0138da0 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.705+0000 7fedceffd700 10 monclient: get_auth_request con 0x7fedbc035fa0 auth_method 0 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.713+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fedc8014e50 con 0x7fedd0138da0 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.713+0000 7fedb77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:15.722 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.713+0000 7fedb77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.893+0000 7fedd529a700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:15.897 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.893+0000 7fedd529a700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.893+0000 7fedd529a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:15.898 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.893+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fedb00053b0 con 0x7fedd0138da0 2024-09-16T06:47:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.973+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v247) v1 ==== 78+0+0 (unknown 1043858077 0 0) 0x7fedc80132c0 con 0x7fedd0138da0 2024-09-16T06:47:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.973+0000 7fedb77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:15.977 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:15.973+0000 7fedb77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.085+0000 7fedd529a700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.085+0000 7fedd529a700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.085+0000 7fedd529a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:16.090 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.085+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fedb0005650 con 0x7fedd0138da0 2024-09-16T06:47:16.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.981+0000 7fedb77fe700 1 -- 172.21.15.189:0/3971681833 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v248) v1 ==== 78+0+0 (unknown 4215291485 0 0) 0x7fedc8014a40 con 0x7fedd0138da0 2024-09-16T06:47:16.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.981+0000 7fedb77fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T06:47:16.987 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.981+0000 7fedb77fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 >> v1:172.21.15.189:6812/3098810674 conn(0x7fedbc035fa0 legacy=0x7fedbc038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 10 monclient: shutdown 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 >> v1:172.21.15.189:6789/0 conn(0x7fedd0138da0 legacy=0x7fedd01d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 shutdown_connections 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 >> 172.21.15.189:0/3971681833 conn(0x7fedd00a03e0 msgr2=0x7fedd009a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.985+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 shutdown_connections 2024-09-16T06:47:16.992 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:16.989+0000 7fedd529a700 1 -- 172.21.15.189:0/3971681833 wait complete. 2024-09-16T06:47:17.006 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T06:47:17.007 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T06:47:17.007 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:47:17.007 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:47:17.063 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-16T06:47:17.065 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-16T06:47:17.065 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-16T06:47:17.075 INFO:tasks.workunit.client.0.smithi189.stderr:+ set -e 2024-09-16T06:47:17.075 INFO:tasks.workunit.client.0.smithi189.stderr:+ uuidgen 2024-09-16T06:47:17.076 INFO:tasks.workunit.client.0.smithi189.stderr:+ ua=12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0 2024-09-16T06:47:17.077 INFO:tasks.workunit.client.0.smithi189.stderr:+ uuidgen 2024-09-16T06:47:17.077 INFO:tasks.workunit.client.0.smithi189.stderr:+ ub=36f68e64-4177-4ce7-bf0b-c78e9b549946 2024-09-16T06:47:17.078 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0 2024-09-16T06:47:17.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: get_monmap_and_config 2024-09-16T06:47:17.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: build_initial_monmap 2024-09-16T06:47:17.190 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: monmap: 2024-09-16T06:47:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:17.190393+0000 2024-09-16T06:47:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:17.190393+0000 2024-09-16T06:47:17.191 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:17.192 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:17.193 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:17.193 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 1 Processor -- start 2024-09-16T06:47:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 1 -- start start 2024-09-16T06:47:17.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: init 2024-09-16T06:47:17.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:17.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:17.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-b con 0x7f7d0c09ce10 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:17.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-a con 0x7f7d0c09d1f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:17.210 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-c con 0x7f7d0c071ee0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c097fb0 con 0x7f7d0c09d1f0 2024-09-16T06:47:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c09d670 con 0x7f7d0c071ee0 2024-09-16T06:47:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c099690 con 0x7f7d0c09ce10 2024-09-16T06:47:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:17.211 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d10902700 10 monclient(hunting): authenticate will time out at 1389.703388s 2024-09-16T06:47:17.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d0b59e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7d0c071ee0 0x7f7d0c1423c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49578/0 (socket says 172.21.15.189:49578) 2024-09-16T06:47:17.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d0b59e700 1 -- 172.21.15.189:0/3784009819 learned_addr learned my addr 172.21.15.189:0/3784009819 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:17.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.185+0000 7f7d0a59c700 1 -- 172.21.15.189:0/3784009819 >> v1:172.21.15.76:6789/0 conn(0x7f7d0c09d1f0 legacy=0x7f7d0c0978a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:17.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662237793 0 0) 0x7f7d0c09d670 con 0x7f7d0c071ee0 2024-09-16T06:47:17.212 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): my global_id is 204223 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf40035d0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966416746 0 0) 0x7f7d0c099690 con 0x7f7d0c09ce10 2024-09-16T06:47:17.213 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:17.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:17.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:17.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient(hunting): my global_id is 194462 2024-09-16T06:47:17.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c09d670 con 0x7f7d0c09ce10 2024-09-16T06:47:17.214 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099487769 0 0) 0x7f7cf40035d0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 >> v1:172.21.15.76:6790/0 conn(0x7f7d0c09ce10 legacy=0x7f7d0c097190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:17.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 >> v1:172.21.15.76:6789/0 conn(0x7f7d0c09d1f0 legacy=0x7f7d0c0978a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:17.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:47:17.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 monclient: found mon.noname-c 2024-09-16T06:47:17.215 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:17.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c098660 con 0x7f7d0c071ee0 2024-09-16T06:47:17.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: _finish_auth 0 2024-09-16T06:47:17.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:47:17.216 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:17.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 5 monclient: authenticate success, global_id 204223 2024-09-16T06:47:17.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d04001990 con 0x7f7d0c071ee0 2024-09-16T06:47:17.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:17.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:17.217 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: dump: 2024-09-16T06:47:17.218 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:17.218 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.218 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.218 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.218 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.219 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d04001c00 con 0x7f7d0c071ee0 2024-09-16T06:47:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 1 -- 172.21.15.189:0/3784009819 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d04001f90 con 0x7f7d0c071ee0 2024-09-16T06:47:17.220 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 4 set_mon_vals no callback set 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d09d9b700 10 monclient: dump: 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.221 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:17.222 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 10 monclient: shutdown 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 1 -- 172.21.15.189:0/3784009819 >> v1:172.21.15.189:6789/0 conn(0x7f7d0c071ee0 legacy=0x7f7d0c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 1 -- 172.21.15.189:0/3784009819 shutdown_connections 2024-09-16T06:47:17.223 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 1 -- 172.21.15.189:0/3784009819 >> 172.21.15.189:0/3784009819 conn(0x7f7d0c09fe70 msgr2=0x7f7d0c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 1 -- 172.21.15.189:0/3784009819 shutdown_connections 2024-09-16T06:47:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 1 -- 172.21.15.189:0/3784009819 wait complete. 2024-09-16T06:47:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 10 monclient: build_initial_monmap 2024-09-16T06:47:17.224 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.189+0000 7f7d10902700 10 monclient: monmap: 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:17.225 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:17.226 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.226 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 Processor -- start 2024-09-16T06:47:17.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- start start 2024-09-16T06:47:17.227 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: init 2024-09-16T06:47:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-c con 0x7f7d0c071ee0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-b con 0x7f7d0c09ce10 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:17.228 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient(hunting): picked mon.noname-a con 0x7f7d0c09d1f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c1af3f0 con 0x7f7d0c09d1f0 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c0715e0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c1ad3d0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient(hunting): authenticate will time out at 1389.709945s 2024-09-16T06:47:17.229 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d0a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7d0c09ce10 0x7f7d0c1cdd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49588/0 (socket says 172.21.15.189:49588) 2024-09-16T06:47:17.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d0b59e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7d0c09d1f0 legacy=0x7f7d0c1d60a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:17.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d0a59c700 1 -- 172.21.15.189:0/350052339 learned_addr learned my addr 172.21.15.189:0/350052339 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:17.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873026800 0 0) 0x7f7d0c0715e0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:17.230 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:17.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:17.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): my global_id is 204226 2024-09-16T06:47:17.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf80035d0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090550896 0 0) 0x7f7d0c1ad3d0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.231 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:17.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:17.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:17.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): my global_id is 194465 2024-09-16T06:47:17.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c0715e0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.232 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3524194869 0 0) 0x7f7cf80035d0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d0c1ad3d0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7cfc004230 con 0x7f7d0c09ce10 2024-09-16T06:47:17.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:17.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:17.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): dump: 2024-09-16T06:47:17.234 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:17.234 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.234 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.234 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.234 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:17.235 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.235 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:17.235 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:17.235 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:17.235 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991045778 0 0) 0x7f7d0c0715e0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf80035d0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d000018b0 con 0x7f7d0c071ee0 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:17.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient(hunting): dump: 2024-09-16T06:47:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.237 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.238 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:17.238 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.238 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:17.238 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:17.238 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:17.239 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605561730 0 0) 0x7f7d0c1ad3d0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 >> v1:172.21.15.76:6790/0 conn(0x7f7d0c071ee0 legacy=0x7f7d0c1c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:17.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 >> v1:172.21.15.76:6789/0 conn(0x7f7d0c09d1f0 legacy=0x7f7d0c1d60a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:17.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 monclient: found mon.b 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c1c8460 con 0x7f7d0c09ce10 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:17.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:17.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:17.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 5 monclient: authenticate success, global_id 204226 2024-09-16T06:47:17.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _renew_subs 2024-09-16T06:47:17.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c1ae7f0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _renew_subs 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c1af110 con 0x7f7d0c09ce10 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cfc004450 con 0x7f7d0c09ce10 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:17.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7cfc0047e0 con 0x7f7d0c09ce10 2024-09-16T06:47:17.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:17.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:17.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7cf37fe700 10 monclient: dump: 2024-09-16T06:47:17.243 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:17.243 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:17.244 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.244 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:17.244 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:17.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.193+0000 7f7d0959a700 4 set_mon_vals no callback set 2024-09-16T06:47:17.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7cfc004940 con 0x7f7d0c09ce10 2024-09-16T06:47:17.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(248..248 src has 151..248) v4 ==== 6745+0+0 (unknown 3814427339 0 0) 0x7f7cfc043d60 con 0x7f7d0c09ce10 2024-09-16T06:47:17.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7d0ad9d700 10 monclient: get_auth_request con 0x7f7cf80362f0 auth_method 0 2024-09-16T06:47:17.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7d10902700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:17.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7d10902700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:17.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7d10902700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.197+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec005100 con 0x7f7d0c09ce10 2024-09-16T06:47:17.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.205+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7cfc005110 con 0x7f7d0c09ce10 2024-09-16T06:47:17.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.205+0000 7f7cf37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:17.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.205+0000 7f7cf37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:17.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.389+0000 7f7d10902700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:17.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.389+0000 7f7d10902700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:17.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.389+0000 7f7d10902700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:17.396 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.389+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f7cec005930 con 0x7f7d0c09ce10 2024-09-16T06:47:17.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.993+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v249) v1 ==== 106+0+3 (unknown 1885847503 0 944643765) 0x7f7cfc012b70 con 0x7f7d0c09ce10 2024-09-16T06:47:17.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.993+0000 7f7cf37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:17.997 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:17.993+0000 7f7cf37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.109+0000 7f7d10902700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.109+0000 7f7d10902700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.109+0000 7f7d10902700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.109+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f7cec0053b0 con 0x7f7d0c09ce10 2024-09-16T06:47:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7cf37fe700 1 -- 172.21.15.189:0/350052339 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v249) v1 ==== 106+0+3 (unknown 1885847503 0 944643765) 0x7f7cfc0142f0 con 0x7f7d0c09ce10 2024-09-16T06:47:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7cf37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7cf37fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7cf80362f0 legacy=0x7f7cf80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 10 monclient: shutdown 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 >> v1:172.21.15.189:6789/0 conn(0x7f7d0c09ce10 legacy=0x7f7d0c1cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 shutdown_connections 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 >> 172.21.15.189:0/350052339 conn(0x7f7d0c09fe70 msgr2=0x7f7d0c0a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 shutdown_connections 2024-09-16T06:47:18.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.113+0000 7f7d10902700 1 -- 172.21.15.189:0/350052339 wait complete. 2024-09-16T06:47:18.130 INFO:tasks.workunit.client.0.smithi189.stderr:+ na=12 2024-09-16T06:47:18.130 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0 2024-09-16T06:47:18.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 10 monclient: get_monmap_and_config 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 10 monclient: build_initial_monmap 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 10 monclient: monmap: 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:18.238285+0000 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:18.238285+0000 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:18.238 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 1 Processor -- start 2024-09-16T06:47:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 1 -- start start 2024-09-16T06:47:18.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.233+0000 7f5bd0568700 10 monclient: init 2024-09-16T06:47:18.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc806a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-c con 0x7f5bc80a1190 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-a con 0x7f5bc80a1990 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc809d5e0 con 0x7f5bc80a1990 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc806b8b0 con 0x7f5bc80a1190 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc81411e0 con 0x7f5bc806a340 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient(hunting): authenticate will time out at 1390.752642s 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bceb05700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5bc80a1990 legacy=0x7f5bc813daa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcdb03700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5bc80a1190 0x7f5bc806ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49596/0 (socket says 172.21.15.189:49596) 2024-09-16T06:47:18.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcdb03700 1 -- 172.21.15.189:0/1459933328 learned_addr learned my addr 172.21.15.189:0/1459933328 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315865394 0 0) 0x7f5bc806b8b0 con 0x7f5bc80a1190 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): my global_id is 204229 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40035d0 con 0x7f5bc80a1190 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175721620 0 0) 0x7f5bc81411e0 con 0x7f5bc806a340 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient(hunting): my global_id is 194468 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc806b8b0 con 0x7f5bc806a340 2024-09-16T06:47:18.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2195126403 0 0) 0x7f5bb40035d0 con 0x7f5bc80a1190 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 >> v1:172.21.15.76:6790/0 conn(0x7f5bc806a340 legacy=0x7f5bc806a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 >> v1:172.21.15.76:6789/0 conn(0x7f5bc80a1990 legacy=0x7f5bc813daa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: _finish_hunting 0 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 monclient: found mon.noname-c 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8142380 con 0x7f5bc80a1190 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: _finish_auth 0 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: _check_auth_tickets 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bb8001610 con 0x7f5bc80a1190 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 5 monclient: authenticate success, global_id 204229 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:18.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: dump: 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bb80018f0 con 0x7f5bc80a1190 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 4 set_mon_vals no callback set 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 1 -- 172.21.15.189:0/1459933328 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bb8001c80 con 0x7f5bc80a1190 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bcd302700 10 monclient: dump: 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.247 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.248 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.248 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 10 monclient: shutdown 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- 172.21.15.189:0/1459933328 >> v1:172.21.15.189:6789/0 conn(0x7f5bc80a1190 legacy=0x7f5bc806ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- 172.21.15.189:0/1459933328 shutdown_connections 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- 172.21.15.189:0/1459933328 >> 172.21.15.189:0/1459933328 conn(0x7f5bc80a21f0 msgr2=0x7f5bc80a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- 172.21.15.189:0/1459933328 shutdown_connections 2024-09-16T06:47:18.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.237+0000 7f5bd0568700 1 -- 172.21.15.189:0/1459933328 wait complete. 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient: build_initial_monmap 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient: monmap: 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 1 Processor -- start 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 1 -- start start 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient: init 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-c con 0x7f5bc806a340 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-a con 0x7f5bc80a1190 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc80a1990 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc8092bd0 con 0x7f5bc80a1190 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc8092eb0 con 0x7f5bc80a1990 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc8093190 con 0x7f5bc806a340 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bd0568700 10 monclient(hunting): authenticate will time out at 1390.759135s 2024-09-16T06:47:18.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bcdb03700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5bc80a1190 legacy=0x7f5bc8090180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bceb05700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5bc80a1990 0x7f5bc81d22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49606/0 (socket says 172.21.15.189:49606) 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bceb05700 1 -- 172.21.15.189:0/4147009556 learned_addr learned my addr 172.21.15.189:0/4147009556 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325092731 0 0) 0x7f5bc8092eb0 con 0x7f5bc80a1990 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): my global_id is 204232 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bac0035d0 con 0x7f5bc80a1990 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575940680 0 0) 0x7f5bc8093190 con 0x7f5bc806a340 2024-09-16T06:47:18.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 10 monclient(hunting): my global_id is 194471 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.241+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc8092eb0 con 0x7f5bc806a340 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674580644 0 0) 0x7f5bac0035d0 con 0x7f5bc80a1990 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc8093190 con 0x7f5bc80a1990 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bc00044f0 con 0x7f5bc80a1990 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): dump: 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.256 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 23997371 0 0) 0x7f5bc8092eb0 con 0x7f5bc806a340 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bac0035d0 con 0x7f5bc806a340 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bc40018b0 con 0x7f5bc806a340 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:18.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient(hunting): dump: 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3707090688 0 0) 0x7f5bc8093190 con 0x7f5bc80a1990 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 >> v1:172.21.15.76:6790/0 conn(0x7f5bc806a340 legacy=0x7f5bc808fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 >> v1:172.21.15.76:6789/0 conn(0x7f5bc80a1190 legacy=0x7f5bc8090180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:18.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 monclient: found mon.b 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc809dbb0 con 0x7f5bc80a1990 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 5 monclient: authenticate success, global_id 204232 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _renew_subs 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc80926d0 con 0x7f5bc80a1990 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _renew_subs 2024-09-16T06:47:18.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc8091300 con 0x7f5bc80a1990 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc0004760 con 0x7f5bc80a1990 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5bc0004af0 con 0x7f5bc80a1990 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bccb01700 4 set_mon_vals no callback set 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 10 monclient: dump: 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.260 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb0005100 con 0x7f5bc80a1990 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5bc0004d10 con 0x7f5bc80a1990 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(249..249 src has 151..249) v4 ==== 6745+0+0 (unknown 2786296018 0 0) 0x7f5bc00441d0 con 0x7f5bc80a1990 2024-09-16T06:47:18.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.245+0000 7f5bce304700 10 monclient: get_auth_request con 0x7f5bac0362a0 auth_method 0 2024-09-16T06:47:18.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.253+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5bc0014b10 con 0x7f5bc80a1990 2024-09-16T06:47:18.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.253+0000 7f5bbeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.253+0000 7f5bbeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:18.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bd0568700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bd0568700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bd0568700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.446 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f5bb0005930 con 0x7f5bc80a1990 2024-09-16T06:47:18.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v249) v1 ==== 106+0+3 (unknown 1885847503 0 944643765) 0x7f5bc0012f80 con 0x7f5bc80a1990 2024-09-16T06:47:18.447 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bbeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.448 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.441+0000 7f5bbeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:18.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bd0568700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bd0568700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bd0568700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f5bb0005b70 con 0x7f5bc80a1990 2024-09-16T06:47:18.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bbeffd700 1 -- 172.21.15.189:0/4147009556 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v249) v1 ==== 106+0+3 (unknown 1885847503 0 944643765) 0x7f5bc0014700 con 0x7f5bc80a1990 2024-09-16T06:47:18.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bbeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:18.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.557+0000 7f5bbeffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5bac0362a0 legacy=0x7f5bac038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 10 monclient: shutdown 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 >> v1:172.21.15.189:6789/0 conn(0x7f5bc80a1990 legacy=0x7f5bc81d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 shutdown_connections 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 >> 172.21.15.189:0/4147009556 conn(0x7f5bc80a21f0 msgr2=0x7f5bc80a3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 shutdown_connections 2024-09-16T06:47:18.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.561+0000 7f5bd0568700 1 -- 172.21.15.189:0/4147009556 wait complete. 2024-09-16T06:47:18.575 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 12 -eq 12 2024-09-16T06:47:18.575 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 36f68e64-4177-4ce7-bf0b-c78e9b549946 2024-09-16T06:47:18.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: get_monmap_and_config 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: build_initial_monmap 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: monmap: 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:18.682142+0000 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:18.682142+0000 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:18.682 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:18.683 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 1 Processor -- start 2024-09-16T06:47:18.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 1 -- start start 2024-09-16T06:47:18.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: init 2024-09-16T06:47:18.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:18.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-c con 0x7f0c980a27a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-a con 0x7f0c980a3d10 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-b con 0x7f0c980a2350 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c98099a40 con 0x7f0c980a3d10 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c98099ef0 con 0x7f0c980a27a0 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9809a1d0 con 0x7f0c980a2350 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.677+0000 7f0c9e29c700 10 monclient(hunting): authenticate will time out at 1391.195962s 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c97fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c980a27a0 0x7f0c980a2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54716/0 (socket says 172.21.15.189:54716) 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c97fff700 1 -- 172.21.15.189:0/2429589689 learned_addr learned my addr 172.21.15.189:0/2429589689 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c8f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0c980a3d10 legacy=0x7f0c9813a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:18.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523983616 0 0) 0x7f0c98099ef0 con 0x7f0c980a27a0 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient(hunting): my global_id is 204235 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c840035d0 con 0x7f0c980a27a0 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147109672 0 0) 0x7f0c840035d0 con 0x7f0c980a27a0 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 >> v1:172.21.15.76:6790/0 conn(0x7f0c980a2350 legacy=0x7f0c9813dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 >> v1:172.21.15.76:6789/0 conn(0x7f0c980a3d10 legacy=0x7f0c9813a5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 monclient: found mon.noname-c 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c980a3290 con 0x7f0c980a27a0 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 5 monclient: authenticate success, global_id 204235 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c88001610 con 0x7f0c980a27a0 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: dump: 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.689 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c88001930 con 0x7f0c980a27a0 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 1 -- 172.21.15.189:0/2429589689 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c88001d00 con 0x7f0c980a27a0 2024-09-16T06:47:18.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 4 set_mon_vals no callback set 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c977fe700 10 monclient: dump: 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.691 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 10 monclient: shutdown 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- 172.21.15.189:0/2429589689 >> v1:172.21.15.189:6789/0 conn(0x7f0c980a27a0 legacy=0x7f0c980a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- 172.21.15.189:0/2429589689 shutdown_connections 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- 172.21.15.189:0/2429589689 >> 172.21.15.189:0/2429589689 conn(0x7f0c980a7d70 msgr2=0x7f0c980a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- 172.21.15.189:0/2429589689 shutdown_connections 2024-09-16T06:47:18.692 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- 172.21.15.189:0/2429589689 wait complete. 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 10 monclient: build_initial_monmap 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 10 monclient: monmap: 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:18.697 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 Processor -- start 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 1 -- start start 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.681+0000 7f0c9e29c700 10 monclient: init 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-a con 0x7f0c980a2350 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-c con 0x7f0c980a27a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient(hunting): picked mon.noname-b con 0x7f0c9813ecf0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:18.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9813dfa0 con 0x7f0c980a2350 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c981c71a0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9806bea0 con 0x7f0c980a27a0 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient(hunting): authenticate will time out at 1391.200856s 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c97fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0c980a2350 legacy=0x7f0c981c6380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9c839700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c9813ecf0 0x7f0c981d1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54720/0 (socket says 172.21.15.189:54720) 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9c839700 1 -- 172.21.15.189:0/483062800 learned_addr learned my addr 172.21.15.189:0/483062800 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224053267 0 0) 0x7f0c981c71a0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): my global_id is 204238 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c780035d0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497133302 0 0) 0x7f0c9806bea0 con 0x7f0c980a27a0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): my global_id is 194477 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c981c71a0 con 0x7f0c980a27a0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802820571 0 0) 0x7f0c780035d0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c9806bea0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c90001c70 con 0x7f0c9813ecf0 2024-09-16T06:47:18.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): dump: 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.701 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2016087030 0 0) 0x7f0c9806bea0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 >> v1:172.21.15.76:6790/0 conn(0x7f0c980a27a0 legacy=0x7f0c981c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 >> v1:172.21.15.76:6789/0 conn(0x7f0c980a2350 legacy=0x7f0c981c6380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 monclient: found mon.b 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9813d5d0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:18.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 5 monclient: authenticate success, global_id 204238 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _renew_subs 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9813dcc0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _renew_subs 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9813c200 con 0x7f0c9813ecf0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c90003a40 con 0x7f0c9813ecf0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0c90003d50 con 0x7f0c9813ecf0 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c96ffd700 4 set_mon_vals no callback set 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:18.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c957fa700 10 monclient: dump: 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.685+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c005100 con 0x7f0c9813ecf0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.689+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0c90003eb0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.689+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(249..249 src has 151..249) v4 ==== 6745+0+0 (unknown 2786296018 0 0) 0x7f0c90044080 con 0x7f0c9813ecf0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.689+0000 7f0c97fff700 10 monclient: get_auth_request con 0x7f0c78036220 auth_method 0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.693+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0c900046b0 con 0x7f0c9813ecf0 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.693+0000 7f0c957fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:18.705 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.693+0000 7f0c957fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:18.877 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.873+0000 7f0c9e29c700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.873+0000 7f0c9e29c700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.873+0000 7f0c9e29c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:18.878 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.873+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"} v 0) v1 -- 0x7f0c7c005930 con 0x7f0c9813ecf0 2024-09-16T06:47:18.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.993+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}]=0 v250) v1 ==== 106+0+3 (unknown 2123768492 0 1367428999) 0x7f0c90012f90 con 0x7f0c9813ecf0 2024-09-16T06:47:18.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.993+0000 7f0c957fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:18.996 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:18.993+0000 7f0c957fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c9e29c700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c9e29c700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c9e29c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.110 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"} v 0) v1 -- 0x7f0c7c0053b0 con 0x7f0c9813ecf0 2024-09-16T06:47:19.111 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c957fa700 1 -- 172.21.15.189:0/483062800 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}]=0 v250) v1 ==== 106+0+3 (unknown 2123768492 0 1367428999) 0x7f0c900148f0 con 0x7f0c9813ecf0 2024-09-16T06:47:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c957fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.112 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.105+0000 7f0c957fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0c78036220 legacy=0x7f0c780386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 10 monclient: shutdown 2024-09-16T06:47:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:19.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 >> v1:172.21.15.189:6789/0 conn(0x7f0c9813ecf0 legacy=0x7f0c981d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 shutdown_connections 2024-09-16T06:47:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 >> 172.21.15.189:0/483062800 conn(0x7f0c980a7d70 msgr2=0x7f0c980a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 shutdown_connections 2024-09-16T06:47:19.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.109+0000 7f0c9e29c700 1 -- 172.21.15.189:0/483062800 wait complete. 2024-09-16T06:47:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:+ nb=14 2024-09-16T06:47:19.124 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 36f68e64-4177-4ce7-bf0b-c78e9b549946 2024-09-16T06:47:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: get_monmap_and_config 2024-09-16T06:47:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: build_initial_monmap 2024-09-16T06:47:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: monmap: 2024-09-16T06:47:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:19.225 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:19.225782+0000 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:19.225782+0000 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:19.226 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 1 Processor -- start 2024-09-16T06:47:19.233 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 1 -- start start 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: init 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-c con 0x7f8b0809bb00 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-b con 0x7f8b0809c5f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-a con 0x7f8b08138da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0809cca0 con 0x7f8b08138da0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0809f750 con 0x7f8b0809bb00 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b08097620 con 0x7f8b0809c5f0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ecf8700 10 monclient(hunting): authenticate will time out at 1391.739086s 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0d295700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8b08138da0 legacy=0x7f8b0813b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:19.234 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ca94700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8b0809bb00 0x7f8b0809bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54736/0 (socket says 172.21.15.189:54736) 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b0ca94700 1 -- 172.21.15.189:0/1431356422 learned_addr learned my addr 172.21.15.189:0/1431356422 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758458036 0 0) 0x7f8b0809f750 con 0x7f8b0809bb00 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 10 monclient(hunting): my global_id is 204241 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af40035d0 con 0x7f8b0809bb00 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.221+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273384561 0 0) 0x7f8b08097620 con 0x7f8b0809c5f0 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient(hunting): my global_id is 194480 2024-09-16T06:47:19.235 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b0809f750 con 0x7f8b0809c5f0 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463277624 0 0) 0x7f8af40035d0 con 0x7f8b0809bb00 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 >> v1:172.21.15.76:6790/0 conn(0x7f8b0809c5f0 legacy=0x7f8b08137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 >> v1:172.21.15.76:6789/0 conn(0x7f8b08138da0 legacy=0x7f8b0813b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 monclient: found mon.noname-c 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08097900 con 0x7f8b0809bb00 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8af0001610 con 0x7f8b0809bb00 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.236 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: dump: 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 5 monclient: authenticate success, global_id 204241 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:19.237 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af00018f0 con 0x7f8b0809bb00 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 4 set_mon_vals no callback set 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 1 -- 172.21.15.189:0/1431356422 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8af0001c80 con 0x7f8b0809bb00 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b077fe700 10 monclient: dump: 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.238 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: shutdown 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1431356422 >> v1:172.21.15.189:6789/0 conn(0x7f8b0809bb00 legacy=0x7f8b0809bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1431356422 shutdown_connections 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1431356422 >> 172.21.15.189:0/1431356422 conn(0x7f8b080a03e0 msgr2=0x7f8b080a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1431356422 shutdown_connections 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1431356422 wait complete. 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: build_initial_monmap 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: monmap: 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:19.239 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 Processor -- start 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- start start 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: init 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:19.240 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-a con 0x7f8b0809bb00 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-b con 0x7f8b0809c5f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient(hunting): picked mon.noname-c con 0x7f8b08138da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b08094eb0 con 0x7f8b0809bb00 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b080a2df0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b080a12f0 con 0x7f8b08138da0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ecf8700 10 monclient(hunting): authenticate will time out at 1391.743021s 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b0ca94700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8b0809bb00 legacy=0x7f8b08137a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b07fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8b0809c5f0 0x7f8b081cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54744/0 (socket says 172.21.15.189:54744) 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b07fff700 1 -- 172.21.15.189:0/1333140774 learned_addr learned my addr 172.21.15.189:0/1333140774 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465266708 0 0) 0x7f8b080a2df0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.241 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): my global_id is 204244 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae80035d0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102388115 0 0) 0x7f8ae80035d0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b080a2df0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8af8001910 con 0x7f8b0809c5f0 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): dump: 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.242 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257083545 0 0) 0x7f8b080a12f0 con 0x7f8b08138da0 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 10 monclient(hunting): my global_id is 194483 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.225+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae80035d0 con 0x7f8b08138da0 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1035041546 0 0) 0x7f8b080a2df0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 >> v1:172.21.15.76:6790/0 conn(0x7f8b08138da0 legacy=0x7f8b081d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.243 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 >> v1:172.21.15.76:6789/0 conn(0x7f8b0809bb00 legacy=0x7f8b08137a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 monclient: found mon.b 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0809a6e0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 5 monclient: authenticate success, global_id 204244 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _renew_subs 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b08094bd0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _renew_subs 2024-09-16T06:47:19.244 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0806d750 con 0x7f8b0809c5f0 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af8001c50 con 0x7f8b0809c5f0 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8af8003b90 con 0x7f8b0809c5f0 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 10 monclient: dump: 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.245 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b067fc700 4 set_mon_vals no callback set 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8af8003d70 con 0x7f8b0809c5f0 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(250..250 src has 151..250) v4 ==== 6745+0+0 (unknown 1774952185 0 0) 0x7f8af8043e60 con 0x7f8b0809c5f0 2024-09-16T06:47:19.246 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ca94700 10 monclient: get_auth_request con 0x7f8ae8036220 auth_method 0 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.229+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec005100 con 0x7f8b0809c5f0 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.237+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8af80148e0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.237+0000 7f8b057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.237+0000 7f8b057fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:19.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b0ecf8700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b0ecf8700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b0ecf8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"} v 0) v1 -- 0x7f8aec005930 con 0x7f8b0809c5f0 2024-09-16T06:47:19.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}]=0 v250) v1 ==== 106+0+3 (unknown 2123768492 0 1367428999) 0x7f8af8012d50 con 0x7f8b0809c5f0 2024-09-16T06:47:19.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.417+0000 7f8b057fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:19.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b0ecf8700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b0ecf8700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b0ecf8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.533 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"} v 0) v1 -- 0x7f8aec0053b0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b057fa700 1 -- 172.21.15.189:0/1333140774 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}]=0 v250) v1 ==== 106+0+3 (unknown 2123768492 0 1367428999) 0x7f8af80144d0 con 0x7f8b0809c5f0 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b057fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "36f68e64-4177-4ce7-bf0b-c78e9b549946"}] 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.529+0000 7f8b057fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8ae8036220 legacy=0x7f8ae80386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 10 monclient: shutdown 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 >> v1:172.21.15.189:6789/0 conn(0x7f8b0809c5f0 legacy=0x7f8b081cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 shutdown_connections 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 >> 172.21.15.189:0/1333140774 conn(0x7f8b080a03e0 msgr2=0x7f8b0809a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 shutdown_connections 2024-09-16T06:47:19.537 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.533+0000 7f8b0ecf8700 1 -- 172.21.15.189:0/1333140774 wait complete. 2024-09-16T06:47:19.546 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 14 -eq 14 2024-09-16T06:47:19.546 INFO:tasks.workunit.client.0.smithi189.stderr:+ test 14 -ne 12 2024-09-16T06:47:19.546 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 12 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: get_monmap_and_config 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: build_initial_monmap 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: monmap: 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:19.638574+0000 2024-09-16T06:47:19.638 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:19.638574+0000 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:19.639 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 1 Processor -- start 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 1 -- start start 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: init 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:19.642 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc09eba0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc09beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc09c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc099ed0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc09d2e0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc09ef80 con 0x7fc2dc09c2e0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e274c700 10 monclient(hunting): authenticate will time out at 1392.151118s 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e174a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc2dc09eba0 0x7fc2dc09b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54750/0 (socket says 172.21.15.189:54750) 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e174a700 1 -- 172.21.15.189:0/2547687037 learned_addr learned my addr 172.21.15.189:0/2547687037 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2e0f49700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc2dc09beb0 legacy=0x7fc2dc137530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320561247 0 0) 0x7fc2dc09d2e0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.643 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 10 monclient(hunting): my global_id is 204247 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.633+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c40035d0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299535726 0 0) 0x7fc2dc09ef80 con 0x7fc2dc09c2e0 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient(hunting): my global_id is 194486 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc09d2e0 con 0x7fc2dc09c2e0 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426562295 0 0) 0x7fc2c40035d0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 >> v1:172.21.15.76:6790/0 conn(0x7fc2dc09c2e0 legacy=0x7fc2dc13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 >> v1:172.21.15.76:6789/0 conn(0x7fc2dc09beb0 legacy=0x7fc2dc137530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:19.644 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 monclient: found mon.noname-c 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc09cc70 con 0x7fc2dc09eba0 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: _finish_auth 0 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 5 monclient: authenticate success, global_id 204247 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2d0001610 con 0x7fc2dc09eba0 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:19.645 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: dump: 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2d0001880 con 0x7fc2dc09eba0 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:19.646 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 4 set_mon_vals no callback set 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 1 -- 172.21.15.189:0/2547687037 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2d0001c10 con 0x7fc2dc09eba0 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2dbfff700 10 monclient: dump: 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: shutdown 2024-09-16T06:47:19.647 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- 172.21.15.189:0/2547687037 >> v1:172.21.15.189:6789/0 conn(0x7fc2dc09eba0 legacy=0x7fc2dc09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- 172.21.15.189:0/2547687037 shutdown_connections 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- 172.21.15.189:0/2547687037 >> 172.21.15.189:0/2547687037 conn(0x7fc2dc09fe70 msgr2=0x7fc2dc0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- 172.21.15.189:0/2547687037 shutdown_connections 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- 172.21.15.189:0/2547687037 wait complete. 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: build_initial_monmap 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: monmap: 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:19.648 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 Processor -- start 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- start start 2024-09-16T06:47:19.649 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: init 2024-09-16T06:47:19.652 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc09beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc09c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc09eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc09e1e0 con 0x7fc2dc09c2e0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc06e850 con 0x7fc2dc09beb0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0a11b0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e274c700 10 monclient(hunting): authenticate will time out at 1392.155332s 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e0f49700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc2dc09c2e0 legacy=0x7fc2dc06e140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e174a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc2dc09beb0 0x7fc2dc06da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54760/0 (socket says 172.21.15.189:54760) 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2e174a700 1 -- 172.21.15.189:0/2432746788 learned_addr learned my addr 172.21.15.189:0/2432746788 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:19.653 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237985853 0 0) 0x7fc2dc06e850 con 0x7fc2dc09beb0 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 10 monclient(hunting): my global_id is 204250 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.637+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c80035d0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846318378 0 0) 0x7fc2dc0a11b0 con 0x7fc2dc09eba0 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): my global_id is 194489 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc06e850 con 0x7fc2dc09eba0 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3273182149 0 0) 0x7fc2c80035d0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.654 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2dc0a11b0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2d0001610 con 0x7fc2dc09beb0 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.655 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093919654 0 0) 0x7fc2dc0a11b0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 >> v1:172.21.15.76:6790/0 conn(0x7fc2dc09eba0 legacy=0x7fc2dc1cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 >> v1:172.21.15.76:6789/0 conn(0x7fc2dc09c2e0 legacy=0x7fc2dc06e140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 monclient: found mon.b 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc09a380 con 0x7fc2dc09beb0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:19.656 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 5 monclient: authenticate success, global_id 204250 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _renew_subs 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc096fe0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _renew_subs 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc06f430 con 0x7fc2dc09beb0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2d00011b0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2d0004e00 con 0x7fc2dc09beb0 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:19.657 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2db7fe700 4 set_mon_vals no callback set 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 10 monclient: dump: 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc2d0004fe0 con 0x7fc2dc09beb0 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.658 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc005100 con 0x7fc2dc09beb0 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(250..250 src has 151..250) v4 ==== 6745+0+0 (unknown 1774952185 0 0) 0x7fc2d0044070 con 0x7fc2dc09beb0 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.641+0000 7fc2e0f49700 10 monclient: get_auth_request con 0x7fc2c8036180 auth_method 0 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.649+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc2d0006600 con 0x7fc2dc09beb0 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.649+0000 7fc2d9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:19.659 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.649+0000 7fc2d9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:19.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.825+0000 7fc2e274c700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:19.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.825+0000 7fc2e274c700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:19.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.825+0000 7fc2e274c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:19.832 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:19.825+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fc2bc005930 con 0x7fc2dc09beb0 2024-09-16T06:47:19.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:19.834+0000 7f779860f700 -1 mon.c@2(leader).osd e250 definitely_dead 0 2024-09-16T06:47:20.014 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.009+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v251) v1 ==== 83+0+0 (unknown 1750214730 0 0) 0x7fc2d0012fc0 con 0x7fc2dc09beb0 2024-09-16T06:47:20.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.009+0000 7fc2d9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.015 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.009+0000 7fc2d9ffb700 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T06:47:20.130 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.125+0000 7fc2e274c700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.125+0000 7fc2e274c700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.125+0000 7fc2e274c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.131 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.125+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fc2bc005420 con 0x7fc2dc09beb0 2024-09-16T06:47:20.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:20.130+0000 7f779860f700 -1 mon.c@2(leader).osd e251 definitely_dead 0 2024-09-16T06:47:20.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2d9ffb700 1 -- 172.21.15.189:0/2432746788 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v251) v1 ==== 92+0+0 (unknown 941693364 0 0) 0x7fc2d0014740 con 0x7fc2dc09beb0 2024-09-16T06:47:20.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2d9ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.132 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2d9ffb700 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T06:47:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc2c8036180 legacy=0x7fc2c8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 10 monclient: shutdown 2024-09-16T06:47:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 >> v1:172.21.15.189:6789/0 conn(0x7fc2dc09beb0 legacy=0x7fc2dc06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 shutdown_connections 2024-09-16T06:47:20.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 >> 172.21.15.189:0/2432746788 conn(0x7fc2dc09fe70 msgr2=0x7fc2dc09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:20.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 shutdown_connections 2024-09-16T06:47:20.136 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.129+0000 7fc2e274c700 1 -- 172.21.15.189:0/2432746788 wait complete. 2024-09-16T06:47:20.136 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T06:47:20.145 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 12 2024-09-16T06:47:20.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.241+0000 7fc0a27a6700 10 monclient: get_monmap_and_config 2024-09-16T06:47:20.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.241+0000 7fc0a27a6700 10 monclient: build_initial_monmap 2024-09-16T06:47:20.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.241+0000 7fc0a27a6700 10 monclient: monmap: 2024-09-16T06:47:20.247 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:20.247741+0000 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:20.247741+0000 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:20.248 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 Processor -- start 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- start start 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient: init 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-a con 0x7fc09c09ce10 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-c con 0x7fc09c09d1f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-b con 0x7fc09c071ee0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c097fb0 con 0x7fc09c09ce10 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c09d670 con 0x7fc09c09d1f0 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c099690 con 0x7fc09c071ee0 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:20.253 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient(hunting): authenticate will time out at 1392.761000s 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a0fa3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc09c09d1f0 0x7fc09c0978a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54768/0 (socket says 172.21.15.189:54768) 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a17a4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc09c09ce10 legacy=0x7fc09c097190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a0fa3700 1 -- 172.21.15.189:0/1921383443 learned_addr learned my addr 172.21.15.189:0/1921383443 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225820749 0 0) 0x7fc09c09d670 con 0x7fc09c09d1f0 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): my global_id is 204253 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0880035d0 con 0x7fc09c09d1f0 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759942104 0 0) 0x7fc09c099690 con 0x7fc09c071ee0 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient(hunting): my global_id is 194492 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09c09d670 con 0x7fc09c071ee0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163770619 0 0) 0x7fc0880035d0 con 0x7fc09c09d1f0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 >> v1:172.21.15.76:6790/0 conn(0x7fc09c071ee0 legacy=0x7fc09c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 >> v1:172.21.15.76:6789/0 conn(0x7fc09c09ce10 legacy=0x7fc09c097190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 monclient: found mon.noname-c 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c098660 con 0x7fc09c09d1f0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: _finish_auth 0 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:20.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 5 monclient: authenticate success, global_id 204253 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc098001990 con 0x7fc09c09d1f0 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: dump: 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.256 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc098001c00 con 0x7fc09c09d1f0 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 1 -- 172.21.15.189:0/1921383443 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc098001f90 con 0x7fc09c09d1f0 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 4 set_mon_vals no callback set 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:20.257 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc093fff700 10 monclient: dump: 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 10 monclient: shutdown 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- 172.21.15.189:0/1921383443 >> v1:172.21.15.189:6789/0 conn(0x7fc09c09d1f0 legacy=0x7fc09c0978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.258 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- 172.21.15.189:0/1921383443 shutdown_connections 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- 172.21.15.189:0/1921383443 >> 172.21.15.189:0/1921383443 conn(0x7fc09c09fe70 msgr2=0x7fc09c0a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- 172.21.15.189:0/1921383443 shutdown_connections 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.245+0000 7fc0a27a6700 1 -- 172.21.15.189:0/1921383443 wait complete. 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: build_initial_monmap 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: monmap: 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:20.259 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:20.260 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 Processor -- start 2024-09-16T06:47:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- start start 2024-09-16T06:47:20.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: init 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-c con 0x7fc09c071ee0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-b con 0x7fc09c09ce10 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient(hunting): picked mon.noname-a con 0x7fc09c09d1f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c1c82f0 con 0x7fc09c09d1f0 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c09a380 con 0x7fc09c09ce10 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c071400 con 0x7fc09c071ee0 2024-09-16T06:47:20.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient(hunting): authenticate will time out at 1392.765661s 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a1fa5700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc09c09d1f0 legacy=0x7fc09c1d6640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a0fa3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc09c09ce10 0x7fc09c1d2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54770/0 (socket says 172.21.15.189:54770) 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a0fa3700 1 -- 172.21.15.189:0/2141055413 learned_addr learned my addr 172.21.15.189:0/2141055413 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621115796 0 0) 0x7fc09c09a380 con 0x7fc09c09ce10 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): my global_id is 204256 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0800035d0 con 0x7fc09c09ce10 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954396979 0 0) 0x7fc09c071400 con 0x7fc09c071ee0 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): my global_id is 194495 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09c09a380 con 0x7fc09c071ee0 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2563136281 0 0) 0x7fc0800035d0 con 0x7fc09c09ce10 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc09c071400 con 0x7fc09c09ce10 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc098001a10 con 0x7fc09c09ce10 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.266 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798264372 0 0) 0x7fc09c09a380 con 0x7fc09c071ee0 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0800035d0 con 0x7fc09c071ee0 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1232385340 0 0) 0x7fc09c071400 con 0x7fc09c09ce10 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 >> v1:172.21.15.76:6790/0 conn(0x7fc09c071ee0 legacy=0x7fc09c1c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 >> v1:172.21.15.76:6789/0 conn(0x7fc09c09d1f0 legacy=0x7fc09c1d6640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:20.267 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 monclient: found mon.b 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0a28a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc091ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 5 monclient: authenticate success, global_id 204256 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _renew_subs 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c1af540 con 0x7fc09c09ce10 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _renew_subs 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.249+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c1ad200 con 0x7fc09c09ce10 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0a27a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0a27a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0a27a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc098005010 con 0x7fc09c09ce10 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c06ce80 con 0x7fc09c09ce10 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0980053a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0937fe700 4 set_mon_vals no callback set 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 10 monclient: dump: 2024-09-16T06:47:20.269 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc098005580 con 0x7fc09c09ce10 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(251..251 src has 151..251) v4 ==== 6745+0+0 (unknown 746820832 0 0) 0x7fc098044480 con 0x7fc09c09ce10 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.253+0000 7fc0a17a4700 10 monclient: get_auth_request con 0x7fc080036200 auth_method 0 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.261+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc098006a50 con 0x7fc09c09ce10 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.261+0000 7fc091ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.261+0000 7fc091ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.437+0000 7fc0a27a6700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.437+0000 7fc0a27a6700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.437+0000 7fc0a27a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.444 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.437+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fc09c1cb5a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:20.446+0000 7f779860f700 -1 mon.c@2(leader).osd e251 definitely_dead 0 2024-09-16T06:47:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.441+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v251) v1 ==== 92+0+0 (unknown 941693364 0 0) 0x7fc09c1cb5a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.441+0000 7fc091ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.445 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.441+0000 7fc091ffb700 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-16T06:47:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.545+0000 7fc0a27a6700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.551 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.545+0000 7fc0a27a6700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.545+0000 7fc0a27a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.545+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7fc09c1cb5a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:20.554+0000 7f779860f700 -1 mon.c@2(leader).osd e251 definitely_dead 0 2024-09-16T06:47:20.552 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc091ffb700 1 -- 172.21.15.189:0/2141055413 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v251) v1 ==== 92+0+0 (unknown 941693364 0 0) 0x7fc09c1cb5a0 con 0x7fc09c09ce10 2024-09-16T06:47:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc091ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T06:47:20.553 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc091ffb700 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc080036200 legacy=0x7fc0800386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 10 monclient: shutdown 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 >> v1:172.21.15.189:6789/0 conn(0x7fc09c09ce10 legacy=0x7fc09c1d2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 shutdown_connections 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 >> 172.21.15.189:0/2141055413 conn(0x7fc09c09fe70 msgr2=0x7fc09c0960d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:20.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 shutdown_connections 2024-09-16T06:47:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.549+0000 7fc0a27a6700 1 -- 172.21.15.189:0/2141055413 wait complete. 2024-09-16T06:47:20.556 INFO:tasks.workunit.client.0.smithi189.stderr:osd.12 does not exist. 2024-09-16T06:47:20.564 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 14 2024-09-16T06:47:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: get_monmap_and_config 2024-09-16T06:47:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: monmap: 2024-09-16T06:47:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:20.666 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:20.666683+0000 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:20.666683+0000 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:20.667 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 1 Processor -- start 2024-09-16T06:47:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 1 -- start start 2024-09-16T06:47:20.674 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: init 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0af009eba0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0af009beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0af009c2e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af0099ed0 con 0x7f0af009c2e0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af009d2e0 con 0x7f0af009beb0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af009ef80 con 0x7f0af009eba0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aef59e700 10 monclient(hunting): authenticate will time out at 1393.179500s 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aeed9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0af009c2e0 legacy=0x7f0af013ac60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aedd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0af009beb0 0x7f0af0137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54772/0 (socket says 172.21.15.189:54772) 2024-09-16T06:47:20.675 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.661+0000 7f0aedd9b700 1 -- 172.21.15.189:0/2729813626 learned_addr learned my addr 172.21.15.189:0/2729813626 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216552541 0 0) 0x7f0af009d2e0 con 0x7f0af009beb0 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): my global_id is 204259 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad80035d0 con 0x7f0af009beb0 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200720518 0 0) 0x7f0af009ef80 con 0x7f0af009eba0 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient(hunting): my global_id is 194498 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af009d2e0 con 0x7f0af009eba0 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2561996350 0 0) 0x7f0ad80035d0 con 0x7f0af009beb0 2024-09-16T06:47:20.676 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 >> v1:172.21.15.76:6790/0 conn(0x7f0af009eba0 legacy=0x7f0af009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 >> v1:172.21.15.76:6789/0 conn(0x7f0af009c2e0 legacy=0x7f0af013ac60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: _finish_hunting 0 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 monclient: found mon.noname-c 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af009cc70 con 0x7f0af009beb0 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: _finish_auth 0 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: _check_auth_tickets 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ae0001610 con 0x7f0af009beb0 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 5 monclient: authenticate success, global_id 204259 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:20.677 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: dump: 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae0001880 con 0x7f0af009beb0 2024-09-16T06:47:20.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 4 set_mon_vals no callback set 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 1 -- 172.21.15.189:0/2729813626 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ae0001c10 con 0x7f0af009beb0 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aed59a700 10 monclient: dump: 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: shutdown 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- 172.21.15.189:0/2729813626 >> v1:172.21.15.189:6789/0 conn(0x7f0af009beb0 legacy=0x7f0af0137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- 172.21.15.189:0/2729813626 shutdown_connections 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- 172.21.15.189:0/2729813626 >> 172.21.15.189:0/2729813626 conn(0x7f0af009fe70 msgr2=0x7f0af00a0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- 172.21.15.189:0/2729813626 shutdown_connections 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- 172.21.15.189:0/2729813626 wait complete. 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: monmap: 2024-09-16T06:47:20.680 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 Processor -- start 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- start start 2024-09-16T06:47:20.681 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: init 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0af009beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0af009c2e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0af009eba0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af013aa70 con 0x7f0af009eba0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af0137930 con 0x7f0af009beb0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af0138c20 con 0x7f0af009c2e0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aef59e700 10 monclient(hunting): authenticate will time out at 1393.183004s 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aee59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0af009beb0 0x7f0af0136b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54784/0 (socket says 172.21.15.189:54784) 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aee59c700 1 -- 172.21.15.189:0/4021141656 learned_addr learned my addr 172.21.15.189:0/4021141656 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0aeed9d700 1 -- 172.21.15.189:0/4021141656 >> v1:172.21.15.76:6789/0 conn(0x7f0af009eba0 legacy=0x7f0af01d2360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:20.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874203338 0 0) 0x7f0af0137930 con 0x7f0af009beb0 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): my global_id is 204262 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad00035d0 con 0x7f0af009beb0 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35117804 0 0) 0x7f0ad00035d0 con 0x7f0af009beb0 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0af0137930 con 0x7f0af009beb0 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ae4001950 con 0x7f0af009beb0 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): dump: 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.683 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351638351 0 0) 0x7f0af0138c20 con 0x7f0af009c2e0 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 10 monclient(hunting): my global_id is 194501 2024-09-16T06:47:20.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad00035d0 con 0x7f0af009c2e0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2841287690 0 0) 0x7f0af0137930 con 0x7f0af009beb0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.665+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 >> v1:172.21.15.76:6790/0 conn(0x7f0af009c2e0 legacy=0x7f0af0137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 >> v1:172.21.15.76:6789/0 conn(0x7f0af009eba0 legacy=0x7f0af01d2360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 monclient: found mon.b 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af00a15e0 con 0x7f0af009beb0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:20.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 5 monclient: authenticate success, global_id 204262 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _renew_subs 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af00a1090 con 0x7f0af009beb0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _renew_subs 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af013a790 con 0x7f0af009beb0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae4001cd0 con 0x7f0af009beb0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ae4003b90 con 0x7f0af009beb0 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adffff700 4 set_mon_vals no callback set 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 10 monclient: dump: 2024-09-16T06:47:20.686 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ae4003cf0 con 0x7f0af009beb0 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4005100 con 0x7f0af009beb0 2024-09-16T06:47:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(251..251 src has 151..251) v4 ==== 6745+0+0 (unknown 746820832 0 0) 0x7f0ae4043f30 con 0x7f0af009beb0 2024-09-16T06:47:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.669+0000 7f0aedd9b700 10 monclient: get_auth_request con 0x7f0ad00361e0 auth_method 0 2024-09-16T06:47:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.673+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0ae40112d0 con 0x7f0af009beb0 2024-09-16T06:47:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.673+0000 7f0adeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:20.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.673+0000 7f0adeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:20.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.853+0000 7f0aef59e700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.853+0000 7f0aef59e700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.853+0000 7f0aef59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:20.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:20.853+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f0ad4005930 con 0x7f0af009beb0 2024-09-16T06:47:20.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:20.858+0000 7f779860f700 -1 mon.c@2(leader).osd e251 definitely_dead 0 2024-09-16T06:47:21.018 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.013+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v252) v1 ==== 83+0+0 (unknown 376526303 0 0) 0x7f0ae4012da0 con 0x7f0af009beb0 2024-09-16T06:47:21.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.013+0000 7f0adeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:21.019 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.013+0000 7f0adeffd700 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T06:47:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0aef59e700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0aef59e700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0aef59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.134 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f0ad4005420 con 0x7f0af009beb0 2024-09-16T06:47:21.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:21.134+0000 7f779860f700 -1 mon.c@2(leader).osd e252 definitely_dead 0 2024-09-16T06:47:21.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0adeffd700 1 -- 172.21.15.189:0/4021141656 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v252) v1 ==== 92+0+0 (unknown 3713259162 0 0) 0x7f0ae4014520 con 0x7f0af009beb0 2024-09-16T06:47:21.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0adeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T06:47:21.135 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.129+0000 7f0adeffd700 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0ad00361e0 legacy=0x7f0ad00386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 10 monclient: shutdown 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 >> v1:172.21.15.189:6789/0 conn(0x7f0af009beb0 legacy=0x7f0af0136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 shutdown_connections 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 >> 172.21.15.189:0/4021141656 conn(0x7f0af009fe70 msgr2=0x7f0af0070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 shutdown_connections 2024-09-16T06:47:21.138 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.133+0000 7f0aef59e700 1 -- 172.21.15.189:0/4021141656 wait complete. 2024-09-16T06:47:21.139 INFO:tasks.workunit.client.0.smithi189.stderr:osd.14 does not exist. 2024-09-16T06:47:21.148 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd rm 1000 2024-09-16T06:47:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: get_monmap_and_config 2024-09-16T06:47:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: build_initial_monmap 2024-09-16T06:47:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: monmap: 2024-09-16T06:47:21.252 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:21.252700+0000 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:21.252700+0000 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:21.253 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 1 Processor -- start 2024-09-16T06:47:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 1 -- start start 2024-09-16T06:47:21.254 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: init 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-b con 0x7f7b600a4710 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-c con 0x7f7b6009b8e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-a con 0x7f7b6009bd30 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6009d4b0 con 0x7f7b6009bd30 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b600a1340 con 0x7f7b6009b8e0 2024-09-16T06:47:21.255 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6009c310 con 0x7f7b600a4710 2024-09-16T06:47:21.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:21.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b66de4700 10 monclient(hunting): authenticate will time out at 1393.767084s 2024-09-16T06:47:21.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b655e1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7b6009b8e0 0x7f7b601374d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54798/0 (socket says 172.21.15.189:54798) 2024-09-16T06:47:21.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b655e1700 1 -- 172.21.15.189:0/2272350961 learned_addr learned my addr 172.21.15.189:0/2272350961 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:21.256 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b665e3700 1 -- 172.21.15.189:0/2272350961 >> v1:172.21.15.76:6789/0 conn(0x7f7b6009bd30 legacy=0x7f7b6013ac00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:21.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214550139 0 0) 0x7f7b600a1340 con 0x7f7b6009b8e0 2024-09-16T06:47:21.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.259 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): my global_id is 204265 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4c0035d0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550929820 0 0) 0x7f7b6009c310 con 0x7f7b600a4710 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 10 monclient(hunting): my global_id is 194504 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.249+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b600a1340 con 0x7f7b600a4710 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1872714983 0 0) 0x7f7b4c0035d0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 >> v1:172.21.15.76:6790/0 conn(0x7f7b600a4710 legacy=0x7f7b6009b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.260 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 >> v1:172.21.15.76:6789/0 conn(0x7f7b6009bd30 legacy=0x7f7b6013ac00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: _finish_hunting 0 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 monclient: found mon.noname-c 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6009c5f0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: _finish_auth 0 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: _check_auth_tickets 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 5 monclient: authenticate success, global_id 204265 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7b50001610 con 0x7f7b6009b8e0 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.261 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: dump: 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:21.262 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b500018f0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 4 set_mon_vals no callback set 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 1 -- 172.21.15.189:0/2272350961 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7b50001c80 con 0x7f7b6009b8e0 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b64de0700 10 monclient: dump: 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.263 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 10 monclient: shutdown 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- 172.21.15.189:0/2272350961 >> v1:172.21.15.189:6789/0 conn(0x7f7b6009b8e0 legacy=0x7f7b601374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- 172.21.15.189:0/2272350961 shutdown_connections 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- 172.21.15.189:0/2272350961 >> 172.21.15.189:0/2272350961 conn(0x7f7b600a4300 msgr2=0x7f7b600a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- 172.21.15.189:0/2272350961 shutdown_connections 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- 172.21.15.189:0/2272350961 wait complete. 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 10 monclient: build_initial_monmap 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 10 monclient: monmap: 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:21.264 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 Processor -- start 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 1 -- start start 2024-09-16T06:47:21.265 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.253+0000 7f7b66de4700 10 monclient: init 2024-09-16T06:47:21.268 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-b con 0x7f7b6009b8e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-a con 0x7f7b6009bd30 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient(hunting): picked mon.noname-c con 0x7f7b600a4710 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b600a3120 con 0x7f7b6009bd30 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b601c0d10 con 0x7f7b6009b8e0 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b65de2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7b6009b8e0 0x7f7b601c0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54800/0 (socket says 172.21.15.189:54800) 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b65de2700 1 -- 172.21.15.189:0/2281525875 learned_addr learned my addr 172.21.15.189:0/2281525875 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:21.269 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b655e1700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7b6009bd30 legacy=0x7f7b601cb930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6006b120 con 0x7f7b600a4710 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient(hunting): authenticate will time out at 1393.773006s 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23749649 0 0) 0x7f7b601c0d10 con 0x7f7b6009b8e0 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): my global_id is 204268 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b440057e0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880438994 0 0) 0x7f7b6006b120 con 0x7f7b600a4710 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.270 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): my global_id is 194507 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b601c0d10 con 0x7f7b600a4710 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263638202 0 0) 0x7f7b440057e0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6006b120 con 0x7f7b6009b8e0 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7b5c001970 con 0x7f7b6009b8e0 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): dump: 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.271 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:21.272 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 888703811 0 0) 0x7f7b6006b120 con 0x7f7b6009b8e0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 >> v1:172.21.15.76:6790/0 conn(0x7f7b600a4710 legacy=0x7f7b601cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 >> v1:172.21.15.76:6789/0 conn(0x7f7b6009bd30 legacy=0x7f7b601cb930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 monclient: found mon.b 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b601a5480 con 0x7f7b6009b8e0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 5 monclient: authenticate success, global_id 204268 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _renew_subs 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b601a7f60 con 0x7f7b6009b8e0 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _renew_subs 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b601a6a10 con 0x7f7b6009b8e0 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b5c001ce0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7b5c003b10 con 0x7f7b6009b8e0 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b567fc700 10 monclient: dump: 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.274 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6009ae10 con 0x7f7b6009b8e0 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.257+0000 7f7b57fff700 4 set_mon_vals no callback set 2024-09-16T06:47:21.275 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.261+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7b5c003d30 con 0x7f7b6009b8e0 2024-09-16T06:47:21.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.261+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(252..252 src has 213..252) v4 ==== 6745+0+0 (unknown 4124253298 0 0) 0x7f7b5c044040 con 0x7f7b6009b8e0 2024-09-16T06:47:21.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.261+0000 7f7b655e1700 10 monclient: get_auth_request con 0x7f7b440383c0 auth_method 0 2024-09-16T06:47:21.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.269+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7b5c014750 con 0x7f7b6009b8e0 2024-09-16T06:47:21.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.269+0000 7f7b567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.276 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.269+0000 7f7b567fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:21.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b66de4700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.449 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b66de4700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b66de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.450 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7b600a0060 con 0x7f7b6009b8e0 2024-09-16T06:47:21.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:21.450+0000 7f779860f700 -1 mon.c@2(leader).osd e252 definitely_dead 0 2024-09-16T06:47:21.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v252) v1 ==== 96+0+0 (unknown 834797404 0 0) 0x7f7b5c012df0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.451 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.445+0000 7f7b567fc700 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-16T06:47:21.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b66de4700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b66de4700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b66de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7b601a78b0 con 0x7f7b6009b8e0 2024-09-16T06:47:21.562 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:21.562+0000 7f779860f700 -1 mon.c@2(leader).osd e252 definitely_dead 0 2024-09-16T06:47:21.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b567fc700 1 -- 172.21.15.189:0/2281525875 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v252) v1 ==== 96+0+0 (unknown 834797404 0 0) 0x7f7b5c014570 con 0x7f7b6009b8e0 2024-09-16T06:47:21.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b567fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T06:47:21.563 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.557+0000 7f7b567fc700 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-16T06:47:21.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7b440383c0 legacy=0x7f7b4403a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 10 monclient: shutdown 2024-09-16T06:47:21.565 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 >> v1:172.21.15.189:6789/0 conn(0x7f7b6009b8e0 legacy=0x7f7b601c0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 shutdown_connections 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 >> 172.21.15.189:0/2281525875 conn(0x7f7b600a4300 msgr2=0x7f7b60070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 shutdown_connections 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.561+0000 7f7b66de4700 1 -- 172.21.15.189:0/2281525875 wait complete. 2024-09-16T06:47:21.566 INFO:tasks.workunit.client.0.smithi189.stderr:osd.1000 does not exist. 2024-09-16T06:47:21.576 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph osd create 12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: get_monmap_and_config 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: build_initial_monmap 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: monmap: 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:21.678443+0000 2024-09-16T06:47:21.678 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:21.678443+0000 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:21.679 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.682 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 1 Processor -- start 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 1 -- start start 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: init 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-c con 0x7f19f81352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-a con 0x7f19f8135db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-b con 0x7f19f80a2030 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f809b050 con 0x7f19f8135db0 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f806ab20 con 0x7f19f81352c0 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f806a840 con 0x7f19f80a2030 2024-09-16T06:47:21.683 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.672+0000 7f19fcce3700 10 monclient(hunting): authenticate will time out at 1394.191998s 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19f81352c0 0x7f19f81356a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54812/0 (socket says 172.21.15.189:54812) 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f6ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f19f8135db0 legacy=0x7f19f813a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f77fe700 1 -- 172.21.15.189:0/882687728 learned_addr learned my addr 172.21.15.189:0/882687728 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694130624 0 0) 0x7f19f806ab20 con 0x7f19f81352c0 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): my global_id is 204271 2024-09-16T06:47:21.684 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e40035d0 con 0x7f19f81352c0 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795386091 0 0) 0x7f19f806a840 con 0x7f19f80a2030 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient(hunting): my global_id is 194510 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f806ab20 con 0x7f19f80a2030 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816870799 0 0) 0x7f19e40035d0 con 0x7f19f81352c0 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 >> v1:172.21.15.76:6790/0 conn(0x7f19f80a2030 legacy=0x7f19f813df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 >> v1:172.21.15.76:6789/0 conn(0x7f19f8135db0 legacy=0x7f19f813a860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:21.685 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 monclient: found mon.noname-c 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f806c1b0 con 0x7f19f81352c0 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:21.686 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 5 monclient: authenticate success, global_id 204271 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19e0001610 con 0x7f19f81352c0 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: dump: 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.687 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e0001830 con 0x7f19f81352c0 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 1 -- 172.21.15.189:0/882687728 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19e0001bc0 con 0x7f19f81352c0 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 4 set_mon_vals no callback set 2024-09-16T06:47:21.688 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19f67fc700 10 monclient: dump: 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.689 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 10 monclient: shutdown 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 1 -- 172.21.15.189:0/882687728 >> v1:172.21.15.189:6789/0 conn(0x7f19f81352c0 legacy=0x7f19f81356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 1 -- 172.21.15.189:0/882687728 shutdown_connections 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 1 -- 172.21.15.189:0/882687728 >> 172.21.15.189:0/882687728 conn(0x7f19f80a4550 msgr2=0x7f19f80a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 1 -- 172.21.15.189:0/882687728 shutdown_connections 2024-09-16T06:47:21.690 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.676+0000 7f19fcce3700 1 -- 172.21.15.189:0/882687728 wait complete. 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: build_initial_monmap 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: monmap: 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:21.695 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 Processor -- start 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- start start 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: init 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-a con 0x7f19f80a2030 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:21.696 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-b con 0x7f19f81352c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient(hunting): picked mon.noname-c con 0x7f19f8135db0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f8130a60 con 0x7f19f80a2030 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f8131020 con 0x7f19f81352c0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19f77fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f19f80a2030 legacy=0x7f19f8131720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f8131300 con 0x7f19f8135db0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19f6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19f81352c0 0x7f19f81ce5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:54822/0 (socket says 172.21.15.189:54822) 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19f6ffd700 1 -- 172.21.15.189:0/3964739913 learned_addr learned my addr 172.21.15.189:0/3964739913 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient(hunting): authenticate will time out at 1394.198368s 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810746826 0 0) 0x7f19f8131020 con 0x7f19f81352c0 2024-09-16T06:47:21.697 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): my global_id is 204274 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d80035d0 con 0x7f19f81352c0 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620564991 0 0) 0x7f19f8131300 con 0x7f19f8135db0 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): my global_id is 194513 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f8131020 con 0x7f19f8135db0 2024-09-16T06:47:21.698 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566073435 0 0) 0x7f19d80035d0 con 0x7f19f81352c0 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f8131300 con 0x7f19f81352c0 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19e8004230 con 0x7f19f81352c0 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): dump: 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.699 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3369123910 0 0) 0x7f19f8131300 con 0x7f19f81352c0 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 >> v1:172.21.15.76:6790/0 conn(0x7f19f8135db0 legacy=0x7f19f81d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 >> v1:172.21.15.76:6789/0 conn(0x7f19f80a2030 legacy=0x7f19f8131720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 monclient: found mon.b 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:21.700 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80a6760 con 0x7f19f81352c0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: _finish_auth 0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 5 monclient: authenticate success, global_id 204274 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _renew_subs 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f80a6230 con 0x7f19f81352c0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _renew_subs 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.701 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f8131e30 con 0x7f19f81352c0 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e80044a0 con 0x7f19f81352c0 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19e8004830 con 0x7f19f81352c0 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19effff700 10 monclient: dump: 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:21.702 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.680+0000 7f19f5ffb700 4 set_mon_vals no callback set 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f19e8004a50 con 0x7f19f81352c0 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19fcce3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19fcce3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19fcce3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(252..252 src has 213..252) v4 ==== 6745+0+0 (unknown 4124253298 0 0) 0x7f19e8043e00 con 0x7f19f81352c0 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f8066d00 con 0x7f19f81352c0 2024-09-16T06:47:21.703 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.684+0000 7f19f77fe700 10 monclient: get_auth_request con 0x7f19d8036180 auth_method 0 2024-09-16T06:47:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.692+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f19e8014820 con 0x7f19f81352c0 2024-09-16T06:47:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.692+0000 7f19effff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:21.704 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.692+0000 7f19effff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.872+0000 7f19fcce3700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.872+0000 7f19fcce3700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.872+0000 7f19fcce3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:21.879 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:21.872+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f19f8130c10 con 0x7f19f81352c0 2024-09-16T06:47:22.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.024+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v253) v1 ==== 106+0+3 (unknown 778697360 0 944643765) 0x7f19f8130c10 con 0x7f19f81352c0 2024-09-16T06:47:22.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.024+0000 7f19effff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:22.028 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.024+0000 7f19effff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19fcce3700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19fcce3700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19fcce3700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:22.145 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"} v 0) v1 -- 0x7f19f8130c10 con 0x7f19f81352c0 2024-09-16T06:47:22.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19effff700 1 -- 172.21.15.189:0/3964739913 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}]=0 v253) v1 ==== 106+0+3 (unknown 778697360 0 944643765) 0x7f19f8130c10 con 0x7f19f81352c0 2024-09-16T06:47:22.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19effff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0"}] 2024-09-16T06:47:22.147 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.140+0000 7f19effff700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:22.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 >> v1:172.21.15.189:6812/3098810674 conn(0x7f19d8036180 legacy=0x7f19d8038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:22.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 10 monclient: shutdown 2024-09-16T06:47:22.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:22.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 >> v1:172.21.15.189:6789/0 conn(0x7f19f81352c0 legacy=0x7f19f81ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:22.149 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 shutdown_connections 2024-09-16T06:47:22.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 >> 172.21.15.189:0/3964739913 conn(0x7f19f80a4550 msgr2=0x7f19f8133ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:22.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 shutdown_connections 2024-09-16T06:47:22.150 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:22.144+0000 7f19fcce3700 1 -- 172.21.15.189:0/3964739913 wait complete. 2024-09-16T06:47:22.160 INFO:tasks.workunit.client.0.smithi189.stderr:+ na2=12 2024-09-16T06:47:22.160 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T06:47:22.161 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T06:47:22.161 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:47:22.161 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:47:22.176 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-16T06:47:22.178 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-16T06:47:22.178 DEBUG:teuthology.orchestra.run.smithi189:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-16T06:47:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-16T06:47:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:+ exit_on_error=1 2024-09-16T06:47:22.191 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ ! -z '' ]] 2024-09-16T06:47:22.192 INFO:tasks.workunit.client.0.smithi189.stderr:++ uname 2024-09-16T06:47:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:+ '[' Linux = FreeBSD ']' 2024-09-16T06:47:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:+ ETIMEDOUT=110 2024-09-16T06:47:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-16T06:47:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-16T06:47:22.193 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:47:22.196 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T06:47:22.196 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T06:47:22.197 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T06:47:22.722 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T06:47:22.723 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T06:47:31.971 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:47:31.971 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:47:31.971 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:47:31.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:47:31.974 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:47:31.974 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:47:31.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: monmap: 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:32.082085+0000 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:32.082085+0000 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.079 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 1 Processor -- start 2024-09-16T06:47:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 1 -- start start 2024-09-16T06:47:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: init 2024-09-16T06:47:32.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient(hunting): picked mon.noname-b con 0x7f1428070ac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.078+0000 7f142e199700 10 monclient(hunting): picked mon.noname-a con 0x7f1428070ea0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): picked mon.noname-c con 0x7f142813b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1428072610 con 0x7f1428070ea0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142809f560 con 0x7f142813b980 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1428072330 con 0x7f1428070ac0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): authenticate will time out at 1399.216069s 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142c996700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1428070ea0 0x7f14280712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54302/0 (socket says 172.21.15.76:54302) 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142c996700 1 -- 172.21.15.76:0/526380988 learned_addr learned my addr 172.21.15.76:0/526380988 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142d197700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1428070ac0 0x7f1428097bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59886/0 (socket says 172.21.15.76:59886) 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219404671 0 0) 0x7f1428072330 con 0x7f1428070ac0 2024-09-16T06:47:32.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): my global_id is 194534 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14140035d0 con 0x7f1428070ac0 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623487530 0 0) 0x7f142809f560 con 0x7f142813b980 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient(hunting): my global_id is 204295 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1428072330 con 0x7f142813b980 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336184445 0 0) 0x7f14140035d0 con 0x7f1428070ac0 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 >> v1:172.21.15.189:6789/0 conn(0x7f142813b980 legacy=0x7f142813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 >> v1:172.21.15.76:6789/0 conn(0x7f1428070ea0 legacy=0x7f14280712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 monclient: found mon.noname-b 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14280974f0 con 0x7f1428070ac0 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1424001610 con 0x7f1428070ac0 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: dump: 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.085 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 5 monclient: authenticate success, global_id 194534 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14240018f0 con 0x7f1428070ac0 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 1 -- 172.21.15.76:0/526380988 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1424001c80 con 0x7f1428070ac0 2024-09-16T06:47:32.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 4 set_mon_vals no callback set 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f141ffff700 10 monclient: dump: 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: shutdown 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- 172.21.15.76:0/526380988 >> v1:172.21.15.76:6790/0 conn(0x7f1428070ac0 legacy=0x7f1428097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- 172.21.15.76:0/526380988 shutdown_connections 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- 172.21.15.76:0/526380988 >> 172.21.15.76:0/526380988 conn(0x7f14280a0030 msgr2=0x7f14280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- 172.21.15.76:0/526380988 shutdown_connections 2024-09-16T06:47:32.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- 172.21.15.76:0/526380988 wait complete. 2024-09-16T06:47:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: monmap: 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 Processor -- start 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- start start 2024-09-16T06:47:32.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: init 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): picked mon.noname-c con 0x7f1428070ac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): picked mon.noname-a con 0x7f1428070ea0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): picked mon.noname-b con 0x7f142813b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142813a9f0 con 0x7f1428070ea0 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1428137770 con 0x7f142813b980 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142809a540 con 0x7f1428070ac0 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.082+0000 7f142e199700 10 monclient(hunting): authenticate will time out at 1399.219929s 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142d197700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1428070ac0 0x7f1428136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59906/0 (socket says 172.21.15.76:59906) 2024-09-16T06:47:32.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142d197700 1 -- 172.21.15.76:0/3272484354 learned_addr learned my addr 172.21.15.76:0/3272484354 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142c996700 1 --1- 172.21.15.76:0/3272484354 >> v1:172.21.15.76:6789/0 conn(0x7f1428070ea0 0x7f1428137060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54312/0 (socket says 172.21.15.76:54312) 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286413990 0 0) 0x7f142809a540 con 0x7f1428070ac0 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): my global_id is 194537 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f140c0035d0 con 0x7f1428070ac0 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402860972 0 0) 0x7f1428137770 con 0x7f142813b980 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): my global_id is 204298 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142809a540 con 0x7f142813b980 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971071875 0 0) 0x7f140c0035d0 con 0x7f1428070ac0 2024-09-16T06:47:32.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1428137770 con 0x7f1428070ac0 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14240015d0 con 0x7f1428070ac0 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): dump: 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1671260278 0 0) 0x7f142809a540 con 0x7f142813b980 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f140c0035d0 con 0x7f142813b980 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1420001c30 con 0x7f142813b980 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient(hunting): dump: 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.096 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2056174457 0 0) 0x7f1428137770 con 0x7f1428070ac0 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 >> v1:172.21.15.189:6789/0 conn(0x7f142813b980 legacy=0x7f14281d2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 >> v1:172.21.15.76:6789/0 conn(0x7f1428070ea0 legacy=0x7f1428137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 monclient: found mon.c 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428099c50 con 0x7f1428070ac0 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 5 monclient: authenticate success, global_id 194537 2024-09-16T06:47:32.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _renew_subs 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428138880 con 0x7f1428070ac0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _renew_subs 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14281394b0 con 0x7f1428070ac0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142809d630 con 0x7f1428070ac0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1424004be0 con 0x7f1428070ac0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1424004f70 con 0x7f1428070ac0 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141dffb700 10 monclient: dump: 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.086+0000 7f141f7fe700 4 set_mon_vals no callback set 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.090+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1424005150 con 0x7f1428070ac0 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.090+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f1424043f20 con 0x7f1428070ac0 2024-09-16T06:47:32.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.090+0000 7f142c996700 10 monclient: get_auth_request con 0x7f140c0362c0 auth_method 0 2024-09-16T06:47:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.094+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1424014ae0 con 0x7f1428070ac0 2024-09-16T06:47:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.094+0000 7f141dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.094+0000 7f141dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f142e199700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f142e199700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f142e199700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f14281353c0 con 0x7f1428070ac0 2024-09-16T06:47:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f141dffb700 1 -- 172.21.15.76:0/3272484354 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3806989343) 0x7f1424012f50 con 0x7f1428070ac0 2024-09-16T06:47:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f141dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.326+0000 7f141dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:32.326 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 >> v1:172.21.15.189:6812/3098810674 conn(0x7f140c0362c0 legacy=0x7f140c038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 10 monclient: shutdown 2024-09-16T06:47:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 >> v1:172.21.15.76:6790/0 conn(0x7f1428070ac0 legacy=0x7f1428136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 shutdown_connections 2024-09-16T06:47:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 >> 172.21.15.76:0/3272484354 conn(0x7f14280a0030 msgr2=0x7f142809a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 shutdown_connections 2024-09-16T06:47:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.330+0000 7f142e199700 1 -- 172.21.15.76:0/3272484354 wait complete. 2024-09-16T06:47:32.346 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:47:32.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:47:32.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:47:32.441 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:47:32.441 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:47:32.441 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:32.441 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.bazar 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: monmap: 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:32.462494+0000 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:32.462494+0000 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.459 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 1 Processor -- start 2024-09-16T06:47:32.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 1 -- start start 2024-09-16T06:47:32.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: init 2024-09-16T06:47:32.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.458+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-a con 0x7fa27c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-b con 0x7fa27c099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-c con 0x7fa27c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c098990 con 0x7fa27c097b60 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c097f40 con 0x7fa27c139500 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c09d610 con 0x7fa27c099090 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient(hunting): authenticate will time out at 1399.596132s 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa27c097b60 0x7fa27c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54328/0 (socket says 172.21.15.76:54328) 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27bfff700 1 -- 172.21.15.76:0/668387638 learned_addr learned my addr 172.21.15.76:0/668387638 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27b7fe700 1 --1- 172.21.15.76:0/668387638 >> v1:172.21.15.76:6790/0 conn(0x7fa27c099090 0x7fa27c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59920/0 (socket says 172.21.15.76:59920) 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911268773 0 0) 0x7fa27c09d610 con 0x7fa27c099090 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): my global_id is 194540 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2680035d0 con 0x7fa27c099090 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902576037 0 0) 0x7fa27c098990 con 0x7fa27c097b60 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): my global_id is 214101 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c09d610 con 0x7fa27c097b60 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054541342 0 0) 0x7fa27c097f40 con 0x7fa27c139500 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient(hunting): my global_id is 204301 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c098990 con 0x7fa27c139500 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754725883 0 0) 0x7fa27c09d610 con 0x7fa27c097b60 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 >> v1:172.21.15.76:6790/0 conn(0x7fa27c099090 legacy=0x7fa27c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 >> v1:172.21.15.189:6789/0 conn(0x7fa27c139500 legacy=0x7fa27c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 monclient: found mon.noname-a 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0986b0 con 0x7fa27c097b60 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 5 monclient: authenticate success, global_id 214101 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa274001970 con 0x7fa27c097b60 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: dump: 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.467 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa274001c10 con 0x7fa27c097b60 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 4 set_mon_vals no callback set 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 1 -- 172.21.15.76:0/668387638 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa274001fa0 con 0x7fa27c097b60 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa27affd700 10 monclient: dump: 2024-09-16T06:47:32.468 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient: shutdown 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- 172.21.15.76:0/668387638 >> v1:172.21.15.76:6789/0 conn(0x7fa27c097b60 legacy=0x7fa27c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- 172.21.15.76:0/668387638 shutdown_connections 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- 172.21.15.76:0/668387638 >> 172.21.15.76:0/668387638 conn(0x7fa27c0a0030 msgr2=0x7fa27c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- 172.21.15.76:0/668387638 shutdown_connections 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 1 -- 172.21.15.76:0/668387638 wait complete. 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.462+0000 7fa2815c3700 10 monclient: monmap: 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.470 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 Processor -- start 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- start start 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: init 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-c con 0x7fa27c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-a con 0x7fa27c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient(hunting): picked mon.noname-b con 0x7fa27c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c06dd00 con 0x7fa27c099090 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c06e620 con 0x7fa27c139500 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c09a540 con 0x7fa27c097b60 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa27bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa27c097b60 0x7fa27c06cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59926/0 (socket says 172.21.15.76:59926) 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa27bfff700 1 -- 172.21.15.76:0/2340827280 learned_addr learned my addr 172.21.15.76:0/2340827280 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient(hunting): authenticate will time out at 1399.601220s 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982713941 0 0) 0x7fa27c09a540 con 0x7fa27c097b60 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): my global_id is 194543 2024-09-16T06:47:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25c0035d0 con 0x7fa27c097b60 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176988206 0 0) 0x7fa27c06dd00 con 0x7fa27c099090 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): my global_id is 214104 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c09a540 con 0x7fa27c099090 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283149686 0 0) 0x7fa27c06e620 con 0x7fa27c139500 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): my global_id is 204304 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c06dd00 con 0x7fa27c139500 2024-09-16T06:47:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 315238329 0 0) 0x7fa25c0035d0 con 0x7fa27c097b60 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa27c06e620 con 0x7fa27c097b60 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa274001f60 con 0x7fa27c097b60 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): dump: 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2846946889 0 0) 0x7fa27c09a540 con 0x7fa27c099090 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa25c0035d0 con 0x7fa27c099090 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa264001190 con 0x7fa27c099090 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient(hunting): dump: 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.479 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 308596111 0 0) 0x7fa27c06e620 con 0x7fa27c097b60 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.189:6789/0 conn(0x7fa27c139500 legacy=0x7fa27c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6789/0 conn(0x7fa27c099090 legacy=0x7fa27c06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 monclient: found mon.c 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c06fa40 con 0x7fa27c097b60 2024-09-16T06:47:32.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 5 monclient: authenticate success, global_id 194543 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _renew_subs 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c06fd20 con 0x7fa27c097b60 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _renew_subs 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c06e010 con 0x7fa27c097b60 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa274004f80 con 0x7fa27c097b60 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa274005310 con 0x7fa27c097b60 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa27a7fc700 4 set_mon_vals no callback set 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa278ff9700 10 monclient: dump: 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.466+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa260006590 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa27400fd80 con 0x7fa27c097b60 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7fa274044240 con 0x7fa27c097b60 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786809855 0 0) 0x7fa260006590 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient(hunting): my global_id is 214107 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c06e620 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa280dc2700 10 monclient: get_auth_request con 0x7fa25c0361b0 auth_method 0 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 114260730 0 0) 0x7fa27c06e620 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa260006590 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa264004c20 con 0x7fa260002830 2024-09-16T06:47:32.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient: dump: 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3612688884 0 0) 0x7fa260006590 con 0x7fa260002830 2024-09-16T06:47:32.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa26000a190 con 0x7fa260002830 2024-09-16T06:47:32.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa264004f10 con 0x7fa260002830 2024-09-16T06:47:32.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6789/0 conn(0x7fa260002830 legacy=0x7fa260004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.470+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6789/0 conn(0x7fa260002830 legacy=0x7fa260004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.478+0000 7fa2815c3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.478+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa260009e50 con 0x7fa260009170 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006262745 0 0) 0x7fa260009e50 con 0x7fa260009170 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient(hunting): my global_id is 214110 2024-09-16T06:47:32.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa260006590 con 0x7fa260009170 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 995486526 0 0) 0x7fa260006590 con 0x7fa260009170 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa260009e50 con 0x7fa260009170 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa2640054e0 con 0x7fa260009170 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient: dump: 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 835041375 0 0) 0x7fa260009e50 con 0x7fa260009170 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa26000c340 con 0x7fa260009170 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 227751875) 0x7fa264005940 con 0x7fa260009170 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6789/0 conn(0x7fa260009170 legacy=0x7fa260009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa278ff9700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6789/0 conn(0x7fa260009170 legacy=0x7fa260009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:32.488 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:32.489 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.490 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:32.491 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:32.492 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:32.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.494 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.495 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.498 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:32.499 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa25c0361b0 legacy=0x7fa25c038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 10 monclient: shutdown 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 >> v1:172.21.15.76:6790/0 conn(0x7fa27c097b60 legacy=0x7fa27c06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa280dc2700 1 -- 172.21.15.76:0/2340827280 reap_dead start 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 shutdown_connections 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 >> 172.21.15.76:0/2340827280 conn(0x7fa27c0a0030 msgr2=0x7fa27c0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 shutdown_connections 2024-09-16T06:47:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.482+0000 7fa2815c3700 1 -- 172.21.15.76:0/2340827280 wait complete. 2024-09-16T06:47:32.502 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:32.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:47:32.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 10 monclient: monmap: 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:47:32.554458+0000 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:47:32.554458+0000 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.554 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 1 Processor -- start 2024-09-16T06:47:32.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 1 -- start start 2024-09-16T06:47:32.555 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.548+0000 7f1284ceb700 10 monclient: init 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-c con 0x7f128009f5e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-b con 0x7f1280071b40 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-a con 0x7f12800a00d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128009ce70 con 0x7f12800a00d0 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128009c5d0 con 0x7f128009f5e0 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128009c8b0 con 0x7f1280071b40 2024-09-16T06:47:32.558 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient(hunting): authenticate will time out at 1405.068807s 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f128009f5e0 0x7f128009f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51916/0 (socket says 172.21.15.189:51916) 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127e59c700 1 -- 172.21.15.189:0/242684369 learned_addr learned my addr 172.21.15.189:0/242684369 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765833983 0 0) 0x7f128009c5d0 con 0x7f128009f5e0 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient(hunting): my global_id is 204307 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c0035d0 con 0x7f128009f5e0 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749439627 0 0) 0x7f126c0035d0 con 0x7f128009f5e0 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 >> v1:172.21.15.76:6790/0 conn(0x7f1280071b40 legacy=0x7f12801373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 >> v1:172.21.15.76:6789/0 conn(0x7f12800a00d0 legacy=0x7f128013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.559 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 monclient: found mon.noname-c 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128009cb90 con 0x7f128009f5e0 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 5 monclient: authenticate success, global_id 204307 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1270001990 con 0x7f128009f5e0 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.560 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: dump: 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1270001bb0 con 0x7f128009f5e0 2024-09-16T06:47:32.561 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 4 set_mon_vals no callback set 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 1 -- 172.21.15.189:0/242684369 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1270001f80 con 0x7f128009f5e0 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f127dd9b700 10 monclient: dump: 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.562 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.563 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.563 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.567 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: shutdown 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- 172.21.15.189:0/242684369 >> v1:172.21.15.189:6789/0 conn(0x7f128009f5e0 legacy=0x7f128009f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- 172.21.15.189:0/242684369 shutdown_connections 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- 172.21.15.189:0/242684369 >> 172.21.15.189:0/242684369 conn(0x7f12800a08b0 msgr2=0x7f12800a0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- 172.21.15.189:0/242684369 shutdown_connections 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 1 -- 172.21.15.189:0/242684369 wait complete. 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.552+0000 7f1284ceb700 10 monclient: monmap: 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:47:32.568 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 Processor -- start 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- start start 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: init 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-b con 0x7f1280071b40 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-c con 0x7f128009f5e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.569 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient(hunting): picked mon.noname-a con 0x7f12800a00d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12801bfa00 con 0x7f12800a00d0 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800722b0 con 0x7f1280071b40 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12801a41c0 con 0x7f128009f5e0 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient(hunting): authenticate will time out at 1405.073582s 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f127e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1280071b40 0x7f12801bf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51920/0 (socket says 172.21.15.189:51920) 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f127e59c700 1 -- 172.21.15.189:0/63340975 learned_addr learned my addr 172.21.15.189:0/63340975 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511182813 0 0) 0x7f12800722b0 con 0x7f1280071b40 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): my global_id is 204310 2024-09-16T06:47:32.570 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12600035d0 con 0x7f1280071b40 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540453493 0 0) 0x7f12801bfa00 con 0x7f12800a00d0 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): my global_id is 214116 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12800722b0 con 0x7f12800a00d0 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931930369 0 0) 0x7f12801a41c0 con 0x7f128009f5e0 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): my global_id is 194549 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12801bfa00 con 0x7f128009f5e0 2024-09-16T06:47:32.571 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510306740 0 0) 0x7f12600035d0 con 0x7f1280071b40 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12801a41c0 con 0x7f1280071b40 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1270001f00 con 0x7f1280071b40 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.572 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426740373 0 0) 0x7f12800722b0 con 0x7f12800a00d0 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12600035d0 con 0x7f12800a00d0 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3123971269 0 0) 0x7f12801a41c0 con 0x7f1280071b40 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 >> v1:172.21.15.76:6790/0 conn(0x7f128009f5e0 legacy=0x7f12801ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 >> v1:172.21.15.76:6789/0 conn(0x7f12800a00d0 legacy=0x7f12801cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 monclient: found mon.b 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12800a0ee0 con 0x7f1280071b40 2024-09-16T06:47:32.573 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 5 monclient: authenticate success, global_id 204310 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _renew_subs 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.576 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12801a6c50 con 0x7f1280071b40 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _renew_subs 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12801a5700 con 0x7f1280071b40 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1270004f40 con 0x7f1280071b40 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1270005250 con 0x7f1280071b40 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f127d59a700 4 set_mon_vals no callback set 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.556+0000 7f12777fe700 10 monclient: dump: 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.577 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f12700053b0 con 0x7f1280071b40 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f1284ceb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f1284ceb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f1284ceb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.578 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264005100 con 0x7f1280071b40 2024-09-16T06:47:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f1270044320 con 0x7f1280071b40 2024-09-16T06:47:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.560+0000 7f1275d9b700 10 monclient: get_auth_request con 0x7f1260036450 auth_method 0 2024-09-16T06:47:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.568+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1270014d20 con 0x7f1280071b40 2024-09-16T06:47:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.568+0000 7f12777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.579 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.568+0000 7f12777fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: monmap: 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:32.629036+0000 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:32.629036+0000 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.626 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 1 Processor -- start 2024-09-16T06:47:32.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 1 -- start start 2024-09-16T06:47:32.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: init 2024-09-16T06:47:32.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:32.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-b con 0x7f876809e800 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-c con 0x7f876809f2f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8768138130 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876809f9a0 con 0x7f8768138130 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876809b740 con 0x7f876809f2f0 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.626+0000 7f8767fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876809ba20 con 0x7f876809e800 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 10 monclient(hunting): authenticate will time out at 1399.764050s 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8766ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f876809e800 0x7f876809ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59928/0 (socket says 172.21.15.76:59928) 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8766ffd700 1 -- 172.21.15.76:0/4134982651 learned_addr learned my addr 172.21.15.76:0/4134982651 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87677fe700 1 --1- 172.21.15.76:0/4134982651 >> v1:172.21.15.76:6789/0 conn(0x7f8768138130 0x7f876813a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54370/0 (socket says 172.21.15.76:54370) 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754393514 0 0) 0x7f876809f9a0 con 0x7f8768138130 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): my global_id is 214119 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87540046e0 con 0x7f8768138130 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790315632 0 0) 0x7f876809b740 con 0x7f876809f2f0 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): my global_id is 204313 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876809f9a0 con 0x7f876809f2f0 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127569454 0 0) 0x7f876809ba20 con 0x7f876809e800 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient(hunting): my global_id is 194552 2024-09-16T06:47:32.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876809b740 con 0x7f876809e800 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842943369 0 0) 0x7f876809b740 con 0x7f876809e800 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 >> v1:172.21.15.189:6789/0 conn(0x7f876809f2f0 legacy=0x7f8768136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 >> v1:172.21.15.76:6789/0 conn(0x7f8768138130 legacy=0x7f876813a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 monclient: found mon.noname-b 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876809bd00 con 0x7f876809e800 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758001530 con 0x7f876809e800 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 5 monclient: authenticate success, global_id 194552 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: dump: 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8758001740 con 0x7f876809e800 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 1 -- 172.21.15.76:0/4134982651 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758001ad0 con 0x7f876809e800 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 4 set_mon_vals no callback set 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f87667fc700 10 monclient: dump: 2024-09-16T06:47:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 10 monclient: shutdown 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 1 -- 172.21.15.76:0/4134982651 >> v1:172.21.15.76:6790/0 conn(0x7f876809e800 legacy=0x7f876809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 1 -- 172.21.15.76:0/4134982651 shutdown_connections 2024-09-16T06:47:32.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 1 -- 172.21.15.76:0/4134982651 >> 172.21.15.76:0/4134982651 conn(0x7f87680a4430 msgr2=0x7f87680a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 1 -- 172.21.15.76:0/4134982651 shutdown_connections 2024-09-16T06:47:32.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.630+0000 7f8767fff700 1 -- 172.21.15.76:0/4134982651 wait complete. 2024-09-16T06:47:32.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: monmap: 2024-09-16T06:47:32.643 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.643 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.643 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 Processor -- start 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- start start 2024-09-16T06:47:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: init 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-b con 0x7f876809e800 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-a con 0x7f876809f2f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8768138130 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876811c280 con 0x7f876809f2f0 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8768136840 con 0x7f876809e800 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8768137610 con 0x7f8768138130 2024-09-16T06:47:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient(hunting): authenticate will time out at 1399.770118s 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f87677fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8768138130 0x7f87681d6380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59930/0 (socket says 172.21.15.76:59930) 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f87677fe700 1 -- 172.21.15.76:0/552561175 learned_addr learned my addr 172.21.15.76:0/552561175 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707360950 0 0) 0x7f8768137610 con 0x7f8768138130 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): my global_id is 194555 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87480035d0 con 0x7f8768138130 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304255750 0 0) 0x7f876811c280 con 0x7f876809f2f0 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): my global_id is 214122 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768137610 con 0x7f876809f2f0 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476173282 0 0) 0x7f8768136840 con 0x7f876809e800 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): my global_id is 204316 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876811c280 con 0x7f876809e800 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030415820 0 0) 0x7f87480035d0 con 0x7f8768138130 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8768136840 con 0x7f8768138130 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87600016c0 con 0x7f8768138130 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537572661 0 0) 0x7f8768137610 con 0x7f876809f2f0 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87480035d0 con 0x7f876809f2f0 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87500019d0 con 0x7f876809f2f0 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952710093 0 0) 0x7f876811c280 con 0x7f876809e800 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8768137610 con 0x7f876809e800 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87580014b0 con 0x7f876809e800 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1732898231 0 0) 0x7f8768136840 con 0x7f8768138130 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.189:6789/0 conn(0x7f876809e800 legacy=0x7f8768135a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6789/0 conn(0x7f876809f2f0 legacy=0x7f8768136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 monclient: found mon.c 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680a2520 con 0x7f8768138130 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f875f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 5 monclient: authenticate success, global_id 194555 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _renew_subs 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876809b0f0 con 0x7f8768138130 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _renew_subs 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.634+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f876811bcc0 con 0x7f8768138130 2024-09-16T06:47:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8760001930 con 0x7f8768138130 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87600020d0 con 0x7f8768138130 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f8765ffb700 4 set_mon_vals no callback set 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: dump: 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f8767fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f874c006590 con 0x7f874c002830 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f876000f970 con 0x7f8768138130 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f8760044170 con 0x7f8768138130 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875ffff700 10 monclient: get_auth_request con 0x7f8748036540 auth_method 0 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294548024 0 0) 0x7f874c006590 con 0x7f874c002830 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient(hunting): my global_id is 194558 2024-09-16T06:47:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8768136840 con 0x7f874c002830 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1858842828 0 0) 0x7f8768136840 con 0x7f874c002830 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f874c006590 con 0x7f874c002830 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8758004a80 con 0x7f874c002830 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: dump: 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 450033778 0 0) 0x7f874c006590 con 0x7f874c002830 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f874c00a190 con 0x7f874c002830 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8758004d20 con 0x7f874c002830 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6790/0 conn(0x7f874c002830 legacy=0x7f874c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.638+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6790/0 conn(0x7f874c002830 legacy=0x7f874c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.646+0000 7f8767fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.646+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f874c009e50 con 0x7f874c009170 2024-09-16T06:47:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214185637 0 0) 0x7f874c009e50 con 0x7f874c009170 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient(hunting): my global_id is 194561 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c006590 con 0x7f874c009170 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2060587106 0 0) 0x7f874c006590 con 0x7f874c009170 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f874c009e50 con 0x7f874c009170 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87580052b0 con 0x7f874c009170 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient: dump: 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 767528125 0 0) 0x7f874c009e50 con 0x7f874c009170 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f874c00c340 con 0x7f874c009170 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1091980397) 0x7f87580066a0 con 0x7f874c009170 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6790/0 conn(0x7f874c009170 legacy=0x7f874c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f875f7fe700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6790/0 conn(0x7f874c009170 legacy=0x7f874c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.662 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:32.663 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.664 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.665 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:32.666 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:32.667 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.668 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.669 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.670 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:32.671 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.674 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.675 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:32.677 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8748036540 legacy=0x7f8748038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 10 monclient: shutdown 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 >> v1:172.21.15.76:6790/0 conn(0x7f8768138130 legacy=0x7f87681d6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f87677fe700 1 -- 172.21.15.76:0/552561175 reap_dead start 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f87677fe700 1 -- 172.21.15.76:0/552561175 reap_dead start 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 shutdown_connections 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 >> 172.21.15.76:0/552561175 conn(0x7f87680a4430 msgr2=0x7f876809c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 shutdown_connections 2024-09-16T06:47:32.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.650+0000 7f8767fff700 1 -- 172.21.15.76:0/552561175 wait complete. 2024-09-16T06:47:32.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:32.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:47:32.680 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:47:32.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: monmap: 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:32.806041+0000 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:32.806041+0000 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.803 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 1 Processor -- start 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 1 -- start start 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: init 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-a con 0x7f0874132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-b con 0x7f087406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-c con 0x7f087406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087406c280 con 0x7f0874132330 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087406bfa0 con 0x7f087406cf50 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087409ace0 con 0x7f087406d480 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.802+0000 7f08788f7700 10 monclient(hunting): authenticate will time out at 1399.939963s 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08737fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0874132330 0x7f0874132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54386/0 (socket says 172.21.15.76:54386) 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08737fe700 1 -- 172.21.15.76:0/610126786 learned_addr learned my addr 172.21.15.76:0/610126786 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882811637 0 0) 0x7f087406c280 con 0x7f0874132330 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): my global_id is 214125 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08580035d0 con 0x7f0874132330 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015887213 0 0) 0x7f087406bfa0 con 0x7f087406cf50 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): my global_id is 204319 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087406c280 con 0x7f087406cf50 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434602546 0 0) 0x7f087409ace0 con 0x7f087406d480 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient(hunting): my global_id is 194564 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087406bfa0 con 0x7f087406d480 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3187230443 0 0) 0x7f08580035d0 con 0x7f0874132330 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 >> v1:172.21.15.76:6790/0 conn(0x7f087406d480 legacy=0x7f087413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 >> v1:172.21.15.189:6789/0 conn(0x7f087406cf50 legacy=0x7f087413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 monclient: found mon.noname-a 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087409ae70 con 0x7f0874132330 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0864001610 con 0x7f0874132330 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 5 monclient: authenticate success, global_id 214125 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: dump: 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.810 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08640018f0 con 0x7f0874132330 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 1 -- 172.21.15.76:0/610126786 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0864001ae0 con 0x7f0874132330 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 4 set_mon_vals no callback set 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08727fc700 10 monclient: dump: 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.811 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.812 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.812 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.800+0000 7f1284ceb700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.800+0000 7f1284ceb700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.800+0000 7f1284ceb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.813 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.800+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f12640053b0 con 0x7f1280071b40 2024-09-16T06:47:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 10 monclient: shutdown 2024-09-16T06:47:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 1 -- 172.21.15.76:0/610126786 >> v1:172.21.15.76:6789/0 conn(0x7f0874132330 legacy=0x7f0874132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 1 -- 172.21.15.76:0/610126786 shutdown_connections 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 1 -- 172.21.15.76:0/610126786 >> 172.21.15.76:0/610126786 conn(0x7f08740a41c0 msgr2=0x7f08740a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 1 -- 172.21.15.76:0/610126786 shutdown_connections 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 1 -- 172.21.15.76:0/610126786 wait complete. 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.806+0000 7f08788f7700 10 monclient: monmap: 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 1 Processor -- start 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 1 -- start start 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient: init 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-b con 0x7f087406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-c con 0x7f087406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient(hunting): picked mon.noname-a con 0x7f087413dd40 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087413e1d0 con 0x7f087413dd40 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0874131800 con 0x7f087406cf50 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08741363d0 con 0x7f087406d480 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f0872ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f087406d480 0x7f0874135cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:59972/0 (socket says 172.21.15.76:59972) 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f0872ffd700 1 -- 172.21.15.76:0/3110703349 learned_addr learned my addr 172.21.15.76:0/3110703349 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.810+0000 7f08788f7700 10 monclient(hunting): authenticate will time out at 1399.946103s 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845855764 0 0) 0x7f08741363d0 con 0x7f087406d480 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): my global_id is 194567 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f085c0035d0 con 0x7f087406d480 2024-09-16T06:47:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1803015018 0 0) 0x7f085c0035d0 con 0x7f087406d480 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08741363d0 con 0x7f087406d480 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0860001930 con 0x7f087406d480 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): dump: 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2342876534 0 0) 0x7f08741363d0 con 0x7f087406d480 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6789/0 conn(0x7f087406cf50 legacy=0x7f08741355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.76:6789/0 conn(0x7f087413dd40 legacy=0x7f08741d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 monclient: found mon.c 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087409e8b0 con 0x7f087406d480 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 5 monclient: authenticate success, global_id 194567 2024-09-16T06:47:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 10 monclient: _renew_subs 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08741371c0 con 0x7f087406d480 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 10 monclient: _renew_subs 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0874138c80 con 0x7f087406d480 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0860001d00 con 0x7f087406d480 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0860003b10 con 0x7f087406d480 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f0871ffb700 4 set_mon_vals no callback set 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 10 monclient: dump: 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0850006590 con 0x7f0850002830 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.814+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0860003cf0 con 0x7f087406d480 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f0860043fe0 con 0x7f087406d480 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f0873fff700 10 monclient: get_auth_request con 0x7f085c036080 auth_method 0 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620786115 0 0) 0x7f0850006590 con 0x7f0850002830 2024-09-16T06:47:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient(hunting): my global_id is 204322 2024-09-16T06:47:32.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08741363d0 con 0x7f0850002830 2024-09-16T06:47:32.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.812+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v26) v1 ==== 82+0+0 (unknown 1890204813 0 0) 0x7f1270013190 con 0x7f1280071b40 2024-09-16T06:47:32.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.812+0000 7f12777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.825 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.812+0000 7f12777fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:32.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1847736103 0 0) 0x7f08741363d0 con 0x7f0850002830 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0850006590 con 0x7f0850002830 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0864001430 con 0x7f0850002830 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient: dump: 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.826 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2170709870 0 0) 0x7f0850006590 con 0x7f0850002830 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f085000a190 con 0x7f0850002830 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0864001610 con 0x7f0850002830 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6789/0 conn(0x7f0850002830 legacy=0x7f0850004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.818+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6789/0 conn(0x7f0850002830 legacy=0x7f0850004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.826+0000 7f08788f7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.826+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0850009e50 con 0x7f0850009170 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083778329 0 0) 0x7f0850009e50 con 0x7f0850009170 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient(hunting): my global_id is 204325 2024-09-16T06:47:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0850006590 con 0x7f0850009170 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2544260872 0 0) 0x7f0850006590 con 0x7f0850009170 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0850009e50 con 0x7f0850009170 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0864004ca0 con 0x7f0850009170 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient: dump: 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4059866624 0 0) 0x7f0850009e50 con 0x7f0850009170 2024-09-16T06:47:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f085000c340 con 0x7f0850009170 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 2655666955) 0x7f0864005fe0 con 0x7f0850009170 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6789/0 conn(0x7f0850009170 legacy=0x7f0850009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f086bfff700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6789/0 conn(0x7f0850009170 legacy=0x7f0850009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:32.832 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:32.834 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:32.836 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:32.837 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.843 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.844 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:32.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:32.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:32.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:32.846 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:32.846 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:32.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.189:6812/3098810674 conn(0x7f085c036080 legacy=0x7f085c038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 10 monclient: shutdown 2024-09-16T06:47:32.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 >> v1:172.21.15.76:6790/0 conn(0x7f087406d480 legacy=0x7f0874135cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f0873fff700 1 -- 172.21.15.76:0/3110703349 reap_dead start 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f0873fff700 1 -- 172.21.15.76:0/3110703349 reap_dead start 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 shutdown_connections 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 >> 172.21.15.76:0/3110703349 conn(0x7f08740a41c0 msgr2=0x7f08740a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 shutdown_connections 2024-09-16T06:47:32.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.830+0000 7f08788f7700 1 -- 172.21.15.76:0/3110703349 wait complete. 2024-09-16T06:47:32.849 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:32.850 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:47:32.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:47:32.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.962+0000 7f92bf59e700 10 monclient: get_monmap_and_config 2024-09-16T06:47:32.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.962+0000 7f92bf59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.962+0000 7f92bf59e700 10 monclient: monmap: 2024-09-16T06:47:32.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:32.966945+0000 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:32.966945+0000 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:32.964 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.962+0000 7f92bf59e700 1 Processor -- start 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- start start 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient: init 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f92c0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f92c006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f92c006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c006c280 con 0x7f92c006d480 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c006bfa0 con 0x7f92c0132330 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c009ace0 con 0x7f92c006cf50 2024-09-16T06:47:32.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient(hunting): authenticate will time out at 1400.100937s 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92b5d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f92c006d480 0x7f92c013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54402/0 (socket says 172.21.15.76:54402) 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92b5d9b700 1 -- 172.21.15.76:0/2090326767 learned_addr learned my addr 172.21.15.76:0/2090326767 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507020442 0 0) 0x7f92c009ace0 con 0x7f92c006cf50 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): my global_id is 194570 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac0035d0 con 0x7f92c006cf50 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637588058 0 0) 0x7f92c006c280 con 0x7f92c006d480 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): my global_id is 214131 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c009ace0 con 0x7f92c006d480 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520078662 0 0) 0x7f92c006bfa0 con 0x7f92c0132330 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient(hunting): my global_id is 204328 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c006c280 con 0x7f92c0132330 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822703432 0 0) 0x7f92ac0035d0 con 0x7f92c006cf50 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 >> v1:172.21.15.189:6789/0 conn(0x7f92c0132330 legacy=0x7f92c0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 >> v1:172.21.15.76:6789/0 conn(0x7f92c006d480 legacy=0x7f92c013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 monclient: found mon.noname-b 2024-09-16T06:47:32.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c009ae70 con 0x7f92c006cf50 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f92b8001990 con 0x7f92c006cf50 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: dump: 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.970 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 5 monclient: authenticate success, global_id 194570 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92b8001c70 con 0x7f92c006cf50 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 1 -- 172.21.15.76:0/2090326767 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f92b8001ea0 con 0x7f92c006cf50 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 4 set_mon_vals no callback set 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bdd9b700 10 monclient: dump: 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.972 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 10 monclient: shutdown 2024-09-16T06:47:32.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- 172.21.15.76:0/2090326767 >> v1:172.21.15.76:6790/0 conn(0x7f92c006cf50 legacy=0x7f92c013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- 172.21.15.76:0/2090326767 shutdown_connections 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- 172.21.15.76:0/2090326767 >> 172.21.15.76:0/2090326767 conn(0x7f92c00a41c0 msgr2=0x7f92c00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- 172.21.15.76:0/2090326767 shutdown_connections 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.966+0000 7f92bf59e700 1 -- 172.21.15.76:0/2090326767 wait complete. 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient: monmap: 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:32.978 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 1 Processor -- start 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 1 -- start start 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient: init 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f92c006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f92c006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f92c0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:32.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c012c4a0 con 0x7f92c006d480 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c01ca600 con 0x7f92c0132330 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c012ff10 con 0x7f92c006cf50 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92bf59e700 10 monclient(hunting): authenticate will time out at 1400.107071s 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92be59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f92c006cf50 0x7f92c01c97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60002/0 (socket says 172.21.15.76:60002) 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b5d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f92c006d480 0x7f92c01c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:54416/0 (socket says 172.21.15.76:54416) 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92be59c700 1 -- 172.21.15.76:0/3636683330 learned_addr learned my addr 172.21.15.76:0/3636683330 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914047141 0 0) 0x7f92c012ff10 con 0x7f92c006cf50 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): my global_id is 194573 2024-09-16T06:47:32.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a00035d0 con 0x7f92c006cf50 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735061392 0 0) 0x7f92c012c4a0 con 0x7f92c006d480 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): my global_id is 214134 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c012ff10 con 0x7f92c006d480 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123437259 0 0) 0x7f92c01ca600 con 0x7f92c0132330 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 10 monclient(hunting): my global_id is 204331 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.970+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c012c4a0 con 0x7f92c0132330 2024-09-16T06:47:32.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251835233 0 0) 0x7f92a00035d0 con 0x7f92c006cf50 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c01ca600 con 0x7f92c006cf50 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f92b0001930 con 0x7f92c006cf50 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.982 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689689188 0 0) 0x7f92c012ff10 con 0x7f92c006d480 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a00035d0 con 0x7f92c006d480 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f92a8001830 con 0x7f92c006d480 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient(hunting): dump: 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.983 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 897092163 0 0) 0x7f92c01ca600 con 0x7f92c006cf50 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 >> v1:172.21.15.189:6789/0 conn(0x7f92c0132330 legacy=0x7f92c01d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 >> v1:172.21.15.76:6789/0 conn(0x7f92c006d480 legacy=0x7f92c01c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 monclient: found mon.c 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c0131620 con 0x7f92c006cf50 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:32.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 5 monclient: authenticate success, global_id 194573 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _renew_subs 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c009e8b0 con 0x7f92c006cf50 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _renew_subs 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c012de40 con 0x7f92c006cf50 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92b0001cc0 con 0x7f92c006cf50 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f92b0003b10 con 0x7f92c006cf50 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bcd99700 4 set_mon_vals no callback set 2024-09-16T06:47:32.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 10 monclient: dump: 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4005100 con 0x7f92c006cf50 2024-09-16T06:47:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f92b0003c70 con 0x7f92c006cf50 2024-09-16T06:47:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f92b0043dc0 con 0x7f92c006cf50 2024-09-16T06:47:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.974+0000 7f92b5d9b700 10 monclient: get_auth_request con 0x7f92a0036230 auth_method 0 2024-09-16T06:47:32.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.982+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f92b00147e0 con 0x7f92c006cf50 2024-09-16T06:47:32.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.982+0000 7f92b77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:32.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:32.982+0000 7f92b77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f1284ceb700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f1284ceb700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f1284ceb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f1264005b90 con 0x7f1280071b40 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f12777fe700 1 -- 172.21.15.189:0/63340975 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v26) v1 ==== 82+0+0 (unknown 1890204813 0 0) 0x7f1270014d20 con 0x7f1280071b40 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f12777fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.980+0000 7f12777fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1260036450 legacy=0x7f1260038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 10 monclient: shutdown 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 >> v1:172.21.15.189:6789/0 conn(0x7f1280071b40 legacy=0x7f12801bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:32.989 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 shutdown_connections 2024-09-16T06:47:32.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 >> 172.21.15.189:0/63340975 conn(0x7f12800a08b0 msgr2=0x7f12800a22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:32.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 shutdown_connections 2024-09-16T06:47:32.990 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:47:32.984+0000 7f1284ceb700 1 -- 172.21.15.189:0/63340975 wait complete. 2024-09-16T06:47:32.999 INFO:tasks.workunit.client.0.smithi189.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-16T06:47:32.999 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-16T06:47:32.999 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-16T06:47:32.999 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:47:33.000 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T06:47:33.000 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T06:47:33.000 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T06:47:33.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92bf59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:47:33.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92bf59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:47:33.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92bf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:33.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f92a4005bc0 con 0x7f92c006cf50 2024-09-16T06:47:33.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92b77fe700 1 -- 172.21.15.76:0/3636683330 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f92b0012c50 con 0x7f92c006cf50 2024-09-16T06:47:33.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92b77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:47:33.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f92b77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=13,logm=46} crc {auth=54333754,config=1097430493,health=889898741,kv=1001081567,logm=4026702551}) 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=1,health=12,kv=13,logm=45} crc {auth=1153123432,config=3009771555,health=2515098687,kv=2283260164,logm=1520461553}) 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.218+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.222+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=13,logm=46} crc {auth=54333754,config=1097430493,health=889898741,kv=1001081567,logm=4026702551}) 2024-09-16T06:47:33.221 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.222+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=1,health=12,kv=13,logm=43} crc {auth=2465799229,config=1228700967,health=2515098687,kv=1001081567,logm=2311616008}) 2024-09-16T06:47:33.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.222+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1311092984}) 2024-09-16T06:47:33.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743634631}) 2024-09-16T06:47:33.224 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1311092984}) 2024-09-16T06:47:33.225 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456929212}) 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 >> v1:172.21.15.189:6812/3098810674 conn(0x7f92a0036230 legacy=0x7f92a00386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 10 monclient: shutdown 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 >> v1:172.21.15.76:6790/0 conn(0x7f92c006cf50 legacy=0x7f92c01c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 shutdown_connections 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 >> 172.21.15.76:0/3636683330 conn(0x7f92c00a41c0 msgr2=0x7f92c009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 shutdown_connections 2024-09-16T06:47:33.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f92bf59e700 1 -- 172.21.15.76:0/3636683330 wait complete. 2024-09-16T06:47:33.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720883416}) 2024-09-16T06:47:33.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=781538933}) 2024-09-16T06:47:33.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.227 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3720883416}) 2024-09-16T06:47:33.228 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.226+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046089048}) 2024-09-16T06:47:33.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.229 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804064758}) 2024-09-16T06:47:33.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452487156}) 2024-09-16T06:47:33.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804064758}) 2024-09-16T06:47:33.230 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130357280}) 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560673838}) 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549815934}) 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560673838}) 2024-09-16T06:47:33.231 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757108117}) 2024-09-16T06:47:33.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653406297}) 2024-09-16T06:47:33.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517571367}) 2024-09-16T06:47:33.232 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.233 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653406297}) 2024-09-16T06:47:33.233 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.230+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205029938}) 2024-09-16T06:47:33.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631596025}) 2024-09-16T06:47:33.234 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306185464}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631596025}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050222780}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250009820}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637276649}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250009820}) 2024-09-16T06:47:33.235 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598861560}) 2024-09-16T06:47:33.237 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.237 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514528327}) 2024-09-16T06:47:33.237 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237007982}) 2024-09-16T06:47:33.237 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514528327}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175195766}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.234+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037043656}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151805778}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037043656}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619829375}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202192657}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997514577}) 2024-09-16T06:47:33.238 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.239 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202192657}) 2024-09-16T06:47:33.240 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2228613463}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167277452}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895864660}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167277452}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461879486}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473192296}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40457192}) 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.242 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473192296}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.238+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071845974}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215715802}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913175502}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215715802}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760897420}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156968883}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769238723}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156968883}) 2024-09-16T06:47:33.243 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972416877}) 2024-09-16T06:47:33.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173827369}) 2024-09-16T06:47:33.244 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494512671}) 2024-09-16T06:47:33.245 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.245 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173827369}) 2024-09-16T06:47:33.245 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.242+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986384032}) 2024-09-16T06:47:33.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185189151}) 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004887607}) 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185189151}) 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3308123227}) 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.252 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323165684}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3710451150}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323165684}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474210894}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713118923}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983717437}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713118923}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.246+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4186525631}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3955529197}) 2024-09-16T06:47:33.253 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481874338}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3955529197}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556812262}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447312943}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508442975}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447312943}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917330245}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679137315}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802323607}) 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.254 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679137315}) 2024-09-16T06:47:33.255 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.250+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2104667385}) 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1493090986}) 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716322358}) 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1493090986}) 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517819269}) 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.257 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545530757}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525328995}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545530757}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4212497430}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201798739}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216699460}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201798739}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.254+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806046999}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226285966}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207825403}) 2024-09-16T06:47:33.258 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.259 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226285966}) 2024-09-16T06:47:33.259 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668219017}) 2024-09-16T06:47:33.260 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.260 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602613920}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850841998}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602613920}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225496817}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505645660}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541371770}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505645660}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.258+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2162309693}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377249943}) 2024-09-16T06:47:33.261 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471625406}) 2024-09-16T06:47:33.262 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.262 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377249943}) 2024-09-16T06:47:33.262 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197411017}) 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270480094}) 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713014000}) 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270480094}) 2024-09-16T06:47:33.263 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085090397}) 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2921706507}) 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951931401}) 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2921706507}) 2024-09-16T06:47:33.264 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=515569916}) 2024-09-16T06:47:33.265 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.265 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321502246}) 2024-09-16T06:47:33.265 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995451402}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321502246}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.262+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189937159}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384689946}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769259306}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1384689946}) 2024-09-16T06:47:33.266 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260446886}) 2024-09-16T06:47:33.267 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195953075}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991520679}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195953075}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837669391}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615569190}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952406683}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615569190}) 2024-09-16T06:47:33.268 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.266+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549708489}) 2024-09-16T06:47:33.269 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.269 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514059688}) 2024-09-16T06:47:33.269 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667573194}) 2024-09-16T06:47:33.269 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.269 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514059688}) 2024-09-16T06:47:33.270 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2564996004}) 2024-09-16T06:47:33.270 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.270 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=300592367}) 2024-09-16T06:47:33.271 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748603789}) 2024-09-16T06:47:33.271 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.271 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=300592367}) 2024-09-16T06:47:33.271 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644101923}) 2024-09-16T06:47:33.272 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.272 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550332972}) 2024-09-16T06:47:33.272 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2562632005}) 2024-09-16T06:47:33.272 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.272 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550332972}) 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.270+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837399032}) 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525707848}) 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725820846}) 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525707848}) 2024-09-16T06:47:33.273 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239622430}) 2024-09-16T06:47:33.274 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.274 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883216393}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40732175}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883216393}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838675866}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259121744}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845214585}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259121744}) 2024-09-16T06:47:33.275 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567863495}) 2024-09-16T06:47:33.276 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.276 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658426572}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228391140}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658426572}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.274+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316217661}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428315910}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981555498}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428315910}) 2024-09-16T06:47:33.277 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2272037317}) 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703203255}) 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103895586}) 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703203255}) 2024-09-16T06:47:33.278 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833575302}) 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492577316}) 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334429130}) 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492577316}) 2024-09-16T06:47:33.279 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271593762}) 2024-09-16T06:47:33.280 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.280 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659657739}) 2024-09-16T06:47:33.280 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=227751496}) 2024-09-16T06:47:33.280 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.280 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659657739}) 2024-09-16T06:47:33.281 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.278+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1006851427}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990197696}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688996866}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990197696}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087364407}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2148807430}) 2024-09-16T06:47:33.282 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708683229}) 2024-09-16T06:47:33.283 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.283 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2148807430}) 2024-09-16T06:47:33.283 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480625610}) 2024-09-16T06:47:33.283 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.284 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802537833}) 2024-09-16T06:47:33.284 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306643055}) 2024-09-16T06:47:33.284 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.284 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802537833}) 2024-09-16T06:47:33.284 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.282+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039450329}) 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=1315880215,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3242701077}) 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4} crc {logm=1552197246,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1298802654,mon_config_key=961595191}) 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=1315880215,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3242701077}) 2024-09-16T06:47:33.285 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=73,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=1986415329,mdsmap=1531151339,mgr=857110920,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3074754928,mon_config_key=938827118}) 2024-09-16T06:47:33.286 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.286 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mon_config_key=1918366733,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=966914556}) 2024-09-16T06:47:33.287 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2300786497}) 2024-09-16T06:47:33.287 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.287 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mon_config_key=1918366733,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=966914556}) 2024-09-16T06:47:33.287 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.286+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=69} crc {mon_config_key=3158179092,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=3951768334}) 2024-09-16T06:47:33.293 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.294 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=10} crc {osdmap=2212598449}) 2024-09-16T06:47:33.294 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=2926680979}) 2024-09-16T06:47:33.294 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:47:33.294 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=10} crc {osdmap=2212598449}) 2024-09-16T06:47:33.294 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:47:33.294+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=7} crc {osdmap=564334273}) 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:47:33.550 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:47:33.551 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:47:33.551 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:47:34.054 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:47:34.054 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T06:47:34.055 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T06:47:34.250 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:47:34.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:34.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.366+0000 7f791e18a700 10 monclient: get_monmap_and_config 2024-09-16T06:47:34.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.366+0000 7f791e18a700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.366+0000 7f791e18a700 10 monclient: monmap: 2024-09-16T06:47:34.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:34.371886+0000 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:34.371886+0000 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:34.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 Processor -- start 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- start start 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient: init 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-a con 0x7f79180964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-c con 0x7f791809b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-b con 0x7f7918137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918098560 con 0x7f79180964f0 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918099ec0 con 0x7f791809b1c0 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791809b5f0 con 0x7f7918137e70 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient(hunting): authenticate will time out at 1401.505147s 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791d188700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f79180964f0 0x7f79180968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52384/0 (socket says 172.21.15.76:52384) 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791d188700 1 -- 172.21.15.76:0/4269176632 learned_addr learned my addr 172.21.15.76:0/4269176632 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916667061 0 0) 0x7f7918098560 con 0x7f79180964f0 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): my global_id is 214155 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79040035d0 con 0x7f79180964f0 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063894369 0 0) 0x7f791809b5f0 con 0x7f7918137e70 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): my global_id is 194594 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7918098560 con 0x7f7918137e70 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110613681 0 0) 0x7f7918099ec0 con 0x7f791809b1c0 2024-09-16T06:47:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient(hunting): my global_id is 204352 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f791809b5f0 con 0x7f791809b1c0 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732436415 0 0) 0x7f79040035d0 con 0x7f79180964f0 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 >> v1:172.21.15.76:6790/0 conn(0x7f7918137e70 legacy=0x7f791813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 >> v1:172.21.15.189:6789/0 conn(0x7f791809b1c0 legacy=0x7f7918136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 monclient: found mon.noname-a 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:34.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918099be0 con 0x7f79180964f0 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 5 monclient: authenticate success, global_id 214155 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7914001610 con 0x7f79180964f0 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: dump: 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79140018f0 con 0x7f79180964f0 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 1 -- 172.21.15.76:0/4269176632 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7914001c80 con 0x7f79180964f0 2024-09-16T06:47:34.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 4 set_mon_vals no callback set 2024-09-16T06:47:34.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:34.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f790bfff700 10 monclient: dump: 2024-09-16T06:47:34.378 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 10 monclient: shutdown 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- 172.21.15.76:0/4269176632 >> v1:172.21.15.76:6789/0 conn(0x7f79180964f0 legacy=0x7f79180968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- 172.21.15.76:0/4269176632 shutdown_connections 2024-09-16T06:47:34.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- 172.21.15.76:0/4269176632 >> 172.21.15.76:0/4269176632 conn(0x7f79180a0030 msgr2=0x7f79180a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- 172.21.15.76:0/4269176632 shutdown_connections 2024-09-16T06:47:34.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.370+0000 7f791e18a700 1 -- 172.21.15.76:0/4269176632 wait complete. 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient: monmap: 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:34.383 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 1 Processor -- start 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 1 -- start start 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient: init 2024-09-16T06:47:34.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-a con 0x7f79180964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-c con 0x7f791809b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient(hunting): picked mon.noname-b con 0x7f7918137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918093560 con 0x7f79180964f0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918093840 con 0x7f7918137e70 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180920b0 con 0x7f791809b1c0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791e18a700 10 monclient(hunting): authenticate will time out at 1401.510401s 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791c987700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f791809b1c0 0x7f79181cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48530/0 (socket says 172.21.15.76:48530) 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.374+0000 7f791c987700 1 -- 172.21.15.76:0/3189065248 learned_addr learned my addr 172.21.15.76:0/3189065248 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524562804 0 0) 0x7f79180920b0 con 0x7f791809b1c0 2024-09-16T06:47:34.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): my global_id is 194597 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc0035d0 con 0x7f791809b1c0 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3720552663 0 0) 0x7f78fc0035d0 con 0x7f791809b1c0 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79180920b0 con 0x7f791809b1c0 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f790c001990 con 0x7f791809b1c0 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:34.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394214296 0 0) 0x7f79180920b0 con 0x7f791809b1c0 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 >> v1:172.21.15.189:6789/0 conn(0x7f7918137e70 legacy=0x7f79181d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 >> v1:172.21.15.76:6789/0 conn(0x7f79180964f0 legacy=0x7f79181ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 monclient: found mon.c 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918097de0 con 0x7f791809b1c0 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 5 monclient: authenticate success, global_id 194597 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _renew_subs 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79180a2d90 con 0x7f791809b1c0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _renew_subs 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7918093280 con 0x7f791809b1c0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c003970 con 0x7f791809b1c0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f790c001d80 con 0x7f791809b1c0 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f790b7fe700 4 set_mon_vals no callback set 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 10 monclient: dump: 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f790c003d00 con 0x7f791809b1c0 2024-09-16T06:47:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f8005100 con 0x7f791809b1c0 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f790c043f30 con 0x7f791809b1c0 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.378+0000 7f791d188700 10 monclient: get_auth_request con 0x7f78fc036150 auth_method 0 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.386+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f790c014810 con 0x7f791809b1c0 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.386+0000 7f7909ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.386+0000 7f7909ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:34.547 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:47:34.547 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T06:47:34.548 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T06:47:34.549 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T06:47:34.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.614+0000 7f791e18a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:34.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.614+0000 7f791e18a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.614+0000 7f791e18a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.614+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78f8005930 con 0x7f791809b1c0 2024-09-16T06:47:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.614+0000 7f7909ffb700 1 -- 172.21.15.76:0/3189065248 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2531574928) 0x7f790c029c80 con 0x7f791809b1c0 2024-09-16T06:47:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.618+0000 7f7909ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.618+0000 7f7909ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:34.617 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 >> v1:172.21.15.189:6812/3098810674 conn(0x7f78fc036150 legacy=0x7f78fc038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 10 monclient: shutdown 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 >> v1:172.21.15.76:6790/0 conn(0x7f791809b1c0 legacy=0x7f79181cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 shutdown_connections 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 >> 172.21.15.76:0/3189065248 conn(0x7f79180a0030 msgr2=0x7f791809bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 shutdown_connections 2024-09-16T06:47:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.622+0000 7f791e18a700 1 -- 172.21.15.76:0/3189065248 wait complete. 2024-09-16T06:47:34.635 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:47:34.635 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:47:34.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:47:34.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:47:34.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.742+0000 7f0dd914c700 10 monclient: get_monmap_and_config 2024-09-16T06:47:34.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.742+0000 7f0dd914c700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.742+0000 7f0dd914c700 10 monclient: monmap: 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:34.746993+0000 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:34.746993+0000 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:34.744 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 Processor -- start 2024-09-16T06:47:34.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- start start 2024-09-16T06:47:34.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: init 2024-09-16T06:47:34.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-c con 0x7f0dd4132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-b con 0x7f0dd406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-a con 0x7f0dd406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd406c280 con 0x7f0dd406cf50 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd406bfa0 con 0x7f0dd4132330 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd409ace0 con 0x7f0dd406d480 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient(hunting): authenticate will time out at 1401.881021s 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd894b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0dd406cf50 0x7f0dd413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52406/0 (socket says 172.21.15.76:52406) 2024-09-16T06:47:34.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd894b700 1 -- 172.21.15.76:0/238158507 learned_addr learned my addr 172.21.15.76:0/238158507 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566374834 0 0) 0x7f0dd406c280 con 0x7f0dd406cf50 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): my global_id is 214164 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0035d0 con 0x7f0dd406cf50 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626151069 0 0) 0x7f0dd409ace0 con 0x7f0dd406d480 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): my global_id is 194606 2024-09-16T06:47:34.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd406c280 con 0x7f0dd406d480 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769019108 0 0) 0x7f0dd406bfa0 con 0x7f0dd4132330 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient(hunting): my global_id is 204361 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd409ace0 con 0x7f0dd4132330 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223331595 0 0) 0x7f0dbc0035d0 con 0x7f0dd406cf50 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 >> v1:172.21.15.76:6790/0 conn(0x7f0dd406d480 legacy=0x7f0dd413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 >> v1:172.21.15.189:6789/0 conn(0x7f0dd4132330 legacy=0x7f0dd4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:34.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd409ae70 con 0x7f0dd406cf50 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dcc001990 con 0x7f0dd406cf50 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 5 monclient: authenticate success, global_id 214164 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: dump: 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.756 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dcc001c70 con 0x7f0dd406cf50 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 1 -- 172.21.15.76:0/238158507 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dcc001e60 con 0x7f0dd406cf50 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 4 set_mon_vals no callback set 2024-09-16T06:47:34.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd2ffd700 10 monclient: dump: 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: shutdown 2024-09-16T06:47:34.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- 172.21.15.76:0/238158507 >> v1:172.21.15.76:6789/0 conn(0x7f0dd406cf50 legacy=0x7f0dd413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- 172.21.15.76:0/238158507 shutdown_connections 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- 172.21.15.76:0/238158507 >> 172.21.15.76:0/238158507 conn(0x7f0dd40a41c0 msgr2=0x7f0dd40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- 172.21.15.76:0/238158507 shutdown_connections 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 1 -- 172.21.15.76:0/238158507 wait complete. 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.746+0000 7f0dd914c700 10 monclient: monmap: 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.759 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 Processor -- start 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- start start 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: init 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-a con 0x7f0dd406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-b con 0x7f0dd406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient(hunting): picked mon.noname-c con 0x7f0dd4132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd412f470 con 0x7f0dd406cf50 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4131800 con 0x7f0dd406d480 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd41312d0 con 0x7f0dd4132330 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient(hunting): authenticate will time out at 1401.884585s 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd894b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0dd4132330 0x7f0dd412e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48548/0 (socket says 172.21.15.76:48548) 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd894b700 1 -- 172.21.15.76:0/3467703300 learned_addr learned my addr 172.21.15.76:0/3467703300 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683061438 0 0) 0x7f0dd41312d0 con 0x7f0dd4132330 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): my global_id is 194609 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc00035d0 con 0x7f0dd4132330 2024-09-16T06:47:34.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563854821 0 0) 0x7f0dd412f470 con 0x7f0dd406cf50 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): my global_id is 214167 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd41312d0 con 0x7f0dd406cf50 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769525438 0 0) 0x7f0dd4131800 con 0x7f0dd406d480 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): my global_id is 204364 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd412f470 con 0x7f0dd406d480 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 757537524 0 0) 0x7f0dc00035d0 con 0x7f0dd4132330 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dd4131800 con 0x7f0dd4132330 2024-09-16T06:47:34.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dcc0013a0 con 0x7f0dd4132330 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): dump: 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3910068735 0 0) 0x7f0dd412f470 con 0x7f0dd406d480 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dc00035d0 con 0x7f0dd406d480 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc80018b0 con 0x7f0dd406d480 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient(hunting): dump: 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.764 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789692577 0 0) 0x7f0dd41312d0 con 0x7f0dd406cf50 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dd412f470 con 0x7f0dd406cf50 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3244782071 0 0) 0x7f0dd4131800 con 0x7f0dd4132330 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.189:6789/0 conn(0x7f0dd406d480 legacy=0x7f0dd412dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6789/0 conn(0x7f0dd406cf50 legacy=0x7f0dd412d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 monclient: found mon.c 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd409e8b0 con 0x7f0dd4132330 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 5 monclient: authenticate success, global_id 194609 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _renew_subs 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4132f00 con 0x7f0dd4132330 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _renew_subs 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd412ef30 con 0x7f0dd4132330 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dcc005000 con 0x7f0dd4132330 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dcc005310 con 0x7f0dd4132330 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd1ffb700 4 set_mon_vals no callback set 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 10 monclient: dump: 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.767 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0db4006590 con 0x7f0db4002830 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.750+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0dcc00fd80 con 0x7f0dd4132330 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f0dcc044280 con 0x7f0dd4132330 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd37fe700 10 monclient: get_auth_request con 0x7f0dc00364a0 auth_method 0 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220100393 0 0) 0x7f0db4006590 con 0x7f0db4002830 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient(hunting): my global_id is 214170 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd4131800 con 0x7f0db4002830 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1674423599 0 0) 0x7f0dd4131800 con 0x7f0db4002830 2024-09-16T06:47:34.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db4006590 con 0x7f0db4002830 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc40036c0 con 0x7f0db4002830 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient: dump: 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2834010496 0 0) 0x7f0db4006590 con 0x7f0db4002830 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0db400a190 con 0x7f0db4002830 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0dc4005880 con 0x7f0db4002830 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6789/0 conn(0x7f0db4002830 legacy=0x7f0db4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.754+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6789/0 conn(0x7f0db4002830 legacy=0x7f0db4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0db4009e30 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218070246 0 0) 0x7f0db4009e30 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient(hunting): my global_id is 214173 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4006590 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3193187077 0 0) 0x7f0db4006590 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db4009e30 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc4003df0 con 0x7f0db4009150 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient: dump: 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1837653210 0 0) 0x7f0db4009e30 con 0x7f0db4009150 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0db400c340 con 0x7f0db4009150 2024-09-16T06:47:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2525494394) 0x7f0dc40059f0 con 0x7f0db4009150 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6789/0 conn(0x7f0db4009150 legacy=0x7f0db40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd0ff9700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6789/0 conn(0x7f0db4009150 legacy=0x7f0db40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:34.785 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:34.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0dc00364a0 legacy=0x7f0dc0038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 10 monclient: shutdown 2024-09-16T06:47:34.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 >> v1:172.21.15.76:6790/0 conn(0x7f0dd4132330 legacy=0x7f0dd412e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd894b700 1 -- 172.21.15.76:0/3467703300 reap_dead start 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd894b700 1 -- 172.21.15.76:0/3467703300 reap_dead start 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 shutdown_connections 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 >> 172.21.15.76:0/3467703300 conn(0x7f0dd40a41c0 msgr2=0x7f0dd409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 shutdown_connections 2024-09-16T06:47:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.766+0000 7f0dd914c700 1 -- 172.21.15.76:0/3467703300 wait complete. 2024-09-16T06:47:34.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:34.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:47:34.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-16T06:47:34.887 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T06:47:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:47:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T06:47:34.888 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 10 monclient: monmap: 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:34.910930+0000 2024-09-16T06:47:34.907 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:34.910930+0000 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:34.908 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 1 Processor -- start 2024-09-16T06:47:34.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 1 -- start start 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.906+0000 7f808f59e700 10 monclient: init 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f8090132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f809006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f809006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809006c280 con 0x7f809006cf50 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809006bfa0 con 0x7f809006d480 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809009ace0 con 0x7f8090132330 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): authenticate will time out at 1402.044211s 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f809006cf50 0x7f809013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52442/0 (socket says 172.21.15.76:52442) 2024-09-16T06:47:34.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808ed9d700 1 -- 172.21.15.76:0/326167748 learned_addr learned my addr 172.21.15.76:0/326167748 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901005590 0 0) 0x7f809006c280 con 0x7f809006cf50 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): my global_id is 214176 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80780035d0 con 0x7f809006cf50 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365212269 0 0) 0x7f809009ace0 con 0x7f8090132330 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): my global_id is 194612 2024-09-16T06:47:34.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809006c280 con 0x7f8090132330 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191005641 0 0) 0x7f809006bfa0 con 0x7f809006d480 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient(hunting): my global_id is 204367 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809009ace0 con 0x7f809006d480 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3465254119 0 0) 0x7f809006c280 con 0x7f8090132330 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 >> v1:172.21.15.189:6789/0 conn(0x7f809006d480 legacy=0x7f809013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 >> v1:172.21.15.76:6789/0 conn(0x7f809006cf50 legacy=0x7f809013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 monclient: found mon.noname-b 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809009ae70 con 0x7f8090132330 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 5 monclient: authenticate success, global_id 194612 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80840015f0 con 0x7f8090132330 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: dump: 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.914 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8084001890 con 0x7f8090132330 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 1 -- 172.21.15.76:0/326167748 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8084001a80 con 0x7f8090132330 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 4 set_mon_vals no callback set 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808d59a700 10 monclient: dump: 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: shutdown 2024-09-16T06:47:34.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- 172.21.15.76:0/326167748 >> v1:172.21.15.76:6790/0 conn(0x7f8090132330 legacy=0x7f8090132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- 172.21.15.76:0/326167748 shutdown_connections 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- 172.21.15.76:0/326167748 >> 172.21.15.76:0/326167748 conn(0x7f80900a41c0 msgr2=0x7f80900a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- 172.21.15.76:0/326167748 shutdown_connections 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- 172.21.15.76:0/326167748 wait complete. 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: build_initial_monmap 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: monmap: 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:34.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:34.918 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:34.918 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:34.918 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 Processor -- start 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- start start 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: init 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f809006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f809006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f8090132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900a1a00 con 0x7f8090132330 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80901ca670 con 0x7f809006cf50 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809012dcf0 con 0x7f809006d480 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.910+0000 7f808f59e700 10 monclient(hunting): authenticate will time out at 1402.047978s 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8090132330 0x7f80901d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52456/0 (socket says 172.21.15.76:52456) 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808ed9d700 1 -- 172.21.15.76:0/3305281843 learned_addr learned my addr 172.21.15.76:0/3305281843 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702627153 0 0) 0x7f80900a1a00 con 0x7f8090132330 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): my global_id is 214179 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80700035d0 con 0x7f8090132330 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992476343 0 0) 0x7f809012dcf0 con 0x7f809006d480 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): my global_id is 194615 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900a1a00 con 0x7f809006d480 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447393070 0 0) 0x7f80901ca670 con 0x7f809006cf50 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): my global_id is 204370 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809012dcf0 con 0x7f809006cf50 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930823077 0 0) 0x7f80700035d0 con 0x7f8090132330 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80901ca670 con 0x7f8090132330 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8088001340 con 0x7f8090132330 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): dump: 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825335903 0 0) 0x7f80901ca670 con 0x7f8090132330 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6790/0 conn(0x7f809006d480 legacy=0x7f80901c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.189:6789/0 conn(0x7f809006cf50 legacy=0x7f80901c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 monclient: found mon.a 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809012e2b0 con 0x7f8090132330 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: _finish_auth 0 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 5 monclient: authenticate success, global_id 214179 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 10 monclient: _renew_subs 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809012dfd0 con 0x7f8090132330 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 10 monclient: _renew_subs 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80900a0f70 con 0x7f8090132330 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8088001d30 con 0x7f8090132330 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8088004b40 con 0x7f8090132330 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807ffff700 4 set_mon_vals no callback set 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: dump: 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f808800f5b0 con 0x7f8090132330 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8074006590 con 0x7f8074002830 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f8088043e50 con 0x7f8090132330 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f808dd9b700 10 monclient: get_auth_request con 0x7f8070036170 auth_method 0 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992885406 0 0) 0x7f8074006590 con 0x7f8074002830 2024-09-16T06:47:34.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient(hunting): my global_id is 194618 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80901ca670 con 0x7f8074002830 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1960898626 0 0) 0x7f80901ca670 con 0x7f8074002830 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8074006590 con 0x7f8074002830 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8084003a30 con 0x7f8074002830 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.914+0000 7f807effd700 10 monclient: dump: 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 793275823 0 0) 0x7f8074006590 con 0x7f8074002830 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f807400a190 con 0x7f8074002830 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8084003cd0 con 0x7f8074002830 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6790/0 conn(0x7f8074002830 legacy=0x7f8074004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.918+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6790/0 conn(0x7f8074002830 legacy=0x7f8074004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:34.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f808f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8074009e30 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908064727 0 0) 0x7f8074009e30 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient(hunting): my global_id is 194621 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074006590 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 452195127 0 0) 0x7f8074006590 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8074009e30 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80840042d0 con 0x7f8074009150 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:34.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient: dump: 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3354230763 0 0) 0x7f8074009e30 con 0x7f8074009150 2024-09-16T06:47:34.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f807400c340 con 0x7f8074009150 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2395154717) 0x7f8084005740 con 0x7f8074009150 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6790/0 conn(0x7f8074009150 legacy=0x7f80740095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.926+0000 7f807effd700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6790/0 conn(0x7f8074009150 legacy=0x7f80740095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:47:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:34.947 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:34.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8070036170 legacy=0x7f8070038630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 10 monclient: shutdown 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 >> v1:172.21.15.76:6789/0 conn(0x7f8090132330 legacy=0x7f80901d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808ed9d700 1 -- 172.21.15.76:0/3305281843 reap_dead start 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808ed9d700 1 -- 172.21.15.76:0/3305281843 reap_dead start 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 shutdown_connections 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 >> 172.21.15.76:0/3305281843 conn(0x7f80900a41c0 msgr2=0x7f8090129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 shutdown_connections 2024-09-16T06:47:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:34.930+0000 7f808f59e700 1 -- 172.21.15.76:0/3305281843 wait complete. 2024-09-16T06:47:34.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:34.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:47:34.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:47:35.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.066+0000 7fd68a1e7700 10 monclient: get_monmap_and_config 2024-09-16T06:47:35.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.066+0000 7fd68a1e7700 10 monclient: build_initial_monmap 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.066+0000 7fd68a1e7700 10 monclient: monmap: 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:35.071264+0000 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:35.071264+0000 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:35.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 Processor -- start 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- start start 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient: init 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-c con 0x7fd684070d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-b con 0x7fd68409e920 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-a con 0x7fd6840a0240 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684073f10 con 0x7fd6840a0240 2024-09-16T06:47:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68409efc0 con 0x7fd684070d10 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68409f2a0 con 0x7fd68409e920 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient(hunting): authenticate will time out at 1402.205327s 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd6889e4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd68409e920 0x7fd68413a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48612/0 (socket says 172.21.15.76:48612) 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd6889e4700 1 -- 172.21.15.76:0/172493608 learned_addr learned my addr 172.21.15.76:0/172493608 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365610247 0 0) 0x7fd684073f10 con 0x7fd6840a0240 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): my global_id is 214188 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6700035d0 con 0x7fd6840a0240 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213540769 0 0) 0x7fd68409f2a0 con 0x7fd68409e920 2024-09-16T06:47:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): my global_id is 194630 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd684073f10 con 0x7fd68409e920 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386313116 0 0) 0x7fd68409efc0 con 0x7fd684070d10 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient(hunting): my global_id is 204379 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd68409f2a0 con 0x7fd684070d10 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426427890 0 0) 0x7fd684073f10 con 0x7fd68409e920 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 >> v1:172.21.15.189:6789/0 conn(0x7fd684070d10 legacy=0x7fd6840710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 >> v1:172.21.15.76:6789/0 conn(0x7fd6840a0240 legacy=0x7fd68413db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 monclient: found mon.noname-b 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68409f580 con 0x7fd68409e920 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: _finish_auth 0 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c0015f0 con 0x7fd68409e920 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: dump: 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 5 monclient: authenticate success, global_id 194630 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd66c0018a0 con 0x7fd68409e920 2024-09-16T06:47:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 1 -- 172.21.15.76:0/172493608 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c001c30 con 0x7fd68409e920 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd683fff700 10 monclient: dump: 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 4 set_mon_vals no callback set 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 10 monclient: shutdown 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- 172.21.15.76:0/172493608 >> v1:172.21.15.76:6790/0 conn(0x7fd68409e920 legacy=0x7fd68413a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- 172.21.15.76:0/172493608 shutdown_connections 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- 172.21.15.76:0/172493608 >> 172.21.15.76:0/172493608 conn(0x7fd6840a4430 msgr2=0x7fd6840a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- 172.21.15.76:0/172493608 shutdown_connections 2024-09-16T06:47:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.070+0000 7fd68a1e7700 1 -- 172.21.15.76:0/172493608 wait complete. 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: build_initial_monmap 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: monmap: 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 Processor -- start 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- start start 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: init 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-c con 0x7fd684070d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-b con 0x7fd6840a0240 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:35.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient(hunting): picked mon.noname-a con 0x7fd684137bc0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68413d750 con 0x7fd684137bc0 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68409b650 con 0x7fd6840a0240 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68409bb80 con 0x7fd684070d10 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient(hunting): authenticate will time out at 1402.210038s 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd6891e5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd684070d10 0x7fd684136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48626/0 (socket says 172.21.15.76:48626) 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd6891e5700 1 -- 172.21.15.76:0/964168330 learned_addr learned my addr 172.21.15.76:0/964168330 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127846774 0 0) 0x7fd68409bb80 con 0x7fd684070d10 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): my global_id is 194633 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6640035d0 con 0x7fd684070d10 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656519494 0 0) 0x7fd68413d750 con 0x7fd684137bc0 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): my global_id is 214191 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd68409bb80 con 0x7fd684137bc0 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2496671701 0 0) 0x7fd6640035d0 con 0x7fd684070d10 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd68413d750 con 0x7fd684070d10 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969664567 0 0) 0x7fd68409b650 con 0x7fd6840a0240 2024-09-16T06:47:35.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): my global_id is 204382 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6640035d0 con 0x7fd6840a0240 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd674001d10 con 0x7fd684070d10 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657108805 0 0) 0x7fd68409bb80 con 0x7fd684137bc0 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd68409b650 con 0x7fd684137bc0 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd67c0018d0 con 0x7fd684137bc0 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient(hunting): dump: 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.088 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3188764832 0 0) 0x7fd68413d750 con 0x7fd684070d10 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6789/0 conn(0x7fd6840a0240 legacy=0x7fd6841374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.76:6789/0 conn(0x7fd684137bc0 legacy=0x7fd6841d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 monclient: found mon.c 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68413a1f0 con 0x7fd684070d10 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: _finish_auth 0 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 5 monclient: authenticate success, global_id 194633 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _renew_subs 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68413a4d0 con 0x7fd684070d10 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _renew_subs 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd684138730 con 0x7fd684070d10 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6740020c0 con 0x7fd684070d10 2024-09-16T06:47:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd674003e90 con 0x7fd684070d10 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd6837fe700 4 set_mon_vals no callback set 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.074+0000 7fd681ffb700 10 monclient: dump: 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.091 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd68a1e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd668006590 con 0x7fd668002830 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd674003ff0 con 0x7fd684070d10 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7fd674044160 con 0x7fd684070d10 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439150071 0 0) 0x7fd668006590 con 0x7fd668002830 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient(hunting): my global_id is 204385 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd68413d750 con 0x7fd668002830 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd6899e6700 10 monclient: get_auth_request con 0x7fd664036240 auth_method 0 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3939026290 0 0) 0x7fd68413d750 con 0x7fd668002830 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd668006590 con 0x7fd668002830 2024-09-16T06:47:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c004ab0 con 0x7fd668002830 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient: dump: 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1223793111 0 0) 0x7fd668006590 con 0x7fd668002830 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd66800a190 con 0x7fd668002830 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd66c004dd0 con 0x7fd668002830 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6789/0 conn(0x7fd668002830 legacy=0x7fd668004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.078+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6789/0 conn(0x7fd668002830 legacy=0x7fd668004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd68a1e7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd668009e30 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741539952 0 0) 0x7fd668009e30 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient(hunting): my global_id is 204388 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd668006590 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 690658614 0 0) 0x7fd668006590 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd668009e30 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c005480 con 0x7fd668009150 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient: dump: 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2012060137 0 0) 0x7fd668009e30 con 0x7fd668009150 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd66800c340 con 0x7fd668009150 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 191976996) 0x7fd66c005750 con 0x7fd668009150 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6789/0 conn(0x7fd668009150 legacy=0x7fd6680095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.094+0000 7fd681ffb700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6789/0 conn(0x7fd668009150 legacy=0x7fd6680095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 86, 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:35.098 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:35.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.100 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:35.101 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.104 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.105 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:35.106 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:35.107 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:35.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:35.109 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:35.109 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:35.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd664036240 legacy=0x7fd664038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 10 monclient: shutdown 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 >> v1:172.21.15.76:6790/0 conn(0x7fd684070d10 legacy=0x7fd684136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd6899e6700 1 -- 172.21.15.76:0/964168330 reap_dead start 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd6899e6700 1 -- 172.21.15.76:0/964168330 reap_dead start 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 shutdown_connections 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 >> 172.21.15.76:0/964168330 conn(0x7fd6840a4430 msgr2=0x7fd68409dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 shutdown_connections 2024-09-16T06:47:35.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.098+0000 7fd68a1e7700 1 -- 172.21.15.76:0/964168330 wait complete. 2024-09-16T06:47:35.111 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:47:35.111 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:47:35.112 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:47:35.112 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:47:35.112 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:47:35.112 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:47:35.112 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:47:35.149 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:47:35.150 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:47:35.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:47:35.224 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T06:47:35.225 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T06:47:35.225 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T06:47:35.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-16T06:47:35.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: get_monmap_and_config 2024-09-16T06:47:35.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: build_initial_monmap 2024-09-16T06:47:35.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: monmap: 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:35.268274+0000 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:35.268274+0000 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:35.265 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 1 Processor -- start 2024-09-16T06:47:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 1 -- start start 2024-09-16T06:47:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: init 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient(hunting): picked mon.noname-c con 0x7f4958132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient(hunting): picked mon.noname-b con 0x7f495806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient(hunting): picked mon.noname-a con 0x7f495806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495806c280 con 0x7f495806cf50 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495806bfa0 con 0x7f4958132330 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495809ace0 con 0x7f495806d480 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient(hunting): authenticate will time out at 1402.402035s 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495659c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4958132330 legacy=0x7f4958132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f4956d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f495806cf50 0x7f495813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52492/0 (socket says 172.21.15.76:52492) 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f4956d9d700 1 -- 172.21.15.76:0/1335514053 learned_addr learned my addr 172.21.15.76:0/1335514053 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799760029 0 0) 0x7f495806c280 con 0x7f495806cf50 2024-09-16T06:47:35.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): my global_id is 214194 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49400035d0 con 0x7f495806cf50 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280239872 0 0) 0x7f495809ace0 con 0x7f495806d480 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient(hunting): my global_id is 194636 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f495806c280 con 0x7f495806d480 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1202094066 0 0) 0x7f49400035d0 con 0x7f495806cf50 2024-09-16T06:47:35.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 >> v1:172.21.15.76:6790/0 conn(0x7f495806d480 legacy=0x7f495813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 >> v1:172.21.15.189:6789/0 conn(0x7f4958132330 legacy=0x7f4958132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: _finish_hunting 0 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 monclient: found mon.noname-a 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495809ae70 con 0x7f495806cf50 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: _finish_auth 0 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: _check_auth_tickets 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 5 monclient: authenticate success, global_id 214194 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4950001990 con 0x7f495806cf50 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:35.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: dump: 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4950001bb0 con 0x7f495806cf50 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 1 -- 172.21.15.76:0/1335514053 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4950001da0 con 0x7f495806cf50 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495759e700 4 set_mon_vals no callback set 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.266+0000 7f495559a700 10 monclient: dump: 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.274 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: shutdown 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- 172.21.15.76:0/1335514053 >> v1:172.21.15.76:6789/0 conn(0x7f495806cf50 legacy=0x7f495813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- 172.21.15.76:0/1335514053 shutdown_connections 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- 172.21.15.76:0/1335514053 >> 172.21.15.76:0/1335514053 conn(0x7f49580a41c0 msgr2=0x7f49580a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- 172.21.15.76:0/1335514053 shutdown_connections 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- 172.21.15.76:0/1335514053 wait complete. 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: build_initial_monmap 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: monmap: 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:35.276 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 Processor -- start 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- start start 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: init 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient(hunting): picked mon.noname-a con 0x7f495806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient(hunting): picked mon.noname-c con 0x7f495806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient(hunting): picked mon.noname-b con 0x7f4958132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4958131260 con 0x7f495806cf50 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495812e580 con 0x7f4958132330 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49581317b0 con 0x7f495806d480 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495759e700 10 monclient(hunting): authenticate will time out at 1402.406760s 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4956d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f4958132330 legacy=0x7f49581d21c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f495806cf50 0x7f495813e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52508/0 (socket says 172.21.15.76:52508) 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f495659c700 1 -- 172.21.15.76:0/1673808355 learned_addr learned my addr 172.21.15.76:0/1673808355 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152025096 0 0) 0x7f4958131260 con 0x7f495806cf50 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): my global_id is 214197 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49380035d0 con 0x7f495806cf50 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253862631 0 0) 0x7f49581317b0 con 0x7f495806d480 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): my global_id is 194639 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4958131260 con 0x7f495806d480 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781034594 0 0) 0x7f49380035d0 con 0x7f495806cf50 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49581317b0 con 0x7f495806cf50 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f494c004230 con 0x7f495806cf50 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): dump: 2024-09-16T06:47:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917894192 0 0) 0x7f4958131260 con 0x7f495806d480 2024-09-16T06:47:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.270+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49380035d0 con 0x7f495806d480 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f49480018b0 con 0x7f495806d480 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient(hunting): dump: 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.284 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 623370692 0 0) 0x7f49581317b0 con 0x7f495806cf50 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 >> v1:172.21.15.76:6790/0 conn(0x7f495806d480 legacy=0x7f49581cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 >> v1:172.21.15.189:6789/0 conn(0x7f4958132330 legacy=0x7f49581d21c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 monclient: found mon.a 2024-09-16T06:47:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495812fef0 con 0x7f495806cf50 2024-09-16T06:47:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: _finish_auth 0 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 5 monclient: authenticate success, global_id 214197 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _renew_subs 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49581301d0 con 0x7f495806cf50 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _renew_subs 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4958130940 con 0x7f495806cf50 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f494c0044a0 con 0x7f495806cf50 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f494c004830 con 0x7f495806cf50 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4954d99700 4 set_mon_vals no callback set 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 10 monclient: dump: 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:35.290 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c005100 con 0x7f495806cf50 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f494c004a50 con 0x7f495806cf50 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f494c043f80 con 0x7f495806cf50 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.274+0000 7f4955d9b700 10 monclient: get_auth_request con 0x7f49380362c0 auth_method 0 2024-09-16T06:47:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.282+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f494c0148c0 con 0x7f495806cf50 2024-09-16T06:47:35.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.282+0000 7f4946ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:35.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.282+0000 7f4946ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f495759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:35.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f495759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f495759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f493c005930 con 0x7f495806cf50 2024-09-16T06:47:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f4946ffd700 1 -- 172.21.15.76:0/1673808355 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 773168047) 0x7f494c012d30 con 0x7f495806cf50 2024-09-16T06:47:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f4946ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:35.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.514+0000 7f4946ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:35.517 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 >> v1:172.21.15.189:6812/3098810674 conn(0x7f49380362c0 legacy=0x7f4938038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 10 monclient: shutdown 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 >> v1:172.21.15.76:6789/0 conn(0x7f495806cf50 legacy=0x7f495813e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 shutdown_connections 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 >> 172.21.15.76:0/1673808355 conn(0x7f49580a41c0 msgr2=0x7f495809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 shutdown_connections 2024-09-16T06:47:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:35.518+0000 7f495759e700 1 -- 172.21.15.76:0/1673808355 wait complete. 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-16T06:47:35.524 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-16T06:47:35.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=110 2024-09-16T06:47:35.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T06:47:35.525 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T06:47:35.525 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T06:47:38.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:38.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.650+0000 7ff911a1c700 10 monclient: get_monmap_and_config 2024-09-16T06:47:38.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.650+0000 7ff911a1c700 10 monclient: build_initial_monmap 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.650+0000 7ff911a1c700 10 monclient: monmap: 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:38.655729+0000 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:38.655729+0000 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:38.653 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 Processor -- start 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- start start 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient: init 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-c con 0x7ff90c132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-b con 0x7ff90c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:38.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-a con 0x7ff90c06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c06c280 con 0x7ff90c06cf50 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c06bfa0 con 0x7ff90c132330 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c09ace0 con 0x7ff90c06d480 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient(hunting): authenticate will time out at 1405.789233s 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff90c06d480 0x7ff90c13a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48664/0 (socket says 172.21.15.76:48664) 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90bfff700 1 -- 172.21.15.76:0/1614144893 learned_addr learned my addr 172.21.15.76:0/1614144893 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff910a1a700 1 -- 172.21.15.76:0/1614144893 >> v1:172.21.15.189:6789/0 conn(0x7ff90c132330 legacy=0x7ff90c132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276999863 0 0) 0x7ff90c09ace0 con 0x7ff90c06d480 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:38.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient(hunting): my global_id is 194654 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80035d0 con 0x7ff90c06d480 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825342502 0 0) 0x7ff8f80035d0 con 0x7ff90c06d480 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 >> v1:172.21.15.189:6789/0 conn(0x7ff90c132330 legacy=0x7ff90c132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 >> v1:172.21.15.76:6789/0 conn(0x7ff90c06cf50 legacy=0x7ff90c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c09ae70 con 0x7ff90c06d480 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8fc001990 con 0x7ff90c06d480 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 5 monclient: authenticate success, global_id 194654 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: dump: 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8fc001cb0 con 0x7ff90c06d480 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 4 set_mon_vals no callback set 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 1 -- 172.21.15.76:0/1614144893 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8fc001ea0 con 0x7ff90c06d480 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff90b7fe700 10 monclient: dump: 2024-09-16T06:47:38.659 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:38.660 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 10 monclient: shutdown 2024-09-16T06:47:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- 172.21.15.76:0/1614144893 >> v1:172.21.15.76:6790/0 conn(0x7ff90c06d480 legacy=0x7ff90c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- 172.21.15.76:0/1614144893 shutdown_connections 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- 172.21.15.76:0/1614144893 >> 172.21.15.76:0/1614144893 conn(0x7ff90c0a41c0 msgr2=0x7ff90c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- 172.21.15.76:0/1614144893 shutdown_connections 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.654+0000 7ff911a1c700 1 -- 172.21.15.76:0/1614144893 wait complete. 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: build_initial_monmap 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: monmap: 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 Processor -- start 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- start start 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: init 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-b con 0x7ff90c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-c con 0x7ff90c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient(hunting): picked mon.noname-a con 0x7ff90c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c12fa20 con 0x7ff90c132330 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c1ca6e0 con 0x7ff90c06cf50 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c12ff50 con 0x7ff90c06d480 2024-09-16T06:47:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient(hunting): authenticate will time out at 1405.794455s 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff910a1a700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff90c06cf50 legacy=0x7ff90c1c98c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff90bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff90c06d480 0x7ff90c1c9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48668/0 (socket says 172.21.15.76:48668) 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff90bfff700 1 -- 172.21.15.76:0/3030315256 learned_addr learned my addr 172.21.15.76:0/3030315256 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff91121b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff90c132330 0x7ff90c1d1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:52528/0 (socket says 172.21.15.76:52528) 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20442613 0 0) 0x7ff90c12ff50 con 0x7ff90c06d480 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): my global_id is 194657 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0035d0 con 0x7ff90c06d480 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3251850097 0 0) 0x7ff8ec0035d0 con 0x7ff90c06d480 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff90c12ff50 con 0x7ff90c06d480 2024-09-16T06:47:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8fc004c10 con 0x7ff90c06d480 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): dump: 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221864851 0 0) 0x7ff90c12ff50 con 0x7ff90c06d480 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 >> v1:172.21.15.189:6789/0 conn(0x7ff90c06cf50 legacy=0x7ff90c1c98c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 >> v1:172.21.15.76:6789/0 conn(0x7ff90c132330 legacy=0x7ff90c1d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 monclient: found mon.c 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c12fc70 con 0x7ff90c06d480 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff9097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 5 monclient: authenticate success, global_id 194657 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _renew_subs 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c12c680 con 0x7ff90c06d480 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _renew_subs 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.658+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c12e180 con 0x7ff90c06d480 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8fc005020 con 0x7ff90c06d480 2024-09-16T06:47:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8fc0053b0 con 0x7ff90c06d480 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff90affd700 4 set_mon_vals no callback set 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 10 monclient: dump: 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:38.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff8fc005510 con 0x7ff90c06d480 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff911a1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff911a1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff911a1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0005100 con 0x7ff90c06d480 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7ff8fc0444f0 con 0x7ff90c06d480 2024-09-16T06:47:38.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.662+0000 7ff910a1a700 10 monclient: get_auth_request con 0x7ff8ec0360f0 auth_method 0 2024-09-16T06:47:38.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.670+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff8fc014cc0 con 0x7ff90c06d480 2024-09-16T06:47:38.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.670+0000 7ff9097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:38.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.670+0000 7ff9097fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff911a1c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff911a1c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff911a1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff8f0005930 con 0x7ff90c06d480 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff9097fa700 1 -- 172.21.15.76:0/3030315256 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2130269582) 0x7ff8fc013360 con 0x7ff90c06d480 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff9097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.906+0000 7ff9097fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:38.907 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:38.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff8ec0360f0 legacy=0x7ff8ec0385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:38.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 10 monclient: shutdown 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 >> v1:172.21.15.76:6790/0 conn(0x7ff90c06d480 legacy=0x7ff90c1c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 shutdown_connections 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 >> 172.21.15.76:0/3030315256 conn(0x7ff90c0a41c0 msgr2=0x7ff90c09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 shutdown_connections 2024-09-16T06:47:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:38.910+0000 7ff911a1c700 1 -- 172.21.15.76:0/3030315256 wait complete. 2024-09-16T06:47:41.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-16T06:47:41.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:47:42.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: get_monmap_and_config 2024-09-16T06:47:42.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: build_initial_monmap 2024-09-16T06:47:42.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: monmap: 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:42.042181+0000 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:42.042181+0000 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 1 Processor -- start 2024-09-16T06:47:42.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 1 -- start start 2024-09-16T06:47:42.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: init 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.038+0000 7f496493d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 10 monclient(hunting): picked mon.noname-b con 0x7f4960132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 10 monclient(hunting): picked mon.noname-a con 0x7f496006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 10 monclient(hunting): picked mon.noname-c con 0x7f496006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496006c280 con 0x7f496006d480 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496006bfa0 con 0x7f496006cf50 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496009ace0 con 0x7f4960132330 2024-09-16T06:47:42.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:42.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f496493d700 10 monclient(hunting): authenticate will time out at 1409.176160s 2024-09-16T06:47:42.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f495f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4960132330 0x7f4960132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48670/0 (socket says 172.21.15.76:48670) 2024-09-16T06:47:42.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f495f7fe700 1 -- 172.21.15.76:0/376827384 learned_addr learned my addr 172.21.15.76:0/376827384 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:42.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.042+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f496013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:42.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.246+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f496013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:42.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:42.650+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f496013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:43.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:43.454+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f496013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): tick 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6790/0 conn(0x7f4960132330 legacy=0x7f4960132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f496013da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6789/0 conn(0x7f496006d480 legacy=0x7f496013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-a con 0x7f4940003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-b con 0x7f4940007330 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-c con 0x7f496006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496006bfa0 con 0x7f4940003740 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494000f8b0 con 0x7f496006cf50 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940010ac0 con 0x7f4940007330 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:45.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.042+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:45.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.245+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:45.649+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:46.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:46.453+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:48.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): tick 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6790/0 conn(0x7f4940007330 legacy=0x7f49400097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6789/0 conn(0x7f4940003740 legacy=0x7f4940005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:48.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-a con 0x7f494000e9b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-b con 0x7f4940010c50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): picked mon.noname-c con 0x7f496006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940011cd0 con 0x7f494000e9b0 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494000f8b0 con 0x7f496006cf50 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940011780 con 0x7f4940010c50 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:48.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.041+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.245+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:48.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:48.649+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:49.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:49.453+0000 7f495ffff700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370309185 0 0) 0x7f4940011cd0 con 0x7f494000e9b0 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient(hunting): my global_id is 214212 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c0035d0 con 0x7f494000e9b0 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892000486 0 0) 0x7f494c0035d0 con 0x7f494000e9b0 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6790/0 conn(0x7f4940010c50 legacy=0x7f4940011070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.189:6789/0 conn(0x7f496006cf50 legacy=0x7f494000d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940011910 con 0x7f494000e9b0 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4950001510 con 0x7f494000e9b0 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:50.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: dump: 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f496493d700 5 monclient: authenticate success, global_id 214212 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f496493d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49500017b0 con 0x7f494000e9b0 2024-09-16T06:47:50.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 1 -- 172.21.15.76:0/376827384 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4950001b80 con 0x7f494000e9b0 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f496493d700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f496493d700 4 set_mon_vals no callback set 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.029+0000 7f495e7fc700 10 monclient: dump: 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.035 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.036 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.036 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.036 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: shutdown 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- 172.21.15.76:0/376827384 >> v1:172.21.15.76:6789/0 conn(0x7f494000e9b0 legacy=0x7f494000ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- 172.21.15.76:0/376827384 shutdown_connections 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- 172.21.15.76:0/376827384 >> 172.21.15.76:0/376827384 conn(0x7f49600a41c0 msgr2=0x7f49600a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- 172.21.15.76:0/376827384 shutdown_connections 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- 172.21.15.76:0/376827384 wait complete. 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: build_initial_monmap 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: monmap: 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:50.038 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 Processor -- start 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- start start 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: init 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:47:50.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient(hunting): picked mon.noname-b con 0x7f494000e9b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient(hunting): picked mon.noname-c con 0x7f4940010c50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient(hunting): picked mon.noname-a con 0x7f496006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4960124540 con 0x7f496006cf50 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4960124820 con 0x7f494000e9b0 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496009e8b0 con 0x7f4940010c50 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f496493d700 10 monclient(hunting): authenticate will time out at 1417.171106s 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f495f7fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f494000e9b0 legacy=0x7f49601c9e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f495ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f496006cf50 0x7f49601d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35224/0 (socket says 172.21.15.76:35224) 2024-09-16T06:47:50.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.033+0000 7f495ffff700 1 -- 172.21.15.76:0/260285537 learned_addr learned my addr 172.21.15.76:0/260285537 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263943320 0 0) 0x7f4960124540 con 0x7f496006cf50 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): my global_id is 214215 2024-09-16T06:47:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4940011cd0 con 0x7f496006cf50 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262746686 0 0) 0x7f4940011cd0 con 0x7f496006cf50 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4960124540 con 0x7f496006cf50 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4958004710 con 0x7f496006cf50 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): dump: 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3708469700 0 0) 0x7f4960124540 con 0x7f496006cf50 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 >> v1:172.21.15.76:6790/0 conn(0x7f4940010c50 legacy=0x7f49601cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 >> v1:172.21.15.189:6789/0 conn(0x7f494000e9b0 legacy=0x7f49601c9e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 monclient: found mon.a 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49600a7320 con 0x7f496006cf50 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 5 monclient: authenticate success, global_id 214215 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 10 monclient: _renew_subs 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4960122e10 con 0x7f496006cf50 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 10 monclient: _renew_subs 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49601224c0 con 0x7f496006cf50 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4958004ba0 con 0x7f496006cf50 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4958004f30 con 0x7f496006cf50 2024-09-16T06:47:50.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f495dffb700 4 set_mon_vals no callback set 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.045+0000 7f4957fff700 10 monclient: dump: 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f496493d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f496493d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f496493d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4944005100 con 0x7f496006cf50 2024-09-16T06:47:50.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4958005110 con 0x7f496006cf50 2024-09-16T06:47:50.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f4958044250 con 0x7f496006cf50 2024-09-16T06:47:50.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.049+0000 7f495f7fe700 10 monclient: get_auth_request con 0x7f4940037a40 auth_method 0 2024-09-16T06:47:50.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.053+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f4958006690 con 0x7f496006cf50 2024-09-16T06:47:50.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.053+0000 7f4957fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.053+0000 7f4957fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:50.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f496493d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:47:50.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f496493d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:50.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f496493d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4944005930 con 0x7f496006cf50 2024-09-16T06:47:50.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f4957fff700 1 -- 172.21.15.76:0/260285537 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 955636762) 0x7f4958013060 con 0x7f496006cf50 2024-09-16T06:47:50.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f4957fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:47:50.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.285+0000 7f4957fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:50.286 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:47:50.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4940037a40 legacy=0x7f4940039f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 10 monclient: shutdown 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 >> v1:172.21.15.76:6789/0 conn(0x7f496006cf50 legacy=0x7f49601d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 shutdown_connections 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 >> 172.21.15.76:0/260285537 conn(0x7f49600a41c0 msgr2=0x7f496009eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 shutdown_connections 2024-09-16T06:47:50.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.289+0000 7f496493d700 1 -- 172.21.15.76:0/260285537 wait complete. 2024-09-16T06:47:50.304 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:47:50.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:47:50.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 10 monclient: get_monmap_and_config 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 10 monclient: build_initial_monmap 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 10 monclient: monmap: 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.419 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:50.422959+0000 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:50.422959+0000 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:50.420 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 1 Processor -- start 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 1 -- start start 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.417+0000 7fa53eff0700 10 monclient: init 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-a con 0x7fa53809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-b con 0x7fa538098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-c con 0x7fa538099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53809d560 con 0x7fa53809ed60 2024-09-16T06:47:50.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53809f140 con 0x7fa538099030 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53809d790 con 0x7fa538098be0 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient(hunting): authenticate will time out at 1417.556151s 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53e7ef700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fa538099030 legacy=0x7fa53813ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53d7ed700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa538098be0 0x7fa5381374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35518/0 (socket says 172.21.15.76:35518) 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53dfee700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa53809ed60 0x7fa5380984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35234/0 (socket says 172.21.15.76:35234) 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53d7ed700 1 -- 172.21.15.76:0/3414738867 learned_addr learned my addr 172.21.15.76:0/3414738867 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777070519 0 0) 0x7fa53809d560 con 0x7fa53809ed60 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): my global_id is 214218 2024-09-16T06:47:50.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5200035d0 con 0x7fa53809ed60 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102430870 0 0) 0x7fa53809d790 con 0x7fa538098be0 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient(hunting): my global_id is 194660 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53809d560 con 0x7fa538098be0 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934767602 0 0) 0x7fa5200035d0 con 0x7fa53809ed60 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 >> v1:172.21.15.76:6790/0 conn(0x7fa538098be0 legacy=0x7fa5381374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 >> v1:172.21.15.189:6789/0 conn(0x7fa538099030 legacy=0x7fa53813ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 monclient: found mon.noname-a 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53809f620 con 0x7fa53809ed60 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 5 monclient: authenticate success, global_id 214218 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa52c001610 con 0x7fa53809ed60 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: dump: 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.427 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa52c0018f0 con 0x7fa53809ed60 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 1 -- 172.21.15.76:0/3414738867 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa52c001c80 con 0x7fa53809ed60 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 4 set_mon_vals no callback set 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53cfec700 10 monclient: dump: 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.429 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 10 monclient: shutdown 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- 172.21.15.76:0/3414738867 >> v1:172.21.15.76:6789/0 conn(0x7fa53809ed60 legacy=0x7fa5380984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- 172.21.15.76:0/3414738867 shutdown_connections 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- 172.21.15.76:0/3414738867 >> 172.21.15.76:0/3414738867 conn(0x7fa5380a0030 msgr2=0x7fa5380a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- 172.21.15.76:0/3414738867 shutdown_connections 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.421+0000 7fa53eff0700 1 -- 172.21.15.76:0/3414738867 wait complete. 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: build_initial_monmap 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: monmap: 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:50.433 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 Processor -- start 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- start start 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: init 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-c con 0x7fa538098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-b con 0x7fa538099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient(hunting): picked mon.noname-a con 0x7fa53809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53808f0a0 con 0x7fa53809ed60 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5380a2260 con 0x7fa538099030 2024-09-16T06:47:50.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53dfee700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa538098be0 0x7fa53808d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35526/0 (socket says 172.21.15.76:35526) 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53dfee700 1 -- 172.21.15.76:0/3745584821 learned_addr learned my addr 172.21.15.76:0/3745584821 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538091470 con 0x7fa538098be0 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53d7ed700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.189:6789/0 conn(0x7fa538099030 legacy=0x7fa53808df40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient(hunting): authenticate will time out at 1417.562695s 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431138314 0 0) 0x7fa53808f0a0 con 0x7fa53809ed60 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): my global_id is 214221 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5240035d0 con 0x7fa53809ed60 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795843663 0 0) 0x7fa538091470 con 0x7fa538098be0 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): my global_id is 194663 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53808f0a0 con 0x7fa538098be0 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2927490507 0 0) 0x7fa5240035d0 con 0x7fa53809ed60 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa538091470 con 0x7fa53809ed60 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa5300040e0 con 0x7fa53809ed60 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): dump: 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304194102 0 0) 0x7fa538091470 con 0x7fa53809ed60 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6790/0 conn(0x7fa538098be0 legacy=0x7fa53808d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.189:6789/0 conn(0x7fa538099030 legacy=0x7fa53808df40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 monclient: found mon.a 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5380a2650 con 0x7fa53809ed60 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 5 monclient: authenticate success, global_id 214221 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _renew_subs 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53809a540 con 0x7fa53809ed60 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _renew_subs 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa530004300 con 0x7fa53809ed60 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa538091190 con 0x7fa53809ed60 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa530004610 con 0x7fa53809ed60 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa5367fc700 10 monclient: dump: 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.425+0000 7fa537fff700 4 set_mon_vals no callback set 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa53eff0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa5300047f0 con 0x7fa53809ed60 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa518006590 con 0x7fa518002830 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7fa530043b70 con 0x7fa53809ed60 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722451625 0 0) 0x7fa518006590 con 0x7fa518002830 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient(hunting): my global_id is 214224 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538091470 con 0x7fa518002830 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa53d7ed700 10 monclient: get_auth_request con 0x7fa5240381b0 auth_method 0 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1531464341 0 0) 0x7fa538091470 con 0x7fa518002830 2024-09-16T06:47:50.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa518006590 con 0x7fa518002830 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa52c004cc0 con 0x7fa518002830 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient: dump: 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.446 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2406824922 0 0) 0x7fa518006590 con 0x7fa518002830 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa51800a190 con 0x7fa518002830 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa52c007050 con 0x7fa518002830 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6789/0 conn(0x7fa518002830 legacy=0x7fa518004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.429+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6789/0 conn(0x7fa518002830 legacy=0x7fa518004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa518009e30 con 0x7fa518009150 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808516947 0 0) 0x7fa518009e30 con 0x7fa518009150 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient(hunting): my global_id is 214227 2024-09-16T06:47:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa518006590 con 0x7fa518009150 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1761160832 0 0) 0x7fa518006590 con 0x7fa518009150 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa518009e30 con 0x7fa518009150 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa52c0054c0 con 0x7fa518009150 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient: dump: 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.448 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2567723683 0 0) 0x7fa518009e30 con 0x7fa518009150 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa51800c340 con 0x7fa518009150 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3868 (unknown 0 0 3261939901) 0x7fa52c006870 con 0x7fa518009150 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6789/0 conn(0x7fa518009150 legacy=0x7fa5180095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa5367fc700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6789/0 conn(0x7fa518009150 legacy=0x7fa5180095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:50.449 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 90, 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:50.460 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:50.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa5240381b0 legacy=0x7fa52403a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 10 monclient: shutdown 2024-09-16T06:47:50.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 >> v1:172.21.15.76:6789/0 conn(0x7fa53809ed60 legacy=0x7fa5381cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53e7ef700 1 -- 172.21.15.76:0/3745584821 reap_dead start 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53e7ef700 1 -- 172.21.15.76:0/3745584821 reap_dead start 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 shutdown_connections 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 >> 172.21.15.76:0/3745584821 conn(0x7fa5380a0030 msgr2=0x7fa5380a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 shutdown_connections 2024-09-16T06:47:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.437+0000 7fa53eff0700 1 -- 172.21.15.76:0/3745584821 wait complete. 2024-09-16T06:47:50.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:47:50.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:47:50.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:47:50.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: get_monmap_and_config 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: build_initial_monmap 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: monmap: 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:47:50.585320+0000 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:47:50.585320+0000 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:47:50.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 1 Processor -- start 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 1 -- start start 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: init 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-a con 0x7f8078132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-b con 0x7f807806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-c con 0x7f807806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807806c280 con 0x7f8078132330 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807806bfa0 con 0x7f807806cf50 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807809ace0 con 0x7f807806d480 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807dda4700 10 monclient(hunting): authenticate will time out at 1417.719547s 2024-09-16T06:47:50.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807d5a3700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f807806cf50 legacy=0x7f807813da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807cda2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8078132330 0x7f8078132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35270/0 (socket says 172.21.15.76:35270) 2024-09-16T06:47:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f8077fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f807806d480 0x7f807813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35536/0 (socket says 172.21.15.76:35536) 2024-09-16T06:47:50.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.581+0000 7f807cda2700 1 -- 172.21.15.76:0/3365141951 learned_addr learned my addr 172.21.15.76:0/3365141951 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676806036 0 0) 0x7f807809ace0 con 0x7f807806d480 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): my global_id is 194666 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80600035d0 con 0x7f807806d480 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079329084 0 0) 0x7f807806c280 con 0x7f8078132330 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient(hunting): my global_id is 214230 2024-09-16T06:47:50.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807809ace0 con 0x7f8078132330 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203772479 0 0) 0x7f80600035d0 con 0x7f807806d480 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 >> v1:172.21.15.189:6789/0 conn(0x7f807806cf50 legacy=0x7f807813da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 >> v1:172.21.15.76:6789/0 conn(0x7f8078132330 legacy=0x7f8078132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 monclient: found mon.noname-b 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807809ae70 con 0x7f807806d480 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 5 monclient: authenticate success, global_id 194666 2024-09-16T06:47:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f806c001610 con 0x7f807806d480 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: dump: 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.589 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f806c001870 con 0x7f807806d480 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: get_monmap_and_config success 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 4 set_mon_vals no callback set 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 1 -- 172.21.15.76:0/3365141951 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f806c001aa0 con 0x7f807806d480 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f80777fe700 10 monclient: dump: 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.590 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: shutdown 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- 172.21.15.76:0/3365141951 >> v1:172.21.15.76:6790/0 conn(0x7f807806d480 legacy=0x7f807813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- 172.21.15.76:0/3365141951 shutdown_connections 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- 172.21.15.76:0/3365141951 >> 172.21.15.76:0/3365141951 conn(0x7f80780a41c0 msgr2=0x7f80780a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- 172.21.15.76:0/3365141951 shutdown_connections 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- 172.21.15.76:0/3365141951 wait complete. 2024-09-16T06:47:50.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: build_initial_monmap 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: monmap: 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 Processor -- start 2024-09-16T06:47:50.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- start start 2024-09-16T06:47:50.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: init 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: _reopen_session rank -1 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-c con 0x7f807806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-b con 0x7f807806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient(hunting): picked mon.noname-a con 0x7f8078132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807812f410 con 0x7f8078132330 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807812df60 con 0x7f807806d480 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8078131490 con 0x7f807806cf50 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient(hunting): _renew_subs 2024-09-16T06:47:50.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807dda4700 10 monclient(hunting): authenticate will time out at 1417.723611s 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807cda2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f807806cf50 0x7f807812d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:35538/0 (socket says 172.21.15.76:35538) 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f807cda2700 1 -- 172.21.15.76:0/10086010 learned_addr learned my addr 172.21.15.76:0/10086010 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.585+0000 7f8077fff700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.189:6789/0 conn(0x7f807806d480 legacy=0x7f807812d850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37527613 0 0) 0x7f8078131490 con 0x7f807806cf50 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): my global_id is 194669 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80640035d0 con 0x7f807806cf50 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58139702 0 0) 0x7f807812f410 con 0x7f8078132330 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): my global_id is 214233 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8078131490 con 0x7f8078132330 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239680036 0 0) 0x7f80640035d0 con 0x7f807806cf50 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f807812f410 con 0x7f807806cf50 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8068001370 con 0x7f807806cf50 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): dump: 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.599 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541571186 0 0) 0x7f8078131490 con 0x7f8078132330 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80640035d0 con 0x7f8078132330 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8070004430 con 0x7f8078132330 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): dump: 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.600 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2220703362 0 0) 0x7f80640035d0 con 0x7f8078132330 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6790/0 conn(0x7f807806cf50 legacy=0x7f807812d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.189:6789/0 conn(0x7f807806d480 legacy=0x7f807812d850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: _finish_hunting 0 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 monclient: found mon.a 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:47:50.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807809e8b0 con 0x7f8078132330 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: _finish_auth 0 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: _check_auth_tickets 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 5 monclient: authenticate success, global_id 214233 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 10 monclient: _renew_subs 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807812ecc0 con 0x7f8078132330 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 10 monclient: _renew_subs 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8078130d40 con 0x7f8078132330 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8070004650 con 0x7f8078132330 2024-09-16T06:47:50.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80700049e0 con 0x7f8078132330 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80767fc700 4 set_mon_vals no callback set 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: dump: 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.603 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8058006590 con 0x7f8058002830 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044350028 0 0) 0x7f8058006590 con 0x7f8058002830 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient(hunting): my global_id is 194672 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80640035d0 con 0x7f8058002830 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8070004c00 con 0x7f8078132330 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f8070043fb0 con 0x7f8078132330 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2131398804 0 0) 0x7f80640035d0 con 0x7f8058002830 2024-09-16T06:47:50.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8058006590 con 0x7f8058002830 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8068001680 con 0x7f8058002830 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f8077fff700 10 monclient: get_auth_request con 0x7f8064036320 auth_method 0 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 10 monclient: dump: 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3699327821 0 0) 0x7f8058006590 con 0x7f8058002830 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.589+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f805800a190 con 0x7f8058002830 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.593+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8068007480 con 0x7f8058002830 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.593+0000 7f80757fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.593+0000 7f80757fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.593+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6790/0 conn(0x7f8058002830 legacy=0x7f8058004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.593+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6790/0 conn(0x7f8058002830 legacy=0x7f8058004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f807dda4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8058009e50 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697072514 0 0) 0x7f8058009e50 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 10 monclient(hunting): _negotiate 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 10 monclient(hunting): my global_id is 194675 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.597+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8058006590 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2297833909 0 0) 0x7f8058006590 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8058009e50 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8068005430 con 0x7f8058009170 2024-09-16T06:47:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 10 monclient: dump: 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:47:50.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 644458129 0 0) 0x7f8058009e50 con 0x7f8058009170 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f805800c340 con 0x7f8058009170 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1877954499) 0x7f8068007670 con 0x7f8058009170 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6790/0 conn(0x7f8058009170 legacy=0x7f8058009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f80757fa700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6790/0 conn(0x7f8058009170 legacy=0x7f8058009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 90, 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:47:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:47:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:47:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:47:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:47:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:47:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:47:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:47:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:47:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:47:50.622 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:47:50.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8064036320 legacy=0x7f80640387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 10 monclient: shutdown 2024-09-16T06:47:50.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 >> v1:172.21.15.76:6789/0 conn(0x7f8078132330 legacy=0x7f80781d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807d5a3700 1 -- 172.21.15.76:0/10086010 reap_dead start 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807d5a3700 1 -- 172.21.15.76:0/10086010 reap_dead start 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 shutdown_connections 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 >> 172.21.15.76:0/10086010 conn(0x7f80780a41c0 msgr2=0x7f807809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 shutdown_connections 2024-09-16T06:47:50.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:47:50.601+0000 7f807dda4700 1 -- 172.21.15.76:0/10086010 wait complete. 2024-09-16T06:47:50.624 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:47:50.624 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:48:10.625 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:48:10.625 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:48:10.626 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:48:10.626 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:48:10.629 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:48:10.629 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:48:10.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:48:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: get_monmap_and_config 2024-09-16T06:48:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: build_initial_monmap 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: monmap: 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:10.753376+0000 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:10.753376+0000 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:10.750 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 1 Processor -- start 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 1 -- start start 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: init 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-b con 0x7f59a409f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-c con 0x7f59a409fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-a con 0x7f59a4139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a409cd80 con 0x7f59a4139b70 2024-09-16T06:48:10.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4099fd0 con 0x7f59a409fd20 2024-09-16T06:48:10.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a409a410 con 0x7f59a409f230 2024-09-16T06:48:10.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:10.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a978f700 10 monclient(hunting): authenticate will time out at 1437.886761s 2024-09-16T06:48:10.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a2ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f59a409f230 0x7f59a409f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43168/0 (socket says 172.21.15.76:43168) 2024-09-16T06:48:10.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.749+0000 7f59a2ffd700 1 -- 172.21.15.76:0/227677309 learned_addr learned my addr 172.21.15.76:0/227677309 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:10.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811699956 0 0) 0x7f59a409cd80 con 0x7f59a4139b70 2024-09-16T06:48:10.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient(hunting): my global_id is 214236 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59900035d0 con 0x7f59a4139b70 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3228414501 0 0) 0x7f59900035d0 con 0x7f59a4139b70 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 >> v1:172.21.15.76:6790/0 conn(0x7f59a409f230 legacy=0x7f59a409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 >> v1:172.21.15.189:6789/0 conn(0x7f59a409fd20 legacy=0x7f59a4138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 monclient: found mon.noname-a 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:48:10.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a409de90 con 0x7f59a4139b70 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f598c001610 con 0x7f59a4139b70 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: dump: 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:10.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 5 monclient: authenticate success, global_id 214236 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f598c0018c0 con 0x7f59a4139b70 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 1 -- 172.21.15.76:0/227677309 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f598c001c50 con 0x7f59a4139b70 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 4 set_mon_vals no callback set 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:10.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a1ffb700 10 monclient: dump: 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:10.782 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 10 monclient: shutdown 2024-09-16T06:48:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 1 -- 172.21.15.76:0/227677309 >> v1:172.21.15.76:6789/0 conn(0x7f59a4139b70 legacy=0x7f59a413bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 1 -- 172.21.15.76:0/227677309 shutdown_connections 2024-09-16T06:48:10.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.777+0000 7f59a978f700 1 -- 172.21.15.76:0/227677309 >> 172.21.15.76:0/227677309 conn(0x7f59a40a3e50 msgr2=0x7f59a40a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- 172.21.15.76:0/227677309 shutdown_connections 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- 172.21.15.76:0/227677309 wait complete. 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient: build_initial_monmap 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient: monmap: 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:10.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 Processor -- start 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- start start 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient: init 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-a con 0x7f59a409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-b con 0x7f59a409fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient(hunting): picked mon.noname-c con 0x7f59a4139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4135250 con 0x7f59a409f230 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4135560 con 0x7f59a409fd20 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a413a100 con 0x7f59a4139b70 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a978f700 10 monclient(hunting): authenticate will time out at 1437.919601s 2024-09-16T06:48:10.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f59a4139b70 0x7f59a41d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43170/0 (socket says 172.21.15.76:43170) 2024-09-16T06:48:10.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f59a409f230 0x7f59a4134430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51006/0 (socket says 172.21.15.76:51006) 2024-09-16T06:48:10.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.781+0000 7f59a37fe700 1 -- 172.21.15.76:0/4106488017 learned_addr learned my addr 172.21.15.76:0/4106488017 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271596099 0 0) 0x7f59a413a100 con 0x7f59a4139b70 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): my global_id is 194678 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59840035d0 con 0x7f59a4139b70 2024-09-16T06:48:10.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866198363 0 0) 0x7f59840035d0 con 0x7f59a4139b70 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59a413a100 con 0x7f59a4139b70 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f598c001690 con 0x7f59a4139b70 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): dump: 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:10.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.785+0000 7f599b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1602105925 0 0) 0x7f59a413a100 con 0x7f59a4139b70 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 >> v1:172.21.15.189:6789/0 conn(0x7f59a409fd20 legacy=0x7f59a4134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 >> v1:172.21.15.76:6789/0 conn(0x7f59a409f230 legacy=0x7f59a4134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 monclient: found mon.c 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a41397a0 con 0x7f59a4139b70 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:10.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 5 monclient: authenticate success, global_id 194678 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _renew_subs 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a4137140 con 0x7f59a4139b70 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _renew_subs 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:10.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a41390c0 con 0x7f59a4139b70 2024-09-16T06:48:10.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f598c003b40 con 0x7f59a4139b70 2024-09-16T06:48:10.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f598c003ed0 con 0x7f59a4139b70 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a0ff9700 4 set_mon_vals no callback set 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 10 monclient: dump: 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:10.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f598c004030 con 0x7f59a4139b70 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5988005100 con 0x7f59a4139b70 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f598c044060 con 0x7f59a4139b70 2024-09-16T06:48:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.789+0000 7f59a2ffd700 10 monclient: get_auth_request con 0x7f5984036150 auth_method 0 2024-09-16T06:48:10.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.797+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f598c015330 con 0x7f59a4139b70 2024-09-16T06:48:10.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.797+0000 7f599b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:10.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:10.797+0000 7f599b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f59a978f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f59a978f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f59a978f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5988005930 con 0x7f59a4139b70 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f599b7fe700 1 -- 172.21.15.76:0/4106488017 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 3514561857) 0x7f598c015330 con 0x7f59a4139b70 2024-09-16T06:48:11.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f599b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:11.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.037+0000 7f599b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:11.039 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:48:11.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5984036150 legacy=0x7f5984038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 10 monclient: shutdown 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 >> v1:172.21.15.76:6790/0 conn(0x7f59a4139b70 legacy=0x7f59a41d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 shutdown_connections 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 >> 172.21.15.76:0/4106488017 conn(0x7f59a40a3e50 msgr2=0x7f59a412d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 shutdown_connections 2024-09-16T06:48:11.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.041+0000 7f59a978f700 1 -- 172.21.15.76:0/4106488017 wait complete. 2024-09-16T06:48:11.056 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:48:11.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:48:11.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.157+0000 7f33bf59e700 10 monclient: get_monmap_and_config 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.157+0000 7f33bf59e700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.157+0000 7f33bf59e700 10 monclient: monmap: 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:11.163010+0000 2024-09-16T06:48:11.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:11.163010+0000 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 Processor -- start 2024-09-16T06:48:11.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 -- start start 2024-09-16T06:48:11.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient: init 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f33c0131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f33c01347f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f33c0138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0132400 con 0x7f33c01347f0 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c009fcd0 con 0x7f33c0138300 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33be59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f33c0131910 0x7f33c0131cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43182/0 (socket says 172.21.15.76:43182) 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c009b300 con 0x7f33c0131910 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33be59c700 1 -- 172.21.15.76:0/3671795120 learned_addr learned my addr 172.21.15.76:0/3671795120 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient(hunting): authenticate will time out at 1438.298149s 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714971128 0 0) 0x7f33c009b300 con 0x7f33c0131910 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): my global_id is 194681 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a80035d0 con 0x7f33c0131910 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536908411 0 0) 0x7f33c0132400 con 0x7f33c01347f0 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient(hunting): my global_id is 214239 2024-09-16T06:48:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c009b300 con 0x7f33c01347f0 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445612748 0 0) 0x7f33a80035d0 con 0x7f33c0131910 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 >> v1:172.21.15.189:6789/0 conn(0x7f33c0138300 legacy=0x7f33c013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 >> v1:172.21.15.76:6789/0 conn(0x7f33c01347f0 legacy=0x7f33c0136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 monclient: found mon.noname-b 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c009b7e0 con 0x7f33c0131910 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 5 monclient: authenticate success, global_id 194681 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b0001610 con 0x7f33c0131910 2024-09-16T06:48:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: dump: 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.167 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33b00018b0 con 0x7f33c0131910 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 1 -- 172.21.15.76:0/3671795120 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b0001c40 con 0x7f33c0131910 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 4 set_mon_vals no callback set 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bd59a700 10 monclient: dump: 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 10 monclient: shutdown 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.161+0000 7f33bf59e700 1 -- 172.21.15.76:0/3671795120 >> v1:172.21.15.76:6790/0 conn(0x7f33c0131910 legacy=0x7f33c0131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3671795120 shutdown_connections 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3671795120 >> 172.21.15.76:0/3671795120 conn(0x7f33c00a4430 msgr2=0x7f33c00a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3671795120 shutdown_connections 2024-09-16T06:48:11.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3671795120 wait complete. 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: monmap: 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:11.175 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 Processor -- start 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- start start 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: init 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f33c0131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f33c01347f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f33c0138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0070f20 con 0x7f33c01347f0 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0138e50 con 0x7f33c0138300 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0139800 con 0x7f33c0131910 2024-09-16T06:48:11.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient(hunting): authenticate will time out at 1438.302003s 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bdd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f33c01347f0 0x7f33c0070810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51038/0 (socket says 172.21.15.76:51038) 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bdd9b700 1 -- 172.21.15.76:0/3643235454 learned_addr learned my addr 172.21.15.76:0/3643235454 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33be59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f33c0131910 0x7f33c0070100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43190/0 (socket says 172.21.15.76:43190) 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182392433 0 0) 0x7f33c0070f20 con 0x7f33c01347f0 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): my global_id is 214242 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a00035d0 con 0x7f33c01347f0 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940079460 0 0) 0x7f33c0139800 con 0x7f33c0131910 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): my global_id is 194684 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c0070f20 con 0x7f33c0131910 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 948502990 0 0) 0x7f33a00035d0 con 0x7f33c01347f0 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33c0139800 con 0x7f33c01347f0 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b40019f0 con 0x7f33c01347f0 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): dump: 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.178 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470522220 0 0) 0x7f33c0070f20 con 0x7f33c0131910 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a00035d0 con 0x7f33c0131910 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b0001bc0 con 0x7f33c0131910 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): dump: 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218926034 0 0) 0x7f33c0138e50 con 0x7f33c0138300 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient(hunting): my global_id is 224101 2024-09-16T06:48:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c0070f20 con 0x7f33c0138300 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3441320551 0 0) 0x7f33c0139800 con 0x7f33c01347f0 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6790/0 conn(0x7f33c0131910 legacy=0x7f33c0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.189:6789/0 conn(0x7f33c0138300 legacy=0x7f33c01cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 monclient: found mon.a 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0139520 con 0x7f33c01347f0 2024-09-16T06:48:11.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 5 monclient: authenticate success, global_id 214242 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _renew_subs 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c0071620 con 0x7f33c01347f0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _renew_subs 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c013f120 con 0x7f33c01347f0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33b40036a0 con 0x7f33c01347f0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b4003a30 con 0x7f33c01347f0 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33bcd99700 4 set_mon_vals no callback set 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.165+0000 7f33aeffd700 10 monclient: dump: 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f33b4003b90 con 0x7f33c01347f0 2024-09-16T06:48:11.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33bf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33a4006590 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f33b4043d10 con 0x7f33c01347f0 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33be59c700 10 monclient: get_auth_request con 0x7f33a0036580 auth_method 0 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255540689 0 0) 0x7f33a4006590 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient(hunting): my global_id is 214245 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c0139800 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4016784004 0 0) 0x7f33c0139800 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a4006590 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b8001a00 con 0x7f33a4002830 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient: dump: 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1731074887 0 0) 0x7f33a4006590 con 0x7f33a4002830 2024-09-16T06:48:11.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33a400a190 con 0x7f33a4002830 2024-09-16T06:48:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f33b8001ce0 con 0x7f33a4002830 2024-09-16T06:48:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6789/0 conn(0x7f33a4002830 legacy=0x7f33a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.169+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6789/0 conn(0x7f33a4002830 legacy=0x7f33a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:11.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f33a4009e50 con 0x7f33a4009170 2024-09-16T06:48:11.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974773701 0 0) 0x7f33a4009e50 con 0x7f33a4009170 2024-09-16T06:48:11.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient(hunting): my global_id is 214248 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a4006590 con 0x7f33a4009170 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2078024225 0 0) 0x7f33a4006590 con 0x7f33a4009170 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a4009e50 con 0x7f33a4009170 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33b8003bc0 con 0x7f33a4009170 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient: dump: 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.188 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3718809837 0 0) 0x7f33a4009e50 con 0x7f33a4009170 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33a400c340 con 0x7f33a4009170 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2450371421) 0x7f33b8004f30 con 0x7f33a4009170 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6789/0 conn(0x7f33a4009170 legacy=0x7f33a4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33aeffd700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6789/0 conn(0x7f33a4009170 legacy=0x7f33a4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.190 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.191 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.192 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:11.193 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.194 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:11.195 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.198 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:11.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.201 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:11.201 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.189:6812/3098810674 conn(0x7f33a0036580 legacy=0x7f33a0038a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 10 monclient: shutdown 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 >> v1:172.21.15.76:6789/0 conn(0x7f33c01347f0 legacy=0x7f33c0070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bed9d700 1 -- 172.21.15.76:0/3643235454 reap_dead start 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bed9d700 1 -- 172.21.15.76:0/3643235454 reap_dead start 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 shutdown_connections 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 >> 172.21.15.76:0/3643235454 conn(0x7f33c00a4430 msgr2=0x7f33c00a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 shutdown_connections 2024-09-16T06:48:11.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.177+0000 7f33bf59e700 1 -- 172.21.15.76:0/3643235454 wait complete. 2024-09-16T06:48:11.203 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:11.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:48:11.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: get_monmap_and_config 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: monmap: 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.317 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:11.321195+0000 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:11.321195+0000 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:11.318 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 1 Processor -- start 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 1 -- start start 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: init 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-b con 0x7f6d780973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-a con 0x7f6d78097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-c con 0x7f6d781400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78099350 con 0x7f6d78097eb0 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7809f690 con 0x7f6d781400d0 2024-09-16T06:48:11.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78099bf0 con 0x7f6d780973c0 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient(hunting): authenticate will time out at 1438.454431s 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7cfb5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6d78097eb0 0x7f6d7813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51072/0 (socket says 172.21.15.76:51072) 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7cfb5700 1 -- 172.21.15.76:0/1212879734 learned_addr learned my addr 172.21.15.76:0/1212879734 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642013281 0 0) 0x7f6d78099350 con 0x7f6d78097eb0 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): my global_id is 214251 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d640035d0 con 0x7f6d78097eb0 2024-09-16T06:48:11.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343112298 0 0) 0x7f6d78099bf0 con 0x7f6d780973c0 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): my global_id is 194687 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78099350 con 0x7f6d780973c0 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484954128 0 0) 0x7f6d7809f690 con 0x7f6d781400d0 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient(hunting): my global_id is 224104 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78099bf0 con 0x7f6d781400d0 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203622980 0 0) 0x7f6d640035d0 con 0x7f6d78097eb0 2024-09-16T06:48:11.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 >> v1:172.21.15.76:6790/0 conn(0x7f6d780973c0 legacy=0x7f6d780977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 >> v1:172.21.15.189:6789/0 conn(0x7f6d781400d0 legacy=0x7f6d781424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 monclient: found mon.noname-a 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78098360 con 0x7f6d78097eb0 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d74001610 con 0x7f6d78097eb0 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:11.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: dump: 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 5 monclient: authenticate success, global_id 214251 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d740018f0 con 0x7f6d78097eb0 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 1 -- 172.21.15.76:0/1212879734 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d74001c80 con 0x7f6d78097eb0 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d7e7b8700 4 set_mon_vals no callback set 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.317+0000 7f6d6ffff700 10 monclient: dump: 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.326 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 10 monclient: shutdown 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1212879734 >> v1:172.21.15.76:6789/0 conn(0x7f6d78097eb0 legacy=0x7f6d7813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1212879734 shutdown_connections 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1212879734 >> 172.21.15.76:0/1212879734 conn(0x7f6d780a0030 msgr2=0x7f6d780a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1212879734 shutdown_connections 2024-09-16T06:48:11.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1212879734 wait complete. 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 10 monclient: monmap: 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:11.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 Processor -- start 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 1 -- start start 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.321+0000 7f6d7e7b8700 10 monclient: init 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-a con 0x7f6d780973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-b con 0x7f6d781400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient(hunting): picked mon.noname-c con 0x7f6d7813d560 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78072660 con 0x7f6d780973c0 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7806c5c0 con 0x7f6d781400d0 2024-09-16T06:48:11.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7813fc00 con 0x7f6d7813d560 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient(hunting): authenticate will time out at 1438.460504s 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7dfb7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6d7813d560 0x7f6d781d66c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43212/0 (socket says 172.21.15.76:43212) 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7dfb7700 1 -- 172.21.15.76:0/1415759239 learned_addr learned my addr 172.21.15.76:0/1415759239 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7d7b6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6d780973c0 0x7f6d78137770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51088/0 (socket says 172.21.15.76:51088) 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006602738 0 0) 0x7f6d7813fc00 con 0x7f6d7813d560 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): my global_id is 194690 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c0035d0 con 0x7f6d7813d560 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540220033 0 0) 0x7f6d78072660 con 0x7f6d780973c0 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): my global_id is 214254 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7813fc00 con 0x7f6d780973c0 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270242547 0 0) 0x7f6d7806c5c0 con 0x7f6d781400d0 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): my global_id is 224107 2024-09-16T06:48:11.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78072660 con 0x7f6d781400d0 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165925533 0 0) 0x7f6d5c0035d0 con 0x7f6d7813d560 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7806c5c0 con 0x7f6d7813d560 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d70001c50 con 0x7f6d7813d560 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): dump: 2024-09-16T06:48:11.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1845238012 0 0) 0x7f6d7806c5c0 con 0x7f6d7813d560 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.189:6789/0 conn(0x7f6d781400d0 legacy=0x7f6d78137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6789/0 conn(0x7f6d780973c0 legacy=0x7f6d78137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 monclient: found mon.c 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78138710 con 0x7f6d7813d560 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 5 monclient: authenticate success, global_id 194690 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _renew_subs 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7809a540 con 0x7f6d7813d560 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _renew_subs 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7813ea10 con 0x7f6d7813d560 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d70001fe0 con 0x7f6d7813d560 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d70003e90 con 0x7f6d7813d560 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 10 monclient: dump: 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.341 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6f7fe700 4 set_mon_vals no callback set 2024-09-16T06:48:11.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6d7806c0e0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6d700040b0 con 0x7f6d7813d560 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.325+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f6d700440b0 con 0x7f6d7813d560 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225569055 0 0) 0x7f6d7806c0e0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient(hunting): my global_id is 194693 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7806c5c0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d7cfb5700 10 monclient: get_auth_request con 0x7f6d5c036390 auth_method 0 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2125383375 0 0) 0x7f6d7806c5c0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7806c0e0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d680016c0 con 0x7f6d7802c820 2024-09-16T06:48:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient: dump: 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1600726952 0 0) 0x7f6d7806c0e0 con 0x7f6d7802c820 2024-09-16T06:48:11.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d78071aa0 con 0x7f6d7802c820 2024-09-16T06:48:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6d68001bd0 con 0x7f6d7802c820 2024-09-16T06:48:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6790/0 conn(0x7f6d7802c820 legacy=0x7f6d78061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.329+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6790/0 conn(0x7f6d7802c820 legacy=0x7f6d78061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6d7802cc00 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843102551 0 0) 0x7f6d7802cc00 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient(hunting): my global_id is 194696 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7806c0e0 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1513073271 0 0) 0x7f6d7806c0e0 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7802cc00 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6d68001970 con 0x7f6d78094810 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient: dump: 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1837590536 0 0) 0x7f6d7802cc00 con 0x7f6d78094810 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6d780724e0 con 0x7f6d78094810 2024-09-16T06:48:11.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1500652059) 0x7f6d68005870 con 0x7f6d78094810 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6790/0 conn(0x7f6d78094810 legacy=0x7f6d781dbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d6dffb700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6790/0 conn(0x7f6d78094810 legacy=0x7f6d781dbe00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:11.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.394 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.395 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:11.396 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:11.398 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.399 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:11.402 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:48:11.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:11.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.405 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:11.405 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6d5c036390 legacy=0x7f6d5c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 10 monclient: shutdown 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 >> v1:172.21.15.76:6790/0 conn(0x7f6d7813d560 legacy=0x7f6d781d66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7dfb7700 1 -- 172.21.15.76:0/1415759239 reap_dead start 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7dfb7700 1 -- 172.21.15.76:0/1415759239 reap_dead start 2024-09-16T06:48:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 shutdown_connections 2024-09-16T06:48:11.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 >> 172.21.15.76:0/1415759239 conn(0x7f6d780a0030 msgr2=0x7f6d7809ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 shutdown_connections 2024-09-16T06:48:11.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.341+0000 7f6d7e7b8700 1 -- 172.21.15.76:0/1415759239 wait complete. 2024-09-16T06:48:11.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:11.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:48:11.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:48:11.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: get_monmap_and_config 2024-09-16T06:48:11.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: monmap: 2024-09-16T06:48:11.521 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:11.525161+0000 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:11.525161+0000 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:11.522 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 1 Processor -- start 2024-09-16T06:48:11.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 1 -- start start 2024-09-16T06:48:11.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: init 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient(hunting): picked mon.noname-a con 0x7f166409b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient(hunting): picked mon.noname-c con 0x7f16640964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient(hunting): picked mon.noname-b con 0x7f1664138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664098560 con 0x7f166409b1c0 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664099ec0 con 0x7f16640964f0 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664097030 con 0x7f1664138870 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f166889b700 10 monclient(hunting): authenticate will time out at 1438.659230s 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16637fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f166409b1c0 0x7f166409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51098/0 (socket says 172.21.15.76:51098) 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16637fe700 1 -- 172.21.15.76:0/2041072277 learned_addr learned my addr 172.21.15.76:0/2041072277 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427514139 0 0) 0x7f1664098560 con 0x7f166409b1c0 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): my global_id is 214257 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c0035d0 con 0x7f166409b1c0 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819663554 0 0) 0x7f1664097030 con 0x7f1664138870 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 10 monclient(hunting): my global_id is 194699 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.521+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1664098560 con 0x7f1664138870 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237556611 0 0) 0x7f1664099ec0 con 0x7f16640964f0 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient(hunting): my global_id is 224110 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1664097030 con 0x7f16640964f0 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2642008487 0 0) 0x7f1664098560 con 0x7f1664138870 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 >> v1:172.21.15.189:6789/0 conn(0x7f16640964f0 legacy=0x7f1664096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 >> v1:172.21.15.76:6789/0 conn(0x7f166409b1c0 legacy=0x7f166409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 monclient: found mon.noname-b 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166409b5a0 con 0x7f1664138870 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f165c0015f0 con 0x7f1664138870 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: dump: 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 5 monclient: authenticate success, global_id 194699 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f165c001910 con 0x7f1664138870 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 1 -- 172.21.15.76:0/2041072277 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f165c001ca0 con 0x7f1664138870 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 4 set_mon_vals no callback set 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f16627fc700 10 monclient: dump: 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.530 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.531 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: shutdown 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- 172.21.15.76:0/2041072277 >> v1:172.21.15.76:6790/0 conn(0x7f1664138870 legacy=0x7f166413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- 172.21.15.76:0/2041072277 shutdown_connections 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- 172.21.15.76:0/2041072277 >> 172.21.15.76:0/2041072277 conn(0x7f16640a0030 msgr2=0x7f16640a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- 172.21.15.76:0/2041072277 shutdown_connections 2024-09-16T06:48:11.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- 172.21.15.76:0/2041072277 wait complete. 2024-09-16T06:48:11.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: monmap: 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 Processor -- start 2024-09-16T06:48:11.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- start start 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: init 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient(hunting): picked mon.noname-a con 0x7f16640964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient(hunting): picked mon.noname-c con 0x7f166409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 10 monclient(hunting): picked mon.noname-b con 0x7f1664138870 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166406f6c0 con 0x7f16640964f0 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.525+0000 7f166889b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664070ee0 con 0x7f1664138870 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f16637fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f16640964f0 0x7f1664095300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51112/0 (socket says 172.21.15.76:51112) 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f16637fe700 1 -- 172.21.15.76:0/1948756793 learned_addr learned my addr 172.21.15.76:0/1948756793 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1664071a20 con 0x7f166409b1c0 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient(hunting): authenticate will time out at 1438.664052s 2024-09-16T06:48:11.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623078359 0 0) 0x7f166406f6c0 con 0x7f16640964f0 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): my global_id is 214260 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16500035d0 con 0x7f16640964f0 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452536609 0 0) 0x7f1664071a20 con 0x7f166409b1c0 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): my global_id is 194702 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166406f6c0 con 0x7f166409b1c0 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331949050 0 0) 0x7f16500035d0 con 0x7f16640964f0 2024-09-16T06:48:11.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1664071a20 con 0x7f16640964f0 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178361784 0 0) 0x7f166406f6c0 con 0x7f166409b1c0 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16500035d0 con 0x7f166409b1c0 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1654001c50 con 0x7f16640964f0 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): dump: 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.540 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f16580018b0 con 0x7f166409b1c0 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): dump: 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.541 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072943584 0 0) 0x7f1664070ee0 con 0x7f1664138870 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): my global_id is 224113 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166406f6c0 con 0x7f1664138870 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504084774 0 0) 0x7f16500035d0 con 0x7f166409b1c0 2024-09-16T06:48:11.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6789/0 conn(0x7f1664138870 legacy=0x7f16641cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.76:6789/0 conn(0x7f16640964f0 legacy=0x7f1664095300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 monclient: found mon.c 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166409be20 con 0x7f166409b1c0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 5 monclient: authenticate success, global_id 194702 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient: _renew_subs 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1664071460 con 0x7f166409b1c0 2024-09-16T06:48:11.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient: _renew_subs 2024-09-16T06:48:11.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f166406fb70 con 0x7f166409b1c0 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16580036c0 con 0x7f166409b1c0 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1658003a50 con 0x7f166409b1c0 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: dump: 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.545 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f16617fa700 4 set_mon_vals no callback set 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1644006590 con 0x7f1644002830 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1658003bb0 con 0x7f166409b1c0 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f1658042b60 con 0x7f166409b1c0 2024-09-16T06:48:11.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f1663fff700 10 monclient: get_auth_request con 0x7f1650038600 auth_method 0 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140270008 0 0) 0x7f1644006590 con 0x7f1644002830 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient(hunting): my global_id is 224116 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16500035d0 con 0x7f1644002830 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2982887758 0 0) 0x7f16500035d0 con 0x7f1644002830 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1644006590 con 0x7f1644002830 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1654005020 con 0x7f1644002830 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:11.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 10 monclient: dump: 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.529+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3537812536 0 0) 0x7f1644006590 con 0x7f1644002830 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f164400a190 con 0x7f1644002830 2024-09-16T06:48:11.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f16540073b0 con 0x7f1644002830 2024-09-16T06:48:11.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:11.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6789/0 conn(0x7f1644002830 legacy=0x7f1644004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.533+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6789/0 conn(0x7f1644002830 legacy=0x7f1644004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:11.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1644009e10 con 0x7f1644009150 2024-09-16T06:48:11.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278366359 0 0) 0x7f1644009e10 con 0x7f1644009150 2024-09-16T06:48:11.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient(hunting): my global_id is 224119 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1644006590 con 0x7f1644009150 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3394572465 0 0) 0x7f1644006590 con 0x7f1644009150 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1644009e10 con 0x7f1644009150 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f16540057e0 con 0x7f1644009150 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient: dump: 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.551 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1708195698 0 0) 0x7f1644009e10 con 0x7f1644009150 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f164400c340 con 0x7f1644009150 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 792718950) 0x7f1654006c40 con 0x7f1644009150 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6789/0 conn(0x7f1644009150 legacy=0x7f16440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f164bfff700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6789/0 conn(0x7f1644009150 legacy=0x7f16440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:11.553 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.555 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:11.556 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:11.557 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.559 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.560 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.561 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.562 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:11.563 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:11.564 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:11.564 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1650038600 legacy=0x7f165003aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 10 monclient: shutdown 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 >> v1:172.21.15.76:6790/0 conn(0x7f166409b1c0 legacy=0x7f16641cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f1663fff700 1 -- 172.21.15.76:0/1948756793 reap_dead start 2024-09-16T06:48:11.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f1663fff700 1 -- 172.21.15.76:0/1948756793 reap_dead start 2024-09-16T06:48:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 shutdown_connections 2024-09-16T06:48:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.545+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 >> 172.21.15.76:0/1948756793 conn(0x7f16640a0030 msgr2=0x7f166409dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.549+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 shutdown_connections 2024-09-16T06:48:11.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.549+0000 7f166889b700 1 -- 172.21.15.76:0/1948756793 wait complete. 2024-09-16T06:48:11.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:11.566 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:48:11.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:48:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: get_monmap_and_config 2024-09-16T06:48:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: monmap: 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:11.686319+0000 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:11.686319+0000 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:11.683 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:11.684 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 1 Processor -- start 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 1 -- start start 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: init 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-a con 0x7fcfc0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-c con 0x7fcfc0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-b con 0x7fcfc0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc0098990 con 0x7fcfc0097b60 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc0097f40 con 0x7fcfc0099090 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc009d610 con 0x7fcfc0139500 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc72fb700 10 monclient(hunting): authenticate will time out at 1438.819547s 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc62f9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcfc0097b60 0x7fcfc009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51124/0 (socket says 172.21.15.76:51124) 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.681+0000 7fcfc62f9700 1 -- 172.21.15.76:0/1549114181 learned_addr learned my addr 172.21.15.76:0/1549114181 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888661409 0 0) 0x7fcfc0098990 con 0x7fcfc0097b60 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): my global_id is 214263 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfac0035d0 con 0x7fcfc0097b60 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201944444 0 0) 0x7fcfc009d610 con 0x7fcfc0139500 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): my global_id is 194705 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc0098990 con 0x7fcfc0139500 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272057632 0 0) 0x7fcfc0097f40 con 0x7fcfc0099090 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient(hunting): my global_id is 224122 2024-09-16T06:48:11.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc009d610 con 0x7fcfc0099090 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3350241246 0 0) 0x7fcfc0098990 con 0x7fcfc0139500 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 >> v1:172.21.15.189:6789/0 conn(0x7fcfc0099090 legacy=0x7fcfc009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 >> v1:172.21.15.76:6789/0 conn(0x7fcfc0097b60 legacy=0x7fcfc009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 monclient: found mon.noname-b 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00986b0 con 0x7fcfc0139500 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 5 monclient: authenticate success, global_id 194705 2024-09-16T06:48:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcfb8001970 con 0x7fcfc0139500 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: dump: 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.695 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfb8001c90 con 0x7fcfc0139500 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 1 -- 172.21.15.76:0/1549114181 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcfb8002020 con 0x7fcfc0139500 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 4 set_mon_vals no callback set 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc52f7700 10 monclient: dump: 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.696 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: shutdown 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- 172.21.15.76:0/1549114181 >> v1:172.21.15.76:6790/0 conn(0x7fcfc0139500 legacy=0x7fcfc013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- 172.21.15.76:0/1549114181 shutdown_connections 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- 172.21.15.76:0/1549114181 >> 172.21.15.76:0/1549114181 conn(0x7fcfc00a0030 msgr2=0x7fcfc00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- 172.21.15.76:0/1549114181 shutdown_connections 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- 172.21.15.76:0/1549114181 wait complete. 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: build_initial_monmap 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: monmap: 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:11.698 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 Processor -- start 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- start start 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: init 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-b con 0x7fcfc0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-c con 0x7fcfc0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient(hunting): picked mon.noname-a con 0x7fcfc0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc0132120 con 0x7fcfc0139500 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc013caa0 con 0x7fcfc0097b60 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc0070f10 con 0x7fcfc0099090 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc72fb700 10 monclient(hunting): authenticate will time out at 1438.823400s 2024-09-16T06:48:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc5af8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcfc0099090 0x7fcfc0070800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43260/0 (socket says 172.21.15.76:43260) 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfc5af8700 1 -- 172.21.15.76:0/2878034774 learned_addr learned my addr 172.21.15.76:0/2878034774 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174172958 0 0) 0x7fcfc0132120 con 0x7fcfc0139500 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.685+0000 7fcfb6ffd700 10 monclient(hunting): my global_id is 214266 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa40035d0 con 0x7fcfc0139500 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293983594 0 0) 0x7fcfc0070f10 con 0x7fcfc0099090 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): my global_id is 194708 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc0132120 con 0x7fcfc0099090 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623757472 0 0) 0x7fcfc013caa0 con 0x7fcfc0097b60 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): my global_id is 224125 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc0070f10 con 0x7fcfc0097b60 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557330844 0 0) 0x7fcfa40035d0 con 0x7fcfc0139500 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfc013caa0 con 0x7fcfc0139500 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893406696 0 0) 0x7fcfc0132120 con 0x7fcfc0099090 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa40035d0 con 0x7fcfc0099090 2024-09-16T06:48:11.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcfb8001fa0 con 0x7fcfc0139500 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): dump: 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcfbc0018f0 con 0x7fcfc0099090 2024-09-16T06:48:11.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient(hunting): dump: 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1716294676 0 0) 0x7fcfc013caa0 con 0x7fcfc0139500 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 >> v1:172.21.15.76:6790/0 conn(0x7fcfc0099090 legacy=0x7fcfc0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 >> v1:172.21.15.189:6789/0 conn(0x7fcfc0097b60 legacy=0x7fcfc00700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 monclient: found mon.a 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0131ab0 con 0x7fcfc0139500 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:11.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 5 monclient: authenticate success, global_id 214266 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _renew_subs 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00a2a20 con 0x7fcfc0139500 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _renew_subs 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc0133700 con 0x7fcfc0139500 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfb8001460 con 0x7fcfc0139500 2024-09-16T06:48:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcfb8003f10 con 0x7fcfc0139500 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc4af6700 4 set_mon_vals no callback set 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 10 monclient: dump: 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:11.706 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcfb800e980 con 0x7fcfc0139500 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7fcfb8042ff0 con 0x7fcfc0139500 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa8005100 con 0x7fcfc0139500 2024-09-16T06:48:11.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.689+0000 7fcfc62f9700 10 monclient: get_auth_request con 0x7fcfa40364a0 auth_method 0 2024-09-16T06:48:11.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.697+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcfb80137c0 con 0x7fcfc0139500 2024-09-16T06:48:11.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.697+0000 7fcfb6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:11.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.697+0000 7fcfb6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.913+0000 7fcfc72fb700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:48:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.913+0000 7fcfc72fb700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:48:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.913+0000 7fcfc72fb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.913+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fcfa8005bc0 con 0x7fcfc0139500 2024-09-16T06:48:11.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.917+0000 7fcfb6ffd700 1 -- 172.21.15.76:0/2878034774 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fcfb8011e60 con 0x7fcfc0139500 2024-09-16T06:48:11.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.917+0000 7fcfb6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:48:11.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.917+0000 7fcfb6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:11.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=454889421,config=1097430493,health=3920756396,kv=790649379,logm=2790847307}) 2024-09-16T06:48:11.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=13,logm=47} crc {auth=4015571301,config=1097430493,health=3920756396,kv=1001081567,logm=2017418379}) 2024-09-16T06:48:11.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=454889421,config=1097430493,health=3920756396,kv=790649379,logm=2790847307}) 2024-09-16T06:48:11.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.917+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=217055303,config=1097430493,health=4126700924,kv=1477047248,logm=627851135}) 2024-09-16T06:48:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcfa40364a0 legacy=0x7fcfa4038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 10 monclient: shutdown 2024-09-16T06:48:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 >> v1:172.21.15.76:6789/0 conn(0x7fcfc0139500 legacy=0x7fcfc01d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 shutdown_connections 2024-09-16T06:48:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 >> 172.21.15.76:0/2878034774 conn(0x7fcfc00a0030 msgr2=0x7fcfc013b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:11.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228510799}) 2024-09-16T06:48:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 shutdown_connections 2024-09-16T06:48:11.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421326756}) 2024-09-16T06:48:11.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228510799}) 2024-09-16T06:48:11.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1347844991}) 2024-09-16T06:48:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:11.921+0000 7fcfc72fb700 1 -- 172.21.15.76:0/2878034774 wait complete. 2024-09-16T06:48:11.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606534569}) 2024-09-16T06:48:11.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126417424}) 2024-09-16T06:48:11.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606534569}) 2024-09-16T06:48:11.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.921+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3534763536}) 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688458408}) 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063921372}) 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688458408}) 2024-09-16T06:48:11.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137146382}) 2024-09-16T06:48:11.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838527751}) 2024-09-16T06:48:11.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724109390}) 2024-09-16T06:48:11.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838527751}) 2024-09-16T06:48:11.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.925+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1865070636}) 2024-09-16T06:48:11.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618365715}) 2024-09-16T06:48:11.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989693072}) 2024-09-16T06:48:11.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618365715}) 2024-09-16T06:48:11.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385783146}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130485604}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416894015}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130485604}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718057159}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2328634545}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727486320}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2328634545}) 2024-09-16T06:48:11.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401683480}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140241066}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9643079}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140241066}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.929+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2954702855}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218830931}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217013937}) 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.933 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218830931}) 2024-09-16T06:48:11.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4276705774}) 2024-09-16T06:48:11.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191493063}) 2024-09-16T06:48:11.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047535898}) 2024-09-16T06:48:11.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1191493063}) 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215212544}) 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407119727}) 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777207139}) 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407119727}) 2024-09-16T06:48:11.936 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755815268}) 2024-09-16T06:48:11.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066560771}) 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.933+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185772839}) 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066560771}) 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388426954}) 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.939 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852475313}) 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051251776}) 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852475313}) 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505116786}) 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639694690}) 2024-09-16T06:48:11.940 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873949250}) 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639694690}) 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3100439846}) 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733970679}) 2024-09-16T06:48:11.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738425768}) 2024-09-16T06:48:11.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733970679}) 2024-09-16T06:48:11.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1061123217}) 2024-09-16T06:48:11.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496001416}) 2024-09-16T06:48:11.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460424801}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496001416}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.937+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478023305}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323493387}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=127149503}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323493387}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138807614}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948134947}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792598545}) 2024-09-16T06:48:11.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948134947}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2361825751}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030982285}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4121625539}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030982285}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611994267}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027918759}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808995451}) 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.951 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027918759}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.941+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1132643545}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035754237}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003038759}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035754237}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2685861660}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564872261}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775402392}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564872261}) 2024-09-16T06:48:11.952 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2597578682}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87785163}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692656879}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87785163}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.945+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560603799}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742407987}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748251773}) 2024-09-16T06:48:11.953 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742407987}) 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583096564}) 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139571403}) 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498245655}) 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139571403}) 2024-09-16T06:48:11.954 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1932139581}) 2024-09-16T06:48:11.957 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.957 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592401504}) 2024-09-16T06:48:11.957 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574573967}) 2024-09-16T06:48:11.957 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.957 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592401504}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=549022680}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603766009}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601218130}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603766009}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.949+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196898626}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251940757}) 2024-09-16T06:48:11.958 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467326828}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251940757}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439587242}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830731043}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273227390}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830731043}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2282734921}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838197382}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=318947827}) 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.959 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838197382}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266140323}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.953+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125918900}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127964262}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125918900}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334056426}) 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008457226}) 2024-09-16T06:48:11.961 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792047332}) 2024-09-16T06:48:11.961 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.961 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008457226}) 2024-09-16T06:48:11.961 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783775927}) 2024-09-16T06:48:11.963 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.963 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1891064014}) 2024-09-16T06:48:11.963 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95237205}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1891064014}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402938895}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617538102}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263748273}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617538102}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338334981}) 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.964 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327791599}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272801608}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327791599}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.957+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=824007897}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248963281}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984663649}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248963281}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3862983435}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048846568}) 2024-09-16T06:48:11.965 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259340532}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048846568}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227980095}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626817996}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415161274}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626817996}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877584703}) 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.966 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732939901}) 2024-09-16T06:48:11.967 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823269954}) 2024-09-16T06:48:11.967 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.967 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732939901}) 2024-09-16T06:48:11.967 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.961+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3045824460}) 2024-09-16T06:48:11.968 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.968 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751200765}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778582275}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751200765}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047521707}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309730229}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835799346}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309730229}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385896460}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797173385}) 2024-09-16T06:48:11.969 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144126802}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797173385}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838238254}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081966292}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077774509}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081966292}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.965+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1677561756}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797272236}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921247102}) 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.970 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797272236}) 2024-09-16T06:48:11.971 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708634737}) 2024-09-16T06:48:11.972 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.972 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260977792}) 2024-09-16T06:48:11.972 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2829737578}) 2024-09-16T06:48:11.972 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260977792}) 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805968716}) 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724946021}) 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721133799}) 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.973 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724946021}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1955486856}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451525582}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758894235}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451525582}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.969+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494388073}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953468050}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776205963}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953468050}) 2024-09-16T06:48:11.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869379787}) 2024-09-16T06:48:11.976 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.976 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549795047}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361937668}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549795047}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648497678}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2337520491}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389203197}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2337520491}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751158890}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041329763}) 2024-09-16T06:48:11.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003103859}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041329763}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.973+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1592298271}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912899008}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226597415}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912899008}) 2024-09-16T06:48:11.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667213022}) 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=3389348057,mdsmap=1531151339,mgr=3454917385,mgr_command_descs=794974419,mgrstat=295676841,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=461221896}) 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=20,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=3635895984,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3849200162,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227}) 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=3389348057,mdsmap=1531151339,mgr=3454917385,mgr_command_descs=794974419,mgrstat=295676841,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=461221896}) 2024-09-16T06:48:11.979 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.977+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=2} crc {logm=711513728,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2624825231,mon_config_key=3144203258,monmap=1887442722,osd_metadata=2882727975,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2685335812}) 2024-09-16T06:48:11.985 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.985 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=1790839250}) 2024-09-16T06:48:11.985 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=2775337577}) 2024-09-16T06:48:11.985 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:48:11.986 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=1790839250}) 2024-09-16T06:48:11.986 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:48:11.985+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=79} crc {osdmap=3150802655}) 2024-09-16T06:48:12.938 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:48:12.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 10 monclient: monmap: 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:13.063045+0000 2024-09-16T06:48:13.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:13.063045+0000 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:13.060 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 1 Processor -- start 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 1 -- start start 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.057+0000 7f082b59e700 10 monclient: init 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:48:13.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f082c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f082c098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f082c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c09d560 con 0x7f082c099030 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c09f140 con 0x7f082c098be0 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c09d790 con 0x7f082c09ed60 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient(hunting): authenticate will time out at 1440.196420s 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082a59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f082c09ed60 0x7f082c0984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43274/0 (socket says 172.21.15.76:43274) 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082a59c700 1 -- 172.21.15.76:0/3440464817 learned_addr learned my addr 172.21.15.76:0/3440464817 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592534227 0 0) 0x7f082c09d790 con 0x7f082c09ed60 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): my global_id is 194711 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f080c0035d0 con 0x7f082c09ed60 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831285554 0 0) 0x7f082c09d560 con 0x7f082c099030 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): my global_id is 214269 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c09d790 con 0x7f082c099030 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693182599 0 0) 0x7f082c09f140 con 0x7f082c098be0 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient(hunting): my global_id is 224128 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c09d560 con 0x7f082c098be0 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2042931131 0 0) 0x7f080c0035d0 con 0x7f082c09ed60 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 >> v1:172.21.15.189:6789/0 conn(0x7f082c098be0 legacy=0x7f082c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 >> v1:172.21.15.76:6789/0 conn(0x7f082c099030 legacy=0x7f082c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 monclient: found mon.noname-b 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c09f620 con 0x7f082c09ed60 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 5 monclient: authenticate success, global_id 194711 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0814001610 con 0x7f082c09ed60 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: dump: 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08140018f0 con 0x7f082c09ed60 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 4 set_mon_vals no callback set 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 1 -- 172.21.15.76:0/3440464817 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0814001c80 con 0x7f082c09ed60 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082959a700 10 monclient: dump: 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: shutdown 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- 172.21.15.76:0/3440464817 >> v1:172.21.15.76:6790/0 conn(0x7f082c09ed60 legacy=0x7f082c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- 172.21.15.76:0/3440464817 shutdown_connections 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- 172.21.15.76:0/3440464817 >> 172.21.15.76:0/3440464817 conn(0x7f082c0a0030 msgr2=0x7f082c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- 172.21.15.76:0/3440464817 shutdown_connections 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 1 -- 172.21.15.76:0/3440464817 wait complete. 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.061+0000 7f082b59e700 10 monclient: monmap: 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.071 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:13.071 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:13.071 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 Processor -- start 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- start start 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: init 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f082c098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f082c099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f082c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c093290 con 0x7f082c098be0 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c092980 con 0x7f082c099030 2024-09-16T06:48:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c092c60 con 0x7f082c09ed60 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient(hunting): authenticate will time out at 1440.201296s 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f082c098be0 0x7f082c096490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51152/0 (socket says 172.21.15.76:51152) 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082a59c700 1 -- 172.21.15.76:0/2657475950 learned_addr learned my addr 172.21.15.76:0/2657475950 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806451267 0 0) 0x7f082c093290 con 0x7f082c098be0 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): my global_id is 214272 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08100035d0 con 0x7f082c098be0 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205333524 0 0) 0x7f082c092c60 con 0x7f082c09ed60 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): my global_id is 194714 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c093290 con 0x7f082c09ed60 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885837916 0 0) 0x7f08100035d0 con 0x7f082c098be0 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f082c092c60 con 0x7f082c098be0 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0814001c00 con 0x7f082c098be0 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): dump: 2024-09-16T06:48:13.077 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265050024 0 0) 0x7f082c092980 con 0x7f082c099030 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient(hunting): my global_id is 224131 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08100035d0 con 0x7f082c099030 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 142773331 0 0) 0x7f082c092c60 con 0x7f082c098be0 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 >> v1:172.21.15.76:6790/0 conn(0x7f082c09ed60 legacy=0x7f082c1d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 >> v1:172.21.15.189:6789/0 conn(0x7f082c099030 legacy=0x7f082c1cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 monclient: found mon.a 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c092f40 con 0x7f082c098be0 2024-09-16T06:48:13.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 5 monclient: authenticate success, global_id 214272 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _renew_subs 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c090510 con 0x7f082c098be0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _renew_subs 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c096ba0 con 0x7f082c098be0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0814004ca0 con 0x7f082c098be0 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0814004fb0 con 0x7f082c098be0 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0823fff700 4 set_mon_vals no callback set 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f0822ffd700 10 monclient: dump: 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.065+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804005100 con 0x7f082c098be0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.069+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f081400fa20 con 0x7f082c098be0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.069+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f0814043f50 con 0x7f082c098be0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.069+0000 7f0829d9b700 10 monclient: get_auth_request con 0x7f08100363e0 auth_method 0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.077+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0814014860 con 0x7f082c098be0 2024-09-16T06:48:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.077+0000 7f0822ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.077+0000 7f0822ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:13.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:48:13.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:13.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0804005930 con 0x7f082c098be0 2024-09-16T06:48:13.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f0822ffd700 1 -- 172.21.15.76:0/2657475950 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 2775924174) 0x7f0814012f00 con 0x7f082c098be0 2024-09-16T06:48:13.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f0822ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:13.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f0822ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:13.302 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 >> v1:172.21.15.189:6812/3098810674 conn(0x7f08100363e0 legacy=0x7f08100388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 10 monclient: shutdown 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.301+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 >> v1:172.21.15.76:6789/0 conn(0x7f082c098be0 legacy=0x7f082c096490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.305+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 shutdown_connections 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.305+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 >> 172.21.15.76:0/2657475950 conn(0x7f082c0a0030 msgr2=0x7f082c0973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.305+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 shutdown_connections 2024-09-16T06:48:13.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.305+0000 7f082b59e700 1 -- 172.21.15.76:0/2657475950 wait complete. 2024-09-16T06:48:13.315 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:48:13.315 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:48:13.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:48:13.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:48:13.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 10 monclient: get_monmap_and_config 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 10 monclient: monmap: 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:13.429789+0000 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:13.429789+0000 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:13.426 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:13.427 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 1 Processor -- start 2024-09-16T06:48:13.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 1 -- start start 2024-09-16T06:48:13.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.425+0000 7f83c92f4700 10 monclient: init 2024-09-16T06:48:13.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:13.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-b con 0x7f83c40a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-a con 0x7f83c40a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-c con 0x7f83c413b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c409b3e0 con 0x7f83c40a24a0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40a3450 con 0x7f83c413b6b0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c409b1a0 con 0x7f83c40a20c0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83bbfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f83c40a24a0 0x7f83c40a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51154/0 (socket says 172.21.15.76:51154) 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83bbfff700 1 -- 172.21.15.76:0/3736692550 learned_addr learned my addr 172.21.15.76:0/3736692550 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): authenticate will time out at 1440.564610s 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182418969 0 0) 0x7f83c409b3e0 con 0x7f83c40a24a0 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): my global_id is 214275 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b00035d0 con 0x7f83c40a24a0 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727885085 0 0) 0x7f83c409b1a0 con 0x7f83c40a20c0 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient(hunting): my global_id is 194717 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c409b3e0 con 0x7f83c40a20c0 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488947117 0 0) 0x7f83b00035d0 con 0x7f83c40a24a0 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 >> v1:172.21.15.76:6790/0 conn(0x7f83c40a20c0 legacy=0x7f83c406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 >> v1:172.21.15.189:6789/0 conn(0x7f83c413b6b0 legacy=0x7f83c413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 monclient: found mon.noname-a 2024-09-16T06:48:13.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c406bd30 con 0x7f83c40a24a0 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 5 monclient: authenticate success, global_id 214275 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83ac001610 con 0x7f83c40a24a0 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: dump: 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.439 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83ac001880 con 0x7f83c40a24a0 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 4 set_mon_vals no callback set 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 1 -- 172.21.15.76:0/3736692550 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83ac001c10 con 0x7f83c40a24a0 2024-09-16T06:48:13.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c37fe700 10 monclient: dump: 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: shutdown 2024-09-16T06:48:13.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- 172.21.15.76:0/3736692550 >> v1:172.21.15.76:6789/0 conn(0x7f83c40a24a0 legacy=0x7f83c40a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- 172.21.15.76:0/3736692550 shutdown_connections 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- 172.21.15.76:0/3736692550 >> 172.21.15.76:0/3736692550 conn(0x7f83c40a41c0 msgr2=0x7f83c40a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- 172.21.15.76:0/3736692550 shutdown_connections 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- 172.21.15.76:0/3736692550 wait complete. 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: monmap: 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.442 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 Processor -- start 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- start start 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: init 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-b con 0x7f83c40a20c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-c con 0x7f83c40a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): picked mon.noname-a con 0x7f83c413b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c413a770 con 0x7f83c413b6b0 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40a72d0 con 0x7f83c40a20c0 2024-09-16T06:48:13.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c409edc0 con 0x7f83c40a24a0 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83c92f4700 10 monclient(hunting): authenticate will time out at 1440.567595s 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83bbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f83c40a24a0 0x7f83c4137510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43294/0 (socket says 172.21.15.76:43294) 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.429+0000 7f83bbfff700 1 -- 172.21.15.76:0/1975071509 learned_addr learned my addr 172.21.15.76:0/1975071509 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054389445 0 0) 0x7f83c413a770 con 0x7f83c413b6b0 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): my global_id is 214278 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a40035d0 con 0x7f83c413b6b0 2024-09-16T06:48:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554249541 0 0) 0x7f83c409edc0 con 0x7f83c40a24a0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): my global_id is 194720 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c413a770 con 0x7f83c40a24a0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023446331 0 0) 0x7f83c40a72d0 con 0x7f83c40a20c0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): my global_id is 224137 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c409edc0 con 0x7f83c40a20c0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030251713 0 0) 0x7f83a40035d0 con 0x7f83c413b6b0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83c40a72d0 con 0x7f83c413b6b0 2024-09-16T06:48:13.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83bc001ae0 con 0x7f83c413b6b0 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): dump: 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.446 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401730866 0 0) 0x7f83c40a72d0 con 0x7f83c413b6b0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6790/0 conn(0x7f83c40a24a0 legacy=0x7f83c4137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.189:6789/0 conn(0x7f83c40a20c0 legacy=0x7f83c4136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 monclient: found mon.a 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c413ad30 con 0x7f83c413b6b0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 5 monclient: authenticate success, global_id 214278 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 10 monclient: _renew_subs 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c413a490 con 0x7f83c413b6b0 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 10 monclient: _renew_subs 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c409e8b0 con 0x7f83c413b6b0 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83bc003a40 con 0x7f83c413b6b0 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c2ffd700 4 set_mon_vals no callback set 2024-09-16T06:48:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83bc003dd0 con 0x7f83c413b6b0 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: dump: 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.449 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f83bc003f30 con 0x7f83c413b6b0 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f83a8006590 con 0x7f83a8002830 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f83bc042ee0 con 0x7f83c413b6b0 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83bbfff700 10 monclient: get_auth_request con 0x7f83a4036110 auth_method 0 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216585598 0 0) 0x7f83a8006590 con 0x7f83a8002830 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient(hunting): my global_id is 214281 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c40a72d0 con 0x7f83a8002830 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1039036882 0 0) 0x7f83c40a72d0 con 0x7f83a8002830 2024-09-16T06:48:13.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83a8006590 con 0x7f83a8002830 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83b40017c0 con 0x7f83a8002830 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 10 monclient: dump: 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4152289454 0 0) 0x7f83a8006590 con 0x7f83a8002830 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.433+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83a800a190 con 0x7f83a8002830 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.437+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f83b4001c20 con 0x7f83a8002830 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.437+0000 7f83c17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.437+0000 7f83c17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.437+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6789/0 conn(0x7f83a8002830 legacy=0x7f83a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.437+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6789/0 conn(0x7f83a8002830 legacy=0x7f83a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f83a8009e50 con 0x7f83a8009170 2024-09-16T06:48:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035802883 0 0) 0x7f83a8009e50 con 0x7f83a8009170 2024-09-16T06:48:13.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient(hunting): my global_id is 214284 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a8006590 con 0x7f83a8009170 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2593952749 0 0) 0x7f83a8006590 con 0x7f83a8009170 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83a8009e50 con 0x7f83a8009170 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f83b4003a30 con 0x7f83a8009170 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient: dump: 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2679521622 0 0) 0x7f83a8009e50 con 0x7f83a8009170 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f83a800c340 con 0x7f83a8009170 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 152664292) 0x7f83b4004f30 con 0x7f83a8009170 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6789/0 conn(0x7f83a8009170 legacy=0x7f83a8009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c17fa700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6789/0 conn(0x7f83a8009170 legacy=0x7f83a8009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.458 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.459 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:13.460 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.461 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.462 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:13.463 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.464 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:13.465 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.466 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:13.467 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.189:6812/3098810674 conn(0x7f83a4036110 legacy=0x7f83a40385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 10 monclient: shutdown 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 >> v1:172.21.15.76:6789/0 conn(0x7f83c413b6b0 legacy=0x7f83c4137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c8af3700 1 -- 172.21.15.76:0/1975071509 reap_dead start 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.445+0000 7f83c8af3700 1 -- 172.21.15.76:0/1975071509 reap_dead start 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.449+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 shutdown_connections 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.449+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 >> 172.21.15.76:0/1975071509 conn(0x7f83c40a41c0 msgr2=0x7f83c40a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.449+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 shutdown_connections 2024-09-16T06:48:13.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.449+0000 7f83c92f4700 1 -- 172.21.15.76:0/1975071509 wait complete. 2024-09-16T06:48:13.471 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:13.471 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:48:13.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:48:13.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: get_monmap_and_config 2024-09-16T06:48:13.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: monmap: 2024-09-16T06:48:13.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:13.589417+0000 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:13.589417+0000 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:13.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 1 Processor -- start 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 1 -- start start 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: init 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-a con 0x7f87d4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-c con 0x7f87d406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-b con 0x7f87d406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d406c280 con 0x7f87d4132330 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d406bfa0 con 0x7f87d406d480 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d409ace0 con 0x7f87d406cf50 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d8e43700 10 monclient(hunting): authenticate will time out at 1440.723328s 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f87d406cf50 0x7f87d413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43306/0 (socket says 172.21.15.76:43306) 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.585+0000 7f87d3fff700 1 -- 172.21.15.76:0/26411354 learned_addr learned my addr 172.21.15.76:0/26411354 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708937469 0 0) 0x7f87d406c280 con 0x7f87d4132330 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): my global_id is 214287 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c00035d0 con 0x7f87d4132330 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541377331 0 0) 0x7f87d409ace0 con 0x7f87d406cf50 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): my global_id is 194723 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d406c280 con 0x7f87d406cf50 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747425946 0 0) 0x7f87d406bfa0 con 0x7f87d406d480 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient(hunting): my global_id is 224140 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d409ace0 con 0x7f87d406d480 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204897156 0 0) 0x7f87d406c280 con 0x7f87d406cf50 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 >> v1:172.21.15.189:6789/0 conn(0x7f87d406d480 legacy=0x7f87d413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 >> v1:172.21.15.76:6789/0 conn(0x7f87d4132330 legacy=0x7f87d4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 monclient: found mon.noname-b 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d409ae70 con 0x7f87d406cf50 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87cc001970 con 0x7f87d406cf50 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 5 monclient: authenticate success, global_id 194723 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: dump: 2024-09-16T06:48:13.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87cc001c90 con 0x7f87d406cf50 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 1 -- 172.21.15.76:0/26411354 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87cc001ec0 con 0x7f87d406cf50 2024-09-16T06:48:13.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d27fc700 10 monclient: dump: 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.595 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 4 set_mon_vals no callback set 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 10 monclient: shutdown 2024-09-16T06:48:13.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 1 -- 172.21.15.76:0/26411354 >> v1:172.21.15.76:6790/0 conn(0x7f87d406cf50 legacy=0x7f87d413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 1 -- 172.21.15.76:0/26411354 shutdown_connections 2024-09-16T06:48:13.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 1 -- 172.21.15.76:0/26411354 >> 172.21.15.76:0/26411354 conn(0x7f87d40a41c0 msgr2=0x7f87d40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 1 -- 172.21.15.76:0/26411354 shutdown_connections 2024-09-16T06:48:13.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 1 -- 172.21.15.76:0/26411354 wait complete. 2024-09-16T06:48:13.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.589+0000 7f87d8e43700 10 monclient: monmap: 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:13.604 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 Processor -- start 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- start start 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: init 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-b con 0x7f87d406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-c con 0x7f87d406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient(hunting): picked mon.noname-a con 0x7f87d4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d409e8b0 con 0x7f87d4132330 2024-09-16T06:48:13.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d4131670 con 0x7f87d406cf50 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d412d820 con 0x7f87d406d480 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient(hunting): authenticate will time out at 1440.729552s 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f87d4132330 0x7f87d41d2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51202/0 (socket says 172.21.15.76:51202) 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d3fff700 1 -- 172.21.15.76:0/3921107348 learned_addr learned my addr 172.21.15.76:0/3921107348 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170660849 0 0) 0x7f87d409e8b0 con 0x7f87d4132330 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): my global_id is 214290 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40035d0 con 0x7f87d4132330 2024-09-16T06:48:13.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352996090 0 0) 0x7f87d412d820 con 0x7f87d406d480 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): my global_id is 194726 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d409e8b0 con 0x7f87d406d480 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035744377 0 0) 0x7f87b40035d0 con 0x7f87d4132330 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d412d820 con 0x7f87d4132330 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87cc001400 con 0x7f87d4132330 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): dump: 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.607 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561266025 0 0) 0x7f87d4131670 con 0x7f87d406cf50 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): my global_id is 224143 2024-09-16T06:48:13.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40035d0 con 0x7f87d406cf50 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2392166404 0 0) 0x7f87d409e8b0 con 0x7f87d406d480 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d4131670 con 0x7f87d406d480 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87c40018b0 con 0x7f87d406d480 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient(hunting): dump: 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.609 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1324302738 0 0) 0x7f87d412d820 con 0x7f87d4132330 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6790/0 conn(0x7f87d406d480 legacy=0x7f87d41ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.189:6789/0 conn(0x7f87d406cf50 legacy=0x7f87d4130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 monclient: found mon.a 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d412db00 con 0x7f87d4132330 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 5 monclient: authenticate success, global_id 214290 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _renew_subs 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d4131100 con 0x7f87d4132330 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _renew_subs 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d412b7d0 con 0x7f87d4132330 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87cc004f10 con 0x7f87d4132330 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87cc0052a0 con 0x7f87d4132330 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d17fa700 4 set_mon_vals no callback set 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:48:13.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87cbfff700 10 monclient: dump: 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.593+0000 7f87d8e43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f87cc005400 con 0x7f87d4132330 2024-09-16T06:48:13.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f87b8006590 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f87cc044340 con 0x7f87d4132330 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802095206 0 0) 0x7f87b8006590 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient(hunting): my global_id is 194729 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d412d820 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87d2ffd700 10 monclient: get_auth_request con 0x7f87b4036430 auth_method 0 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3316034991 0 0) 0x7f87d412d820 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b8006590 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87bc001850 con 0x7f87b8002830 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient: dump: 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3794876 0 0) 0x7f87b8006590 con 0x7f87b8002830 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87b800a190 con 0x7f87b8002830 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f87bc001a70 con 0x7f87b8002830 2024-09-16T06:48:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6790/0 conn(0x7f87b8002830 legacy=0x7f87b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.597+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6790/0 conn(0x7f87b8002830 legacy=0x7f87b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:13.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f87b8009e30 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507635739 0 0) 0x7f87b8009e30 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient(hunting): my global_id is 194732 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8006590 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1888800155 0 0) 0x7f87b8006590 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b8009e30 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87bc003840 con 0x7f87b8009150 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient: dump: 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2586105361 0 0) 0x7f87b8009e30 con 0x7f87b8009150 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f87b800c340 con 0x7f87b8009150 2024-09-16T06:48:13.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 1395041300) 0x7f87bc005ac0 con 0x7f87b8009150 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6790/0 conn(0x7f87b8009150 legacy=0x7f87b80095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87cbfff700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6790/0 conn(0x7f87b8009150 legacy=0x7f87b80095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:13.619 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.620 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:13.622 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.623 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.624 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.625 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.626 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:13.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.628 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.629 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.630 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:13.631 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.189:6812/3098810674 conn(0x7f87b4036430 legacy=0x7f87b40388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 10 monclient: shutdown 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 >> v1:172.21.15.76:6789/0 conn(0x7f87d4132330 legacy=0x7f87d41d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d3fff700 1 -- 172.21.15.76:0/3921107348 reap_dead start 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d3fff700 1 -- 172.21.15.76:0/3921107348 reap_dead start 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 shutdown_connections 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 >> 172.21.15.76:0/3921107348 conn(0x7f87d40a41c0 msgr2=0x7f87d4129fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 shutdown_connections 2024-09-16T06:48:13.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.609+0000 7f87d8e43700 1 -- 172.21.15.76:0/3921107348 wait complete. 2024-09-16T06:48:13.634 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:13.634 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:48:13.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.753+0000 7f0abce68700 10 monclient: get_monmap_and_config 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.753+0000 7f0abce68700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.753+0000 7f0abce68700 10 monclient: monmap: 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:13.759879+0000 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:13.759879+0000 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 Processor -- start 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- start start 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient: init 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:13.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-b con 0x7f0ab00207b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-a con 0x7f0ab00229f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-c con 0x7f0ab0021e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab0021bf0 con 0x7f0ab00229f0 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab001c340 con 0x7f0ab0021e70 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab001d500 con 0x7f0ab00207b0 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient(hunting): authenticate will time out at 1440.893233s 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ab00207b0 0x7f0ab0020b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43326/0 (socket says 172.21.15.76:43326) 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab77fe700 1 -- 172.21.15.76:0/483075831 learned_addr learned my addr 172.21.15.76:0/483075831 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab6ffd700 1 --1- 172.21.15.76:0/483075831 >> v1:172.21.15.76:6789/0 conn(0x7f0ab00229f0 0x7f0ab00bcde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51212/0 (socket says 172.21.15.76:51212) 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293650449 0 0) 0x7f0ab001d500 con 0x7f0ab00207b0 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): my global_id is 194735 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa80035d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837912234 0 0) 0x7f0ab0021bf0 con 0x7f0ab00229f0 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient(hunting): my global_id is 214293 2024-09-16T06:48:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab001d500 con 0x7f0ab00229f0 2024-09-16T06:48:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224196843 0 0) 0x7f0aa80035d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 >> v1:172.21.15.189:6789/0 conn(0x7f0ab0021e70 legacy=0x7f0ab00c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 >> v1:172.21.15.76:6789/0 conn(0x7f0ab00229f0 legacy=0x7f0ab00bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 monclient: found mon.noname-b 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab001d920 con 0x7f0ab00207b0 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab8069400 con 0x7f0ab00207b0 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 5 monclient: authenticate success, global_id 194735 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: dump: 2024-09-16T06:48:13.769 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.770 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ab804ec20 con 0x7f0ab00207b0 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 4 set_mon_vals no callback set 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 1 -- 172.21.15.76:0/483075831 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab80008d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0ab67fc700 10 monclient: dump: 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 10 monclient: shutdown 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- 172.21.15.76:0/483075831 >> v1:172.21.15.76:6790/0 conn(0x7f0ab00207b0 legacy=0x7f0ab0020b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- 172.21.15.76:0/483075831 shutdown_connections 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- 172.21.15.76:0/483075831 >> 172.21.15.76:0/483075831 conn(0x7f0ab0005b40 msgr2=0x7f0ab0005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- 172.21.15.76:0/483075831 shutdown_connections 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.757+0000 7f0abce68700 1 -- 172.21.15.76:0/483075831 wait complete. 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: build_initial_monmap 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: monmap: 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:13.773 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 Processor -- start 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- start start 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: init 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-c con 0x7f0ab00207b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-a con 0x7f0ab0021e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient(hunting): picked mon.noname-b con 0x7f0ab00229f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab001f7a0 con 0x7f0ab0021e70 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab0148010 con 0x7f0ab00229f0 2024-09-16T06:48:13.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00204a0 con 0x7f0ab00207b0 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient(hunting): authenticate will time out at 1440.897606s 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0ab77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0ab00207b0 0x7f0ab0147900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43338/0 (socket says 172.21.15.76:43338) 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0ab77fe700 1 -- 172.21.15.76:0/3041218622 learned_addr learned my addr 172.21.15.76:0/3041218622 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838512451 0 0) 0x7f0ab00204a0 con 0x7f0ab00207b0 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): my global_id is 194738 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c0035d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033222933 0 0) 0x7f0ab001f7a0 con 0x7f0ab0021e70 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): my global_id is 214296 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab00204a0 con 0x7f0ab0021e70 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418245466 0 0) 0x7f0ab0148010 con 0x7f0ab00229f0 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): my global_id is 224149 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab001f7a0 con 0x7f0ab00229f0 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883397295 0 0) 0x7f0a9c0035d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ab0148010 con 0x7f0ab00207b0 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab80008d0 con 0x7f0ab00207b0 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): dump: 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:13.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365823549 0 0) 0x7f0ab001f7a0 con 0x7f0ab00229f0 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a9c0035d0 con 0x7f0ab00229f0 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa40017b0 con 0x7f0ab00229f0 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient(hunting): dump: 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132460371 0 0) 0x7f0ab0148010 con 0x7f0ab00207b0 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6789/0 conn(0x7f0ab00229f0 legacy=0x7f0ab0156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.76:6789/0 conn(0x7f0ab0021e70 legacy=0x7f0ab0152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 monclient: found mon.c 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab012c730 con 0x7f0ab00207b0 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: _finish_auth 0 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:13.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 5 monclient: authenticate success, global_id 194738 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _renew_subs 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab012f260 con 0x7f0ab00207b0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _renew_subs 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab012dd10 con 0x7f0ab00207b0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ab8069400 con 0x7f0ab00207b0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ab804f5c0 con 0x7f0ab00207b0 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0ab57fa700 4 set_mon_vals no callback set 2024-09-16T06:48:13.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.761+0000 7f0aa3fff700 10 monclient: dump: 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ab8067aa0 con 0x7f0ab00207b0 2024-09-16T06:48:13.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0abce68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f0ab80bdc40 con 0x7f0ab00207b0 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a98006590 con 0x7f0a98002830 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0ab6ffd700 10 monclient: get_auth_request con 0x7f0a9c036250 auth_method 0 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7620278 0 0) 0x7f0a98006590 con 0x7f0a98002830 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient(hunting): my global_id is 224152 2024-09-16T06:48:13.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab0148010 con 0x7f0a98002830 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 741761205 0 0) 0x7f0ab0148010 con 0x7f0a98002830 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a98006590 con 0x7f0a98002830 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa40036a0 con 0x7f0a98002830 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient: dump: 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.784 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1617678188 0 0) 0x7f0a98006590 con 0x7f0a98002830 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a9800a190 con 0x7f0a98002830 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0aa4005a50 con 0x7f0a98002830 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6789/0 conn(0x7f0a98002830 legacy=0x7f0a98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.765+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6789/0 conn(0x7f0a98002830 legacy=0x7f0a98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0abce68700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a98009e20 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920767708 0 0) 0x7f0a98009e20 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient(hunting): my global_id is 224155 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a98006590 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 601652873 0 0) 0x7f0a98006590 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a98009e20 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0aa4003e70 con 0x7f0a98009170 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:13.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient: dump: 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3967112788 0 0) 0x7f0a98009e20 con 0x7f0a98009170 2024-09-16T06:48:13.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a9800c340 con 0x7f0a98009170 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2069330811) 0x7f0aa4005bc0 con 0x7f0a98009170 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6789/0 conn(0x7f0a98009170 legacy=0x7f0a98009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.781+0000 7f0aa3fff700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6789/0 conn(0x7f0a98009170 legacy=0x7f0a98009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:13.788 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 92, 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.790 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:13.792 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:13.793 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:13.794 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:13.795 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.796 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:13.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.798 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:13.799 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:13.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0a9c036250 legacy=0x7f0a9c038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 10 monclient: shutdown 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 >> v1:172.21.15.76:6790/0 conn(0x7f0ab00207b0 legacy=0x7f0ab0147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0ab7fff700 1 -- 172.21.15.76:0/3041218622 reap_dead start 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0ab7fff700 1 -- 172.21.15.76:0/3041218622 reap_dead start 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 shutdown_connections 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 >> 172.21.15.76:0/3041218622 conn(0x7f0ab0005b40 msgr2=0x7f0ab0004c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 shutdown_connections 2024-09-16T06:48:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:13.785+0000 7f0abce68700 1 -- 172.21.15.76:0/3041218622 wait complete. 2024-09-16T06:48:13.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:48:13.802 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:48:13.802 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T06:48:13.802 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:48:13.802 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:48:13.802 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:48:13.802 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:48:13.836 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:48:13.836 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:48:13.836 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:48:13.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:48:33.837 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:48:33.837 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:48:33.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:48:33.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.952+0000 7f7945083700 10 monclient: get_monmap_and_config 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.952+0000 7f7945083700 10 monclient: build_initial_monmap 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.952+0000 7f7945083700 10 monclient: monmap: 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:33.955 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:33.959302+0000 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:33.959302+0000 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:33.956 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:33.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 1 Processor -- start 2024-09-16T06:48:33.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 1 -- start start 2024-09-16T06:48:33.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient: init 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient(hunting): picked mon.noname-a con 0x7f7940131910 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient(hunting): picked mon.noname-b con 0x7f79401347f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f793ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940131cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:33.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient(hunting): picked mon.noname-c con 0x7f7940138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940132400 con 0x7f7940131910 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009fcd0 con 0x7f7940138300 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009b300 con 0x7f79401347f0 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f7945083700 10 monclient(hunting): authenticate will time out at 1461.094510s 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f793f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 0x7f7940136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41774/0 (socket says 172.21.15.76:41774) 2024-09-16T06:48:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:33.956+0000 7f793f7fe700 1 -- 172.21.15.76:0/3752593611 learned_addr learned my addr 172.21.15.76:0/3752593611 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:34.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:34.160+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940131cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:34.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:34.564+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940131cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:35.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.368+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940131cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:35.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.988+0000 7f793f7fe700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 legacy=0x7f7940136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T06:48:35.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.988+0000 7f793f7fe700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 legacy=0x7f7940136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:48:35.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.988+0000 7f793f7fe700 1 --1- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 0x7f7940136be0 :-1 s=OPENED pgs=1293 cs=1 l=1).handle_message read tag failed 2024-09-16T06:48:35.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.988+0000 7f793f7fe700 1 --1- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 0x7f7940136be0 :-1 s=OPENED pgs=1293 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:48:35.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:35.988+0000 7f793effd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): tick 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 legacy=0x7f7940136be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.189:6789/0 conn(0x7f7940138300 legacy=0x7f794013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940131cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7940131910 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): picked mon.noname-c con 0x7f7940138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): picked mon.noname-b con 0x7f79401347f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009b7e0 con 0x7f7940131910 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f792c0067d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940132400 con 0x7f7940138300 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c010f20 con 0x7f79401347f0 2024-09-16T06:48:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:36.956+0000 7f793dffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:37.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:37.160+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f792c0067d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:37.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:37.564+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f792c0067d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.368+0000 7f793ffff700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f792c0067d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:38.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925986326 0 0) 0x7f7940132400 con 0x7f7940138300 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): my global_id is 224158 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79280035d0 con 0x7f7940138300 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380515166 0 0) 0x7f792c010f20 con 0x7f79401347f0 2024-09-16T06:48:38.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:38.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:38.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:38.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 10 monclient(hunting): my global_id is 194741 2024-09-16T06:48:38.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.888+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940132400 con 0x7f79401347f0 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615690401 0 0) 0x7f7940132400 con 0x7f79401347f0 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.189:6789/0 conn(0x7f7940138300 legacy=0x7f792c009f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f792c0067d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: _finish_hunting 0 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 monclient: found mon.noname-b 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:38.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c0120c0 con 0x7f79401347f0 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: _finish_auth 0 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: _check_auth_tickets 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7938001950 con 0x7f79401347f0 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f7945083700 5 monclient: authenticate success, global_id 194741 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f7945083700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f793effd700 10 monclient: dump: 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.900 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.896+0000 7f7945083700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:38.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7938001bb0 con 0x7f79401347f0 2024-09-16T06:48:38.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:38.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 1 -- 172.21.15.76:0/3752593611 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7938001f40 con 0x7f79401347f0 2024-09-16T06:48:38.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:38.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f793effd700 10 monclient: dump: 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f7945083700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:38.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.900+0000 7f7945083700 4 set_mon_vals no callback set 2024-09-16T06:48:38.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: shutdown 2024-09-16T06:48:38.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:38.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- 172.21.15.76:0/3752593611 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 legacy=0x7f792c00d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:38.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- 172.21.15.76:0/3752593611 shutdown_connections 2024-09-16T06:48:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- 172.21.15.76:0/3752593611 >> 172.21.15.76:0/3752593611 conn(0x7f79400a4430 msgr2=0x7f79400a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- 172.21.15.76:0/3752593611 shutdown_connections 2024-09-16T06:48:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- 172.21.15.76:0/3752593611 wait complete. 2024-09-16T06:48:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: build_initial_monmap 2024-09-16T06:48:38.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: monmap: 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 Processor -- start 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- start start 2024-09-16T06:48:38.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: init 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient(hunting): picked mon.noname-a con 0x7f7940131910 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient(hunting): picked mon.noname-c con 0x7f79401347f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient(hunting): picked mon.noname-b con 0x7f7940138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009d040 con 0x7f7940131910 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940137e70 con 0x7f7940138300 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009b0f0 con 0x7f79401347f0 2024-09-16T06:48:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f7945083700 10 monclient(hunting): authenticate will time out at 1466.043866s 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.904+0000 7f793ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940130bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 0x7f79401cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41830/0 (socket says 172.21.15.76:41830) 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793f7fe700 1 -- 172.21.15.76:0/4140785861 learned_addr learned my addr 172.21.15.76:0/4140785861 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136721475 0 0) 0x7f794009b0f0 con 0x7f79401347f0 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): my global_id is 194744 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c006010 con 0x7f79401347f0 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602231747 0 0) 0x7f7940137e70 con 0x7f7940138300 2024-09-16T06:48:38.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): my global_id is 224161 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f794009b0f0 con 0x7f7940138300 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139022252 0 0) 0x7f792c006010 con 0x7f79401347f0 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7940137e70 con 0x7f79401347f0 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f79340018b0 con 0x7f79401347f0 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): dump: 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659716020 0 0) 0x7f794009b0f0 con 0x7f7940138300 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f792c006010 con 0x7f7940138300 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f79380018d0 con 0x7f7940138300 2024-09-16T06:48:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient(hunting): dump: 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2370609993 0 0) 0x7f7940137e70 con 0x7f79401347f0 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 >> v1:172.21.15.189:6789/0 conn(0x7f7940138300 legacy=0x7f79401d23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 >> v1:172.21.15.76:6789/0 conn(0x7f7940131910 legacy=0x7f7940130bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 monclient: found mon.c 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794011f2e0 con 0x7f79401347f0 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: _finish_auth 0 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 5 monclient: authenticate success, global_id 194744 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _renew_subs 2024-09-16T06:48:38.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794011f5c0 con 0x7f79401347f0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _renew_subs 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940070420 con 0x7f79401347f0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7934004890 con 0x7f79401347f0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7934004ba0 con 0x7f79401347f0 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:38.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 10 monclient: dump: 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793e7fc700 4 set_mon_vals no callback set 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:38.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f793400f610 con 0x7f79401347f0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7920005100 con 0x7f79401347f0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f7934043d60 con 0x7f79401347f0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.908+0000 7f793ffff700 10 monclient: get_auth_request con 0x7f792c036640 auth_method 0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.916+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7934010b10 con 0x7f79401347f0 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.916+0000 7f793cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:38.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:38.916+0000 7f793cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:39.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f7945083700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:48:39.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f7945083700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:39.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f7945083700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7920005930 con 0x7f79401347f0 2024-09-16T06:48:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f793cff9700 1 -- 172.21.15.76:0/4140785861 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1383 (unknown 254746657 0 1354125208) 0x7f7934012b70 con 0x7f79401347f0 2024-09-16T06:48:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f793cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:48:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.144+0000 7f793cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:39.147 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":94,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 >> v1:172.21.15.189:6812/3098810674 conn(0x7f792c036640 legacy=0x7f792c038b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 10 monclient: shutdown 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 >> v1:172.21.15.76:6790/0 conn(0x7f79401347f0 legacy=0x7f79401cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 shutdown_connections 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 >> 172.21.15.76:0/4140785861 conn(0x7f79400a4430 msgr2=0x7f794009d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 shutdown_connections 2024-09-16T06:48:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.148+0000 7f7945083700 1 -- 172.21.15.76:0/4140785861 wait complete. 2024-09-16T06:48:39.161 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:48:39.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:48:39.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: get_monmap_and_config 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: build_initial_monmap 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: monmap: 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.274 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:39.278483+0000 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:39.278483+0000 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:39.275 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 1 Processor -- start 2024-09-16T06:48:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 1 -- start start 2024-09-16T06:48:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: init 2024-09-16T06:48:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-b con 0x7f2ea409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-a con 0x7f2ea4098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ea4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea409d560 con 0x7f2ea4098be0 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea409f140 con 0x7f2ea4099030 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea409d790 con 0x7f2ea409ed60 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea359e700 10 monclient(hunting): authenticate will time out at 1466.411942s 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.272+0000 7f2ea1d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2ea4098be0 legacy=0x7f2ea41374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2ea409ed60 0x7f2ea40984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41842/0 (socket says 172.21.15.76:41842) 2024-09-16T06:48:39.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea259c700 1 -- 172.21.15.76:0/3707209711 learned_addr learned my addr 172.21.15.76:0/3707209711 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474560812 0 0) 0x7f2ea409d790 con 0x7f2ea409ed60 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): my global_id is 194747 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e840035d0 con 0x7f2ea409ed60 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339210755 0 0) 0x7f2ea409f140 con 0x7f2ea4099030 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient(hunting): my global_id is 224164 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea409d790 con 0x7f2ea4099030 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 796150999 0 0) 0x7f2e840035d0 con 0x7f2ea409ed60 2024-09-16T06:48:39.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 >> v1:172.21.15.189:6789/0 conn(0x7f2ea4099030 legacy=0x7f2ea413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 >> v1:172.21.15.76:6789/0 conn(0x7f2ea4098be0 legacy=0x7f2ea41374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: _finish_hunting 0 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 monclient: found mon.noname-b 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea409f620 con 0x7f2ea409ed60 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: _finish_auth 0 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: _check_auth_tickets 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 5 monclient: authenticate success, global_id 194747 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e94001990 con 0x7f2ea409ed60 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:39.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: dump: 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e94001c70 con 0x7f2ea409ed60 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 1 -- 172.21.15.76:0/3707209711 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e94002000 con 0x7f2ea409ed60 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 4 set_mon_vals no callback set 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea159a700 10 monclient: dump: 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.283 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 10 monclient: shutdown 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 1 -- 172.21.15.76:0/3707209711 >> v1:172.21.15.76:6790/0 conn(0x7f2ea409ed60 legacy=0x7f2ea40984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 1 -- 172.21.15.76:0/3707209711 shutdown_connections 2024-09-16T06:48:39.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 1 -- 172.21.15.76:0/3707209711 >> 172.21.15.76:0/3707209711 conn(0x7f2ea40a0030 msgr2=0x7f2ea40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:39.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 1 -- 172.21.15.76:0/3707209711 shutdown_connections 2024-09-16T06:48:39.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 1 -- 172.21.15.76:0/3707209711 wait complete. 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 10 monclient: build_initial_monmap 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.276+0000 7f2ea359e700 10 monclient: monmap: 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:39.290 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 Processor -- start 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- start start 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: init 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ea4098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:39.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-b con 0x7f2ea4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient(hunting): picked mon.noname-a con 0x7f2ea409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea4071b00 con 0x7f2ea409ed60 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea4092c10 con 0x7f2ea4099030 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea409a540 con 0x7f2ea4098be0 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea2d9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2ea409ed60 legacy=0x7f2ea41d0290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient(hunting): authenticate will time out at 1466.418212s 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2ea4098be0 0x7f2ea4096720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41852/0 (socket says 172.21.15.76:41852) 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea259c700 1 -- 172.21.15.76:0/75217465 learned_addr learned my addr 172.21.15.76:0/75217465 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569147161 0 0) 0x7f2ea409a540 con 0x7f2ea4098be0 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): my global_id is 194750 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e880035d0 con 0x7f2ea4098be0 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996308242 0 0) 0x7f2ea4092c10 con 0x7f2ea4099030 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): my global_id is 224167 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea409a540 con 0x7f2ea4099030 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662594075 0 0) 0x7f2e880035d0 con 0x7f2ea4098be0 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ea4092c10 con 0x7f2ea4098be0 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e94001950 con 0x7f2ea4098be0 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): dump: 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475512373 0 0) 0x7f2ea4092c10 con 0x7f2ea4098be0 2024-09-16T06:48:39.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.189:6789/0 conn(0x7f2ea4099030 legacy=0x7f2ea41ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6789/0 conn(0x7f2ea409ed60 legacy=0x7f2ea41d0290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: _finish_hunting 0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 monclient: found mon.c 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4092930 con 0x7f2ea4098be0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: _finish_auth 0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: _check_auth_tickets 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 5 monclient: authenticate success, global_id 194750 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _renew_subs 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea4098070 con 0x7f2ea4098be0 2024-09-16T06:48:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _renew_subs 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea4091a40 con 0x7f2ea4098be0 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e94005020 con 0x7f2ea4098be0 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e940053b0 con 0x7f2ea4098be0 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2ea0d99700 4 set_mon_vals no callback set 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.280+0000 7f2e9affd700 10 monclient: dump: 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2ea359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e7c006590 con 0x7f2e7c002830 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2e940055e0 con 0x7f2ea4098be0 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7f2e94044370 con 0x7f2ea4098be0 2024-09-16T06:48:39.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502461456 0 0) 0x7f2e7c006590 con 0x7f2e7c002830 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient(hunting): my global_id is 194753 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4092c10 con 0x7f2e7c002830 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2ea2d9d700 10 monclient: get_auth_request con 0x7f2e88035fe0 auth_method 0 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3430904692 0 0) 0x7f2ea4092c10 con 0x7f2e7c002830 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e7c006590 con 0x7f2e7c002830 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e8c004090 con 0x7f2e7c002830 2024-09-16T06:48:39.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient: dump: 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 785334857 0 0) 0x7f2e7c006590 con 0x7f2e7c002830 2024-09-16T06:48:39.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e7c00a190 con 0x7f2e7c002830 2024-09-16T06:48:39.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2e8c0055f0 con 0x7f2e7c002830 2024-09-16T06:48:39.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:39.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:39.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6790/0 conn(0x7f2e7c002830 legacy=0x7f2e7c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.284+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6790/0 conn(0x7f2e7c002830 legacy=0x7f2e7c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:39.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2ea359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2e7c009e30 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181146913 0 0) 0x7f2e7c009e30 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient(hunting): my global_id is 194756 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c006590 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2159901315 0 0) 0x7f2e7c006590 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e7c009e30 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2e8c004830 con 0x7f2e7c009150 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 10 monclient: dump: 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2712291451 0 0) 0x7f2e7c009e30 con 0x7f2e7c009150 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.296+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e7c00c340 con 0x7f2e7c009150 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4202 (unknown 0 0 1805290167) 0x7f2e8c005760 con 0x7f2e7c009150 2024-09-16T06:48:39.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2e9affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2e9affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6790/0 conn(0x7f2e7c009150 legacy=0x7f2e7c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2e9affd700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6790/0 conn(0x7f2e7c009150 legacy=0x7f2e7c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 94, 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:39.304 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.305 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.306 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:39.307 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.308 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.309 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:39.311 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:39.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:39.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.315 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:39.316 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2e88035fe0 legacy=0x7f2e880384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 10 monclient: shutdown 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 >> v1:172.21.15.76:6790/0 conn(0x7f2ea4098be0 legacy=0x7f2ea4096720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea2d9d700 1 -- 172.21.15.76:0/75217465 reap_dead start 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea2d9d700 1 -- 172.21.15.76:0/75217465 reap_dead start 2024-09-16T06:48:39.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 shutdown_connections 2024-09-16T06:48:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 >> 172.21.15.76:0/75217465 conn(0x7f2ea40a0030 msgr2=0x7f2ea406e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 shutdown_connections 2024-09-16T06:48:39.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.300+0000 7f2ea359e700 1 -- 172.21.15.76:0/75217465 wait complete. 2024-09-16T06:48:39.318 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:48:39.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:48:39.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:48:39.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: get_monmap_and_config 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: build_initial_monmap 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: monmap: 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:48:39.441886+0000 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:48:39.441886+0000 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:48:39.438 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:48:39.439 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 1 Processor -- start 2024-09-16T06:48:39.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 1 -- start start 2024-09-16T06:48:39.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: init 2024-09-16T06:48:39.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:39.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-a con 0x7fce5006beb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-b con 0x7fce50131bb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-c con 0x7fce50132060 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5009f630 con 0x7fce5006beb0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5009f8a0 con 0x7fce50132060 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50099350 con 0x7fce50131bb0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce56df6700 10 monclient(hunting): authenticate will time out at 1466.575090s 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce55df4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fce5006beb0 legacy=0x7fce501314a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce555f3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fce50131bb0 0x7fce501376f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41880/0 (socket says 172.21.15.76:41880) 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce555f3700 1 -- 172.21.15.76:0/1952870352 learned_addr learned my addr 172.21.15.76:0/1952870352 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:39.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763018029 0 0) 0x7fce50099350 con 0x7fce50131bb0 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): my global_id is 194759 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c0035d0 con 0x7fce50131bb0 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811928317 0 0) 0x7fce5009f8a0 con 0x7fce50132060 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 10 monclient(hunting): my global_id is 224170 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.436+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50099350 con 0x7fce50132060 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034355479 0 0) 0x7fce3c0035d0 con 0x7fce50131bb0 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 >> v1:172.21.15.189:6789/0 conn(0x7fce50132060 legacy=0x7fce5013ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 >> v1:172.21.15.76:6789/0 conn(0x7fce5006beb0 legacy=0x7fce501314a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: _finish_hunting 0 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 monclient: found mon.noname-b 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50099630 con 0x7fce50131bb0 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: _finish_auth 0 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: _check_auth_tickets 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce40001610 con 0x7fce50131bb0 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 5 monclient: authenticate success, global_id 194759 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: dump: 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce400018f0 con 0x7fce50131bb0 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 1 -- 172.21.15.76:0/1952870352 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce40001c80 con 0x7fce50131bb0 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce54df2700 10 monclient: dump: 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: get_monmap_and_config success 2024-09-16T06:48:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 4 set_mon_vals no callback set 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: shutdown 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- 172.21.15.76:0/1952870352 >> v1:172.21.15.76:6790/0 conn(0x7fce50131bb0 legacy=0x7fce501376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- 172.21.15.76:0/1952870352 shutdown_connections 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- 172.21.15.76:0/1952870352 >> 172.21.15.76:0/1952870352 conn(0x7fce500a0030 msgr2=0x7fce500a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- 172.21.15.76:0/1952870352 shutdown_connections 2024-09-16T06:48:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- 172.21.15.76:0/1952870352 wait complete. 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: build_initial_monmap 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: monmap: 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:48:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 Processor -- start 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 1 -- start start 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: init 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: _reopen_session rank -1 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.440+0000 7fce56df6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-b con 0x7fce5006beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-a con 0x7fce50131bb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient(hunting): picked mon.noname-c con 0x7fce50132060 addr v1:172.21.15.76:6790/0 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50096f50 con 0x7fce50131bb0 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce500a2680 con 0x7fce5006beb0 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50130590 con 0x7fce50132060 2024-09-16T06:48:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient(hunting): _renew_subs 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce555f3700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fce50131bb0 legacy=0x7fce5012fe80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient(hunting): authenticate will time out at 1466.580151s 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce565f5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fce50132060 0x7fce501d1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41894/0 (socket says 172.21.15.76:41894) 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce565f5700 1 -- 172.21.15.76:0/512927968 learned_addr learned my addr 172.21.15.76:0/512927968 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019178584 0 0) 0x7fce50130590 con 0x7fce50132060 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): my global_id is 194762 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340035d0 con 0x7fce50132060 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610573515 0 0) 0x7fce500a2680 con 0x7fce5006beb0 2024-09-16T06:48:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): my global_id is 224173 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50130590 con 0x7fce5006beb0 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836166003 0 0) 0x7fce340035d0 con 0x7fce50132060 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce500a2680 con 0x7fce50132060 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce48001c50 con 0x7fce50132060 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): dump: 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2007541914 0 0) 0x7fce500a2680 con 0x7fce50132060 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6789/0 conn(0x7fce5006beb0 legacy=0x7fce5012f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.76:6789/0 conn(0x7fce50131bb0 legacy=0x7fce5012fe80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: _finish_hunting 0 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 monclient: found mon.c 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:48:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5006c660 con 0x7fce50132060 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: _finish_auth 0 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: _check_auth_tickets 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 5 monclient: authenticate success, global_id 194762 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient: _renew_subs 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5006def0 con 0x7fce50132060 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient: _renew_subs 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5006ce40 con 0x7fce50132060 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce500008d0 con 0x7fce50128410 2024-09-16T06:48:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce48001fe0 con 0x7fce50132060 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce48003e90 con 0x7fce50132060 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 10 monclient: dump: 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce47fff700 4 set_mon_vals no callback set 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.444+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fce48004070 con 0x7fce50132060 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 213..255) v4 ==== 6745+0+0 (unknown 973800025 0 0) 0x7fce480440e0 con 0x7fce50132060 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760173991 0 0) 0x7fce500008d0 con 0x7fce50128410 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient(hunting): my global_id is 224176 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce500a2680 con 0x7fce50128410 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce555f3700 10 monclient: get_auth_request con 0x7fce34036180 auth_method 0 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1363206063 0 0) 0x7fce500a2680 con 0x7fce50128410 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce500008d0 con 0x7fce50128410 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce4c004150 con 0x7fce50128410 2024-09-16T06:48:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient: dump: 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3793340271 0 0) 0x7fce500008d0 con 0x7fce50128410 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce500a2350 con 0x7fce50128410 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fce4c006520 con 0x7fce50128410 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6789/0 conn(0x7fce50128410 legacy=0x7fce501287f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.448+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6789/0 conn(0x7fce50128410 legacy=0x7fce501287f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:48:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce501292e0 con 0x7fce50128f00 2024-09-16T06:48:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532851957 0 0) 0x7fce501292e0 con 0x7fce50128f00 2024-09-16T06:48:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient(hunting): _negotiate 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient(hunting): my global_id is 224179 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce500008d0 con 0x7fce50128f00 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2893965914 0 0) 0x7fce500008d0 con 0x7fce50128f00 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce501292e0 con 0x7fce50128f00 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fce4c0049f0 con 0x7fce50128f00 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient: dump: 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1376759852 0 0) 0x7fce501292e0 con 0x7fce50128f00 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce5009d3b0 con 0x7fce50128f00 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4034 (unknown 0 0 1852909223) 0x7fce4c005e20 con 0x7fce50128f00 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6789/0 conn(0x7fce50128f00 legacy=0x7fce5002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce467fc700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6789/0 conn(0x7fce50128f00 legacy=0x7fce5002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 94, 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:48:39.465 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.466 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.467 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:48:39.468 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:48:39.469 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:48:39.470 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.471 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.472 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:48:39.473 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.474 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.475 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:48:39.476 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:48:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.189:6812/3098810674 conn(0x7fce34036180 legacy=0x7fce34038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 10 monclient: shutdown 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 >> v1:172.21.15.76:6790/0 conn(0x7fce50132060 legacy=0x7fce501d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce565f5700 1 -- 172.21.15.76:0/512927968 reap_dead start 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce565f5700 1 -- 172.21.15.76:0/512927968 reap_dead start 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 shutdown_connections 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 >> 172.21.15.76:0/512927968 conn(0x7fce500a0030 msgr2=0x7fce5009b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 shutdown_connections 2024-09-16T06:48:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:48:39.460+0000 7fce56df6700 1 -- 172.21.15.76:0/512927968 wait complete. 2024-09-16T06:48:39.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:48:39.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:48:59.479 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:48:59.480 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:48:59.480 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:48:59.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:48:59.483 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:48:59.483 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:48:59.483 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:48:59.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:49:19.484 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:49:19.485 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:49:19.485 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:49:19.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:49:19.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: get_monmap_and_config 2024-09-16T06:49:19.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: build_initial_monmap 2024-09-16T06:49:19.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: monmap: 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:19.609826+0000 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:19.609826+0000 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:19.606 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 1 Processor -- start 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 1 -- start start 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: init 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient(hunting): picked mon.noname-c con 0x7f3714132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient(hunting): picked mon.noname-b con 0x7f371406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient(hunting): picked mon.noname-a con 0x7f371406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371406c280 con 0x7f371406cf50 2024-09-16T06:49:19.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371406bfa0 con 0x7f3714132330 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371409ace0 con 0x7f371406d480 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f371914a700 10 monclient(hunting): authenticate will time out at 1506.743424s 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f37137fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f371406d480 0x7f371413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36336/0 (socket says 172.21.15.76:36336) 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f3718949700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f371406cf50 0x7f371413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35904/0 (socket says 172.21.15.76:35904) 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.607+0000 7f37137fe700 1 -- 172.21.15.76:0/1408349590 learned_addr learned my addr 172.21.15.76:0/1408349590 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188612748 0 0) 0x7f371406c280 con 0x7f371406cf50 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): my global_id is 234099 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc0035d0 con 0x7f371406cf50 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052750163 0 0) 0x7f371409ace0 con 0x7f371406d480 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): my global_id is 194765 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371406c280 con 0x7f371406d480 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355654770 0 0) 0x7f371406bfa0 con 0x7f3714132330 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient(hunting): my global_id is 224182 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371409ace0 con 0x7f3714132330 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3364478800 0 0) 0x7f36fc0035d0 con 0x7f371406cf50 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 >> v1:172.21.15.76:6790/0 conn(0x7f371406d480 legacy=0x7f371413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 >> v1:172.21.15.189:6789/0 conn(0x7f3714132330 legacy=0x7f3714132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 monclient: found mon.noname-a 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371409ae70 con 0x7f371406cf50 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: _finish_auth 0 2024-09-16T06:49:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001610 con 0x7f371406cf50 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 5 monclient: authenticate success, global_id 234099 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: dump: 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37040018f0 con 0x7f371406cf50 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 1 -- 172.21.15.76:0/1408349590 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001ae0 con 0x7f371406cf50 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 4 set_mon_vals no callback set 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3712ffd700 10 monclient: dump: 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: shutdown 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- 172.21.15.76:0/1408349590 >> v1:172.21.15.76:6789/0 conn(0x7f371406cf50 legacy=0x7f371413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- 172.21.15.76:0/1408349590 shutdown_connections 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- 172.21.15.76:0/1408349590 >> 172.21.15.76:0/1408349590 conn(0x7f37140a41c0 msgr2=0x7f37140a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- 172.21.15.76:0/1408349590 shutdown_connections 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- 172.21.15.76:0/1408349590 wait complete. 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: build_initial_monmap 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: monmap: 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:19.619 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 Processor -- start 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- start start 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: init 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:19.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient(hunting): picked mon.noname-a con 0x7f371406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient(hunting): picked mon.noname-c con 0x7f371406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient(hunting): picked mon.noname-b con 0x7f3714132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371413df40 con 0x7f371406cf50 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714138fb0 con 0x7f3714132330 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714136170 con 0x7f371406d480 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f371914a700 10 monclient(hunting): authenticate will time out at 1506.747170s 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f37137fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f371406d480 0x7f3714135a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36340/0 (socket says 172.21.15.76:36340) 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f37137fe700 1 -- 172.21.15.76:0/2585532306 learned_addr learned my addr 172.21.15.76:0/2585532306 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3713fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f371406cf50 0x7f3714135350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35920/0 (socket says 172.21.15.76:35920) 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980958752 0 0) 0x7f3714136170 con 0x7f371406d480 2024-09-16T06:49:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): my global_id is 194768 2024-09-16T06:49:19.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37000035d0 con 0x7f371406d480 2024-09-16T06:49:19.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194207197 0 0) 0x7f3714138fb0 con 0x7f3714132330 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): my global_id is 224185 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714136170 con 0x7f3714132330 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267056253 0 0) 0x7f371413df40 con 0x7f371406cf50 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 10 monclient(hunting): my global_id is 234102 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.611+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714138fb0 con 0x7f371406cf50 2024-09-16T06:49:19.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3144080667 0 0) 0x7f37000035d0 con 0x7f371406d480 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f371413df40 con 0x7f371406d480 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3708001870 con 0x7f371406d480 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.627 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359178277 0 0) 0x7f3714138fb0 con 0x7f371406cf50 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37000035d0 con 0x7f371406cf50 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f370c001c20 con 0x7f371406cf50 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.628 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843529056 0 0) 0x7f3714136170 con 0x7f3714132330 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3714138fb0 con 0x7f3714132330 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3704001a60 con 0x7f3714132330 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.629 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3594159168 0 0) 0x7f37000035d0 con 0x7f371406cf50 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 >> v1:172.21.15.76:6790/0 conn(0x7f371406d480 legacy=0x7f3714135a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 >> v1:172.21.15.189:6789/0 conn(0x7f3714132330 legacy=0x7f37141d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:49:19.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 monclient: found mon.a 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3714137b00 con 0x7f371406cf50 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: _finish_auth 0 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 5 monclient: authenticate success, global_id 234102 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _renew_subs 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37140a70e0 con 0x7f371406cf50 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _renew_subs 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3714137880 con 0x7f371406cf50 2024-09-16T06:49:19.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f370c0020f0 con 0x7f371406cf50 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f370c003be0 con 0x7f371406cf50 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3711ffb700 4 set_mon_vals no callback set 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 10 monclient: dump: 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.632 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f370c003dc0 con 0x7f371406cf50 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f370c0441a0 con 0x7f371406cf50 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f4005100 con 0x7f371406cf50 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.615+0000 7f37137fe700 10 monclient: get_auth_request con 0x7f37000365a0 auth_method 0 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.623+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f370c014a80 con 0x7f371406cf50 2024-09-16T06:49:19.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.623+0000 7f3710ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:19.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.623+0000 7f3710ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:19.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.855+0000 7f371914a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:49:19.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.855+0000 7f371914a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:19.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.855+0000 7f371914a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:19.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.855+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f36f4005930 con 0x7f371406cf50 2024-09-16T06:49:19.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f3710ff9700 1 -- 172.21.15.76:0/2585532306 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 1993128523) 0x7f370c012ef0 con 0x7f371406cf50 2024-09-16T06:49:19.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f3710ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:19.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f3710ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:19.856 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:49:19.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 >> v1:172.21.15.189:6812/3098810674 conn(0x7f37000365a0 legacy=0x7f3700038a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 10 monclient: shutdown 2024-09-16T06:49:19.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:19.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 >> v1:172.21.15.76:6789/0 conn(0x7f371406cf50 legacy=0x7f3714135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:19.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 shutdown_connections 2024-09-16T06:49:19.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 >> 172.21.15.76:0/2585532306 conn(0x7f37140a41c0 msgr2=0x7f3714131800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:19.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 shutdown_connections 2024-09-16T06:49:19.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.859+0000 7f371914a700 1 -- 172.21.15.76:0/2585532306 wait complete. 2024-09-16T06:49:19.870 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:49:19.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:49:19.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: get_monmap_and_config 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: build_initial_monmap 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: monmap: 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:19.988643+0000 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:19.988643+0000 2024-09-16T06:49:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:19.985 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:19.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 1 Processor -- start 2024-09-16T06:49:19.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 1 -- start start 2024-09-16T06:49:19.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: init 2024-09-16T06:49:19.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:19.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8c40964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8c409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8c4137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c4098560 con 0x7fd8c4137e70 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c4099ec0 con 0x7fd8c40964f0 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c409b5f0 con 0x7fd8c409b1c0 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 10 monclient(hunting): authenticate will time out at 1507.122797s 2024-09-16T06:49:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c945c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd8c409b1c0 0x7fd8c4136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36346/0 (socket says 172.21.15.76:36346) 2024-09-16T06:49:19.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c945c700 1 -- 172.21.15.76:0/3104252607 learned_addr learned my addr 172.21.15.76:0/3104252607 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:19.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485971047 0 0) 0x7fd8c409b5f0 con 0x7fd8c409b1c0 2024-09-16T06:49:19.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _negotiate 2024-09-16T06:49:19.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:19.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): my global_id is 194771 2024-09-16T06:49:20.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b00035d0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104169350 0 0) 0x7fd8c4098560 con 0x7fd8c4137e70 2024-09-16T06:49:20.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): my global_id is 234105 2024-09-16T06:49:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c409b5f0 con 0x7fd8c4137e70 2024-09-16T06:49:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082045864 0 0) 0x7fd8c4099ec0 con 0x7fd8c40964f0 2024-09-16T06:49:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient(hunting): my global_id is 224188 2024-09-16T06:49:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c4098560 con 0x7fd8c40964f0 2024-09-16T06:49:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243484409 0 0) 0x7fd8b00035d0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 >> v1:172.21.15.189:6789/0 conn(0x7fd8c40964f0 legacy=0x7fd8c40968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 >> v1:172.21.15.76:6789/0 conn(0x7fd8c4137e70 legacy=0x7fd8c413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 monclient: found mon.noname-b 2024-09-16T06:49:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c4099be0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8b8001610 con 0x7fd8c409b1c0 2024-09-16T06:49:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 5 monclient: authenticate success, global_id 194771 2024-09-16T06:49:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:49:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8c8c5b700 10 monclient: dump: 2024-09-16T06:49:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.007 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.008 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.987+0000 7fd8cac5f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b80018f0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 1 -- 172.21.15.76:0/3104252607 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8b8001c80 con 0x7fd8c409b1c0 2024-09-16T06:49:20.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:20.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 4 set_mon_vals no callback set 2024-09-16T06:49:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8c8c5b700 10 monclient: dump: 2024-09-16T06:49:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.010 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.011 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.011 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.011 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.011 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.011 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.012 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.012 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.012 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: shutdown 2024-09-16T06:49:20.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- 172.21.15.76:0/3104252607 >> v1:172.21.15.76:6790/0 conn(0x7fd8c409b1c0 legacy=0x7fd8c4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- 172.21.15.76:0/3104252607 shutdown_connections 2024-09-16T06:49:20.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- 172.21.15.76:0/3104252607 >> 172.21.15.76:0/3104252607 conn(0x7fd8c40a0030 msgr2=0x7fd8c40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- 172.21.15.76:0/3104252607 shutdown_connections 2024-09-16T06:49:20.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- 172.21.15.76:0/3104252607 wait complete. 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: monmap: 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:20.014 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:20.015 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.015 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:20.015 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.015 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:20.015 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:20.016 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 Processor -- start 2024-09-16T06:49:20.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- start start 2024-09-16T06:49:20.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: init 2024-09-16T06:49:20.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:20.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8c40964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8c409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8c4137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c4095310 con 0x7fd8c4137e70 2024-09-16T06:49:20.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c408d7b0 con 0x7fd8c40964f0 2024-09-16T06:49:20.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c40a2e70 con 0x7fd8c409b1c0 2024-09-16T06:49:20.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8cac5f700 10 monclient(hunting): authenticate will time out at 1507.127693s 2024-09-16T06:49:20.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8ca45e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd8c4137e70 0x7fd8c408c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35928/0 (socket says 172.21.15.76:35928) 2024-09-16T06:49:20.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.991+0000 7fd8ca45e700 1 -- 172.21.15.76:0/4242028205 learned_addr learned my addr 172.21.15.76:0/4242028205 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123840 0 0) 0x7fd8c4095310 con 0x7fd8c4137e70 2024-09-16T06:49:20.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): my global_id is 234108 2024-09-16T06:49:20.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a80035d0 con 0x7fd8c4137e70 2024-09-16T06:49:20.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237222779 0 0) 0x7fd8c40a2e70 con 0x7fd8c409b1c0 2024-09-16T06:49:20.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): my global_id is 194774 2024-09-16T06:49:20.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c4095310 con 0x7fd8c409b1c0 2024-09-16T06:49:20.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936402089 0 0) 0x7fd8a80035d0 con 0x7fd8c4137e70 2024-09-16T06:49:20.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c40a2e70 con 0x7fd8c4137e70 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292175652 0 0) 0x7fd8c408d7b0 con 0x7fd8c40964f0 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): my global_id is 224191 2024-09-16T06:49:20.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a80035d0 con 0x7fd8c40964f0 2024-09-16T06:49:20.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8bc001c30 con 0x7fd8c4137e70 2024-09-16T06:49:20.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:20.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): dump: 2024-09-16T06:49:20.023 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:20.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1178935308 0 0) 0x7fd8c4095310 con 0x7fd8c409b1c0 2024-09-16T06:49:20.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c408d7b0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8b80015d0 con 0x7fd8c409b1c0 2024-09-16T06:49:20.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): dump: 2024-09-16T06:49:20.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.027 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.027 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.028 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.028 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.028 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2853418983 0 0) 0x7fd8c40a2e70 con 0x7fd8c4137e70 2024-09-16T06:49:20.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6790/0 conn(0x7fd8c409b1c0 legacy=0x7fd8c408bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.189:6789/0 conn(0x7fd8c40964f0 legacy=0x7fd8c408b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 monclient: found mon.a 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c4097ea0 con 0x7fd8c4137e70 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 5 monclient: authenticate success, global_id 234108 2024-09-16T06:49:20.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 10 monclient: _renew_subs 2024-09-16T06:49:20.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c409a540 con 0x7fd8c4137e70 2024-09-16T06:49:20.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 10 monclient: _renew_subs 2024-09-16T06:49:20.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c408f600 con 0x7fd8c4137e70 2024-09-16T06:49:20.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8bc003a40 con 0x7fd8c4137e70 2024-09-16T06:49:20.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8bc003d50 con 0x7fd8c4137e70 2024-09-16T06:49:20.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b7fff700 4 set_mon_vals no callback set 2024-09-16T06:49:20.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: dump: 2024-09-16T06:49:20.033 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.034 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.034 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.034 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.034 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.034 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.035 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.035 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.035 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.035 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd8ac006590 con 0x7fd8ac002830 2024-09-16T06:49:20.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd8bc003f30 con 0x7fd8c4137e70 2024-09-16T06:49:20.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd8bc044130 con 0x7fd8c4137e70 2024-09-16T06:49:20.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775273796 0 0) 0x7fd8ac006590 con 0x7fd8ac002830 2024-09-16T06:49:20.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient(hunting): my global_id is 234111 2024-09-16T06:49:20.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c40a2e70 con 0x7fd8ac002830 2024-09-16T06:49:20.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8c945c700 10 monclient: get_auth_request con 0x7fd8a80364c0 auth_method 0 2024-09-16T06:49:20.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 900468110 0 0) 0x7fd8c40a2e70 con 0x7fd8ac002830 2024-09-16T06:49:20.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8ac006590 con 0x7fd8ac002830 2024-09-16T06:49:20.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8c0001740 con 0x7fd8ac002830 2024-09-16T06:49:20.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 10 monclient: dump: 2024-09-16T06:49:20.039 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.039 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.040 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.040 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.040 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.040 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.040 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.041 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.041 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.041 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3101116584 0 0) 0x7fd8ac006590 con 0x7fd8ac002830 2024-09-16T06:49:20.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.995+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8ac00a190 con 0x7fd8ac002830 2024-09-16T06:49:20.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.999+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd8c0001c10 con 0x7fd8ac002830 2024-09-16T06:49:20.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.999+0000 7fd8b67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.999+0000 7fd8b67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:20.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.999+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6789/0 conn(0x7fd8ac002830 legacy=0x7fd8ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:19.999+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6789/0 conn(0x7fd8ac002830 legacy=0x7fd8ac004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8cac5f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:20.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd8ac009e30 con 0x7fd8ac009150 2024-09-16T06:49:20.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532649468 0 0) 0x7fd8ac009e30 con 0x7fd8ac009150 2024-09-16T06:49:20.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient(hunting): my global_id is 234114 2024-09-16T06:49:20.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac006590 con 0x7fd8ac009150 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 61548631 0 0) 0x7fd8ac006590 con 0x7fd8ac009150 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8ac009e30 con 0x7fd8ac009150 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd8c0001d80 con 0x7fd8ac009150 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient: dump: 2024-09-16T06:49:20.049 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.050 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.050 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 478131151 0 0) 0x7fd8ac009e30 con 0x7fd8ac009150 2024-09-16T06:49:20.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd8ac00c340 con 0x7fd8ac009150 2024-09-16T06:49:20.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 3883374456) 0x7fd8c00046f0 con 0x7fd8ac009150 2024-09-16T06:49:20.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:20.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:20.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6789/0 conn(0x7fd8ac009150 legacy=0x7fd8ac0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.007+0000 7fd8b67fc700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6789/0 conn(0x7fd8ac009150 legacy=0x7fd8ac0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.052 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:20.053 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:49:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.055 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:20.055 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:20.055 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.055 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.055 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.056 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.056 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.056 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.057 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.058 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:20.058 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:20.058 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.058 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.058 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:20.059 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:20.060 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.061 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:20.062 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.063 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.064 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.066 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.067 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:20.068 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:20.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd8a80364c0 legacy=0x7fd8a8038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 10 monclient: shutdown 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 >> v1:172.21.15.76:6789/0 conn(0x7fd8c4137e70 legacy=0x7fd8c408c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8ca45e700 1 -- 172.21.15.76:0/4242028205 reap_dead start 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8ca45e700 1 -- 172.21.15.76:0/4242028205 reap_dead start 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 shutdown_connections 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 >> 172.21.15.76:0/4242028205 conn(0x7fd8c40a0030 msgr2=0x7fd8c409b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 shutdown_connections 2024-09-16T06:49:20.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.011+0000 7fd8cac5f700 1 -- 172.21.15.76:0/4242028205 wait complete. 2024-09-16T06:49:20.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:20.071 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:49:20.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:49:20.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: get_monmap_and_config 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: monmap: 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:20.198220+0000 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:20.198220+0000 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:20.194 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:20.195 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 1 Processor -- start 2024-09-16T06:49:20.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 1 -- start start 2024-09-16T06:49:20.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: init 2024-09-16T06:49:20.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-a con 0x7f7d6c135500 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-b con 0x7f7d6c09a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.195+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-c con 0x7f7d6c13b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0a67b0 con 0x7f7d6c135500 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0a7320 con 0x7f7d6c13b980 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c09ade0 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient(hunting): authenticate will time out at 1507.332086s 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d711fa700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7d6c135500 0x7f7d6c1358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35966/0 (socket says 172.21.15.76:35966) 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d709f9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7d6c09a9d0 0x7f7d6c13a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36356/0 (socket says 172.21.15.76:36356) 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d711fa700 1 -- 172.21.15.76:0/1000067786 learned_addr learned my addr 172.21.15.76:0/1000067786 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542104771 0 0) 0x7f7d6c0a67b0 con 0x7f7d6c135500 2024-09-16T06:49:20.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): my global_id is 234117 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d580035d0 con 0x7f7d6c135500 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777544926 0 0) 0x7f7d6c09ade0 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): my global_id is 194777 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c0a67b0 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633478041 0 0) 0x7f7d6c0a7320 con 0x7f7d6c13b980 2024-09-16T06:49:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient(hunting): my global_id is 224194 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c09ade0 con 0x7f7d6c13b980 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233097483 0 0) 0x7f7d580035d0 con 0x7f7d6c135500 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 >> v1:172.21.15.76:6790/0 conn(0x7f7d6c09a9d0 legacy=0x7f7d6c13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 >> v1:172.21.15.189:6789/0 conn(0x7f7d6c13b980 legacy=0x7f7d6c13dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 monclient: found mon.noname-a 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c099d10 con 0x7f7d6c135500 2024-09-16T06:49:20.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 5 monclient: authenticate success, global_id 234117 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d5c001610 con 0x7f7d6c135500 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: dump: 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d5c0018f0 con 0x7f7d6c135500 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 4 set_mon_vals no callback set 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 1 -- 172.21.15.76:0/1000067786 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d5c001cc0 con 0x7f7d6c135500 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d63fff700 10 monclient: dump: 2024-09-16T06:49:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient: shutdown 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- 172.21.15.76:0/1000067786 >> v1:172.21.15.76:6789/0 conn(0x7f7d6c135500 legacy=0x7f7d6c1358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- 172.21.15.76:0/1000067786 shutdown_connections 2024-09-16T06:49:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- 172.21.15.76:0/1000067786 >> 172.21.15.76:0/1000067786 conn(0x7f7d6c0a42f0 msgr2=0x7f7d6c0a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- 172.21.15.76:0/1000067786 shutdown_connections 2024-09-16T06:49:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 1 -- 172.21.15.76:0/1000067786 wait complete. 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.199+0000 7f7d7345e700 10 monclient: monmap: 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:20.210 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 Processor -- start 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- start start 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: init 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-b con 0x7f7d6c09a9d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-c con 0x7f7d6c135500 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient(hunting): picked mon.noname-a con 0x7f7d6c13b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c09bd90 con 0x7f7d6c13b980 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c13ee20 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c134a10 con 0x7f7d6c135500 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient(hunting): authenticate will time out at 1507.336854s 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d709f9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7d6c135500 0x7f7d6c09b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36368/0 (socket says 172.21.15.76:36368) 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d709f9700 1 -- 172.21.15.76:0/1387597328 learned_addr learned my addr 172.21.15.76:0/1387597328 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d719fb700 1 --1- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6789/0 conn(0x7f7d6c13b980 0x7f7d6c1d24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35968/0 (socket says 172.21.15.76:35968) 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757398180 0 0) 0x7f7d6c134a10 con 0x7f7d6c135500 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): my global_id is 194780 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d500035d0 con 0x7f7d6c135500 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790262109 0 0) 0x7f7d6c09bd90 con 0x7f7d6c13b980 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): my global_id is 234120 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c134a10 con 0x7f7d6c13b980 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771226811 0 0) 0x7f7d6c13ee20 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): my global_id is 224197 2024-09-16T06:49:20.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c09bd90 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1099680870 0 0) 0x7f7d500035d0 con 0x7f7d6c135500 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d6c13ee20 con 0x7f7d6c135500 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d68001d50 con 0x7f7d6c135500 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.214 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175923915 0 0) 0x7f7d6c134a10 con 0x7f7d6c13b980 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d500035d0 con 0x7f7d6c13b980 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d640018b0 con 0x7f7d6c13b980 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.215 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714596962 0 0) 0x7f7d6c09bd90 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d6c134a10 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d5c001430 con 0x7f7d6c09a9d0 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2923675884 0 0) 0x7f7d6c13ee20 con 0x7f7d6c135500 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.189:6789/0 conn(0x7f7d6c09a9d0 legacy=0x7f7d6c09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6789/0 conn(0x7f7d6c13b980 legacy=0x7f7d6c1d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 monclient: found mon.c 2024-09-16T06:49:20.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c06d430 con 0x7f7d6c135500 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 5 monclient: authenticate success, global_id 194780 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _renew_subs 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c09dcd0 con 0x7f7d6c135500 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _renew_subs 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c09c5a0 con 0x7f7d6c135500 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d68004c10 con 0x7f7d6c135500 2024-09-16T06:49:20.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d637fe700 4 set_mon_vals no callback set 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d68004f20 con 0x7f7d6c135500 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d61ffb700 10 monclient: dump: 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.203+0000 7f7d7345e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d4c006590 con 0x7f7d4c002830 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7d6800f990 con 0x7f7d6c135500 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7d68044030 con 0x7f7d6c135500 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087917607 0 0) 0x7f7d4c006590 con 0x7f7d4c002830 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient(hunting): my global_id is 194783 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c13ee20 con 0x7f7d4c002830 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d719fb700 10 monclient: get_auth_request con 0x7f7d50036520 auth_method 0 2024-09-16T06:49:20.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3787099903 0 0) 0x7f7d6c13ee20 con 0x7f7d4c002830 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c006590 con 0x7f7d4c002830 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d5c004be0 con 0x7f7d4c002830 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient: dump: 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.221 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 976763093 0 0) 0x7f7d4c006590 con 0x7f7d4c002830 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d4c00a190 con 0x7f7d4c002830 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7d5c007480 con 0x7f7d4c002830 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c002830 legacy=0x7f7d4c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.207+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c002830 legacy=0x7f7d4c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d7345e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d4c009e50 con 0x7f7d4c009170 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856764653 0 0) 0x7f7d4c009e50 con 0x7f7d4c009170 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient(hunting): my global_id is 194786 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c006590 con 0x7f7d4c009170 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1527212246 0 0) 0x7f7d4c006590 con 0x7f7d4c009170 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c009e50 con 0x7f7d4c009170 2024-09-16T06:49:20.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d5c005490 con 0x7f7d4c009170 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 10 monclient: dump: 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.225 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4004375506 0 0) 0x7f7d4c009e50 con 0x7f7d4c009170 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.215+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d4c00c340 con 0x7f7d4c009170 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 2442595101) 0x7f7d5c0075f0 con 0x7f7d4c009170 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d61ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d61ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c009170 legacy=0x7f7d4c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d61ffb700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c009170 legacy=0x7f7d4c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:20.226 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.227 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:20.229 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:20.230 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:49:20.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:20.238 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7d50036520 legacy=0x7f7d500389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 10 monclient: shutdown 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 >> v1:172.21.15.76:6790/0 conn(0x7f7d6c135500 legacy=0x7f7d6c09b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d719fb700 1 -- 172.21.15.76:0/1387597328 reap_dead start 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d719fb700 1 -- 172.21.15.76:0/1387597328 reap_dead start 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 shutdown_connections 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 >> 172.21.15.76:0/1387597328 conn(0x7f7d6c0a42f0 msgr2=0x7f7d6c13de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 shutdown_connections 2024-09-16T06:49:20.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.219+0000 7f7d7345e700 1 -- 172.21.15.76:0/1387597328 wait complete. 2024-09-16T06:49:20.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:20.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:49:20.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: get_monmap_and_config 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: monmap: 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:20.368591+0000 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:20.368591+0000 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:20.366 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 1 Processor -- start 2024-09-16T06:49:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 1 -- start start 2024-09-16T06:49:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: init 2024-09-16T06:49:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-c con 0x7f95e406beb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-b con 0x7f95e4131bb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-a con 0x7f95e4132060 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e409f630 con 0x7f95e4132060 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e409f8a0 con 0x7f95e406beb0 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e4099350 con 0x7f95e4131bb0 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient(hunting): authenticate will time out at 1507.502019s 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95e9f63700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f95e4132060 0x7f95e413ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35978/0 (socket says 172.21.15.76:35978) 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95e9f63700 1 -- 172.21.15.76:0/1600698771 learned_addr learned my addr 172.21.15.76:0/1600698771 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839081127 0 0) 0x7f95e409f630 con 0x7f95e4132060 2024-09-16T06:49:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): my global_id is 234123 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d00057e0 con 0x7f95e4132060 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079680921 0 0) 0x7f95e4099350 con 0x7f95e4131bb0 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): my global_id is 194789 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e409f630 con 0x7f95e4131bb0 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984762991 0 0) 0x7f95e409f8a0 con 0x7f95e406beb0 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient(hunting): my global_id is 224200 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4099350 con 0x7f95e406beb0 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384612308 0 0) 0x7f95d00057e0 con 0x7f95e4132060 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 >> v1:172.21.15.76:6790/0 conn(0x7f95e4131bb0 legacy=0x7f95e41376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 >> v1:172.21.15.189:6789/0 conn(0x7f95e406beb0 legacy=0x7f95e41314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 monclient: found mon.noname-a 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4099630 con 0x7f95e4132060 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95dc001990 con 0x7f95e4132060 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: dump: 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.380 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 5 monclient: authenticate success, global_id 234123 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95dc001c70 con 0x7f95e4132060 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 1 -- 172.21.15.76:0/1600698771 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95dc002040 con 0x7f95e4132060 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95ea764700 4 set_mon_vals no callback set 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.367+0000 7f95dbfff700 10 monclient: dump: 2024-09-16T06:49:20.381 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: shutdown 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- 172.21.15.76:0/1600698771 >> v1:172.21.15.76:6789/0 conn(0x7f95e4132060 legacy=0x7f95e413ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- 172.21.15.76:0/1600698771 shutdown_connections 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- 172.21.15.76:0/1600698771 >> 172.21.15.76:0/1600698771 conn(0x7f95e40a0030 msgr2=0x7f95e40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- 172.21.15.76:0/1600698771 shutdown_connections 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- 172.21.15.76:0/1600698771 wait complete. 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: monmap: 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:20.383 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 Processor -- start 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- start start 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: init 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-c con 0x7f95e406beb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-b con 0x7f95e4131bb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient(hunting): picked mon.noname-a con 0x7f95e4132060 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e409bf40 con 0x7f95e4132060 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e4131810 con 0x7f95e4131bb0 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e409a540 con 0x7f95e406beb0 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95ea764700 10 monclient(hunting): authenticate will time out at 1507.506864s 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95e9f63700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f95e4132060 0x7f95e41bdae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:35992/0 (socket says 172.21.15.76:35992) 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95e9f63700 1 -- 172.21.15.76:0/2860089041 learned_addr learned my addr 172.21.15.76:0/2860089041 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090252605 0 0) 0x7f95e409bf40 con 0x7f95e4132060 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): my global_id is 234126 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c80035d0 con 0x7f95e4132060 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093497349 0 0) 0x7f95e409a540 con 0x7f95e406beb0 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): my global_id is 194792 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e409bf40 con 0x7f95e406beb0 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349958778 0 0) 0x7f95e4131810 con 0x7f95e4131bb0 2024-09-16T06:49:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): my global_id is 224203 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e409a540 con 0x7f95e4131bb0 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4217064951 0 0) 0x7f95c80035d0 con 0x7f95e4132060 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e4131810 con 0x7f95e4132060 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95dc001950 con 0x7f95e4132060 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1744836520 0 0) 0x7f95e409bf40 con 0x7f95e406beb0 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c80035d0 con 0x7f95e406beb0 2024-09-16T06:49:20.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95e0001870 con 0x7f95e406beb0 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.371+0000 7f95d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4165076838 0 0) 0x7f95e409a540 con 0x7f95e4131bb0 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e409bf40 con 0x7f95e4131bb0 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95d40019f0 con 0x7f95e4131bb0 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3510104262 0 0) 0x7f95e4131810 con 0x7f95e4132060 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.76:6790/0 conn(0x7f95e406beb0 legacy=0x7f95e41bccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6789/0 conn(0x7f95e4131bb0 legacy=0x7f95e41bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 monclient: found mon.a 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4128820 con 0x7f95e4132060 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 5 monclient: authenticate success, global_id 234126 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 10 monclient: _renew_subs 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e412a7a0 con 0x7f95e4132060 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 10 monclient: _renew_subs 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e41293c0 con 0x7f95e4132060 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95dc005020 con 0x7f95e4132060 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95dc005330 con 0x7f95e4132060 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95db7fe700 4 set_mon_vals no callback set 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: dump: 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.393 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f95dc00fda0 con 0x7f95e4132060 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f95dc044410 con 0x7f95e4132060 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95e9762700 10 monclient: get_auth_request con 0x7f95c8036630 auth_method 0 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95c4006590 con 0x7f95c4002830 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979238760 0 0) 0x7f95c4006590 con 0x7f95c4002830 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient(hunting): my global_id is 224206 2024-09-16T06:49:20.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4131810 con 0x7f95c4002830 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1013951932 0 0) 0x7f95e4131810 con 0x7f95c4002830 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c4006590 con 0x7f95c4002830 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95d4003840 con 0x7f95c4002830 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.375+0000 7f95d9ffb700 10 monclient: dump: 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3493363674 0 0) 0x7f95c4006590 con 0x7f95c4002830 2024-09-16T06:49:20.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95c400a190 con 0x7f95c4002830 2024-09-16T06:49:20.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f95d40046d0 con 0x7f95c4002830 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6789/0 conn(0x7f95c4002830 legacy=0x7f95c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.379+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6789/0 conn(0x7f95c4002830 legacy=0x7f95c4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95ea764700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95c4009e50 con 0x7f95c4009170 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187300472 0 0) 0x7f95c4009e50 con 0x7f95c4009170 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient(hunting): my global_id is 224209 2024-09-16T06:49:20.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c4006590 con 0x7f95c4009170 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3266788002 0 0) 0x7f95c4006590 con 0x7f95c4009170 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c4009e50 con 0x7f95c4009170 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95d4003df0 con 0x7f95c4009170 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient: dump: 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.400 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3341839484 0 0) 0x7f95c4009e50 con 0x7f95c4009170 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95c400c340 con 0x7f95c4009170 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 3319009883) 0x7f95d4004170 con 0x7f95c4009170 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6789/0 conn(0x7f95c4009170 legacy=0x7f95c4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.387+0000 7f95d9ffb700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6789/0 conn(0x7f95c4009170 legacy=0x7f95c4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:20.402 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.403 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:20.404 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.405 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.406 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:20.408 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:20.409 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:20.411 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:20.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:20.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:20.415 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.189:6812/3098810674 conn(0x7f95c8036630 legacy=0x7f95c8038af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 10 monclient: shutdown 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 >> v1:172.21.15.76:6789/0 conn(0x7f95e4132060 legacy=0x7f95e41bdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95e9f63700 1 -- 172.21.15.76:0/2860089041 reap_dead start 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95e9f63700 1 -- 172.21.15.76:0/2860089041 reap_dead start 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 shutdown_connections 2024-09-16T06:49:20.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 >> 172.21.15.76:0/2860089041 conn(0x7f95e40a0030 msgr2=0x7f95e4070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 shutdown_connections 2024-09-16T06:49:20.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.391+0000 7f95ea764700 1 -- 172.21.15.76:0/2860089041 wait complete. 2024-09-16T06:49:20.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:20.418 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:49:20.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:49:20.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.535+0000 7f3a8e740700 10 monclient: get_monmap_and_config 2024-09-16T06:49:20.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.535+0000 7f3a8e740700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.535+0000 7f3a8e740700 10 monclient: monmap: 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:20.539921+0000 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:20.539921+0000 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:20.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 Processor -- start 2024-09-16T06:49:20.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- start start 2024-09-16T06:49:20.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient: init 2024-09-16T06:49:20.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-c con 0x7f3a8809b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-a con 0x7f3a8809f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-b con 0x7f3a88138840 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809bfb0 con 0x7f3a8809f230 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809d3c0 con 0x7f3a8809b6a0 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8809c220 con 0x7f3a88138840 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient(hunting): authenticate will time out at 1507.673250s 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8ccdd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3a88138840 0x7f3a8813ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36422/0 (socket says 172.21.15.76:36422) 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8ccdd700 1 -- 172.21.15.76:0/102204617 learned_addr learned my addr 172.21.15.76:0/102204617 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a877fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a8809f230 0x7f3a88137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:36002/0 (socket says 172.21.15.76:36002) 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694733817 0 0) 0x7f3a8809c220 con 0x7f3a88138840 2024-09-16T06:49:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): my global_id is 194795 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a740057e0 con 0x7f3a88138840 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253216511 0 0) 0x7f3a8809bfb0 con 0x7f3a8809f230 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): my global_id is 234129 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8809c220 con 0x7f3a8809f230 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730223819 0 0) 0x7f3a8809d3c0 con 0x7f3a8809b6a0 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient(hunting): my global_id is 224212 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8809bfb0 con 0x7f3a8809b6a0 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085306149 0 0) 0x7f3a8809c220 con 0x7f3a8809f230 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 >> v1:172.21.15.76:6790/0 conn(0x7f3a88138840 legacy=0x7f3a8813ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 >> v1:172.21.15.189:6789/0 conn(0x7f3a8809b6a0 legacy=0x7f3a880a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 monclient: found mon.noname-a 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8809c500 con 0x7f3a8809f230 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a700018a0 con 0x7f3a8809f230 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: dump: 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 5 monclient: authenticate success, global_id 234129 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a70001ac0 con 0x7f3a8809f230 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 1 -- 172.21.15.76:0/102204617 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a70001ed0 con 0x7f3a8809f230 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a86ffd700 10 monclient: dump: 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 4 set_mon_vals no callback set 2024-09-16T06:49:20.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 10 monclient: shutdown 2024-09-16T06:49:20.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- 172.21.15.76:0/102204617 >> v1:172.21.15.76:6789/0 conn(0x7f3a8809f230 legacy=0x7f3a88137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- 172.21.15.76:0/102204617 shutdown_connections 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.539+0000 7f3a8e740700 1 -- 172.21.15.76:0/102204617 >> 172.21.15.76:0/102204617 conn(0x7f3a880a3e50 msgr2=0x7f3a880a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- 172.21.15.76:0/102204617 shutdown_connections 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- 172.21.15.76:0/102204617 wait complete. 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient: build_initial_monmap 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient: monmap: 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:20.549 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 Processor -- start 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- start start 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient: init 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-a con 0x7f3a8809b6a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-c con 0x7f3a8809f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient(hunting): picked mon.noname-b con 0x7f3a88138840 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:20.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a88133840 con 0x7f3a8809b6a0 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8811c020 con 0x7f3a88138840 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8811d9f0 con 0x7f3a8809f230 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a8e740700 10 monclient(hunting): authenticate will time out at 1507.679108s 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a87fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a8809b6a0 0x7f3a88132a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:36004/0 (socket says 172.21.15.76:36004) 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a87fff700 1 -- 172.21.15.76:0/1578906828 learned_addr learned my addr 172.21.15.76:0/1578906828 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204772563 0 0) 0x7f3a8811d9f0 con 0x7f3a8809f230 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): my global_id is 194798 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a680035d0 con 0x7f3a8809f230 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331812067 0 0) 0x7f3a88133840 con 0x7f3a8809b6a0 2024-09-16T06:49:20.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): my global_id is 234132 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8811d9f0 con 0x7f3a8809b6a0 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032303465 0 0) 0x7f3a8811c020 con 0x7f3a88138840 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 10 monclient(hunting): my global_id is 224215 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.543+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a88133840 con 0x7f3a88138840 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880856127 0 0) 0x7f3a680035d0 con 0x7f3a8809f230 2024-09-16T06:49:20.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a8811c020 con 0x7f3a8809f230 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a70001dd0 con 0x7f3a8809f230 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 793605076 0 0) 0x7f3a8811c020 con 0x7f3a8809f230 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 >> v1:172.21.15.189:6789/0 conn(0x7f3a88138840 legacy=0x7f3a881ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 >> v1:172.21.15.76:6789/0 conn(0x7f3a8809b6a0 legacy=0x7f3a88132a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 monclient: found mon.c 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8811d710 con 0x7f3a8809f230 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: _finish_auth 0 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:49:20.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 5 monclient: authenticate success, global_id 194798 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _renew_subs 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a880a2dd0 con 0x7f3a8809f230 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _renew_subs 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a88072450 con 0x7f3a8809f230 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a70004ec0 con 0x7f3a8809f230 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a70005250 con 0x7f3a8809f230 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 10 monclient: dump: 2024-09-16T06:49:20.555 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a867fc700 4 set_mon_vals no callback set 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3a700053b0 con 0x7f3a8809f230 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3a70044330 con 0x7f3a8809f230 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c005100 con 0x7f3a8809f230 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.547+0000 7f3a87fff700 10 monclient: get_auth_request con 0x7f3a68036110 auth_method 0 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.555+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3a70014d70 con 0x7f3a8809f230 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.555+0000 7f3a84ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:20.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.555+0000 7f3a84ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:20.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.787+0000 7f3a8e740700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:49:20.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.787+0000 7f3a8e740700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:49:20.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.787+0000 7f3a8e740700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:20.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.787+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3a6c005bc0 con 0x7f3a8809f230 2024-09-16T06:49:20.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f3a84ff9700 1 -- 172.21.15.76:0/1578906828 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3a700131e0 con 0x7f3a8809f230 2024-09-16T06:49:20.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f3a84ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:49:20.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f3a84ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:20.789 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.789 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=2083279389,config=1097430493,health=3595141338,kv=2335881212,logm=3001828537}) 2024-09-16T06:49:20.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=10,kv=12,logm=49} crc {auth=3212873691,config=1097430493,health=542606653,kv=1588170259,logm=4230334910}) 2024-09-16T06:49:20.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=2083279389,config=1097430493,health=3595141338,kv=2335881212,logm=3001828537}) 2024-09-16T06:49:20.790 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.791+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=1653195022,config=1097430493,health=1350913232,kv=876538035,logm=2524722341}) 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298364496}) 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920413354}) 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298364496}) 2024-09-16T06:49:20.792 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543025020}) 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3a68036110 legacy=0x7f3a680385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 10 monclient: shutdown 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 >> v1:172.21.15.76:6790/0 conn(0x7f3a8809f230 legacy=0x7f3a881cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 shutdown_connections 2024-09-16T06:49:20.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 >> 172.21.15.76:0/1578906828 conn(0x7f3a880a3e50 msgr2=0x7f3a8809a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:20.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 shutdown_connections 2024-09-16T06:49:20.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f3a8e740700 1 -- 172.21.15.76:0/1578906828 wait complete. 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644872128}) 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3653869523}) 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644872128}) 2024-09-16T06:49:20.795 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.795+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902788658}) 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795821195}) 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715116062}) 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795821195}) 2024-09-16T06:49:20.797 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883959935}) 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435366742}) 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123092932}) 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435366742}) 2024-09-16T06:49:20.799 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.799+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417032863}) 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277960803}) 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074185639}) 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277960803}) 2024-09-16T06:49:20.801 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444168641}) 2024-09-16T06:49:20.802 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186463299}) 2024-09-16T06:49:20.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140900102}) 2024-09-16T06:49:20.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186463299}) 2024-09-16T06:49:20.803 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.803+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321795750}) 2024-09-16T06:49:20.804 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.804 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716828133}) 2024-09-16T06:49:20.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256021174}) 2024-09-16T06:49:20.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716828133}) 2024-09-16T06:49:20.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930578026}) 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830199036}) 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880881066}) 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830199036}) 2024-09-16T06:49:20.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.807+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247913543}) 2024-09-16T06:49:20.807 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651282684}) 2024-09-16T06:49:20.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097712456}) 2024-09-16T06:49:20.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651282684}) 2024-09-16T06:49:20.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560525830}) 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399617354}) 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592591070}) 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399617354}) 2024-09-16T06:49:20.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=41210023}) 2024-09-16T06:49:20.810 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501556368}) 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058225248}) 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501556368}) 2024-09-16T06:49:20.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726980253}) 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881010861}) 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244568007}) 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881010861}) 2024-09-16T06:49:20.812 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.811+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3556659174}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211647349}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718821378}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211647349}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900155069}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559895623}) 2024-09-16T06:49:20.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395316834}) 2024-09-16T06:49:20.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559895623}) 2024-09-16T06:49:20.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209268252}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=508231842}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320768473}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=508231842}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710480028}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452565569}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011803339}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452565569}) 2024-09-16T06:49:20.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.815+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178379890}) 2024-09-16T06:49:20.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682714985}) 2024-09-16T06:49:20.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572428574}) 2024-09-16T06:49:20.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682714985}) 2024-09-16T06:49:20.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230147568}) 2024-09-16T06:49:20.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334054708}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763457375}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334054708}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389308225}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=927531382}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216889855}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=927531382}) 2024-09-16T06:49:20.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167097454}) 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737045739}) 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665064033}) 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737045739}) 2024-09-16T06:49:20.819 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.819+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519026938}) 2024-09-16T06:49:20.820 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.820 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646146187}) 2024-09-16T06:49:20.820 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934505013}) 2024-09-16T06:49:20.820 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.820 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646146187}) 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255815274}) 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796090152}) 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698637360}) 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796090152}) 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599033409}) 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942428421}) 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678785531}) 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942428421}) 2024-09-16T06:49:20.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158928106}) 2024-09-16T06:49:20.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839078280}) 2024-09-16T06:49:20.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934355907}) 2024-09-16T06:49:20.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839078280}) 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.823+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489172690}) 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731302430}) 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512036178}) 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731302430}) 2024-09-16T06:49:20.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2840892871}) 2024-09-16T06:49:20.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145372666}) 2024-09-16T06:49:20.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063769639}) 2024-09-16T06:49:20.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145372666}) 2024-09-16T06:49:20.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=303261850}) 2024-09-16T06:49:20.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434961665}) 2024-09-16T06:49:20.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932018622}) 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434961665}) 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097370521}) 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=795585505}) 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1028245006}) 2024-09-16T06:49:20.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=795585505}) 2024-09-16T06:49:20.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.827+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835687666}) 2024-09-16T06:49:20.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126980518}) 2024-09-16T06:49:20.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878159497}) 2024-09-16T06:49:20.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126980518}) 2024-09-16T06:49:20.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723052890}) 2024-09-16T06:49:20.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723858068}) 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473248731}) 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723858068}) 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1474749631}) 2024-09-16T06:49:20.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833970922}) 2024-09-16T06:49:20.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087387434}) 2024-09-16T06:49:20.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833970922}) 2024-09-16T06:49:20.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447344142}) 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467932178}) 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103670907}) 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.831+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467932178}) 2024-09-16T06:49:20.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1481129518}) 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102445849}) 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185384950}) 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102445849}) 2024-09-16T06:49:20.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875430941}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348482253}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470924918}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348482253}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1700366209}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022654171}) 2024-09-16T06:49:20.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2334632656}) 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022654171}) 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194733681}) 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283171143}) 2024-09-16T06:49:20.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879947690}) 2024-09-16T06:49:20.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283171143}) 2024-09-16T06:49:20.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.835+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265001455}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969702220}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838153854}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969702220}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111442746}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4212786644}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741391212}) 2024-09-16T06:49:20.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4212786644}) 2024-09-16T06:49:20.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232539210}) 2024-09-16T06:49:20.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334709923}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549901959}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334709923}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4285024542}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695212935}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565693815}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695212935}) 2024-09-16T06:49:20.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.839+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1981979689}) 2024-09-16T06:49:20.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471523287}) 2024-09-16T06:49:20.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409686770}) 2024-09-16T06:49:20.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471523287}) 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449251329}) 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935686284}) 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650970176}) 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935686284}) 2024-09-16T06:49:20.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680000349}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407656850}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229837421}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407656850}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298553049}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625362110}) 2024-09-16T06:49:20.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582404924}) 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625362110}) 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1927893881}) 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240316640}) 2024-09-16T06:49:20.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879173805}) 2024-09-16T06:49:20.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240316640}) 2024-09-16T06:49:20.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.843+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869220899}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414658250}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2423169457}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414658250}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=526960089}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860509339}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289877634}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860509339}) 2024-09-16T06:49:20.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067589159}) 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990547939}) 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215402784}) 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2990547939}) 2024-09-16T06:49:20.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1506279520}) 2024-09-16T06:49:20.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563715524}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750371444}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563715524}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.847+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174461280}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158684847}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2915720298}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158684847}) 2024-09-16T06:49:20.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417601926}) 2024-09-16T06:49:20.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711723853}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792683059}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711723853}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1764539557}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409968157}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094481047}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409968157}) 2024-09-16T06:49:20.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582772301}) 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494759500}) 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371548894}) 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494759500}) 2024-09-16T06:49:20.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.851+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468227299}) 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918723610}) 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312512073}) 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918723610}) 2024-09-16T06:49:20.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877206158}) 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774542661}) 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3313550815}) 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774542661}) 2024-09-16T06:49:20.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502624400}) 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416737142}) 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340577573}) 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416737142}) 2024-09-16T06:49:20.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.855+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577900980}) 2024-09-16T06:49:20.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2293705054,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=558189638}) 2024-09-16T06:49:20.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1742516989,mdsmap=1531151339,mgr=2591763412,mgr_command_descs=794974420,mgr_metadata=1347778423,mgrstat=2265765824}) 2024-09-16T06:49:20.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2293705054,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=558189638}) 2024-09-16T06:49:20.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.859+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3647516814,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1086472424}) 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=2015876677,mon_config_key=3671709268,monmap=968649221,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=213298684}) 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=5,mon_config_key=20,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=3101172053,mon_config_key=3733271562,monmap=3140514009,osd_metadata=1269664135,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4132858537}) 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=5,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=2015876677,mon_config_key=3671709268,monmap=968649221,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=213298684}) 2024-09-16T06:49:20.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:49:20.867+0000 7f779860f700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=5,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=2015876677,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1883579448}) 2024-09-16T06:49:21.812 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:49:21.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:49:21.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.931+0000 7f2b8a197700 10 monclient: get_monmap_and_config 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.931+0000 7f2b8a197700 10 monclient: build_initial_monmap 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: monmap: 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:21.935976+0000 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:21.935976+0000 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:21.932 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 Processor -- start 2024-09-16T06:49:21.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- start start 2024-09-16T06:49:21.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: init 2024-09-16T06:49:21.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:21.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-b con 0x7f2b84132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-c con 0x7f2b8406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-a con 0x7f2b8406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8406c280 con 0x7f2b8406cf50 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8406bfa0 con 0x7f2b8406d480 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8409ace0 con 0x7f2b84132330 2024-09-16T06:49:21.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient(hunting): authenticate will time out at 1509.069458s 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b89195700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2b84132330 0x7f2b84132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36438/0 (socket says 172.21.15.76:36438) 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b89195700 1 -- 172.21.15.76:0/857970780 learned_addr learned my addr 172.21.15.76:0/857970780 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b89996700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b8406cf50 0x7f2b8413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:36016/0 (socket says 172.21.15.76:36016) 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175868360 0 0) 0x7f2b8406c280 con 0x7f2b8406cf50 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): my global_id is 234135 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b700035d0 con 0x7f2b8406cf50 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615995470 0 0) 0x7f2b8409ace0 con 0x7f2b84132330 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): my global_id is 194801 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8406c280 con 0x7f2b84132330 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404143057 0 0) 0x7f2b8406bfa0 con 0x7f2b8406d480 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient(hunting): my global_id is 224218 2024-09-16T06:49:21.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8409ace0 con 0x7f2b8406d480 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256618807 0 0) 0x7f2b700035d0 con 0x7f2b8406cf50 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 >> v1:172.21.15.76:6790/0 conn(0x7f2b84132330 legacy=0x7f2b84132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 >> v1:172.21.15.189:6789/0 conn(0x7f2b8406d480 legacy=0x7f2b8413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: _finish_hunting 0 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 monclient: found mon.noname-a 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8409ae70 con 0x7f2b8406cf50 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: _finish_auth 0 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: _check_auth_tickets 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:21.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 5 monclient: authenticate success, global_id 234135 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b80001540 con 0x7f2b8406cf50 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: dump: 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b80001820 con 0x7f2b8406cf50 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 4 set_mon_vals no callback set 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 1 -- 172.21.15.76:0/857970780 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b80001a50 con 0x7f2b8406cf50 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b77fff700 10 monclient: dump: 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:21.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 10 monclient: shutdown 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- 172.21.15.76:0/857970780 >> v1:172.21.15.76:6789/0 conn(0x7f2b8406cf50 legacy=0x7f2b8413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- 172.21.15.76:0/857970780 shutdown_connections 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- 172.21.15.76:0/857970780 >> 172.21.15.76:0/857970780 conn(0x7f2b840a41c0 msgr2=0x7f2b840a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:21.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- 172.21.15.76:0/857970780 shutdown_connections 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.935+0000 7f2b8a197700 1 -- 172.21.15.76:0/857970780 wait complete. 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: build_initial_monmap 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: monmap: 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:21.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:21.945 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 Processor -- start 2024-09-16T06:49:21.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- start start 2024-09-16T06:49:21.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: init 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-c con 0x7f2b8406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-a con 0x7f2b8406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient(hunting): picked mon.noname-b con 0x7f2b84132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:21.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8409e8b0 con 0x7f2b8406d480 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b841316b0 con 0x7f2b84132330 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b841311f0 con 0x7f2b8406cf50 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient(hunting): authenticate will time out at 1509.074042s 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b88994700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b8406d480 0x7f2b841ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:36022/0 (socket says 172.21.15.76:36022) 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b88994700 1 -- 172.21.15.76:0/4080688487 learned_addr learned my addr 172.21.15.76:0/4080688487 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607362788 0 0) 0x7f2b8409e8b0 con 0x7f2b8406d480 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): my global_id is 234138 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b680035d0 con 0x7f2b8406d480 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242791071 0 0) 0x7f2b841311f0 con 0x7f2b8406cf50 2024-09-16T06:49:21.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): my global_id is 194804 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8409e8b0 con 0x7f2b8406cf50 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165370904 0 0) 0x7f2b841316b0 con 0x7f2b84132330 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): my global_id is 224221 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b841311f0 con 0x7f2b84132330 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3550770090 0 0) 0x7f2b680035d0 con 0x7f2b8406d480 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b841316b0 con 0x7f2b8406d480 2024-09-16T06:49:21.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b7c001cb0 con 0x7f2b8406d480 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:21.953 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696061170 0 0) 0x7f2b8409e8b0 con 0x7f2b8406cf50 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b680035d0 con 0x7f2b8406cf50 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b78001830 con 0x7f2b8406cf50 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3683349710 0 0) 0x7f2b841316b0 con 0x7f2b8406d480 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 >> v1:172.21.15.76:6790/0 conn(0x7f2b8406cf50 legacy=0x7f2b8413e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 >> v1:172.21.15.189:6789/0 conn(0x7f2b84132330 legacy=0x7f2b841d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 monclient: found mon.a 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:21.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8412ece0 con 0x7f2b8406d480 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: _finish_auth 0 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 5 monclient: authenticate success, global_id 234138 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _renew_subs 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8412f430 con 0x7f2b8406d480 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _renew_subs 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b84130dd0 con 0x7f2b8406d480 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b7c003a40 con 0x7f2b8406d480 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:21.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2b7c003d50 con 0x7f2b8406d480 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b777fe700 4 set_mon_vals no callback set 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.939+0000 7f2b75ffb700 10 monclient: dump: 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:21.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b8a197700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b8a197700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b8a197700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b64005100 con 0x7f2b8406d480 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2b7c003f30 con 0x7f2b8406d480 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2b7c0442a0 con 0x7f2b8406d480 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.943+0000 7f2b89195700 10 monclient: get_auth_request con 0x7f2b680364c0 auth_method 0 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.951+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2b7c014be0 con 0x7f2b8406d480 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.951+0000 7f2b75ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:21.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:21.951+0000 7f2b75ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:22.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.179+0000 7f2b8a197700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:49:22.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.179+0000 7f2b8a197700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:22.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.179+0000 7f2b8a197700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.179+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2b64005930 con 0x7f2b8406d480 2024-09-16T06:49:22.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b75ffb700 1 -- 172.21.15.76:0/4080688487 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1389 (unknown 254746657 0 46297796) 0x7f2b7c013050 con 0x7f2b8406d480 2024-09-16T06:49:22.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b75ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:22.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b75ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2b680364c0 legacy=0x7f2b68038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 10 monclient: shutdown 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 >> v1:172.21.15.76:6789/0 conn(0x7f2b8406d480 legacy=0x7f2b841ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 shutdown_connections 2024-09-16T06:49:22.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 >> 172.21.15.76:0/4080688487 conn(0x7f2b840a41c0 msgr2=0x7f2b8409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 shutdown_connections 2024-09-16T06:49:22.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.183+0000 7f2b8a197700 1 -- 172.21.15.76:0/4080688487 wait complete. 2024-09-16T06:49:22.194 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:49:22.194 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:49:22.195 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:49:22.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:49:22.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: get_monmap_and_config 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: monmap: 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:22.310171+0000 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:22.310171+0000 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:22.306 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 1 Processor -- start 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 1 -- start start 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: init 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-a con 0x7f7a4c0973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-b con 0x7f7a4c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c099350 con 0x7f7a4c0973c0 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c09f690 con 0x7f7a4c097eb0 2024-09-16T06:49:22.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c099bf0 con 0x7f7a4c1400d0 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a520d4700 10 monclient(hunting): authenticate will time out at 1509.443414s 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a518d3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7a4c1400d0 0x7f7a4c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36462/0 (socket says 172.21.15.76:36462) 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.307+0000 7f7a518d3700 1 -- 172.21.15.76:0/1183914114 learned_addr learned my addr 172.21.15.76:0/1183914114 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324955922 0 0) 0x7f7a4c099350 con 0x7f7a4c0973c0 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): my global_id is 234141 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a380035d0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457011675 0 0) 0x7f7a4c099bf0 con 0x7f7a4c1400d0 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): my global_id is 194807 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c099350 con 0x7f7a4c1400d0 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744700174 0 0) 0x7f7a4c09f690 con 0x7f7a4c097eb0 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient(hunting): my global_id is 224224 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c099bf0 con 0x7f7a4c097eb0 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500473753 0 0) 0x7f7a380035d0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 >> v1:172.21.15.76:6790/0 conn(0x7f7a4c1400d0 legacy=0x7f7a4c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 >> v1:172.21.15.189:6789/0 conn(0x7f7a4c097eb0 legacy=0x7f7a4c13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 monclient: found mon.noname-a 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c098360 con 0x7f7a4c0973c0 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a3c001610 con 0x7f7a4c0973c0 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 5 monclient: authenticate success, global_id 234141 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: dump: 2024-09-16T06:49:22.313 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a3c0018f0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 1 -- 172.21.15.76:0/1183914114 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a3c001c80 con 0x7f7a4c0973c0 2024-09-16T06:49:22.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a43fff700 10 monclient: dump: 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 4 set_mon_vals no callback set 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 10 monclient: shutdown 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 1 -- 172.21.15.76:0/1183914114 >> v1:172.21.15.76:6789/0 conn(0x7f7a4c0973c0 legacy=0x7f7a4c0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 1 -- 172.21.15.76:0/1183914114 shutdown_connections 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 1 -- 172.21.15.76:0/1183914114 >> 172.21.15.76:0/1183914114 conn(0x7f7a4c0a0030 msgr2=0x7f7a4c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 1 -- 172.21.15.76:0/1183914114 shutdown_connections 2024-09-16T06:49:22.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 1 -- 172.21.15.76:0/1183914114 wait complete. 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.311+0000 7f7a520d4700 10 monclient: monmap: 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 Processor -- start 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- start start 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: init 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-b con 0x7f7a4c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient(hunting): picked mon.noname-a con 0x7f7a4c1400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c072660 con 0x7f7a4c1400d0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c06c500 con 0x7f7a4c097eb0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c13fae0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient(hunting): authenticate will time out at 1509.449182s 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a510d2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7a4c0973c0 0x7f7a4c137770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36474/0 (socket says 172.21.15.76:36474) 2024-09-16T06:49:22.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a510d2700 1 -- 172.21.15.76:0/64326035 learned_addr learned my addr 172.21.15.76:0/64326035 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643895112 0 0) 0x7f7a4c13fae0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): my global_id is 194810 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a300035d0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900623714 0 0) 0x7f7a4c072660 con 0x7f7a4c1400d0 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): my global_id is 234144 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c13fae0 con 0x7f7a4c1400d0 2024-09-16T06:49:22.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649373925 0 0) 0x7f7a4c06c500 con 0x7f7a4c097eb0 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): my global_id is 224227 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c072660 con 0x7f7a4c097eb0 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776029440 0 0) 0x7f7a4c13fae0 con 0x7f7a4c1400d0 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4c06c500 con 0x7f7a4c1400d0 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a440018b0 con 0x7f7a4c1400d0 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3648956932 0 0) 0x7f7a300035d0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4c13fae0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a3c0015d0 con 0x7f7a4c0973c0 2024-09-16T06:49:22.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient(hunting): dump: 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1871772588 0 0) 0x7f7a4c06c500 con 0x7f7a4c1400d0 2024-09-16T06:49:22.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6790/0 conn(0x7f7a4c0973c0 legacy=0x7f7a4c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.189:6789/0 conn(0x7f7a4c097eb0 legacy=0x7f7a4c1d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 monclient: found mon.a 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c06ca30 con 0x7f7a4c1400d0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 5 monclient: authenticate success, global_id 234144 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _renew_subs 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c06be10 con 0x7f7a4c1400d0 2024-09-16T06:49:22.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _renew_subs 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c06c190 con 0x7f7a4c1400d0 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a44004890 con 0x7f7a4c1400d0 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a437fe700 4 set_mon_vals no callback set 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a44004a00 con 0x7f7a4c1400d0 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 10 monclient: dump: 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a2c006590 con 0x7f7a2c002830 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.315+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438265787 0 0) 0x7f7a2c006590 con 0x7f7a2c002830 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient(hunting): my global_id is 234147 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c06c500 con 0x7f7a2c002830 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3684462235 0 0) 0x7f7a4c06c500 con 0x7f7a2c002830 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a2c006590 con 0x7f7a2c002830 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a3c003a70 con 0x7f7a2c002830 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient: dump: 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.330 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 484310024 0 0) 0x7f7a2c006590 con 0x7f7a2c002830 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a2c00a190 con 0x7f7a2c002830 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7a4400f470 con 0x7f7a4c1400d0 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7a44043b60 con 0x7f7a4c1400d0 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7a3c005830 con 0x7f7a2c002830 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6789/0 conn(0x7f7a2c002830 legacy=0x7f7a2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6789/0 conn(0x7f7a2c002830 legacy=0x7f7a2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.319+0000 7f7a508d1700 10 monclient: get_auth_request con 0x7f7a30036660 auth_method 0 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7a2c00c4e0 con 0x7f7a2c009150 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287136051 0 0) 0x7f7a2c00c4e0 con 0x7f7a2c009150 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient(hunting): my global_id is 234150 2024-09-16T06:49:22.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c006590 con 0x7f7a2c009150 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4282258638 0 0) 0x7f7a2c006590 con 0x7f7a2c009150 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a2c00c4e0 con 0x7f7a2c009150 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7a3c004360 con 0x7f7a2c009150 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient: dump: 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.334 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1416105406 0 0) 0x7f7a2c00c4e0 con 0x7f7a2c009150 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a2c009cf0 con 0x7f7a2c009150 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4045 (unknown 0 0 1917616727) 0x7f7a3c005a20 con 0x7f7a2c009150 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6789/0 conn(0x7f7a2c009150 legacy=0x7f7a2c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a41ffb700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6789/0 conn(0x7f7a2c009150 legacy=0x7f7a2c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:49:22.336 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.337 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.338 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:22.339 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.340 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:22.342 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.343 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.344 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.345 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:49:22.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.347 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:22.347 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7a30036660 legacy=0x7f7a30038b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 10 monclient: shutdown 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 >> v1:172.21.15.76:6789/0 conn(0x7f7a4c1400d0 legacy=0x7f7a4c1d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a518d3700 1 -- 172.21.15.76:0/64326035 reap_dead start 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a518d3700 1 -- 172.21.15.76:0/64326035 reap_dead start 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 shutdown_connections 2024-09-16T06:49:22.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 >> 172.21.15.76:0/64326035 conn(0x7f7a4c0a0030 msgr2=0x7f7a4c09c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 shutdown_connections 2024-09-16T06:49:22.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.327+0000 7f7a520d4700 1 -- 172.21.15.76:0/64326035 wait complete. 2024-09-16T06:49:22.349 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:22.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:49:22.349 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 10 monclient: get_monmap_and_config 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 10 monclient: monmap: 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:22.470424+0000 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:22.470424+0000 2024-09-16T06:49:22.466 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.467 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.467 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.467 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:22.467 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:22.467 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 1 Processor -- start 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 1 -- start start 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.467+0000 7f2ad52f0700 10 monclient: init 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-b con 0x7f2ad0097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-a con 0x7f2ad0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-c con 0x7f2ad0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad0098990 con 0x7f2ad0099090 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad0097f40 con 0x7f2ad0139500 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad009d610 con 0x7f2ad0097b60 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient(hunting): authenticate will time out at 1509.604306s 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2acffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2ad0097b60 0x7f2ad009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36490/0 (socket says 172.21.15.76:36490) 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2acffff700 1 -- 172.21.15.76:0/766628081 learned_addr learned my addr 172.21.15.76:0/766628081 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207867947 0 0) 0x7f2ad0098990 con 0x7f2ad0099090 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): my global_id is 234153 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab80035d0 con 0x7f2ad0099090 2024-09-16T06:49:22.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364795015 0 0) 0x7f2ad009d610 con 0x7f2ad0097b60 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): my global_id is 194813 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad0098990 con 0x7f2ad0097b60 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297122572 0 0) 0x7f2ad0097f40 con 0x7f2ad0139500 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient(hunting): my global_id is 224230 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad009d610 con 0x7f2ad0139500 2024-09-16T06:49:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3209673120 0 0) 0x7f2ab80035d0 con 0x7f2ad0099090 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 >> v1:172.21.15.76:6790/0 conn(0x7f2ad0097b60 legacy=0x7f2ad009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 >> v1:172.21.15.189:6789/0 conn(0x7f2ad0139500 legacy=0x7f2ad013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 monclient: found mon.noname-a 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad00986b0 con 0x7f2ad0099090 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 5 monclient: authenticate success, global_id 234153 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac4001610 con 0x7f2ad0099090 2024-09-16T06:49:22.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: dump: 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac4001830 con 0x7f2ad0099090 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 1 -- 172.21.15.76:0/766628081 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac4001c00 con 0x7f2ad0099090 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 4 set_mon_vals no callback set 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2aceffd700 10 monclient: dump: 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: shutdown 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- 172.21.15.76:0/766628081 >> v1:172.21.15.76:6789/0 conn(0x7f2ad0099090 legacy=0x7f2ad009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- 172.21.15.76:0/766628081 shutdown_connections 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- 172.21.15.76:0/766628081 >> 172.21.15.76:0/766628081 conn(0x7f2ad00a0030 msgr2=0x7f2ad00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- 172.21.15.76:0/766628081 shutdown_connections 2024-09-16T06:49:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 1 -- 172.21.15.76:0/766628081 wait complete. 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.471+0000 7f2ad52f0700 10 monclient: monmap: 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 Processor -- start 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- start start 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: init 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-a con 0x7f2ad0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-c con 0x7f2ad0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient(hunting): picked mon.noname-b con 0x7f2ad0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad006ee90 con 0x7f2ad0097b60 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad006d100 con 0x7f2ad0139500 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad006ccb0 con 0x7f2ad0099090 2024-09-16T06:49:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient(hunting): authenticate will time out at 1509.608953s 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2acf7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2ad0099090 0x7f2ad006c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36500/0 (socket says 172.21.15.76:36500) 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2acf7fe700 1 -- 172.21.15.76:0/2941490860 learned_addr learned my addr 172.21.15.76:0/2941490860 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641991092 0 0) 0x7f2ad006ccb0 con 0x7f2ad0099090 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): my global_id is 194816 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc0035d0 con 0x7f2ad0099090 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465371398 0 0) 0x7f2ad006ee90 con 0x7f2ad0097b60 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): my global_id is 234156 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad006ccb0 con 0x7f2ad0097b60 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2393278393 0 0) 0x7f2abc0035d0 con 0x7f2ad0099090 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad006ee90 con 0x7f2ad0099090 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac4001b80 con 0x7f2ad0099090 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): dump: 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386802895 0 0) 0x7f2ad006d100 con 0x7f2ad0139500 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): my global_id is 224233 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc0035d0 con 0x7f2ad0139500 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936506703 0 0) 0x7f2ad006ccb0 con 0x7f2ad0097b60 2024-09-16T06:49:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad006d100 con 0x7f2ad0097b60 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac00019d0 con 0x7f2ad0097b60 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient(hunting): dump: 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.486 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3716038275 0 0) 0x7f2ad006ee90 con 0x7f2ad0099090 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.189:6789/0 conn(0x7f2ad0139500 legacy=0x7f2ad01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6789/0 conn(0x7f2ad0097b60 legacy=0x7f2ad006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 monclient: found mon.c 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad009a540 con 0x7f2ad0099090 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 5 monclient: authenticate success, global_id 194816 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _renew_subs 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad006ebb0 con 0x7f2ad0099090 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _renew_subs 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad0138e30 con 0x7f2ad0099090 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac40011b0 con 0x7f2ad0099090 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac4003b70 con 0x7f2ad0099090 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ace7fc700 4 set_mon_vals no callback set 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 10 monclient: dump: 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.489 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ab0006590 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.475+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2ac4003d50 con 0x7f2ad0099090 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379379369 0 0) 0x7f2ab0006590 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient(hunting): my global_id is 194819 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad006ee90 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2ac4043ff0 con 0x7f2ad0099090 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1637348222 0 0) 0x7f2ad006ee90 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ab0006590 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac0004a30 con 0x7f2ab0002830 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient: dump: 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2ad4aef700 10 monclient: get_auth_request con 0x7f2abc0364b0 auth_method 0 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3105561322 0 0) 0x7f2ab0006590 con 0x7f2ab0002830 2024-09-16T06:49:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ab000a190 con 0x7f2ab0002830 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2ac0004cd0 con 0x7f2ab0002830 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6790/0 conn(0x7f2ab0002830 legacy=0x7f2ab0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.479+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6790/0 conn(0x7f2ab0002830 legacy=0x7f2ab0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ab0009e50 con 0x7f2ab0009170 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516274120 0 0) 0x7f2ab0009e50 con 0x7f2ab0009170 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient(hunting): my global_id is 194822 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0006590 con 0x7f2ab0009170 2024-09-16T06:49:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1310681318 0 0) 0x7f2ab0006590 con 0x7f2ab0009170 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ab0009e50 con 0x7f2ab0009170 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2ac0005430 con 0x7f2ab0009170 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient: dump: 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1242164115 0 0) 0x7f2ab0009e50 con 0x7f2ab0009170 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2ab000c340 con 0x7f2ab0009170 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4213 (unknown 0 0 3291362479) 0x7f2ac00056d0 con 0x7f2ab0009170 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6790/0 conn(0x7f2ab0009170 legacy=0x7f2ab0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2accff9700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6790/0 conn(0x7f2ab0009170 legacy=0x7f2ab0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:22.496 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.497 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.498 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.499 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:22.500 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.505 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.506 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.507 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:22.508 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:22.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2abc0364b0 legacy=0x7f2abc038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 10 monclient: shutdown 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 >> v1:172.21.15.76:6790/0 conn(0x7f2ad0099090 legacy=0x7f2ad006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad4aef700 1 -- 172.21.15.76:0/2941490860 reap_dead start 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad4aef700 1 -- 172.21.15.76:0/2941490860 reap_dead start 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 shutdown_connections 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 >> 172.21.15.76:0/2941490860 conn(0x7f2ad00a0030 msgr2=0x7f2ad00a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 shutdown_connections 2024-09-16T06:49:22.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.487+0000 7f2ad52f0700 1 -- 172.21.15.76:0/2941490860 wait complete. 2024-09-16T06:49:22.510 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:22.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:49:22.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:49:22.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: get_monmap_and_config 2024-09-16T06:49:22.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: monmap: 2024-09-16T06:49:22.628 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:22.632900+0000 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:22.632900+0000 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:22.629 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 1 Processor -- start 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 1 -- start start 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: init 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-a con 0x7fd4d809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-b con 0x7fd4d8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-c con 0x7fd4d8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d809d560 con 0x7fd4d809ed60 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d809f140 con 0x7fd4d8099030 2024-09-16T06:49:22.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d809d790 con 0x7fd4d8098be0 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4dd149700 10 monclient(hunting): authenticate will time out at 1509.766961s 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd4d8098be0 0x7fd4d81374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36530/0 (socket says 172.21.15.76:36530) 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d77fe700 1 -- 172.21.15.76:0/814406339 learned_addr learned my addr 172.21.15.76:0/814406339 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d7fff700 1 --1- 172.21.15.76:0/814406339 >> v1:172.21.15.76:6789/0 conn(0x7fd4d809ed60 0x7fd4d80984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:36088/0 (socket says 172.21.15.76:36088) 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722838896 0 0) 0x7fd4d809d790 con 0x7fd4d8098be0 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): my global_id is 194825 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c00035d0 con 0x7fd4d8098be0 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140119102 0 0) 0x7fd4d809d560 con 0x7fd4d809ed60 2024-09-16T06:49:22.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): my global_id is 234159 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d809d790 con 0x7fd4d809ed60 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968412671 0 0) 0x7fd4d809f140 con 0x7fd4d8099030 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 10 monclient(hunting): my global_id is 224236 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d809d560 con 0x7fd4d8099030 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1601713113 0 0) 0x7fd4c00035d0 con 0x7fd4d8098be0 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 >> v1:172.21.15.189:6789/0 conn(0x7fd4d8099030 legacy=0x7fd4d813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.631+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 >> v1:172.21.15.76:6789/0 conn(0x7fd4d809ed60 legacy=0x7fd4d80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 1 monclient: found mon.noname-b 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d809f620 con 0x7fd4d8098be0 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 5 monclient: authenticate success, global_id 194825 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4c8001990 con 0x7fd4d8098be0 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: dump: 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c8001bb0 con 0x7fd4d8098be0 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 4 set_mon_vals no callback set 2024-09-16T06:49:22.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 1 -- 172.21.15.76:0/814406339 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4c8001f40 con 0x7fd4d8098be0 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4d6ffd700 10 monclient: dump: 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.637 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 10 monclient: shutdown 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- 172.21.15.76:0/814406339 >> v1:172.21.15.76:6790/0 conn(0x7fd4d8098be0 legacy=0x7fd4d81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- 172.21.15.76:0/814406339 shutdown_connections 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- 172.21.15.76:0/814406339 >> 172.21.15.76:0/814406339 conn(0x7fd4d80a0030 msgr2=0x7fd4d80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- 172.21.15.76:0/814406339 shutdown_connections 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- 172.21.15.76:0/814406339 wait complete. 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 10 monclient: build_initial_monmap 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 10 monclient: monmap: 2024-09-16T06:49:22.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 Processor -- start 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 1 -- start start 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.635+0000 7fd4dd149700 10 monclient: init 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:22.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-b con 0x7fd4d8098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-c con 0x7fd4d8099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient(hunting): picked mon.noname-a con 0x7fd4d809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d809a540 con 0x7fd4d809ed60 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd4d8099030 0x7fd4d81c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36544/0 (socket says 172.21.15.76:36544) 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d77fe700 1 -- 172.21.15.76:0/2909632621 learned_addr learned my addr 172.21.15.76:0/2909632621 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d8090ea0 con 0x7fd4d8098be0 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d8091da0 con 0x7fd4d8099030 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient(hunting): authenticate will time out at 1509.772602s 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732900949 0 0) 0x7fd4d8091da0 con 0x7fd4d8099030 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): my global_id is 194828 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c40035d0 con 0x7fd4d8099030 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596799004 0 0) 0x7fd4d809a540 con 0x7fd4d809ed60 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): my global_id is 234162 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d8091da0 con 0x7fd4d809ed60 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182333160 0 0) 0x7fd4d8090ea0 con 0x7fd4d8098be0 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): my global_id is 224239 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d809a540 con 0x7fd4d8098be0 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283744516 0 0) 0x7fd4c40035d0 con 0x7fd4d8099030 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4d8090ea0 con 0x7fd4d8099030 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4c8001890 con 0x7fd4d8099030 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165713553 0 0) 0x7fd4d8091da0 con 0x7fd4d809ed60 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4c40035d0 con 0x7fd4d809ed60 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4d00018b0 con 0x7fd4d809ed60 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient(hunting): dump: 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.647 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4268826409 0 0) 0x7fd4d8090ea0 con 0x7fd4d8099030 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6789/0 conn(0x7fd4d8098be0 legacy=0x7fd4d80968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.76:6789/0 conn(0x7fd4d809ed60 legacy=0x7fd4d81d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 monclient: found mon.c 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8096fd0 con 0x7fd4d8099030 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 5 monclient: authenticate success, global_id 194828 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _renew_subs 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d80901d0 con 0x7fd4d8099030 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _renew_subs 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d81c75b0 con 0x7fd4d8099030 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c8004ec0 con 0x7fd4d8099030 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4c80051d0 con 0x7fd4d8099030 2024-09-16T06:49:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 10 monclient: dump: 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d5ffb700 4 set_mon_vals no callback set 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd4d80754d0 con 0x7fd4d808aff0 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd4c800fc40 con 0x7fd4d8099030 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.639+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd4c8044040 con 0x7fd4d8099030 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4dc948700 10 monclient: get_auth_request con 0x7fd4c40361b0 auth_method 0 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095815234 0 0) 0x7fd4d80754d0 con 0x7fd4d808aff0 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient(hunting): my global_id is 224242 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d8090ea0 con 0x7fd4d808aff0 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2616430287 0 0) 0x7fd4d8090ea0 con 0x7fd4d808aff0 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4d80754d0 con 0x7fd4d808aff0 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4cc004890 con 0x7fd4d808aff0 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient: dump: 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 358907143 0 0) 0x7fd4d80754d0 con 0x7fd4d808aff0 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4d80702a0 con 0x7fd4d808aff0 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd4cc004c40 con 0x7fd4d808aff0 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6789/0 conn(0x7fd4d808aff0 legacy=0x7fd4d808b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.643+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6789/0 conn(0x7fd4d808aff0 legacy=0x7fd4d808b3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4dd149700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:22.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd4d81c7860 con 0x7fd4d808bae0 2024-09-16T06:49:22.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142310611 0 0) 0x7fd4d81c7860 con 0x7fd4d808bae0 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient(hunting): my global_id is 224245 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d80754d0 con 0x7fd4d808bae0 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 474128470 0 0) 0x7fd4d80754d0 con 0x7fd4d808bae0 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4d81c7860 con 0x7fd4d808bae0 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd4cc005300 con 0x7fd4d808bae0 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 10 monclient: dump: 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:22.655 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 165572012 0 0) 0x7fd4d81c7860 con 0x7fd4d808bae0 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.651+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd4d809aaa0 con 0x7fd4d808bae0 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3879 (unknown 0 0 2445051718) 0x7fd4cc006620 con 0x7fd4d808bae0 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4d4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4d4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:22.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6789/0 conn(0x7fd4d808bae0 legacy=0x7fd4d802c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4d4ff9700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6789/0 conn(0x7fd4d808bae0 legacy=0x7fd4d802c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 98, 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:49:22.657 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.658 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.659 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:22.660 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:22.663 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.665 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:22.667 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd4c40361b0 legacy=0x7fd4c4038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 10 monclient: shutdown 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 >> v1:172.21.15.76:6790/0 conn(0x7fd4d8099030 legacy=0x7fd4d81c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dc948700 1 -- 172.21.15.76:0/2909632621 reap_dead start 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dc948700 1 -- 172.21.15.76:0/2909632621 reap_dead start 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 shutdown_connections 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 >> 172.21.15.76:0/2909632621 conn(0x7fd4d80a0030 msgr2=0x7fd4d80972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 shutdown_connections 2024-09-16T06:49:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:22.655+0000 7fd4dd149700 1 -- 172.21.15.76:0/2909632621 wait complete. 2024-09-16T06:49:22.670 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:49:22.670 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:49:22.670 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T06:49:22.670 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:49:22.670 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:49:22.670 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:49:22.670 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:49:22.718 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:49:22.718 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:49:22.719 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:49:22.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:49:42.720 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:49:42.721 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:49:42.721 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:49:42.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:49:42.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: get_monmap_and_config 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: build_initial_monmap 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: monmap: 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:42.845364+0000 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:42.845364+0000 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:42.841 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:42.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 1 Processor -- start 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 1 -- start start 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: init 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-a con 0x7f75480964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-b con 0x7f754809b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-c con 0x7f7548137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:42.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548098560 con 0x7f75480964f0 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548099ec0 con 0x7f7548137e70 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754809b5f0 con 0x7f754809b1c0 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754e6a6700 10 monclient(hunting): authenticate will time out at 1529.978946s 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754cea3700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f7548136750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754d6a4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 0x7f75480968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:53414/0 (socket says 172.21.15.76:53414) 2024-09-16T06:49:42.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:42.843+0000 7f754d6a4700 1 -- 172.21.15.76:0/2239467325 learned_addr learned my addr 172.21.15.76:0/2239467325 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:43.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:43.047+0000 7f754cea3700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f7548136750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:43.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:43.451+0000 7f754cea3700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f7548136750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:44.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.255+0000 7f754cea3700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f7548136750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:44.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.591+0000 7f754d6a4700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 legacy=0x7f75480968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T06:49:44.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.591+0000 7f754d6a4700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 legacy=0x7f75480968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:49:44.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.591+0000 7f754d6a4700 1 --1- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 0x7f75480968d0 :-1 s=OPENED pgs=1017 cs=1 l=1).handle_message read tag failed 2024-09-16T06:49:44.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.591+0000 7f754d6a4700 1 --1- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 0x7f75480968d0 :-1 s=OPENED pgs=1017 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:49:44.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:44.591+0000 7f753ffff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): tick 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 monclient(hunting): continuing hunt 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f7548136750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.189:6789/0 conn(0x7f7548137e70 legacy=0x7f754813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 legacy=0x7f75480968d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:49:45.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): picked mon.noname-a con 0x7f75480964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): picked mon.noname-c con 0x7f7548137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): picked mon.noname-b con 0x7f754809b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548099be0 con 0x7f75480964f0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754809b5f0 con 0x7f7548137e70 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f754dea5700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f752c00e8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c012080 con 0x7f754809b1c0 2024-09-16T06:49:45.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:45.843+0000 7f753effd700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:46.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:46.047+0000 7f754dea5700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f752c00e8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:46.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:46.451+0000 7f754dea5700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f752c00e8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:47.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.255+0000 7f754dea5700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f752c00e8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:47.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747530950 0 0) 0x7f7548099be0 con 0x7f75480964f0 2024-09-16T06:49:47.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:47.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:47.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:47.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 10 monclient(hunting): my global_id is 234165 2024-09-16T06:49:47.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.759+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75340035d0 con 0x7f75480964f0 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687711536 0 0) 0x7f754809b5f0 con 0x7f7548137e70 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 10 monclient(hunting): my global_id is 224248 2024-09-16T06:49:47.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.771+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548099be0 con 0x7f7548137e70 2024-09-16T06:49:47.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4177842805 0 0) 0x7f7548099be0 con 0x7f7548137e70 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6790/0 conn(0x7f754809b1c0 legacy=0x7f752c00e8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 legacy=0x7f752c007900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 monclient: found mon.noname-c 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c013290 con 0x7f7548137e70 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: _finish_auth 0 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7538004280 con 0x7f7548137e70 2024-09-16T06:49:47.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.775+0000 7f753ffff700 10 monclient: dump: 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 5 monclient: authenticate success, global_id 224248 2024-09-16T06:49:47.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75380044a0 con 0x7f7548137e70 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 1 -- 172.21.15.76:0/2239467325 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7538004830 con 0x7f7548137e70 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 4 set_mon_vals no callback set 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f753ffff700 10 monclient: dump: 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.778 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:47.779 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 10 monclient: shutdown 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 1 -- 172.21.15.76:0/2239467325 >> v1:172.21.15.189:6789/0 conn(0x7f7548137e70 legacy=0x7f752c00b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 1 -- 172.21.15.76:0/2239467325 shutdown_connections 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 1 -- 172.21.15.76:0/2239467325 >> 172.21.15.76:0/2239467325 conn(0x7f75480a0030 msgr2=0x7f75480a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 1 -- 172.21.15.76:0/2239467325 shutdown_connections 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 1 -- 172.21.15.76:0/2239467325 wait complete. 2024-09-16T06:49:47.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 10 monclient: build_initial_monmap 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.779+0000 7f754e6a6700 10 monclient: monmap: 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 Processor -- start 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- start start 2024-09-16T06:49:47.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: init 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-a con 0x7f75480964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-b con 0x7f754809b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient(hunting): picked mon.noname-c con 0x7f7548137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548070cc0 con 0x7f75480964f0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548070900 con 0x7f754809b1c0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754809a540 con 0x7f7548137e70 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient(hunting): authenticate will time out at 1534.917752s 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754dea5700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f7548137e70 legacy=0x7f75481d1780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754d6a4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 0x7f75480954d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33730/0 (socket says 172.21.15.76:33730) 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754d6a4700 1 -- 172.21.15.76:0/730607549 learned_addr learned my addr 172.21.15.76:0/730607549 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:47.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754cea3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f754809b1c0 0x7f75481ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:46934/0 (socket says 172.21.15.76:46934) 2024-09-16T06:49:47.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025390983 0 0) 0x7f7548070900 con 0x7f754809b1c0 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): my global_id is 224251 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0083d0 con 0x7f754809b1c0 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674572840 0 0) 0x7f7548070cc0 con 0x7f75480964f0 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): my global_id is 234168 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548070900 con 0x7f75480964f0 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1432326983 0 0) 0x7f752c0083d0 con 0x7f754809b1c0 2024-09-16T06:49:47.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7548070cc0 con 0x7f754809b1c0 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7538004610 con 0x7f754809b1c0 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): dump: 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:47.788 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1066492420 0 0) 0x7f7548070900 con 0x7f75480964f0 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c0083d0 con 0x7f75480964f0 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7544001860 con 0x7f75480964f0 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient(hunting): dump: 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2551439779 0 0) 0x7f7548070cc0 con 0x7f754809b1c0 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 >> v1:172.21.15.76:6790/0 conn(0x7f7548137e70 legacy=0x7f75481d1780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 >> v1:172.21.15.76:6789/0 conn(0x7f75480964f0 legacy=0x7f75480954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:49:47.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 monclient: found mon.b 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548091860 con 0x7f754809b1c0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: _finish_auth 0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 5 monclient: authenticate success, global_id 224251 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _renew_subs 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7548095f20 con 0x7f754809b1c0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _renew_subs 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:49:47.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7548092d50 con 0x7f754809b1c0 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7538004a20 con 0x7f754809b1c0 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7538004d30 con 0x7f754809b1c0 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753f7fe700 4 set_mon_vals no callback set 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.783+0000 7f753dffb700 10 monclient: dump: 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:47.792 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f753800f820 con 0x7f754809b1c0 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f754e6a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f754e6a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f754e6a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754808c7b0 con 0x7f754809b1c0 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7538043e10 con 0x7f754809b1c0 2024-09-16T06:49:47.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.787+0000 7f754d6a4700 10 monclient: get_auth_request con 0x7f752c036670 auth_method 0 2024-09-16T06:49:47.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.795+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7538014810 con 0x7f754809b1c0 2024-09-16T06:49:47.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.795+0000 7f753dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:47.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:47.795+0000 7f753dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:48.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.031+0000 7f754e6a6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:49:48.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.031+0000 7f754e6a6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:48.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.031+0000 7f754e6a6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:49:48.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.031+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f75480672a0 con 0x7f754809b1c0 2024-09-16T06:49:48.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f753dffb700 1 -- 172.21.15.76:0/730607549 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3835611585) 0x7f7538012c80 con 0x7f754809b1c0 2024-09-16T06:49:48.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f753dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:49:48.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f753dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:48.032 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":100,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 >> v1:172.21.15.189:6812/3098810674 conn(0x7f752c036670 legacy=0x7f752c038b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 10 monclient: shutdown 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 >> v1:172.21.15.189:6789/0 conn(0x7f754809b1c0 legacy=0x7f75481ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 shutdown_connections 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 >> 172.21.15.76:0/730607549 conn(0x7f75480a0030 msgr2=0x7f7548071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 shutdown_connections 2024-09-16T06:49:48.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.035+0000 7f754e6a6700 1 -- 172.21.15.76:0/730607549 wait complete. 2024-09-16T06:49:48.047 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:49:48.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:49:48.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: get_monmap_and_config 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: build_initial_monmap 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: monmap: 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:48.161655+0000 2024-09-16T06:49:48.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:48.161655+0000 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:48.158 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 1 Processor -- start 2024-09-16T06:49:48.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 1 -- start start 2024-09-16T06:49:48.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: init 2024-09-16T06:49:48.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:48.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-b con 0x7fd2b809ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-a con 0x7fd2b8098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-c con 0x7fd2b8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b809d560 con 0x7fd2b8098be0 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b809f140 con 0x7fd2b8099030 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b809d790 con 0x7fd2b809ed60 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient(hunting): authenticate will time out at 1535.294841s 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b7fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fd2b809ed60 legacy=0x7fd2b80984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2affff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2b8098be0 0x7fd2b81374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33740/0 (socket says 172.21.15.76:33740) 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2affff700 1 -- 172.21.15.76:0/3352352314 learned_addr learned my addr 172.21.15.76:0/3352352314 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:48.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029216573 0 0) 0x7fd2b809d560 con 0x7fd2b8098be0 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): my global_id is 234171 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a40035d0 con 0x7fd2b8098be0 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932713384 0 0) 0x7fd2b809f140 con 0x7fd2b8099030 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient(hunting): my global_id is 224254 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b809d560 con 0x7fd2b8099030 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3922205425 0 0) 0x7fd2a40035d0 con 0x7fd2b8098be0 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 >> v1:172.21.15.76:6790/0 conn(0x7fd2b809ed60 legacy=0x7fd2b80984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 >> v1:172.21.15.189:6789/0 conn(0x7fd2b8099030 legacy=0x7fd2b813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 monclient: found mon.noname-a 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b809f620 con 0x7fd2b8098be0 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: _finish_auth 0 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a0001610 con 0x7fd2b8098be0 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 5 monclient: authenticate success, global_id 234171 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: dump: 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.165 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2a00018f0 con 0x7fd2b8098be0 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 1 -- 172.21.15.76:0/3352352314 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a0001c80 con 0x7fd2b8098be0 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:48.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2bd138700 4 set_mon_vals no callback set 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.159+0000 7fd2b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2b77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2b77fe700 10 monclient: dump: 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.167 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 10 monclient: shutdown 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- 172.21.15.76:0/3352352314 >> v1:172.21.15.76:6789/0 conn(0x7fd2b8098be0 legacy=0x7fd2b81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- 172.21.15.76:0/3352352314 shutdown_connections 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- 172.21.15.76:0/3352352314 >> 172.21.15.76:0/3352352314 conn(0x7fd2b80a0030 msgr2=0x7fd2b80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- 172.21.15.76:0/3352352314 shutdown_connections 2024-09-16T06:49:48.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- 172.21.15.76:0/3352352314 wait complete. 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 10 monclient: build_initial_monmap 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 10 monclient: monmap: 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:48.173 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 Processor -- start 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 1 -- start start 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.163+0000 7fd2bd138700 10 monclient: init 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-a con 0x7fd2b8098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-c con 0x7fd2b8099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:48.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient(hunting): picked mon.noname-b con 0x7fd2b809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b8090bd0 con 0x7fd2b8098be0 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b80a3070 con 0x7fd2b809ed60 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b8091100 con 0x7fd2b8099030 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2affff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fd2b8099030 legacy=0x7fd2b81ceef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient(hunting): authenticate will time out at 1535.300553s 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2b8098be0 0x7fd2b8095940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33742/0 (socket says 172.21.15.76:33742) 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b7fff700 1 -- 172.21.15.76:0/1932595618 learned_addr learned my addr 172.21.15.76:0/1932595618 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771193789 0 0) 0x7fd2b8090bd0 con 0x7fd2b8098be0 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): my global_id is 234174 2024-09-16T06:49:48.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2980035d0 con 0x7fd2b8098be0 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22106986 0 0) 0x7fd2b80a3070 con 0x7fd2b809ed60 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): my global_id is 224257 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b8090bd0 con 0x7fd2b809ed60 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763765370 0 0) 0x7fd2980035d0 con 0x7fd2b8098be0 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b80a3070 con 0x7fd2b8098be0 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a8004200 con 0x7fd2b8098be0 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:48.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): dump: 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722812923 0 0) 0x7fd2b8090bd0 con 0x7fd2b809ed60 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2980035d0 con 0x7fd2b809ed60 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1598095388 0 0) 0x7fd2b80a3070 con 0x7fd2b8098be0 2024-09-16T06:49:48.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6790/0 conn(0x7fd2b8099030 legacy=0x7fd2b81ceef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.189:6789/0 conn(0x7fd2b809ed60 legacy=0x7fd2b81d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 monclient: found mon.a 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8071d30 con 0x7fd2b8098be0 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: _finish_auth 0 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 5 monclient: authenticate success, global_id 234174 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _renew_subs 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b80a1500 con 0x7fd2b8098be0 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _renew_subs 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b80913e0 con 0x7fd2b8098be0 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2a8004470 con 0x7fd2b8098be0 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b6ffd700 4 set_mon_vals no callback set 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a8004780 con 0x7fd2b8098be0 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient: dump: 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd29c006590 con 0x7fd29c002830 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd2a8004960 con 0x7fd2b8098be0 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd2a8043ec0 con 0x7fd2b8098be0 2024-09-16T06:49:48.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869567337 0 0) 0x7fd29c006590 con 0x7fd29c002830 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 10 monclient(hunting): my global_id is 234177 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b80a3070 con 0x7fd29c002830 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.167+0000 7fd2bc937700 10 monclient: get_auth_request con 0x7fd298036250 auth_method 0 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 793926628 0 0) 0x7fd2b80a3070 con 0x7fd29c002830 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd29c006590 con 0x7fd29c002830 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a0001620 con 0x7fd29c002830 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 10 monclient: dump: 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4042829013 0 0) 0x7fd29c006590 con 0x7fd29c002830 2024-09-16T06:49:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd29c00a190 con 0x7fd29c002830 2024-09-16T06:49:48.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd2a0001c10 con 0x7fd29c002830 2024-09-16T06:49:48.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:48.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:48.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6789/0 conn(0x7fd29c002830 legacy=0x7fd29c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.171+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6789/0 conn(0x7fd29c002830 legacy=0x7fd29c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd29c009e30 con 0x7fd29c009150 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416687301 0 0) 0x7fd29c009e30 con 0x7fd29c009150 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient(hunting): my global_id is 234180 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29c006590 con 0x7fd29c009150 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1339175535 0 0) 0x7fd29c006590 con 0x7fd29c009150 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd29c009e30 con 0x7fd29c009150 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd2a0004d60 con 0x7fd29c009150 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient: dump: 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 445845511 0 0) 0x7fd29c009e30 con 0x7fd29c009150 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd29c00c340 con 0x7fd29c009150 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4035 (unknown 0 0 3585371355) 0x7fd2a00050a0 con 0x7fd29c009150 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6789/0 conn(0x7fd29c009150 legacy=0x7fd29c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2b57fa700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6789/0 conn(0x7fd29c009150 legacy=0x7fd29c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 100, 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:49:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:48.198 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:48.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd298036250 legacy=0x7fd298038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 10 monclient: shutdown 2024-09-16T06:49:48.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 >> v1:172.21.15.76:6789/0 conn(0x7fd2b8098be0 legacy=0x7fd2b8095940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bc937700 1 -- 172.21.15.76:0/1932595618 reap_dead start 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bc937700 1 -- 172.21.15.76:0/1932595618 reap_dead start 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 shutdown_connections 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 >> 172.21.15.76:0/1932595618 conn(0x7fd2b80a0030 msgr2=0x7fd2b81bd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 shutdown_connections 2024-09-16T06:49:48.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.183+0000 7fd2bd138700 1 -- 172.21.15.76:0/1932595618 wait complete. 2024-09-16T06:49:48.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:49:48.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:49:48.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:49:48.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: get_monmap_and_config 2024-09-16T06:49:48.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: build_initial_monmap 2024-09-16T06:49:48.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: monmap: 2024-09-16T06:49:48.321 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:48.321 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:49:48.325971+0000 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:49:48.325971+0000 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:49:48.322 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 1 Processor -- start 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 1 -- start start 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: init 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-b con 0x7fed3009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-a con 0x7fed3009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-c con 0x7fed30139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3009cd80 con 0x7fed3009fd20 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed30099fd0 con 0x7fed30139b70 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3009a410 con 0x7fed3009f230 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed3830a700 10 monclient(hunting): authenticate will time out at 1535.459434s 2024-09-16T06:49:48.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed360a6700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fed3009f230 legacy=0x7fed3009f610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed358a5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed3009fd20 0x7fed30138450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33774/0 (socket says 172.21.15.76:33774) 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.323+0000 7fed358a5700 1 -- 172.21.15.76:0/412274303 learned_addr learned my addr 172.21.15.76:0/412274303 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368376036 0 0) 0x7fed3009cd80 con 0x7fed3009fd20 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): my global_id is 234183 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c0035d0 con 0x7fed3009fd20 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597485288 0 0) 0x7fed30099fd0 con 0x7fed30139b70 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient(hunting): my global_id is 224260 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3009cd80 con 0x7fed30139b70 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 698550333 0 0) 0x7fed1c0035d0 con 0x7fed3009fd20 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 >> v1:172.21.15.76:6790/0 conn(0x7fed3009f230 legacy=0x7fed3009f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 >> v1:172.21.15.189:6789/0 conn(0x7fed30139b70 legacy=0x7fed3013bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: _finish_hunting 0 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 monclient: found mon.noname-a 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3009de90 con 0x7fed3009fd20 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: _finish_auth 0 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: _check_auth_tickets 2024-09-16T06:49:48.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 5 monclient: authenticate success, global_id 234183 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed20001610 con 0x7fed3009fd20 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: dump: 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.329 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed20001880 con 0x7fed3009fd20 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: get_monmap_and_config success 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 1 -- 172.21.15.76:0/412274303 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed20001c10 con 0x7fed3009fd20 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 4 set_mon_vals no callback set 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed350a4700 10 monclient: dump: 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: shutdown 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- 172.21.15.76:0/412274303 >> v1:172.21.15.76:6789/0 conn(0x7fed3009fd20 legacy=0x7fed30138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- 172.21.15.76:0/412274303 shutdown_connections 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- 172.21.15.76:0/412274303 >> 172.21.15.76:0/412274303 conn(0x7fed300a3e50 msgr2=0x7fed300a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- 172.21.15.76:0/412274303 shutdown_connections 2024-09-16T06:49:48.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- 172.21.15.76:0/412274303 wait complete. 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: build_initial_monmap 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: monmap: 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:49:48.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 Processor -- start 2024-09-16T06:49:48.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 1 -- start start 2024-09-16T06:49:48.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: init 2024-09-16T06:49:48.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: _reopen_session rank -1 2024-09-16T06:49:48.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.327+0000 7fed3830a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-c con 0x7fed3009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-a con 0x7fed3009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient(hunting): picked mon.noname-b con 0x7fed30139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3009c6f0 con 0x7fed3009fd20 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed360a6700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fed3009f230 legacy=0x7fed301c9a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3013d3e0 con 0x7fed30139b70 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed301ca870 con 0x7fed3009f230 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient(hunting): _renew_subs 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient(hunting): authenticate will time out at 1535.464179s 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed358a5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed3009fd20 0x7fed301ca160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33788/0 (socket says 172.21.15.76:33788) 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed358a5700 1 -- 172.21.15.76:0/1708199620 learned_addr learned my addr 172.21.15.76:0/1708199620 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707421962 0 0) 0x7fed3009c6f0 con 0x7fed3009fd20 2024-09-16T06:49:48.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): my global_id is 234186 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed140035d0 con 0x7fed3009fd20 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206927340 0 0) 0x7fed3013d3e0 con 0x7fed30139b70 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): my global_id is 224263 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3009c6f0 con 0x7fed30139b70 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560641227 0 0) 0x7fed140035d0 con 0x7fed3009fd20 2024-09-16T06:49:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed3013d3e0 con 0x7fed3009fd20 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed20001610 con 0x7fed3009fd20 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): dump: 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.340 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2523275649 0 0) 0x7fed3013d3e0 con 0x7fed3009fd20 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.76:6790/0 conn(0x7fed3009f230 legacy=0x7fed301c9a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6789/0 conn(0x7fed30139b70 legacy=0x7fed301d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 monclient: found mon.a 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30072450 con 0x7fed3009fd20 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: _finish_auth 0 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:49:48.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 5 monclient: authenticate success, global_id 234186 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient: _renew_subs 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3006f2e0 con 0x7fed3009fd20 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient: _renew_subs 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed300704f0 con 0x7fed3009fd20 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed20004be0 con 0x7fed3009fd20 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed348a3700 4 set_mon_vals no callback set 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed20004f70 con 0x7fed3009fd20 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient: dump: 2024-09-16T06:49:48.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed18006590 con 0x7fed18002830 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fed200050d0 con 0x7fed3009fd20 2024-09-16T06:49:48.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fed20043fd0 con 0x7fed3009fd20 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40859837 0 0) 0x7fed18006590 con 0x7fed18002830 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 10 monclient(hunting): my global_id is 224266 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3013d3e0 con 0x7fed18002830 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.331+0000 7fed368a7700 10 monclient: get_auth_request con 0x7fed14035fa0 auth_method 0 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3058811319 0 0) 0x7fed3013d3e0 con 0x7fed18002830 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed18006590 con 0x7fed18002830 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed2c003f90 con 0x7fed18002830 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:48.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 10 monclient: dump: 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2998973348 0 0) 0x7fed18006590 con 0x7fed18002830 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed1800a190 con 0x7fed18002830 2024-09-16T06:49:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fed2c004250 con 0x7fed18002830 2024-09-16T06:49:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:49:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:49:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6789/0 conn(0x7fed18002830 legacy=0x7fed18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.335+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6789/0 conn(0x7fed18002830 legacy=0x7fed18004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:48.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed3830a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:49:48.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed18009e50 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316566448 0 0) 0x7fed18009e50 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 10 monclient(hunting): my global_id is 224269 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.343+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed18006590 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3605754905 0 0) 0x7fed18006590 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed18009e50 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed2c0049b0 con 0x7fed18009170 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 10 monclient: dump: 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:49:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2406029025 0 0) 0x7fed18009e50 con 0x7fed18009170 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed1800c340 con 0x7fed18009170 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 4095567334) 0x7fed2c006d00 con 0x7fed18009170 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:49:48.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6789/0 conn(0x7fed18009170 legacy=0x7fed18009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed26ffd700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6789/0 conn(0x7fed18009170 legacy=0x7fed18009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 100, 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:49:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:49:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:49:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:49:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:49:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.189:6812/3098810674 conn(0x7fed14035fa0 legacy=0x7fed14038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 10 monclient: shutdown 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 >> v1:172.21.15.76:6789/0 conn(0x7fed3009fd20 legacy=0x7fed301ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed368a7700 1 -- 172.21.15.76:0/1708199620 reap_dead start 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed368a7700 1 -- 172.21.15.76:0/1708199620 reap_dead start 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 shutdown_connections 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 >> 172.21.15.76:0/1708199620 conn(0x7fed300a3e50 msgr2=0x7fed30138840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 shutdown_connections 2024-09-16T06:49:48.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:49:48.347+0000 7fed3830a700 1 -- 172.21.15.76:0/1708199620 wait complete. 2024-09-16T06:49:48.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:49:48.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:50:08.366 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:50:08.366 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:50:08.366 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:50:08.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:50:08.370 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:50:08.370 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:50:08.370 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:50:08.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:50:28.372 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:50:28.372 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:50:28.372 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:50:28.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: get_monmap_and_config 2024-09-16T06:50:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: build_initial_monmap 2024-09-16T06:50:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: monmap: 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:28.494163+0000 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:28.494163+0000 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:28.490 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 1 Processor -- start 2024-09-16T06:50:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 1 -- start start 2024-09-16T06:50:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: init 2024-09-16T06:50:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-a con 0x7f136c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-c con 0x7f136c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-b con 0x7f136c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c098990 con 0x7f136c097b60 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c097f40 con 0x7f136c099090 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c09d610 con 0x7f136c139500 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f13708e3700 10 monclient(hunting): authenticate will time out at 1575.627579s 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f136b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f136c097b60 0x7f136c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:32976/0 (socket says 172.21.15.76:32976) 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.490+0000 7f136b7fe700 1 -- 172.21.15.76:0/3534857674 learned_addr learned my addr 172.21.15.76:0/3534857674 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019883260 0 0) 0x7f136c09d610 con 0x7f136c139500 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient(hunting): my global_id is 244100 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13580035d0 con 0x7f136c139500 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2371073598 0 0) 0x7f13580035d0 con 0x7f136c139500 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 >> v1:172.21.15.189:6789/0 conn(0x7f136c099090 legacy=0x7f136c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 >> v1:172.21.15.76:6789/0 conn(0x7f136c097b60 legacy=0x7f136c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: _finish_hunting 0 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 monclient: found mon.noname-b 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0986b0 con 0x7f136c139500 2024-09-16T06:50:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: _finish_auth 0 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: _check_auth_tickets 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1364001990 con 0x7f136c139500 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 5 monclient: authenticate success, global_id 244100 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: dump: 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.496 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1364001c40 con 0x7f136c139500 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 4 set_mon_vals no callback set 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 1 -- 172.21.15.76:0/3534857674 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1364002010 con 0x7f136c139500 2024-09-16T06:50:28.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136affd700 10 monclient: dump: 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: shutdown 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- 172.21.15.76:0/3534857674 >> v1:172.21.15.76:6790/0 conn(0x7f136c139500 legacy=0x7f136c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- 172.21.15.76:0/3534857674 shutdown_connections 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- 172.21.15.76:0/3534857674 >> 172.21.15.76:0/3534857674 conn(0x7f136c0a0030 msgr2=0x7f136c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- 172.21.15.76:0/3534857674 shutdown_connections 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- 172.21.15.76:0/3534857674 wait complete. 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: build_initial_monmap 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: monmap: 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:28.499 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 Processor -- start 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- start start 2024-09-16T06:50:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: init 2024-09-16T06:50:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-c con 0x7f136c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-a con 0x7f136c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient(hunting): picked mon.noname-b con 0x7f136c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c09a540 con 0x7f136c099090 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c09b8d0 con 0x7f136c139500 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c09bbb0 con 0x7f136c097b60 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f13708e3700 10 monclient(hunting): authenticate will time out at 1575.631476s 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f136c097b60 0x7f136c0702a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51454/0 (socket says 172.21.15.76:51454) 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f136b7fe700 1 -- 172.21.15.76:0/1354370077 learned_addr learned my addr 172.21.15.76:0/1354370077 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021122581 0 0) 0x7f136c09bbb0 con 0x7f136c097b60 2024-09-16T06:50:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 10 monclient(hunting): my global_id is 244103 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.494+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c0035d0 con 0x7f136c097b60 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 150505301 0 0) 0x7f134c0035d0 con 0x7f136c097b60 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f136c09bbb0 con 0x7f136c097b60 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f135c0017a0 con 0x7f136c097b60 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient(hunting): dump: 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2578857474 0 0) 0x7f136c09bbb0 con 0x7f136c097b60 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 >> v1:172.21.15.189:6789/0 conn(0x7f136c139500 legacy=0x7f136c1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 >> v1:172.21.15.76:6789/0 conn(0x7f136c099090 legacy=0x7f136c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 monclient: found mon.c 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0710c0 con 0x7f136c097b60 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: _finish_auth 0 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 5 monclient: authenticate success, global_id 244103 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _renew_subs 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c09a970 con 0x7f136c097b60 2024-09-16T06:50:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _renew_subs 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c138f40 con 0x7f136c097b60 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f135c004890 con 0x7f136c097b60 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f135c004c20 con 0x7f136c097b60 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f136a7fc700 4 set_mon_vals no callback set 2024-09-16T06:50:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 10 monclient: dump: 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f135c004e00 con 0x7f136c097b60 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:28.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1350005100 con 0x7f136c097b60 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f135c043f00 con 0x7f136c097b60 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.498+0000 7f1363fff700 10 monclient: get_auth_request con 0x7f134c036070 auth_method 0 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.506+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f135c0148a0 con 0x7f136c097b60 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.506+0000 7f1368ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:28.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.506+0000 7f1368ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:28.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f13708e3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:28.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f13708e3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:28.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f13708e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:28.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1350005930 con 0x7f136c097b60 2024-09-16T06:50:28.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f1368ff9700 1 -- 172.21.15.76:0/1354370077 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 172355316) 0x7f135c012d10 con 0x7f136c097b60 2024-09-16T06:50:28.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f1368ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:28.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.742+0000 7f1368ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:28.741 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":102,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:28.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 >> v1:172.21.15.189:6812/3098810674 conn(0x7f134c036070 legacy=0x7f134c038530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 10 monclient: shutdown 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 >> v1:172.21.15.76:6790/0 conn(0x7f136c097b60 legacy=0x7f136c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 shutdown_connections 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 >> 172.21.15.76:0/1354370077 conn(0x7f136c0a0030 msgr2=0x7f136c132520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 shutdown_connections 2024-09-16T06:50:28.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:28.746+0000 7f13708e3700 1 -- 172.21.15.76:0/1354370077 wait complete. 2024-09-16T06:50:31.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:31.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:50:31.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: build_initial_monmap 2024-09-16T06:50:31.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: monmap: 2024-09-16T06:50:31.875 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:31.880417+0000 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:31.880417+0000 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:31.876 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 1 Processor -- start 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 1 -- start start 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: init 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fba840207b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fba840229f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fba84021e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba84021bf0 con 0x7fba840207b0 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8401c340 con 0x7fba84021e70 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8401d500 con 0x7fba840229f0 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient(hunting): authenticate will time out at 1579.014269s 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8a7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fba840229f0 0x7fba840bcde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51472/0 (socket says 172.21.15.76:51472) 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8a7fc700 1 -- 172.21.15.76:0/1600110320 learned_addr learned my addr 172.21.15.76:0/1600110320 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994337913 0 0) 0x7fba8401d500 con 0x7fba840229f0 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient(hunting): my global_id is 244106 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba740034f0 con 0x7fba840229f0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 29859831 0 0) 0x7fba740034f0 con 0x7fba840229f0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 >> v1:172.21.15.189:6789/0 conn(0x7fba84021e70 legacy=0x7fba840c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 >> v1:172.21.15.76:6789/0 conn(0x7fba840207b0 legacy=0x7fba84020b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 monclient: found mon.noname-b 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8401d920 con 0x7fba840229f0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: _finish_auth 0 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:50:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba80001610 con 0x7fba840229f0 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 5 monclient: authenticate success, global_id 244106 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: dump: 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba80001900 con 0x7fba840229f0 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 1 -- 172.21.15.76:0/1600110320 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba80001d10 con 0x7fba840229f0 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba8bfff700 4 set_mon_vals no callback set 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.878+0000 7fba89ffb700 10 monclient: dump: 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: shutdown 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1600110320 >> v1:172.21.15.76:6790/0 conn(0x7fba840229f0 legacy=0x7fba840bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1600110320 shutdown_connections 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1600110320 >> 172.21.15.76:0/1600110320 conn(0x7fba84005b40 msgr2=0x7fba84005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1600110320 shutdown_connections 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1600110320 wait complete. 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: build_initial_monmap 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: monmap: 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 Processor -- start 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- start start 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: init 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fba840207b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:31.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fba84021e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fba840229f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8401f7b0 con 0x7fba840207b0 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840204a0 con 0x7fba840229f0 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840a62d0 con 0x7fba84021e70 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient(hunting): authenticate will time out at 1579.018046s 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 0x7fba84155130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:51484/0 (socket says 172.21.15.76:51484) 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 -- 172.21.15.76:0/1842957524 learned_addr learned my addr 172.21.15.76:0/1842957524 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8affd700 1 --1- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6789/0 conn(0x7fba840207b0 0x7fba84150540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:32988/0 (socket says 172.21.15.76:32988) 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504396581 0 0) 0x7fba840a62d0 con 0x7fba84021e70 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:31.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): my global_id is 244109 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba780035d0 con 0x7fba84021e70 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 240617874 0 0) 0x7fba780035d0 con 0x7fba84021e70 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba840a62d0 con 0x7fba84021e70 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba80001430 con 0x7fba84021e70 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): dump: 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.897 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1949871439 0 0) 0x7fba840a62d0 con 0x7fba84021e70 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.189:6789/0 conn(0x7fba840229f0 legacy=0x7fba84158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6789/0 conn(0x7fba840207b0 legacy=0x7fba84150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 monclient: found mon.c 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84028cc0 con 0x7fba84021e70 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient: _finish_auth 0 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba73fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 5 monclient: authenticate success, global_id 244109 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _renew_subs 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba840a83b0 con 0x7fba84021e70 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _renew_subs 2024-09-16T06:50:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 legacy=0x7fba84155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 legacy=0x7fba84155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 --1- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 0x7fba84155130 :-1 s=OPENED pgs=1344 cs=1 l=1).handle_message read tag failed 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8a7fc700 1 --1- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 0x7fba84155130 :-1 s=OPENED pgs=1344 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.882+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba840c1300 con 0x7fba84021e70 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6790/0 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba84021e70 legacy=0x7fba84155130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): picked mon.c con 0x7fba78004870 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): picked mon.b con 0x7fba840229f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): picked mon.a con 0x7fba840207b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fba840a62d0 con 0x7fba840207b0 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fba780035d0 con 0x7fba840229f0 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fba78011f90 con 0x7fba78004870 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681915082 0 0) 0x7fba78011f90 con 0x7fba78004870 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fba78012120 con 0x7fba78004870 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba8bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba8bfff700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2881134202 0 0) 0x7fba78012120 con 0x7fba78004870 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.189:6789/0 conn(0x7fba840229f0 legacy=0x7fba7800aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6789/0 conn(0x7fba840207b0 legacy=0x7fba7800e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 monclient: found mon.c 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fba780132d0 con 0x7fba78004870 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba840c1300 con 0x7fba78004870 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba78011f90 con 0x7fba78004870 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _finish_auth 0 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fba7c001550 con 0x7fba78004870 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: dump: 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba7c0017f0 con 0x7fba78004870 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba897fa700 4 set_mon_vals no callback set 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fba7c0019e0 con 0x7fba78004870 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 6 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fba7c043290 con 0x7fba78004870 2024-09-16T06:50:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.886+0000 7fba8a7fc700 10 monclient: get_auth_request con 0x7fba84021e70 auth_method 0 2024-09-16T06:50:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.894+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fba7c013d10 con 0x7fba78004870 2024-09-16T06:50:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.894+0000 7fba73fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.894+0000 7fba73fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.898+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fba7c012180 con 0x7fba78004870 2024-09-16T06:50:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:31.898+0000 7fba73fff700 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T06:50:32.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba8bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:32.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba8bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:32.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba8bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:32.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fba84150d10 con 0x7fba78004870 2024-09-16T06:50:32.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba73fff700 1 -- 172.21.15.76:0/1842957524 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 932129575) 0x7fba7c013900 con 0x7fba78004870 2024-09-16T06:50:32.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba73fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:32.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.126+0000 7fba73fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:32.127 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":102,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.189:6812/3098810674 conn(0x7fba84021e70 legacy=0x7fba78044170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 10 monclient: shutdown 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 >> v1:172.21.15.76:6790/0 conn(0x7fba78004870 legacy=0x7fba78004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 shutdown_connections 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 >> 172.21.15.76:0/1842957524 conn(0x7fba84005b40 msgr2=0x7fba8401dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 shutdown_connections 2024-09-16T06:50:32.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:32.130+0000 7fba8bfff700 1 -- 172.21.15.76:0/1842957524 wait complete. 2024-09-16T06:50:35.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:35.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.265+0000 7fa91b3b2700 10 monclient: get_monmap_and_config 2024-09-16T06:50:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.265+0000 7fa91b3b2700 10 monclient: build_initial_monmap 2024-09-16T06:50:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.265+0000 7fa91b3b2700 10 monclient: monmap: 2024-09-16T06:50:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:35.271597+0000 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:35.271597+0000 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:35.269 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 Processor -- start 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- start start 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient: init 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:35.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-b con 0x7fa914132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-a con 0x7fa91406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-c con 0x7fa91406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91406c280 con 0x7fa91406d480 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91406bfa0 con 0x7fa91406cf50 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91409ace0 con 0x7fa914132330 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient(hunting): authenticate will time out at 1582.405267s 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91a3b0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa914132330 0x7fa914132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41394/0 (socket says 172.21.15.76:41394) 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91a3b0700 1 -- 172.21.15.76:0/4212051273 learned_addr learned my addr 172.21.15.76:0/4212051273 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534689013 0 0) 0x7fa91406c280 con 0x7fa91406d480 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _negotiate 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:35.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): my global_id is 234189 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9000035d0 con 0x7fa91406d480 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438537771 0 0) 0x7fa91406bfa0 con 0x7fa91406cf50 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _negotiate 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient(hunting): my global_id is 224272 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa91406c280 con 0x7fa91406cf50 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498567758 0 0) 0x7fa9000035d0 con 0x7fa91406d480 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 >> v1:172.21.15.76:6790/0 conn(0x7fa914132330 legacy=0x7fa914132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 >> v1:172.21.15.189:6789/0 conn(0x7fa91406cf50 legacy=0x7fa91413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: _finish_hunting 0 2024-09-16T06:50:35.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 monclient: found mon.noname-a 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91409ae70 con 0x7fa91406d480 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: _finish_auth 0 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: _check_auth_tickets 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 5 monclient: authenticate success, global_id 234189 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa904001610 con 0x7fa91406d480 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:35.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: dump: 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9040018f0 con 0x7fa91406d480 2024-09-16T06:50:35.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 4 set_mon_vals no callback set 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 1 -- 172.21.15.76:0/4212051273 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa904001ae0 con 0x7fa91406d480 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa9193ae700 10 monclient: dump: 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:35.282 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 10 monclient: shutdown 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- 172.21.15.76:0/4212051273 >> v1:172.21.15.76:6789/0 conn(0x7fa91406d480 legacy=0x7fa91413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- 172.21.15.76:0/4212051273 shutdown_connections 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- 172.21.15.76:0/4212051273 >> 172.21.15.76:0/4212051273 conn(0x7fa9140a41c0 msgr2=0x7fa9140a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- 172.21.15.76:0/4212051273 shutdown_connections 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.269+0000 7fa91b3b2700 1 -- 172.21.15.76:0/4212051273 wait complete. 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: build_initial_monmap 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: monmap: 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:35.283 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 Processor -- start 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- start start 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: init 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:35.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-a con 0x7fa91406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-c con 0x7fa91406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient(hunting): picked mon.noname-b con 0x7fa914132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa914139020 con 0x7fa91406cf50 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91413df90 con 0x7fa914132330 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140a6bb0 con 0x7fa91406d480 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient(hunting): authenticate will time out at 1582.409457s 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa919baf700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa91406d480 0x7fa914135b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41404/0 (socket says 172.21.15.76:41404) 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa919baf700 1 -- 172.21.15.76:0/2035659038 learned_addr learned my addr 172.21.15.76:0/2035659038 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064037507 0 0) 0x7fa914139020 con 0x7fa91406cf50 2024-09-16T06:50:35.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): my global_id is 234192 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f80035d0 con 0x7fa91406cf50 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141115479 0 0) 0x7fa91413df90 con 0x7fa914132330 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): my global_id is 224275 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914139020 con 0x7fa914132330 2024-09-16T06:50:35.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226545844 0 0) 0x7fa8f80035d0 con 0x7fa91406cf50 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa91413df90 con 0x7fa91406cf50 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9100017e0 con 0x7fa91406cf50 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): dump: 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:35.287 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769146235 0 0) 0x7fa91413df90 con 0x7fa91406cf50 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 >> v1:172.21.15.76:6790/0 conn(0x7fa91406d480 legacy=0x7fa914135b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 >> v1:172.21.15.189:6789/0 conn(0x7fa914132330 legacy=0x7fa9141d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: _finish_hunting 0 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 monclient: found mon.a 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9141362a0 con 0x7fa91406cf50 2024-09-16T06:50:35.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: _finish_auth 0 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: _check_auth_tickets 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 5 monclient: authenticate success, global_id 234192 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _renew_subs 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9140a7100 con 0x7fa91406cf50 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _renew_subs 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914136f30 con 0x7fa91406cf50 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa910001d50 con 0x7fa91406cf50 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa910004ab0 con 0x7fa91406cf50 2024-09-16T06:50:35.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa918bad700 4 set_mon_vals no callback set 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.273+0000 7fa90affd700 10 monclient: dump: 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:35.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa910004c10 con 0x7fa91406cf50 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa91b3b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa91b3b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa91b3b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc005100 con 0x7fa91406cf50 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa910043de0 con 0x7fa91406cf50 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.277+0000 7fa919baf700 10 monclient: get_auth_request con 0x7fa8f80361b0 auth_method 0 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.285+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa910005430 con 0x7fa91406cf50 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.285+0000 7fa90affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:35.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.285+0000 7fa90affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:35.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:35.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:35.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:35.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8fc005930 con 0x7fa91406cf50 2024-09-16T06:50:35.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa90affd700 1 -- 172.21.15.76:0/2035659038 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 770921579) 0x7fa910012b90 con 0x7fa91406cf50 2024-09-16T06:50:35.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa90affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:35.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa90affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:35.524 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":104,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa8f80361b0 legacy=0x7fa8f8038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 10 monclient: shutdown 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 >> v1:172.21.15.76:6789/0 conn(0x7fa91406cf50 legacy=0x7fa914135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 shutdown_connections 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 >> 172.21.15.76:0/2035659038 conn(0x7fa9140a41c0 msgr2=0x7fa914131770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:35.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 shutdown_connections 2024-09-16T06:50:35.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:35.525+0000 7fa91b3b2700 1 -- 172.21.15.76:0/2035659038 wait complete. 2024-09-16T06:50:38.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: get_monmap_and_config 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: build_initial_monmap 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: monmap: 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.657 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:38.662363+0000 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:38.662363+0000 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:38.658 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 1 Processor -- start 2024-09-16T06:50:38.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 1 -- start start 2024-09-16T06:50:38.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: init 2024-09-16T06:50:38.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-a con 0x7f7c840973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-b con 0x7f7c84097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-c con 0x7f7c841400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c84099350 con 0x7f7c840973c0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8409f690 con 0x7f7c841400d0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c84099bf0 con 0x7f7c84097eb0 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c8a0bc700 10 monclient(hunting): authenticate will time out at 1585.795743s 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c888b9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7c84097eb0 0x7f7c8413e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41418/0 (socket says 172.21.15.76:41418) 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c890ba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7c840973c0 0x7f7c840977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45370/0 (socket says 172.21.15.76:45370) 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.657+0000 7f7c888b9700 1 -- 172.21.15.76:0/2053872813 learned_addr learned my addr 172.21.15.76:0/2053872813 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:38.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818756394 0 0) 0x7f7c84099bf0 con 0x7f7c84097eb0 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): my global_id is 244112 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c700035d0 con 0x7f7c84097eb0 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501518268 0 0) 0x7f7c84099350 con 0x7f7c840973c0 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): my global_id is 234195 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c84099bf0 con 0x7f7c840973c0 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637760985 0 0) 0x7f7c8409f690 con 0x7f7c841400d0 2024-09-16T06:50:38.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient(hunting): my global_id is 224278 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c84099350 con 0x7f7c841400d0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905999221 0 0) 0x7f7c700035d0 con 0x7f7c84097eb0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 >> v1:172.21.15.189:6789/0 conn(0x7f7c841400d0 legacy=0x7f7c841424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 >> v1:172.21.15.76:6789/0 conn(0x7f7c840973c0 legacy=0x7f7c840977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 monclient: found mon.noname-b 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c84098360 con 0x7f7c84097eb0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: _finish_auth 0 2024-09-16T06:50:38.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c80001610 con 0x7f7c84097eb0 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 5 monclient: authenticate success, global_id 244112 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: dump: 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c800018f0 con 0x7f7c84097eb0 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 1 -- 172.21.15.76:0/2053872813 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c80001c80 con 0x7f7c84097eb0 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c7bfff700 10 monclient: dump: 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.667 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 4 set_mon_vals no callback set 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 10 monclient: shutdown 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2053872813 >> v1:172.21.15.76:6790/0 conn(0x7f7c84097eb0 legacy=0x7f7c8413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2053872813 shutdown_connections 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2053872813 >> 172.21.15.76:0/2053872813 conn(0x7f7c840a0030 msgr2=0x7f7c840a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2053872813 shutdown_connections 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2053872813 wait complete. 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 10 monclient: build_initial_monmap 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 10 monclient: monmap: 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:38.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:38.670 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.661+0000 7f7c8a0bc700 1 Processor -- start 2024-09-16T06:50:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- start start 2024-09-16T06:50:38.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: init 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-c con 0x7f7c840973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-a con 0x7f7c84097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient(hunting): picked mon.noname-b con 0x7f7c841400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c84072660 con 0x7f7c84097eb0 2024-09-16T06:50:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8413f450 con 0x7f7c841400d0 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8406c5a0 con 0x7f7c840973c0 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient(hunting): authenticate will time out at 1585.801364s 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c890ba700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7c840973c0 0x7f7c8413c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41424/0 (socket says 172.21.15.76:41424) 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c888b9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7c84097eb0 0x7f7c8413cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45384/0 (socket says 172.21.15.76:45384) 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c890ba700 1 -- 172.21.15.76:0/2749647512 learned_addr learned my addr 172.21.15.76:0/2749647512 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940181987 0 0) 0x7f7c8406c5a0 con 0x7f7c840973c0 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): my global_id is 244115 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c680035d0 con 0x7f7c840973c0 2024-09-16T06:50:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456532707 0 0) 0x7f7c84072660 con 0x7f7c84097eb0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): my global_id is 234198 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c8406c5a0 con 0x7f7c84097eb0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838691577 0 0) 0x7f7c8413f450 con 0x7f7c841400d0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): my global_id is 224281 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c84072660 con 0x7f7c841400d0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087866098 0 0) 0x7f7c680035d0 con 0x7f7c840973c0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c8413f450 con 0x7f7c840973c0 2024-09-16T06:50:38.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c740019d0 con 0x7f7c840973c0 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:38.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2746516315 0 0) 0x7f7c8406c5a0 con 0x7f7c84097eb0 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c680035d0 con 0x7f7c84097eb0 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c800015d0 con 0x7f7c84097eb0 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2703055060 0 0) 0x7f7c8413f450 con 0x7f7c840973c0 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 >> v1:172.21.15.189:6789/0 conn(0x7f7c841400d0 legacy=0x7f7c841d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 >> v1:172.21.15.76:6789/0 conn(0x7f7c84097eb0 legacy=0x7f7c8413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 monclient: found mon.c 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8413fda0 con 0x7f7c840973c0 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: _finish_auth 0 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:50:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 5 monclient: authenticate success, global_id 244115 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _renew_subs 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8409a540 con 0x7f7c840973c0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _renew_subs 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8413f930 con 0x7f7c840973c0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c74001c40 con 0x7f7c840973c0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7c74004ce0 con 0x7f7c840973c0 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c79ffb700 10 monclient: dump: 2024-09-16T06:50:38.680 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.665+0000 7f7c7b7fe700 4 set_mon_vals no callback set 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c8a0bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c8a0bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:38.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c8a0bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8409bef0 con 0x7f7c840973c0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7c7400f750 con 0x7f7c840973c0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7c74043f70 con 0x7f7c840973c0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.669+0000 7f7c888b9700 10 monclient: get_auth_request con 0x7f7c68036400 auth_method 0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.673+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7c74010ca0 con 0x7f7c840973c0 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.673+0000 7f7c79ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.673+0000 7f7c79ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c8a0bc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c8a0bc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c8a0bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7c840672a0 con 0x7f7c840973c0 2024-09-16T06:50:38.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c79ffb700 1 -- 172.21.15.76:0/2749647512 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 454719177) 0x7f7c740144a0 con 0x7f7c840973c0 2024-09-16T06:50:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c79ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:38.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.909+0000 7f7c79ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:38.911 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7c68036400 legacy=0x7f7c680388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 10 monclient: shutdown 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 >> v1:172.21.15.76:6790/0 conn(0x7f7c840973c0 legacy=0x7f7c8413c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 shutdown_connections 2024-09-16T06:50:38.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 >> 172.21.15.76:0/2749647512 conn(0x7f7c840a0030 msgr2=0x7f7c84096260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 shutdown_connections 2024-09-16T06:50:38.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:38.913+0000 7f7c8a0bc700 1 -- 172.21.15.76:0/2749647512 wait complete. 2024-09-16T06:50:38.925 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:50:38.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:50:38.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:50:39.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: get_monmap_and_config 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: monmap: 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:39.048308+0000 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:39.048308+0000 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 1 Processor -- start 2024-09-16T06:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 1 -- start start 2024-09-16T06:50:39.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: init 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-c con 0x7f4f6409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-a con 0x7f4f64098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-b con 0x7f4f64099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6409d560 con 0x7f4f64098be0 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6409f140 con 0x7f4f6409ed60 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6409d790 con 0x7f4f64099030 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient(hunting): authenticate will time out at 1586.181798s 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f69993700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4f64099030 0x7f4f6413ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41436/0 (socket says 172.21.15.76:41436) 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f68991700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4f64098be0 0x7f4f641374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45396/0 (socket says 172.21.15.76:45396) 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f69993700 1 -- 172.21.15.76:0/1652956197 learned_addr learned my addr 172.21.15.76:0/1652956197 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362147941 0 0) 0x7f4f6409d790 con 0x7f4f64099030 2024-09-16T06:50:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): my global_id is 244118 2024-09-16T06:50:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500057e0 con 0x7f4f64099030 2024-09-16T06:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868858247 0 0) 0x7f4f6409d560 con 0x7f4f64098be0 2024-09-16T06:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): my global_id is 234201 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6409d790 con 0x7f4f64098be0 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189789958 0 0) 0x7f4f6409f140 con 0x7f4f6409ed60 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient(hunting): my global_id is 224284 2024-09-16T06:50:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6409d560 con 0x7f4f6409ed60 2024-09-16T06:50:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 307759148 0 0) 0x7f4f500057e0 con 0x7f4f64099030 2024-09-16T06:50:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 >> v1:172.21.15.189:6789/0 conn(0x7f4f6409ed60 legacy=0x7f4f640984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 >> v1:172.21.15.76:6789/0 conn(0x7f4f64098be0 legacy=0x7f4f641374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 monclient: found mon.noname-b 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6409f620 con 0x7f4f64099030 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f5c001990 con 0x7f4f64099030 2024-09-16T06:50:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 5 monclient: authenticate success, global_id 244118 2024-09-16T06:50:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: dump: 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.056 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.057 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.057 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.057 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.057 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.057 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f5c001c70 con 0x7f4f64099030 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 1 -- 172.21.15.76:0/1652956197 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f5c002000 con 0x7f4f64099030 2024-09-16T06:50:39.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:39.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f6a194700 4 set_mon_vals no callback set 2024-09-16T06:50:39.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.045+0000 7f4f5bfff700 10 monclient: dump: 2024-09-16T06:50:39.059 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.060 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.061 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.061 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.061 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: shutdown 2024-09-16T06:50:39.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1652956197 >> v1:172.21.15.76:6790/0 conn(0x7f4f64099030 legacy=0x7f4f6413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1652956197 shutdown_connections 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1652956197 >> 172.21.15.76:0/1652956197 conn(0x7f4f640a0030 msgr2=0x7f4f640a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1652956197 shutdown_connections 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1652956197 wait complete. 2024-09-16T06:50:39.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: monmap: 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:39.063 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:39.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:39.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 Processor -- start 2024-09-16T06:50:39.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- start start 2024-09-16T06:50:39.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: init 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-a con 0x7f4f64098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-b con 0x7f4f64099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient(hunting): picked mon.noname-c con 0x7f4f6409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f640924f0 con 0x7f4f64098be0 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f64093200 con 0x7f4f64099030 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f640985f0 con 0x7f4f6409ed60 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient(hunting): authenticate will time out at 1586.186469s 2024-09-16T06:50:39.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f69993700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4f6409ed60 0x7f4f641d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41440/0 (socket says 172.21.15.76:41440) 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f69993700 1 -- 172.21.15.76:0/1066876903 learned_addr learned my addr 172.21.15.76:0/1066876903 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679568514 0 0) 0x7f4f640985f0 con 0x7f4f6409ed60 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 244121 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f480035d0 con 0x7f4f6409ed60 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352062509 0 0) 0x7f4f640924f0 con 0x7f4f64098be0 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 234204 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f640985f0 con 0x7f4f64098be0 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411641612 0 0) 0x7f4f480035d0 con 0x7f4f6409ed60 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f640924f0 con 0x7f4f6409ed60 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f5c001f80 con 0x7f4f6409ed60 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759564565 0 0) 0x7f4f64093200 con 0x7f4f64099030 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 224287 2024-09-16T06:50:39.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f480035d0 con 0x7f4f64099030 2024-09-16T06:50:39.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801042505 0 0) 0x7f4f640985f0 con 0x7f4f64098be0 2024-09-16T06:50:39.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f64093200 con 0x7f4f64098be0 2024-09-16T06:50:39.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f600017b0 con 0x7f4f64098be0 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 98825765 0 0) 0x7f4f640924f0 con 0x7f4f6409ed60 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.189:6789/0 conn(0x7f4f64099030 legacy=0x7f4f64091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f64098be0 legacy=0x7f4f640916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 monclient: found mon.c 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640a1250 con 0x7f4f6409ed60 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 5 monclient: authenticate success, global_id 244121 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _renew_subs 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f64095540 con 0x7f4f6409ed60 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _renew_subs 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f64095260 con 0x7f4f6409ed60 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f5c001460 con 0x7f4f6409ed60 2024-09-16T06:50:39.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f5c0051c0 con 0x7f4f6409ed60 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f5affd700 4 set_mon_vals no callback set 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.049+0000 7f4f59ffb700 10 monclient: dump: 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4f5c00fc30 con 0x7f4f6409ed60 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f6a194700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f4f5c0442a0 con 0x7f4f6409ed60 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f44006590 con 0x7f4f44002830 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f69192700 10 monclient: get_auth_request con 0x7f4f48036530 auth_method 0 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213163489 0 0) 0x7f4f44006590 con 0x7f4f44002830 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 234207 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f640924f0 con 0x7f4f44002830 2024-09-16T06:50:39.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 244482470 0 0) 0x7f4f640924f0 con 0x7f4f44002830 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f44006590 con 0x7f4f44002830 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f54001730 con 0x7f4f44002830 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: dump: 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.088 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f68991700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 legacy=0x7f4f44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f68991700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 legacy=0x7f4f44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f68991700 1 --1- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 0x7f4f44004cd0 :-1 s=OPENED pgs=1065 cs=1 l=1).handle_message read tag failed 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f68991700 1 --1- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 0x7f4f44004cd0 :-1 s=OPENED pgs=1065 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 legacy=0x7f4f44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44002830 legacy=0x7f4f44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f640924f0 con 0x7f4f48041d60 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106010630 0 0) 0x7f4f640924f0 con 0x7f4f48041d60 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 234210 2024-09-16T06:50:39.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f640985f0 con 0x7f4f48041d60 2024-09-16T06:50:39.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 517380694 0 0) 0x7f4f640985f0 con 0x7f4f48041d60 2024-09-16T06:50:39.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f640924f0 con 0x7f4f48041d60 2024-09-16T06:50:39.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f54001570 con 0x7f4f48041d60 2024-09-16T06:50:39.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: dump: 2024-09-16T06:50:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4158240636 0 0) 0x7f4f640924f0 con 0x7f4f48041d60 2024-09-16T06:50:39.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f48042850 con 0x7f4f48041d60 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4f540069b0 con 0x7f4f48041d60 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f48041d60 legacy=0x7f4f48042140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.053+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f48041d60 legacy=0x7f4f48042140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.065+0000 7f4f6a194700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.065+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4f44009d90 con 0x7f4f44009150 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970277218 0 0) 0x7f4f44009d90 con 0x7f4f44009150 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient(hunting): my global_id is 234213 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f640924f0 con 0x7f4f44009150 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 783079301 0 0) 0x7f4f640924f0 con 0x7f4f44009150 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f44009d90 con 0x7f4f44009150 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4f54004d60 con 0x7f4f44009150 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient: dump: 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.095 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1500587483 0 0) 0x7f4f44009d90 con 0x7f4f44009150 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4f4400c340 con 0x7f4f44009150 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 246745179) 0x7f4f54006b20 con 0x7f4f44009150 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44009150 legacy=0x7f4f440095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f69993700 1 -- 172.21.15.76:0/1066876903 reap_dead start 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f59ffb700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6789/0 conn(0x7f4f44009150 legacy=0x7f4f440095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:39.116 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:39.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4f48036530 legacy=0x7f4f480389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 10 monclient: shutdown 2024-09-16T06:50:39.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 >> v1:172.21.15.76:6790/0 conn(0x7f4f6409ed60 legacy=0x7f4f641d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 shutdown_connections 2024-09-16T06:50:39.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 >> 172.21.15.76:0/1066876903 conn(0x7f4f640a0030 msgr2=0x7f4f6406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 shutdown_connections 2024-09-16T06:50:39.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.069+0000 7f4f6a194700 1 -- 172.21.15.76:0/1066876903 wait complete. 2024-09-16T06:50:39.118 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:39.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:50:39.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:50:39.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: get_monmap_and_config 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: monmap: 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:39.244625+0000 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:39.244625+0000 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:39.240 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 1 Processor -- start 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 1 -- start start 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: init 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:39.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-c con 0x7fb2b4132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b406c280 con 0x7fb2b406cf50 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b406bfa0 con 0x7fb2b4132330 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b409ace0 con 0x7fb2b406d480 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 10 monclient(hunting): authenticate will time out at 1586.378787s 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2b9ce9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb2b406cf50 0x7fb2b413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45424/0 (socket says 172.21.15.76:45424) 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2b9ce9700 1 -- 172.21.15.76:0/1590878540 learned_addr learned my addr 172.21.15.76:0/1590878540 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626887308 0 0) 0x7fb2b406c280 con 0x7fb2b406cf50 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): my global_id is 234216 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a00035d0 con 0x7fb2b406cf50 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859329537 0 0) 0x7fb2b409ace0 con 0x7fb2b406d480 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): my global_id is 244124 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b406c280 con 0x7fb2b406d480 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290503658 0 0) 0x7fb2b406bfa0 con 0x7fb2b4132330 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient(hunting): my global_id is 224290 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b409ace0 con 0x7fb2b4132330 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4287294431 0 0) 0x7fb2a00035d0 con 0x7fb2b406cf50 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 >> v1:172.21.15.76:6790/0 conn(0x7fb2b406d480 legacy=0x7fb2b413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 >> v1:172.21.15.189:6789/0 conn(0x7fb2b4132330 legacy=0x7fb2b4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 monclient: found mon.noname-a 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b409ae70 con 0x7fb2b406cf50 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 5 monclient: authenticate success, global_id 234216 2024-09-16T06:50:39.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2ba4ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2ac001990 con 0x7fb2b406cf50 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.241+0000 7fb2abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: dump: 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.250 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2ac001bb0 con 0x7fb2b406cf50 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 4 set_mon_vals no callback set 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 1 -- 172.21.15.76:0/1590878540 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2ac001da0 con 0x7fb2b406cf50 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2abfff700 10 monclient: dump: 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: shutdown 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/1590878540 >> v1:172.21.15.76:6789/0 conn(0x7fb2b406cf50 legacy=0x7fb2b413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/1590878540 shutdown_connections 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/1590878540 >> 172.21.15.76:0/1590878540 conn(0x7fb2b40a41c0 msgr2=0x7fb2b40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/1590878540 shutdown_connections 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/1590878540 wait complete. 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: monmap: 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 Processor -- start 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- start start 2024-09-16T06:50:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: init 2024-09-16T06:50:39.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:39.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-c con 0x7fb2b406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b4132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b412f850 con 0x7fb2b406cf50 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b412f190 con 0x7fb2b4132330 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b41316b0 con 0x7fb2b406d480 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2ba4ea700 10 monclient(hunting): authenticate will time out at 1586.383536s 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2b94e8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb2b406cf50 0x7fb2b413e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45436/0 (socket says 172.21.15.76:45436) 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2b94e8700 1 -- 172.21.15.76:0/598403653 learned_addr learned my addr 172.21.15.76:0/598403653 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721387935 0 0) 0x7fb2b412f850 con 0x7fb2b406cf50 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 10 monclient(hunting): my global_id is 234219 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.245+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2980035d0 con 0x7fb2b406cf50 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302077324 0 0) 0x7fb2b41316b0 con 0x7fb2b406d480 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): my global_id is 244127 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b412f850 con 0x7fb2b406d480 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545733852 0 0) 0x7fb2980035d0 con 0x7fb2b406cf50 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b41316b0 con 0x7fb2b406cf50 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2b0001990 con 0x7fb2b406cf50 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317220281 0 0) 0x7fb2b412f190 con 0x7fb2b4132330 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): my global_id is 224293 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2980035d0 con 0x7fb2b4132330 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000147761 0 0) 0x7fb2b412f850 con 0x7fb2b406d480 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b412f190 con 0x7fb2b406d480 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a40019d0 con 0x7fb2b406d480 2024-09-16T06:50:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): dump: 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1648196186 0 0) 0x7fb2b41316b0 con 0x7fb2b406cf50 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6790/0 conn(0x7fb2b406d480 legacy=0x7fb2b41cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.189:6789/0 conn(0x7fb2b4132330 legacy=0x7fb2b41d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 monclient: found mon.a 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4130a20 con 0x7fb2b406cf50 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 5 monclient: authenticate success, global_id 234219 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 10 monclient: _renew_subs 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b4130d00 con 0x7fb2b406cf50 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 10 monclient: _renew_subs 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b413eb00 con 0x7fb2b406cf50 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b0001d40 con 0x7fb2b406cf50 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2b0003b10 con 0x7fb2b406cf50 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ab7fe700 4 set_mon_vals no callback set 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: dump: 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.268 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb294006590 con 0x7fb294002830 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781513240 0 0) 0x7fb294006590 con 0x7fb294002830 2024-09-16T06:50:39.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient(hunting): my global_id is 244130 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b41316b0 con 0x7fb294002830 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb2b0003cf0 con 0x7fb2b406cf50 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fb2b0043ea0 con 0x7fb2b406cf50 2024-09-16T06:50:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2748754107 0 0) 0x7fb2b41316b0 con 0x7fb294002830 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb294006590 con 0x7fb294002830 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a4003840 con 0x7fb294002830 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 10 monclient: dump: 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.271 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2b9ce9700 10 monclient: get_auth_request con 0x7fb298036450 auth_method 0 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4243958426 0 0) 0x7fb294006590 con 0x7fb294002830 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.249+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb29400a190 con 0x7fb294002830 2024-09-16T06:50:39.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.253+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb2a40039a0 con 0x7fb294002830 2024-09-16T06:50:39.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.253+0000 7fb2a9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.253+0000 7fb2a9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:39.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.253+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6790/0 conn(0x7fb294002830 legacy=0x7fb294004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.253+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6790/0 conn(0x7fb294002830 legacy=0x7fb294004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2ba4ea700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb294009e50 con 0x7fb294009170 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857726287 0 0) 0x7fb294009e50 con 0x7fb294009170 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient(hunting): my global_id is 244133 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb294006590 con 0x7fb294009170 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3190270466 0 0) 0x7fb294006590 con 0x7fb294009170 2024-09-16T06:50:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb294009e50 con 0x7fb294009170 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb2a4003fa0 con 0x7fb294009170 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient: dump: 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.277 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4096557525 0 0) 0x7fb294009e50 con 0x7fb294009170 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb29400c340 con 0x7fb294009170 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3434660962) 0x7fb2a4004270 con 0x7fb294009170 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6790/0 conn(0x7fb294009170 legacy=0x7fb294009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.261+0000 7fb2a9ffb700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6790/0 conn(0x7fb294009170 legacy=0x7fb294009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:39.278 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:50:39.279 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.280 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.281 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:39.282 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.284 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.285 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:39.286 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.287 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.288 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.288 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:39.288 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.289 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.290 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.292 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:39.293 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb298036450 legacy=0x7fb298038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 10 monclient: shutdown 2024-09-16T06:50:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 >> v1:172.21.15.76:6789/0 conn(0x7fb2b406cf50 legacy=0x7fb2b413e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2b9ce9700 1 -- 172.21.15.76:0/598403653 reap_dead start 2024-09-16T06:50:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2b9ce9700 1 -- 172.21.15.76:0/598403653 reap_dead start 2024-09-16T06:50:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 shutdown_connections 2024-09-16T06:50:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 >> 172.21.15.76:0/598403653 conn(0x7fb2b40a41c0 msgr2=0x7fb2b409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 shutdown_connections 2024-09-16T06:50:39.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.265+0000 7fb2ba4ea700 1 -- 172.21.15.76:0/598403653 wait complete. 2024-09-16T06:50:39.296 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:39.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:50:39.297 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:50:39.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.417+0000 7f182cf58700 10 monclient: get_monmap_and_config 2024-09-16T06:50:39.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.417+0000 7f182cf58700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.417+0000 7f182cf58700 10 monclient: monmap: 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:39.423319+0000 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:39.423319+0000 2024-09-16T06:50:39.419 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.420 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.420 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.420 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:39.420 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:39.420 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 Processor -- start 2024-09-16T06:50:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- start start 2024-09-16T06:50:39.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: init 2024-09-16T06:50:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-c con 0x7f1828097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-b con 0x7f1828099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-a con 0x7f1828139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828098990 con 0x7f1828139500 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828097f40 con 0x7f1828097b60 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182809d610 con 0x7f1828099090 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient(hunting): authenticate will time out at 1586.556787s 2024-09-16T06:50:39.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f1827fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 0x7f182813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45446/0 (socket says 172.21.15.76:45446) 2024-09-16T06:50:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f1827fff700 1 -- 172.21.15.76:0/2344241254 learned_addr learned my addr 172.21.15.76:0/2344241254 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544802101 0 0) 0x7f182809d610 con 0x7f1828099090 2024-09-16T06:50:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): my global_id is 244136 2024-09-16T06:50:39.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18100035d0 con 0x7f1828099090 2024-09-16T06:50:39.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442837031 0 0) 0x7f1828098990 con 0x7f1828139500 2024-09-16T06:50:39.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): my global_id is 234222 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f182809d610 con 0x7f1828139500 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598677459 0 0) 0x7f1828097f40 con 0x7f1828097b60 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient(hunting): my global_id is 224296 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828098990 con 0x7f1828097b60 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144751572 0 0) 0x7f182809d610 con 0x7f1828139500 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 >> v1:172.21.15.76:6790/0 conn(0x7f1828099090 legacy=0x7f182809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 >> v1:172.21.15.189:6789/0 conn(0x7f1828097b60 legacy=0x7f182809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 monclient: found mon.noname-a 2024-09-16T06:50:39.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280986b0 con 0x7f1828139500 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1820001970 con 0x7f1828139500 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 5 monclient: authenticate success, global_id 234222 2024-09-16T06:50:39.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: dump: 2024-09-16T06:50:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.453 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.454 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.455 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1820001c90 con 0x7f1828139500 2024-09-16T06:50:39.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 1 -- 172.21.15.76:0/2344241254 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1820002060 con 0x7f1828139500 2024-09-16T06:50:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 4 set_mon_vals no callback set 2024-09-16T06:50:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f18267fc700 10 monclient: dump: 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.457 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: shutdown 2024-09-16T06:50:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- 172.21.15.76:0/2344241254 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 legacy=0x7f182813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- 172.21.15.76:0/2344241254 shutdown_connections 2024-09-16T06:50:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- 172.21.15.76:0/2344241254 >> 172.21.15.76:0/2344241254 conn(0x7f18280a0030 msgr2=0x7f18280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- 172.21.15.76:0/2344241254 shutdown_connections 2024-09-16T06:50:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 1 -- 172.21.15.76:0/2344241254 wait complete. 2024-09-16T06:50:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.421+0000 7f182cf58700 10 monclient: monmap: 2024-09-16T06:50:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.461 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:39.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 Processor -- start 2024-09-16T06:50:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- start start 2024-09-16T06:50:39.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: init 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-c con 0x7f1828097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-b con 0x7f1828099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient(hunting): picked mon.noname-a con 0x7f1828139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828138750 con 0x7f1828139500 2024-09-16T06:50:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280a1250 con 0x7f1828099090 2024-09-16T06:50:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18281368d0 con 0x7f1828097b60 2024-09-16T06:50:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient(hunting): authenticate will time out at 1586.560921s 2024-09-16T06:50:39.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 0x7f18281d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45452/0 (socket says 172.21.15.76:45452) 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 -- 172.21.15.76:0/1545973877 learned_addr learned my addr 172.21.15.76:0/1545973877 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353914767 0 0) 0x7f1828138750 con 0x7f1828139500 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): my global_id is 234225 2024-09-16T06:50:39.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18140035d0 con 0x7f1828139500 2024-09-16T06:50:39.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496756343 0 0) 0x7f18281368d0 con 0x7f1828097b60 2024-09-16T06:50:39.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): my global_id is 244139 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828138750 con 0x7f1828097b60 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239894673 0 0) 0x7f18280a1250 con 0x7f1828099090 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): my global_id is 224299 2024-09-16T06:50:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18281368d0 con 0x7f1828099090 2024-09-16T06:50:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013149725 0 0) 0x7f18140035d0 con 0x7f1828139500 2024-09-16T06:50:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18280a1250 con 0x7f1828139500 2024-09-16T06:50:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1820001930 con 0x7f1828139500 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): dump: 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.469 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.471 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:39.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714696213 0 0) 0x7f1828138750 con 0x7f1828097b60 2024-09-16T06:50:39.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18140035d0 con 0x7f1828097b60 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1818001870 con 0x7f1828097b60 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): dump: 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.474 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 374825241 0 0) 0x7f18280a1250 con 0x7f1828139500 2024-09-16T06:50:39.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6790/0 conn(0x7f1828097b60 legacy=0x7f1828135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f1828099090 legacy=0x7f18281361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 monclient: found mon.a 2024-09-16T06:50:39.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828136be0 con 0x7f1828139500 2024-09-16T06:50:39.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 5 monclient: authenticate success, global_id 234225 2024-09-16T06:50:39.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _renew_subs 2024-09-16T06:50:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182809a540 con 0x7f1828139500 2024-09-16T06:50:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _renew_subs 2024-09-16T06:50:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:39.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1828138a30 con 0x7f1828139500 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 legacy=0x7f18281d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 legacy=0x7f18281d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 --1- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 0x7f18281d1f10 :-1 s=OPENED pgs=1072 cs=1 l=1).handle_message read tag failed 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f1827fff700 1 --1- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 0x7f18281d1f10 :-1 s=OPENED pgs=1072 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6789/0 2024-09-16T06:50:39.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1828139500 legacy=0x7f18281d1f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:39.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): picked mon.a con 0x7f1814004df0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): picked mon.c con 0x7f1828097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): picked mon.b con 0x7f1828099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18280a1250 con 0x7f1814004df0 2024-09-16T06:50:39.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1828138750 con 0x7f1828099090 2024-09-16T06:50:39.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1814012280 con 0x7f1828097b60 2024-09-16T06:50:39.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625396503 0 0) 0x7f18280a1250 con 0x7f1814004df0 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1814010220 con 0x7f1814004df0 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011269158 0 0) 0x7f1814012280 con 0x7f1828097b60 2024-09-16T06:50:39.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f18280a1250 con 0x7f1828097b60 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501277308 0 0) 0x7f1828138750 con 0x7f1828099090 2024-09-16T06:50:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18280994e0 con 0x7f1828138e10 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1814012280 con 0x7f1828099090 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2983858166 0 0) 0x7f18280a1250 con 0x7f1828097b60 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f1828099090 legacy=0x7f181400e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6789/0 conn(0x7f1814004df0 legacy=0x7f1814007a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 monclient: found mon.c 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f18140135c0 con 0x7f1828097b60 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f181c001850 con 0x7f1828097b60 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.425+0000 7f180ffff700 10 monclient: dump: 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f181c001a30 con 0x7f1828097b60 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f1825ffb700 4 set_mon_vals no callback set 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118341675 0 0) 0x7f18140035d0 con 0x7f1828138e10 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient(hunting): my global_id is 224302 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18280a1250 con 0x7f1828138e10 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f181c001c20 con 0x7f1828097b60 2024-09-16T06:50:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.2 v1:172.21.15.76:6790/0 6 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f181c0423c0 con 0x7f1828097b60 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3405232253 0 0) 0x7f18280a1250 con 0x7f1828138e10 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18140035d0 con 0x7f1828138e10 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f1827fff700 10 monclient: get_auth_request con 0x7f1828139500 auth_method 0 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1818003a50 con 0x7f1828138e10 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: dump: 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3490174334 0 0) 0x7f18140035d0 con 0x7f1828138e10 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f182809b860 con 0x7f1828138e10 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1818003ea0 con 0x7f1828138e10 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f1828138e10 legacy=0x7f1828135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.429+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f1828138e10 legacy=0x7f1828135ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f182cf58700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18280994a0 con 0x7f18281361c0 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845864096 0 0) 0x7f18280994a0 con 0x7f18281361c0 2024-09-16T06:50:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient(hunting): my global_id is 224305 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18140035d0 con 0x7f18281361c0 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1583890516 0 0) 0x7f18140035d0 con 0x7f18281361c0 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18280994a0 con 0x7f18281361c0 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f18180044b0 con 0x7f18281361c0 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 10 monclient: dump: 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2073274675 0 0) 0x7f18280994a0 con 0x7f18281361c0 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.441+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f18280973c0 con 0x7f18281361c0 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 456142832) 0x7f1818005820 con 0x7f18281361c0 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f180ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f180ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f18281361c0 legacy=0x7f182802b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f180ffff700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6789/0 conn(0x7f18281361c0 legacy=0x7f182802b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:39.490 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 1, 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.491 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.492 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:39.493 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:39.494 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:39.497 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.498 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.499 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:39.500 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:39.501 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:39.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1828139500 legacy=0x7f181404c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 10 monclient: shutdown 2024-09-16T06:50:39.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 >> v1:172.21.15.76:6790/0 conn(0x7f1828097b60 legacy=0x7f181400b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f1827fff700 1 -- 172.21.15.76:0/1545973877 reap_dead start 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f1827fff700 1 -- 172.21.15.76:0/1545973877 reap_dead start 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 shutdown_connections 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 >> 172.21.15.76:0/1545973877 conn(0x7f18280a0030 msgr2=0x7f18280702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 shutdown_connections 2024-09-16T06:50:39.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.445+0000 7f182cf58700 1 -- 172.21.15.76:0/1545973877 wait complete. 2024-09-16T06:50:39.503 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:39.503 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:50:39.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: get_monmap_and_config 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: monmap: 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:39.628911+0000 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:39.628911+0000 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.624 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:39.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:39.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 1 Processor -- start 2024-09-16T06:50:39.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 1 -- start start 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: init 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-b con 0x7f55c809ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-c con 0x7f55c8098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-a con 0x7f55c8099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c809d560 con 0x7f55c8099030 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c809f140 con 0x7f55c8098be0 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c809d790 con 0x7f55c809ed60 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient(hunting): authenticate will time out at 1586.762151s 2024-09-16T06:50:39.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cd4e1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f55c8099030 0x7f55c813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45480/0 (socket says 172.21.15.76:45480) 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cd4e1700 1 -- 172.21.15.76:0/3455694077 learned_addr learned my addr 172.21.15.76:0/3455694077 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983842248 0 0) 0x7f55c809d560 con 0x7f55c8099030 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): my global_id is 234228 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b00035d0 con 0x7f55c8099030 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976683456 0 0) 0x7f55c809d790 con 0x7f55c809ed60 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): my global_id is 244142 2024-09-16T06:50:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c809d560 con 0x7f55c809ed60 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040409786 0 0) 0x7f55c809f140 con 0x7f55c8098be0 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient(hunting): my global_id is 224308 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c809d790 con 0x7f55c8098be0 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958912665 0 0) 0x7f55c809d560 con 0x7f55c809ed60 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 >> v1:172.21.15.189:6789/0 conn(0x7f55c8098be0 legacy=0x7f55c81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 >> v1:172.21.15.76:6789/0 conn(0x7f55c8099030 legacy=0x7f55c813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 monclient: found mon.noname-b 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c809f620 con 0x7f55c809ed60 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55bc0015f0 con 0x7f55c809ed60 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: dump: 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.631 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 5 monclient: authenticate success, global_id 244142 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc001890 con 0x7f55c809ed60 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 1 -- 172.21.15.76:0/3455694077 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55bc001c60 con 0x7f55c809ed60 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55c77fe700 10 monclient: dump: 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.632 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.625+0000 7f55cdce2700 4 set_mon_vals no callback set 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 10 monclient: shutdown 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- 172.21.15.76:0/3455694077 >> v1:172.21.15.76:6790/0 conn(0x7f55c809ed60 legacy=0x7f55c80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- 172.21.15.76:0/3455694077 shutdown_connections 2024-09-16T06:50:39.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- 172.21.15.76:0/3455694077 >> 172.21.15.76:0/3455694077 conn(0x7f55c80a0030 msgr2=0x7f55c80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- 172.21.15.76:0/3455694077 shutdown_connections 2024-09-16T06:50:39.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- 172.21.15.76:0/3455694077 wait complete. 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 10 monclient: build_initial_monmap 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 10 monclient: monmap: 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:39.638 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 Processor -- start 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 1 -- start start 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.629+0000 7f55cdce2700 10 monclient: init 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-c con 0x7f55c8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-b con 0x7f55c8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient(hunting): picked mon.noname-a con 0x7f55c809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c81ca660 con 0x7f55c809ed60 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c809a540 con 0x7f55c8099030 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c8094990 con 0x7f55c8098be0 2024-09-16T06:50:39.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient(hunting): authenticate will time out at 1586.768755s 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cd4e1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f55c809ed60 0x7f55c81d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45484/0 (socket says 172.21.15.76:45484) 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cd4e1700 1 -- 172.21.15.76:0/2415861408 learned_addr learned my addr 172.21.15.76:0/2415861408 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242252969 0 0) 0x7f55c8094990 con 0x7f55c8098be0 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): my global_id is 244145 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b40035d0 con 0x7f55c8098be0 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155599446 0 0) 0x7f55c81ca660 con 0x7f55c809ed60 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): my global_id is 234231 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c8094990 con 0x7f55c809ed60 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736151018 0 0) 0x7f55c809a540 con 0x7f55c8099030 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): my global_id is 224311 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c81ca660 con 0x7f55c8099030 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228833643 0 0) 0x7f55b40035d0 con 0x7f55c8098be0 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c809a540 con 0x7f55c8098be0 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55bc001ba0 con 0x7f55c8098be0 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:39.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): dump: 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646862760 0 0) 0x7f55c8094990 con 0x7f55c809ed60 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b40035d0 con 0x7f55c809ed60 2024-09-16T06:50:39.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55c0001680 con 0x7f55c809ed60 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient(hunting): dump: 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.643 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719051325 0 0) 0x7f55c81ca660 con 0x7f55c8099030 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c8094990 con 0x7f55c8099030 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1866218897 0 0) 0x7f55c809a540 con 0x7f55c8098be0 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 >> v1:172.21.15.189:6789/0 conn(0x7f55c8099030 legacy=0x7f55c81cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 >> v1:172.21.15.76:6789/0 conn(0x7f55c809ed60 legacy=0x7f55c81d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 monclient: found mon.c 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c809e880 con 0x7f55c8098be0 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: _finish_auth 0 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:50:39.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 5 monclient: authenticate success, global_id 244145 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _renew_subs 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c80a2f70 con 0x7f55c8098be0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _renew_subs 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c8092f10 con 0x7f55c8098be0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc0039d0 con 0x7f55c8098be0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55bc003d60 con 0x7f55c8098be0 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c6ffd700 4 set_mon_vals no callback set 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:39.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 10 monclient: dump: 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f55bc003ec0 con 0x7f55c8098be0 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f55bc042d40 con 0x7f55c8098be0 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.633+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8005100 con 0x7f55c8098be0 2024-09-16T06:50:39.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.637+0000 7f55c7fff700 10 monclient: get_auth_request con 0x7f55b4036410 auth_method 0 2024-09-16T06:50:39.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.641+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f55bc005550 con 0x7f55c8098be0 2024-09-16T06:50:39.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.641+0000 7f55c57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:39.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.641+0000 7f55c57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:39.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.869+0000 7f55cdce2700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:50:39.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.869+0000 7f55cdce2700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:50:39.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.869+0000 7f55cdce2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:39.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.869+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f55a8005bc0 con 0x7f55c8098be0 2024-09-16T06:50:39.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f55c57fa700 1 -- 172.21.15.76:0/2415861408 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f55bc011d10 con 0x7f55c8098be0 2024-09-16T06:50:39.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f55c57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:50:39.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f55c57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=13,logm=44} crc {auth=256697198,config=1097430493,health=514036659,kv=344456112,logm=2482921163}) 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=13,logm=46} crc {auth=2607467671,config=1097430493,health=1373778773,kv=1001081567,logm=612675897}) 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=13,logm=44} crc {auth=256697198,config=1097430493,health=514036659,kv=344456112,logm=2482921163}) 2024-09-16T06:50:39.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.873+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=13,logm=43} crc {auth=261274776,config=1097430493,health=514036659,kv=1001081567,logm=2393315328}) 2024-09-16T06:50:39.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 >> v1:172.21.15.189:6812/3098810674 conn(0x7f55b4036410 legacy=0x7f55b40388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 10 monclient: shutdown 2024-09-16T06:50:39.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:39.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 >> v1:172.21.15.76:6790/0 conn(0x7f55c8098be0 legacy=0x7f55c8095ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:39.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 shutdown_connections 2024-09-16T06:50:39.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 >> 172.21.15.76:0/2415861408 conn(0x7f55c80a0030 msgr2=0x7f55c80973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:39.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 shutdown_connections 2024-09-16T06:50:39.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f55cdce2700 1 -- 172.21.15.76:0/2415861408 wait complete. 2024-09-16T06:50:39.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925524270}) 2024-09-16T06:50:39.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059621946}) 2024-09-16T06:50:39.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925524270}) 2024-09-16T06:50:39.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.877+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=938939837}) 2024-09-16T06:50:39.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698187593}) 2024-09-16T06:50:39.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347162363}) 2024-09-16T06:50:39.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698187593}) 2024-09-16T06:50:39.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360798491}) 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916241110}) 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248918170}) 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916241110}) 2024-09-16T06:50:39.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830584312}) 2024-09-16T06:50:39.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954323695}) 2024-09-16T06:50:39.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147812520}) 2024-09-16T06:50:39.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954323695}) 2024-09-16T06:50:39.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.881+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185878547}) 2024-09-16T06:50:39.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634873982}) 2024-09-16T06:50:39.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096198268}) 2024-09-16T06:50:39.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634873982}) 2024-09-16T06:50:39.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737163205}) 2024-09-16T06:50:39.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197419688}) 2024-09-16T06:50:39.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283026679}) 2024-09-16T06:50:39.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197419688}) 2024-09-16T06:50:39.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=743437181}) 2024-09-16T06:50:39.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980249107}) 2024-09-16T06:50:39.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918213748}) 2024-09-16T06:50:39.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980249107}) 2024-09-16T06:50:39.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.885+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035364798}) 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290980391}) 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714908241}) 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290980391}) 2024-09-16T06:50:39.888 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080472521}) 2024-09-16T06:50:39.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3233890702}) 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702335024}) 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3233890702}) 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115280134}) 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776613165}) 2024-09-16T06:50:39.891 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198295591}) 2024-09-16T06:50:39.892 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.892 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776613165}) 2024-09-16T06:50:39.892 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.889+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81038026}) 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287960105}) 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649954474}) 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287960105}) 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616333202}) 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936783557}) 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989173740}) 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936783557}) 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111534226}) 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300424691}) 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234885280}) 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300424691}) 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.893+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765210533}) 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702593378}) 2024-09-16T06:50:39.895 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501562187}) 2024-09-16T06:50:39.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702593378}) 2024-09-16T06:50:39.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200070616}) 2024-09-16T06:50:39.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.896 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228777028}) 2024-09-16T06:50:39.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285725510}) 2024-09-16T06:50:39.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228777028}) 2024-09-16T06:50:39.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865823768}) 2024-09-16T06:50:39.897 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027268592}) 2024-09-16T06:50:39.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1904427874}) 2024-09-16T06:50:39.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027268592}) 2024-09-16T06:50:39.898 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.897+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945705223}) 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15007985}) 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851058945}) 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15007985}) 2024-09-16T06:50:39.899 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766235458}) 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005926710}) 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38657668}) 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005926710}) 2024-09-16T06:50:39.900 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383293401}) 2024-09-16T06:50:39.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440583879}) 2024-09-16T06:50:39.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496718178}) 2024-09-16T06:50:39.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.901 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440583879}) 2024-09-16T06:50:39.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856324111}) 2024-09-16T06:50:39.902 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084748558}) 2024-09-16T06:50:39.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462399176}) 2024-09-16T06:50:39.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084748558}) 2024-09-16T06:50:39.903 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.901+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008215473}) 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712152973}) 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164406252}) 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712152973}) 2024-09-16T06:50:39.904 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227903213}) 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082955552}) 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298373538}) 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082955552}) 2024-09-16T06:50:39.905 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712367961}) 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733853916}) 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=112035838}) 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733853916}) 2024-09-16T06:50:39.906 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.905+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675170395}) 2024-09-16T06:50:39.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263734559}) 2024-09-16T06:50:39.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208430379}) 2024-09-16T06:50:39.907 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263734559}) 2024-09-16T06:50:39.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616424464}) 2024-09-16T06:50:39.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529473862}) 2024-09-16T06:50:39.908 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681147958}) 2024-09-16T06:50:39.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529473862}) 2024-09-16T06:50:39.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950002876}) 2024-09-16T06:50:39.909 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123492376}) 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679246346}) 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123492376}) 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.909+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=240232804}) 2024-09-16T06:50:39.910 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.911 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877142855}) 2024-09-16T06:50:39.911 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285449669}) 2024-09-16T06:50:39.911 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.911 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877142855}) 2024-09-16T06:50:39.911 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461885122}) 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166413207}) 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619846648}) 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166413207}) 2024-09-16T06:50:39.912 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110539069}) 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105884948}) 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=591920521}) 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105884948}) 2024-09-16T06:50:39.913 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104830104}) 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228444316}) 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066374975}) 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228444316}) 2024-09-16T06:50:39.914 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.913+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023359718}) 2024-09-16T06:50:39.915 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.915 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643427091}) 2024-09-16T06:50:39.915 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755730454}) 2024-09-16T06:50:39.915 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.915 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643427091}) 2024-09-16T06:50:39.916 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472120833}) 2024-09-16T06:50:39.916 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.917 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738560620}) 2024-09-16T06:50:39.917 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949993244}) 2024-09-16T06:50:39.917 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.917 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738560620}) 2024-09-16T06:50:39.917 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2903809013}) 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572184255}) 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135881012}) 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572184255}) 2024-09-16T06:50:39.918 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.917+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307666719}) 2024-09-16T06:50:39.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690638395}) 2024-09-16T06:50:39.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166617187}) 2024-09-16T06:50:39.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.919 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690638395}) 2024-09-16T06:50:39.920 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127171585}) 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2868546358}) 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265139804}) 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2868546358}) 2024-09-16T06:50:39.921 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974619842}) 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2038630057}) 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67399959}) 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2038630057}) 2024-09-16T06:50:39.922 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.921+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231390264}) 2024-09-16T06:50:39.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479235189}) 2024-09-16T06:50:39.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601833218}) 2024-09-16T06:50:39.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.923 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479235189}) 2024-09-16T06:50:39.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941728578}) 2024-09-16T06:50:39.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.924 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2926799892}) 2024-09-16T06:50:39.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598526552}) 2024-09-16T06:50:39.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2926799892}) 2024-09-16T06:50:39.925 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4222878476}) 2024-09-16T06:50:39.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550061236}) 2024-09-16T06:50:39.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160841761}) 2024-09-16T06:50:39.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.926 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550061236}) 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937960509}) 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103572008}) 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167609111}) 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103572008}) 2024-09-16T06:50:39.927 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.925+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026681500}) 2024-09-16T06:50:39.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=811025078}) 2024-09-16T06:50:39.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238818207}) 2024-09-16T06:50:39.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.928 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=811025078}) 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3736098364}) 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317062573}) 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284401874}) 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317062573}) 2024-09-16T06:50:39.929 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343000282}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810900523}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659953222}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810900523}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895411118}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887266109}) 2024-09-16T06:50:39.931 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688430793}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887266109}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.929+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929949424}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525629996}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3710674580}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525629996}) 2024-09-16T06:50:39.932 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172880976}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545489041}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931218265}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545489041}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491588688}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626155929}) 2024-09-16T06:50:39.934 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886872076}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626155929}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716310234}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645129371}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933323735}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645129371}) 2024-09-16T06:50:39.935 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.933+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263527068}) 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750444846}) 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135678404}) 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750444846}) 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172511958}) 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.937 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745745722}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553141061}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745745722}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437996856}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859666389}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911696526}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859666389}) 2024-09-16T06:50:39.938 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1876477225}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394513005}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379976175}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394513005}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.937+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=752053105}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2159589329}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685742734}) 2024-09-16T06:50:39.941 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2159589329}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511456362}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910638015}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409107083}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910638015}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453917396}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674246718}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908410305}) 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.942 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674246718}) 2024-09-16T06:50:39.943 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997761766}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=433845100}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770046184}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=433845100}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032072040}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821681954}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164892609}) 2024-09-16T06:50:39.945 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.941+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821681954}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137884392}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503075080}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500229175}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503075080}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3836128780}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631483317}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959652690}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631483317}) 2024-09-16T06:50:39.946 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=265440020}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485870642}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245279569}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485870642}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272942872}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864567317}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537968563}) 2024-09-16T06:50:39.948 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.945+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864567317}) 2024-09-16T06:50:39.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682399835}) 2024-09-16T06:50:39.949 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=560338464,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2307282664,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1395902275}) 2024-09-16T06:50:39.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=2514758294,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974420,mgr_metadata=1347778423,mgrstat=1643834192,mon_config_key=3366812847,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4200930638}) 2024-09-16T06:50:39.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=560338464,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2307282664,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1395902275}) 2024-09-16T06:50:39.950 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.949+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=41,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=799435143,mdsmap=3585761405,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2307282664,mon_config_key=3972503601,monmap=1820574642,osd_metadata=3249661422,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=3606355792}) 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=2886762832}) 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1315121456}) 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=2886762832}) 2024-09-16T06:50:39.960 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:50:39.961+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=3500399092}) 2024-09-16T06:50:40.890 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:50:40.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:41.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: get_monmap_and_config 2024-09-16T06:50:41.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: monmap: 2024-09-16T06:50:41.008 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.008 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:41.013472+0000 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:41.013472+0000 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:41.009 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 1 Processor -- start 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 1 -- start start 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: init 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc3e80a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc3e813b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e809b3e0 con 0x7fc3e813b6b0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80a3450 con 0x7fc3e80a24a0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e809b1a0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e759e700 10 monclient(hunting): authenticate will time out at 1588.146870s 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc3e80a20c0 0x7fc3e806cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41544/0 (socket says 172.21.15.76:41544) 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e6d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc3e813b6b0 0x7fc3e813daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45490/0 (socket says 172.21.15.76:45490) 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e659c700 1 -- 172.21.15.76:0/2256228543 learned_addr learned my addr 172.21.15.76:0/2256228543 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354144705 0 0) 0x7fc3e809b3e0 con 0x7fc3e813b6b0 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): my global_id is 234234 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d00035d0 con 0x7fc3e813b6b0 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280333206 0 0) 0x7fc3e809b1a0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): my global_id is 244148 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e809b3e0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842330146 0 0) 0x7fc3e80a3450 con 0x7fc3e80a24a0 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 10 monclient(hunting): my global_id is 224314 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.009+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e809b1a0 con 0x7fc3e80a24a0 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315467035 0 0) 0x7fc3e809b3e0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 >> v1:172.21.15.189:6789/0 conn(0x7fc3e80a24a0 legacy=0x7fc3e80a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 >> v1:172.21.15.76:6789/0 conn(0x7fc3e813b6b0 legacy=0x7fc3e813daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 monclient: found mon.noname-b 2024-09-16T06:50:41.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e806bd30 con 0x7fc3e80a20c0 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3dc0015f0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: dump: 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.016 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 5 monclient: authenticate success, global_id 244148 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3dc001820 con 0x7fc3e80a20c0 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 1 -- 172.21.15.76:0/2256228543 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3dc001bf0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 4 set_mon_vals no callback set 2024-09-16T06:50:41.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e559a700 10 monclient: dump: 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: shutdown 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- 172.21.15.76:0/2256228543 >> v1:172.21.15.76:6790/0 conn(0x7fc3e80a20c0 legacy=0x7fc3e806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- 172.21.15.76:0/2256228543 shutdown_connections 2024-09-16T06:50:41.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- 172.21.15.76:0/2256228543 >> 172.21.15.76:0/2256228543 conn(0x7fc3e80a41c0 msgr2=0x7fc3e80a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- 172.21.15.76:0/2256228543 shutdown_connections 2024-09-16T06:50:41.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- 172.21.15.76:0/2256228543 wait complete. 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: monmap: 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.023 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 Processor -- start 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- start start 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: init 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 10 monclient(hunting): picked mon.noname-b con 0x7fc3e813b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.013+0000 7fc3e759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8138660 con 0x7fc3e80a24a0 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e809e8b0 con 0x7fc3e813b6b0 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80a6f90 con 0x7fc3e80a20c0 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient(hunting): authenticate will time out at 1588.152077s 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc3e80a20c0 0x7fc3e81352d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41560/0 (socket says 172.21.15.76:41560) 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e659c700 1 -- 172.21.15.76:0/3533713229 learned_addr learned my addr 172.21.15.76:0/3533713229 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281108698 0 0) 0x7fc3e80a6f90 con 0x7fc3e80a20c0 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): my global_id is 244151 2024-09-16T06:50:41.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c80035d0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271602717 0 0) 0x7fc3e8138660 con 0x7fc3e80a24a0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): my global_id is 234237 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80a6f90 con 0x7fc3e80a24a0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956862236 0 0) 0x7fc3e809e8b0 con 0x7fc3e813b6b0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): my global_id is 224317 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8138660 con 0x7fc3e813b6b0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2735361859 0 0) 0x7fc3c80035d0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e809e8b0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3dc001b70 con 0x7fc3e80a20c0 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): dump: 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.027 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232796686 0 0) 0x7fc3e80a6f90 con 0x7fc3e80a24a0 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c80035d0 con 0x7fc3e80a24a0 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3d8001b70 con 0x7fc3e80a24a0 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient(hunting): dump: 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2282240709 0 0) 0x7fc3e809e8b0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 >> v1:172.21.15.189:6789/0 conn(0x7fc3e813b6b0 legacy=0x7fc3e81d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 >> v1:172.21.15.76:6789/0 conn(0x7fc3e80a24a0 legacy=0x7fc3e81ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 monclient: found mon.c 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e813a0e0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 5 monclient: authenticate success, global_id 244151 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _renew_subs 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e813a3c0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _renew_subs 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e813b110 con 0x7fc3e80a20c0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3dc004bd0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc3dc004f60 con 0x7fc3e80a20c0 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 10 monclient: dump: 2024-09-16T06:50:41.030 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e4d99700 4 set_mon_vals no callback set 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc005100 con 0x7fc3e80a20c0 2024-09-16T06:50:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc3dc005140 con 0x7fc3e80a20c0 2024-09-16T06:50:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.017+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc3dc044090 con 0x7fc3e80a20c0 2024-09-16T06:50:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.021+0000 7fc3e5d9b700 10 monclient: get_auth_request con 0x7fc3c8036470 auth_method 0 2024-09-16T06:50:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.025+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc3dc014b70 con 0x7fc3e80a20c0 2024-09-16T06:50:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.025+0000 7fc3d6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.025+0000 7fc3d6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:41.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.261+0000 7fc3e759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:41.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.261+0000 7fc3e759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:41.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.261+0000 7fc3e759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.261+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3cc005930 con 0x7fc3e80a20c0 2024-09-16T06:50:41.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3d6ffd700 1 -- 172.21.15.76:0/3533713229 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1863630406) 0x7fc3dc012fe0 con 0x7fc3e80a20c0 2024-09-16T06:50:41.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3d6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:41.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3d6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:41.263 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:41.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc3c8036470 legacy=0x7fc3c8038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 10 monclient: shutdown 2024-09-16T06:50:41.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 >> v1:172.21.15.76:6790/0 conn(0x7fc3e80a20c0 legacy=0x7fc3e81352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 shutdown_connections 2024-09-16T06:50:41.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 >> 172.21.15.76:0/3533713229 conn(0x7fc3e80a41c0 msgr2=0x7fc3e80a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 shutdown_connections 2024-09-16T06:50:41.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.265+0000 7fc3e759e700 1 -- 172.21.15.76:0/3533713229 wait complete. 2024-09-16T06:50:41.277 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:50:41.277 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:50:41.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:50:41.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: monmap: 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:41.393022+0000 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:41.393022+0000 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:41.389 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 1 Processor -- start 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 1 -- start start 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: init 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7f2c0a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c0a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7f2c13b6b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c09b3e0 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c0a3450 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c09b1a0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2b59e700 10 monclient(hunting): authenticate will time out at 1588.527111s 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2ad9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7f2c13b6b0 0x7f7f2c13daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41568/0 (socket says 172.21.15.76:41568) 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2ad9d700 1 -- 172.21.15.76:0/989849383 learned_addr learned my addr 172.21.15.76:0/989849383 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634158814 0 0) 0x7f7f2c09b3e0 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): my global_id is 234240 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f180035d0 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805656042 0 0) 0x7f7f2c09b1a0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): my global_id is 244154 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c09b3e0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897798292 0 0) 0x7f7f2c0a3450 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.389+0000 7f7f2959a700 10 monclient(hunting): my global_id is 224320 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c09b1a0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3296993956 0 0) 0x7f7f2c09b3e0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 >> v1:172.21.15.189:6789/0 conn(0x7f7f2c0a24a0 legacy=0x7f7f2c0a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 >> v1:172.21.15.76:6789/0 conn(0x7f7f2c0a20c0 legacy=0x7f7f2c06cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 monclient: found mon.noname-b 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c06bd30 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 5 monclient: authenticate success, global_id 244154 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f24001970 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: dump: 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f24001b90 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 4 set_mon_vals no callback set 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 1 -- 172.21.15.76:0/989849383 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f24001ea0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2959a700 10 monclient: dump: 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.398 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: shutdown 2024-09-16T06:50:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- 172.21.15.76:0/989849383 >> v1:172.21.15.76:6790/0 conn(0x7f7f2c13b6b0 legacy=0x7f7f2c13daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- 172.21.15.76:0/989849383 shutdown_connections 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- 172.21.15.76:0/989849383 >> 172.21.15.76:0/989849383 conn(0x7f7f2c0a41c0 msgr2=0x7f7f2c0a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- 172.21.15.76:0/989849383 shutdown_connections 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- 172.21.15.76:0/989849383 wait complete. 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: monmap: 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:41.401 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 Processor -- start 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- start start 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: init 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7f2c0a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7f2c0a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7f2c13b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c0a19e0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c09e8b0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c139200 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f2b59e700 10 monclient(hunting): authenticate will time out at 1588.531349s 2024-09-16T06:50:41.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f29d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7f2c0a24a0 0x7f7f2c1ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45520/0 (socket says 172.21.15.76:45520) 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f29d9b700 1 -- 172.21.15.76:0/356099918 learned_addr learned my addr 172.21.15.76:0/356099918 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474094527 0 0) 0x7f7f2c09b1a0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 10 monclient(hunting): my global_id is 234243 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.393+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0c0035d0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270358328 0 0) 0x7f7f2c139200 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): my global_id is 244157 2024-09-16T06:50:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c09b1a0 con 0x7f7f2c0a20c0 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607986582 0 0) 0x7f7f0c0035d0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f2c139200 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f1c0017f0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): dump: 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905186991 0 0) 0x7f7f2c09e8b0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): my global_id is 224323 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0c0035d0 con 0x7f7f2c13b6b0 2024-09-16T06:50:41.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910378543 0 0) 0x7f7f2c139200 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6790/0 conn(0x7f7f2c0a20c0 legacy=0x7f7f2c1353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.189:6789/0 conn(0x7f7f2c13b6b0 legacy=0x7f7f2c1d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 monclient: found mon.a 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c1394e0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 5 monclient: authenticate success, global_id 234243 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 10 monclient: _renew_subs 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:41.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c0a1560 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 10 monclient: _renew_subs 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c09dc00 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f1c0036c0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f1c003a50 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f28d99700 4 set_mon_vals no callback set 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: dump: 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.412 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7f1c003c30 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f10006590 con 0x7f7f10002830 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7f1c043cb0 con 0x7f7f2c0a24a0 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f2ad9d700 10 monclient: get_auth_request con 0x7f7f0c0361d0 auth_method 0 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144036353 0 0) 0x7f7f10006590 con 0x7f7f10002830 2024-09-16T06:50:41.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient(hunting): my global_id is 234246 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c139200 con 0x7f7f10002830 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4076201552 0 0) 0x7f7f2c139200 con 0x7f7f10002830 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f10006590 con 0x7f7f10002830 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f14001d60 con 0x7f7f10002830 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 10 monclient: dump: 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.414 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2449896016 0 0) 0x7f7f10006590 con 0x7f7f10002830 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.397+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f1000a190 con 0x7f7f10002830 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.401+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7f14005ad0 con 0x7f7f10002830 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.401+0000 7f7f22ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.401+0000 7f7f22ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.401+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6789/0 conn(0x7f7f10002830 legacy=0x7f7f10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.401+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6789/0 conn(0x7f7f10002830 legacy=0x7f7f10004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:41.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f10009e30 con 0x7f7f10009150 2024-09-16T06:50:41.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075953510 0 0) 0x7f7f10009e30 con 0x7f7f10009150 2024-09-16T06:50:41.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient(hunting): my global_id is 234249 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10006590 con 0x7f7f10009150 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3959303496 0 0) 0x7f7f10006590 con 0x7f7f10009150 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f10009e30 con 0x7f7f10009150 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7f14003f50 con 0x7f7f10009150 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient: dump: 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.418 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2877714379 0 0) 0x7f7f10009e30 con 0x7f7f10009150 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f1000c340 con 0x7f7f10009150 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1398265656) 0x7f7f14005330 con 0x7f7f10009150 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:41.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6789/0 conn(0x7f7f10009150 legacy=0x7f7f100095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f22ffd700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6789/0 conn(0x7f7f10009150 legacy=0x7f7f100095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:41.420 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.421 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.422 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:41.423 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:41.424 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.425 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.426 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.428 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.429 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.430 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.432 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:41.432 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:41.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7f0c0361d0 legacy=0x7f7f0c038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 10 monclient: shutdown 2024-09-16T06:50:41.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 >> v1:172.21.15.76:6789/0 conn(0x7f7f2c0a24a0 legacy=0x7f7f2c1ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2ad9d700 1 -- 172.21.15.76:0/356099918 reap_dead start 2024-09-16T06:50:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2ad9d700 1 -- 172.21.15.76:0/356099918 reap_dead start 2024-09-16T06:50:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 shutdown_connections 2024-09-16T06:50:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 >> 172.21.15.76:0/356099918 conn(0x7f7f2c0a41c0 msgr2=0x7f7f2c13aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 shutdown_connections 2024-09-16T06:50:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.413+0000 7f7f2b59e700 1 -- 172.21.15.76:0/356099918 wait complete. 2024-09-16T06:50:41.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:41.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:50:41.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:50:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.553+0000 7f6e2ca1c700 10 monclient: get_monmap_and_config 2024-09-16T06:50:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.553+0000 7f6e2ca1c700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.553+0000 7f6e2ca1c700 10 monclient: monmap: 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:41.558652+0000 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:41.558652+0000 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:41.554 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 Processor -- start 2024-09-16T06:50:41.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- start start 2024-09-16T06:50:41.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: init 2024-09-16T06:50:41.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-b con 0x7f6e28132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-a con 0x7f6e2806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-c con 0x7f6e2806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2806c280 con 0x7f6e2806d480 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2806bfa0 con 0x7f6e2806cf50 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2809ace0 con 0x7f6e28132330 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient(hunting): authenticate will time out at 1588.692722s 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e26d9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6e28132330 0x7f6e28132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41580/0 (socket says 172.21.15.76:41580) 2024-09-16T06:50:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e26d9d700 1 -- 172.21.15.76:0/3546142501 learned_addr learned my addr 172.21.15.76:0/3546142501 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2659c700 1 --1- 172.21.15.76:0/3546142501 >> v1:172.21.15.76:6789/0 conn(0x7f6e2806d480 0x7f6e2813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45552/0 (socket says 172.21.15.76:45552) 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426558328 0 0) 0x7f6e2806c280 con 0x7f6e2806d480 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): my global_id is 234252 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e100035d0 con 0x7f6e2806d480 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237443895 0 0) 0x7f6e2809ace0 con 0x7f6e28132330 2024-09-16T06:50:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): my global_id is 244160 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2806c280 con 0x7f6e28132330 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159802002 0 0) 0x7f6e2806bfa0 con 0x7f6e2806cf50 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient(hunting): my global_id is 224326 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2809ace0 con 0x7f6e2806cf50 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2131282173 0 0) 0x7f6e100035d0 con 0x7f6e2806d480 2024-09-16T06:50:41.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 >> v1:172.21.15.76:6790/0 conn(0x7f6e28132330 legacy=0x7f6e28132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 >> v1:172.21.15.189:6789/0 conn(0x7f6e2806cf50 legacy=0x7f6e2813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 monclient: found mon.noname-a 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2809ae70 con 0x7f6e2806d480 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e1c001610 con 0x7f6e2806d480 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 5 monclient: authenticate success, global_id 234252 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: dump: 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e1c0018f0 con 0x7f6e2806d480 2024-09-16T06:50:41.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 1 -- 172.21.15.76:0/3546142501 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e1c001b20 con 0x7f6e2806d480 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 4 set_mon_vals no callback set 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e25d9b700 10 monclient: dump: 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.567 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: shutdown 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/3546142501 >> v1:172.21.15.76:6789/0 conn(0x7f6e2806d480 legacy=0x7f6e2813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/3546142501 shutdown_connections 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/3546142501 >> 172.21.15.76:0/3546142501 conn(0x7f6e280a41c0 msgr2=0x7f6e280a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/3546142501 shutdown_connections 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/3546142501 wait complete. 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.557+0000 7f6e2ca1c700 10 monclient: monmap: 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:41.568 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 Processor -- start 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- start start 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: init 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-b con 0x7f6e2806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-a con 0x7f6e2806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient(hunting): picked mon.noname-c con 0x7f6e28132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e28139310 con 0x7f6e2806d480 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e281361d0 con 0x7f6e2806cf50 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2809e8b0 con 0x7f6e28132330 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient(hunting): authenticate will time out at 1588.696640s 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2759e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6e28132330 0x7f6e281d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41584/0 (socket says 172.21.15.76:41584) 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2759e700 1 -- 172.21.15.76:0/1580852846 learned_addr learned my addr 172.21.15.76:0/1580852846 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2659c700 1 --1- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6789/0 conn(0x7f6e2806d480 0x7f6e28135ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45560/0 (socket says 172.21.15.76:45560) 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356246038 0 0) 0x7f6e2809e8b0 con 0x7f6e28132330 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): my global_id is 244163 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e140035d0 con 0x7f6e28132330 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414496558 0 0) 0x7f6e28139310 con 0x7f6e2806d480 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): my global_id is 234255 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2809e8b0 con 0x7f6e2806d480 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 142502778 0 0) 0x7f6e140035d0 con 0x7f6e28132330 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e28139310 con 0x7f6e28132330 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e20001c30 con 0x7f6e28132330 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:41.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799643181 0 0) 0x7f6e281361d0 con 0x7f6e2806cf50 2024-09-16T06:50:41.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): my global_id is 224329 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e140035d0 con 0x7f6e2806cf50 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390714100 0 0) 0x7f6e2809e8b0 con 0x7f6e2806d480 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e281361d0 con 0x7f6e2806d480 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e1c001000 con 0x7f6e2806d480 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): dump: 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2361195870 0 0) 0x7f6e28139310 con 0x7f6e28132330 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.189:6789/0 conn(0x7f6e2806cf50 legacy=0x7f6e281353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6789/0 conn(0x7f6e2806d480 legacy=0x7f6e28135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 monclient: found mon.c 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e281316c0 con 0x7f6e28132330 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 5 monclient: authenticate success, global_id 244163 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _renew_subs 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e28137900 con 0x7f6e28132330 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _renew_subs 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e28136fb0 con 0x7f6e28132330 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e20001fc0 con 0x7f6e28132330 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e20003e90 con 0x7f6e28132330 2024-09-16T06:50:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2559a700 4 set_mon_vals no callback set 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient: dump: 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e08006590 con 0x7f6e08002830 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6e200040b0 con 0x7f6e28132330 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6e200440a0 con 0x7f6e28132330 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020892333 0 0) 0x7f6e08006590 con 0x7f6e08002830 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 10 monclient(hunting): my global_id is 244166 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e28139310 con 0x7f6e08002830 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.561+0000 7f6e2659c700 10 monclient: get_auth_request con 0x7f6e14036450 auth_method 0 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2207267555 0 0) 0x7f6e28139310 con 0x7f6e08002830 2024-09-16T06:50:41.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e08006590 con 0x7f6e08002830 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e18001830 con 0x7f6e08002830 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 10 monclient: dump: 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.578 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2079451919 0 0) 0x7f6e08006590 con 0x7f6e08002830 2024-09-16T06:50:41.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e0800a190 con 0x7f6e08002830 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6e18001c00 con 0x7f6e08002830 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6790/0 conn(0x7f6e08002830 legacy=0x7f6e08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.565+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6790/0 conn(0x7f6e08002830 legacy=0x7f6e08004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e2ca1c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e08009e30 con 0x7f6e08009150 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956401526 0 0) 0x7f6e08009e30 con 0x7f6e08009150 2024-09-16T06:50:41.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient(hunting): my global_id is 244169 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08006590 con 0x7f6e08009150 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3078282318 0 0) 0x7f6e08006590 con 0x7f6e08009150 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e08009e30 con 0x7f6e08009150 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6e18004c20 con 0x7f6e08009150 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 10 monclient: dump: 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.582 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2051403645 0 0) 0x7f6e08009e30 con 0x7f6e08009150 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.573+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e0800c340 con 0x7f6e08009150 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 187312126) 0x7f6e18004e90 con 0x7f6e08009150 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e0f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e0f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6790/0 conn(0x7f6e08009150 legacy=0x7f6e080095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e0f7fe700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6790/0 conn(0x7f6e08009150 legacy=0x7f6e080095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.584 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.585 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.586 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:41.587 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.588 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:41.589 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:41.592 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.593 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:50:41.594 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.595 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:41.595 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:41.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6e14036450 legacy=0x7f6e14038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 10 monclient: shutdown 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 >> v1:172.21.15.76:6790/0 conn(0x7f6e28132330 legacy=0x7f6e281d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2759e700 1 -- 172.21.15.76:0/1580852846 reap_dead start 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2759e700 1 -- 172.21.15.76:0/1580852846 reap_dead start 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 shutdown_connections 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 >> 172.21.15.76:0/1580852846 conn(0x7f6e280a41c0 msgr2=0x7f6e280a0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 shutdown_connections 2024-09-16T06:50:41.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.577+0000 7f6e2ca1c700 1 -- 172.21.15.76:0/1580852846 wait complete. 2024-09-16T06:50:41.597 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:41.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:50:41.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:50:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: get_monmap_and_config 2024-09-16T06:50:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: monmap: 2024-09-16T06:50:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.715 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:41.720050+0000 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:41.720050+0000 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:41.716 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 Processor -- start 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- start start 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: init 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-c con 0x7f3d6009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-a con 0x7f3d6009d520 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-b con 0x7f3d6009fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6009b520 con 0x7f3d6009d520 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d60071f30 con 0x7f3d6009f230 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6009b0e0 con 0x7f3d6009fd20 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient(hunting): authenticate will time out at 1588.853368s 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3d6009fd20 0x7f3d6013ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41616/0 (socket says 172.21.15.76:41616) 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5f7fe700 1 -- 172.21.15.76:0/2337818709 learned_addr learned my addr 172.21.15.76:0/2337818709 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418115282 0 0) 0x7f3d6009b0e0 con 0x7f3d6009fd20 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): my global_id is 244172 2024-09-16T06:50:41.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d4c0035d0 con 0x7f3d6009fd20 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763599461 0 0) 0x7f3d6009b520 con 0x7f3d6009d520 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): my global_id is 234258 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6009b0e0 con 0x7f3d6009d520 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257863618 0 0) 0x7f3d60071f30 con 0x7f3d6009f230 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient(hunting): my global_id is 224332 2024-09-16T06:50:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6009b520 con 0x7f3d6009f230 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943811477 0 0) 0x7f3d4c0035d0 con 0x7f3d6009fd20 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 >> v1:172.21.15.189:6789/0 conn(0x7f3d6009f230 legacy=0x7f3d6009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 >> v1:172.21.15.76:6789/0 conn(0x7f3d6009d520 legacy=0x7f3d60137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6009c7d0 con 0x7f3d6009fd20 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d58001990 con 0x7f3d6009fd20 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: dump: 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 5 monclient: authenticate success, global_id 244172 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d58001c00 con 0x7f3d6009fd20 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 4 set_mon_vals no callback set 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 1 -- 172.21.15.76:0/2337818709 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d58001f90 con 0x7f3d6009fd20 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d5e7fc700 10 monclient: dump: 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.728 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 10 monclient: shutdown 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- 172.21.15.76:0/2337818709 >> v1:172.21.15.76:6790/0 conn(0x7f3d6009fd20 legacy=0x7f3d6013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- 172.21.15.76:0/2337818709 shutdown_connections 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- 172.21.15.76:0/2337818709 >> 172.21.15.76:0/2337818709 conn(0x7f3d600a3e50 msgr2=0x7f3d600a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- 172.21.15.76:0/2337818709 shutdown_connections 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.717+0000 7f3d657c4700 1 -- 172.21.15.76:0/2337818709 wait complete. 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: monmap: 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 Processor -- start 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- start start 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: init 2024-09-16T06:50:41.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-c con 0x7f3d6009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-a con 0x7f3d6009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient(hunting): picked mon.noname-b con 0x7f3d6011af00 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6011d790 con 0x7f3d6009fd20 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6011d1d0 con 0x7f3d6011af00 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6011bf90 con 0x7f3d6009f230 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient(hunting): authenticate will time out at 1588.857188s 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d5effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3d6009f230 0x7f3d6011a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41618/0 (socket says 172.21.15.76:41618) 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d5effd700 1 -- 172.21.15.76:0/3346390205 learned_addr learned my addr 172.21.15.76:0/3346390205 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d57fff700 1 --1- 172.21.15.76:0/3346390205 >> v1:172.21.15.76:6789/0 conn(0x7f3d6009fd20 0x7f3d6011a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45580/0 (socket says 172.21.15.76:45580) 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154429291 0 0) 0x7f3d6011bf90 con 0x7f3d6009f230 2024-09-16T06:50:41.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): my global_id is 244175 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d400035d0 con 0x7f3d6009f230 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804833638 0 0) 0x7f3d6011d790 con 0x7f3d6009fd20 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): my global_id is 234261 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6011bf90 con 0x7f3d6009fd20 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2495767301 0 0) 0x7f3d400035d0 con 0x7f3d6009f230 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d6011d790 con 0x7f3d6009f230 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d50001990 con 0x7f3d6009f230 2024-09-16T06:50:41.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): dump: 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:41.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479089035 0 0) 0x7f3d6011d1d0 con 0x7f3d6011af00 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): my global_id is 224335 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d400035d0 con 0x7f3d6011af00 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2458884158 0 0) 0x7f3d6011bf90 con 0x7f3d6009fd20 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d6011d1d0 con 0x7f3d6009fd20 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d48001870 con 0x7f3d6009fd20 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient(hunting): dump: 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.734 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 337904850 0 0) 0x7f3d6011d790 con 0x7f3d6009f230 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6789/0 conn(0x7f3d6011af00 legacy=0x7f3d601d21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.76:6789/0 conn(0x7f3d6009fd20 legacy=0x7f3d6011a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 monclient: found mon.c 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6013b530 con 0x7f3d6009f230 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 5 monclient: authenticate success, global_id 244175 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _renew_subs 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6011cef0 con 0x7f3d6009f230 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _renew_subs 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6011bcd0 con 0x7f3d6009f230 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d50001d40 con 0x7f3d6009f230 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d50003b10 con 0x7f3d6009f230 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d5dffb700 4 set_mon_vals no callback set 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 10 monclient: dump: 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.737 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3d50003c70 con 0x7f3d6009f230 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3d50043e50 con 0x7f3d6009f230 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d44006590 con 0x7f3d44002830 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.721+0000 7f3d57fff700 10 monclient: get_auth_request con 0x7f3d40036450 auth_method 0 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995515388 0 0) 0x7f3d44006590 con 0x7f3d44002830 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient(hunting): my global_id is 224338 2024-09-16T06:50:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6011d790 con 0x7f3d44002830 2024-09-16T06:50:41.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1618830498 0 0) 0x7f3d6011d790 con 0x7f3d44002830 2024-09-16T06:50:41.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d44006590 con 0x7f3d44002830 2024-09-16T06:50:41.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d580017b0 con 0x7f3d44002830 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient: dump: 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.741 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 415328042 0 0) 0x7f3d44006590 con 0x7f3d44002830 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d4400a190 con 0x7f3d44002830 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3d58007480 con 0x7f3d44002830 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6789/0 conn(0x7f3d44002830 legacy=0x7f3d44004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.725+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6789/0 conn(0x7f3d44002830 legacy=0x7f3d44004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d657c4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d44009e50 con 0x7f3d44009170 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527721296 0 0) 0x7f3d44009e50 con 0x7f3d44009170 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient(hunting): my global_id is 224341 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d44006590 con 0x7f3d44009170 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1699173801 0 0) 0x7f3d44006590 con 0x7f3d44009170 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d44009e50 con 0x7f3d44009170 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d580053d0 con 0x7f3d44009170 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:41.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 10 monclient: dump: 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.745 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2836398761 0 0) 0x7f3d44009e50 con 0x7f3d44009170 2024-09-16T06:50:41.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.737+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d4400c340 con 0x7f3d44009170 2024-09-16T06:50:41.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 4240556720) 0x7f3d580075f0 con 0x7f3d44009170 2024-09-16T06:50:41.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d577fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:41.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d577fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:41.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6789/0 conn(0x7f3d44009170 legacy=0x7f3d44009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d577fe700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6789/0 conn(0x7f3d44009170 legacy=0x7f3d44009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:41.748 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:41.748 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 108, 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:41.749 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 4, 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.750 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.751 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:41.752 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.753 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.754 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:41.755 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:41.756 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:41.757 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:41.758 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.759 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:41.760 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:41.761 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:41.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:41.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:41.766 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:41.767 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3d40036450 legacy=0x7f3d40038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 10 monclient: shutdown 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 >> v1:172.21.15.76:6790/0 conn(0x7f3d6009f230 legacy=0x7f3d6011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d5f7fe700 1 -- 172.21.15.76:0/3346390205 reap_dead start 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 shutdown_connections 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 >> 172.21.15.76:0/3346390205 conn(0x7f3d600a3e50 msgr2=0x7f3d600a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 shutdown_connections 2024-09-16T06:50:41.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.741+0000 7f3d657c4700 1 -- 172.21.15.76:0/3346390205 wait complete. 2024-09-16T06:50:41.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:50:41.769 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:50:41.769 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:50:41.769 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:50:41.769 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:50:41.769 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:50:41.770 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:50:41.822 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:50:41.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:50:41.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: get_monmap_and_config 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: monmap: 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:41.941881+0000 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:41.941881+0000 2024-09-16T06:50:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:41.938 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 1 Processor -- start 2024-09-16T06:50:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 1 -- start start 2024-09-16T06:50:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: init 2024-09-16T06:50:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-c con 0x7fa19c0973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-a con 0x7fa19c097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-b con 0x7fa19c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c099350 con 0x7fa19c097eb0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c09f690 con 0x7fa19c0973c0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c099bf0 con 0x7fa19c1400d0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a1e5d700 10 monclient(hunting): authenticate will time out at 1589.075178s 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa19c097eb0 legacy=0x7fa19c13e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a165c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa19c1400d0 0x7fa19c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41654/0 (socket says 172.21.15.76:41654) 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa1a165c700 1 -- 172.21.15.76:0/2970290027 learned_addr learned my addr 172.21.15.76:0/2970290027 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525398238 0 0) 0x7fa19c099bf0 con 0x7fa19c1400d0 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient(hunting): my global_id is 244178 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1840035d0 con 0x7fa19c1400d0 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 872027407 0 0) 0x7fa1840035d0 con 0x7fa19c1400d0 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 >> v1:172.21.15.189:6789/0 conn(0x7fa19c0973c0 legacy=0x7fa19c0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 >> v1:172.21.15.76:6789/0 conn(0x7fa19c097eb0 legacy=0x7fa19c13e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:41.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c098360 con 0x7fa19c1400d0 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.937+0000 7fa19b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa194001610 con 0x7fa19c1400d0 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: dump: 2024-09-16T06:50:41.944 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.946 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 5 monclient: authenticate success, global_id 244178 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa194001930 con 0x7fa19c1400d0 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 1 -- 172.21.15.76:0/2970290027 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa194001cc0 con 0x7fa19c1400d0 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa19b7fe700 10 monclient: dump: 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 4 set_mon_vals no callback set 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 10 monclient: shutdown 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/2970290027 >> v1:172.21.15.76:6790/0 conn(0x7fa19c1400d0 legacy=0x7fa19c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/2970290027 shutdown_connections 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/2970290027 >> 172.21.15.76:0/2970290027 conn(0x7fa19c0a0030 msgr2=0x7fa19c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/2970290027 shutdown_connections 2024-09-16T06:50:41.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/2970290027 wait complete. 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 10 monclient: build_initial_monmap 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 10 monclient: monmap: 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:41.949 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 Processor -- start 2024-09-16T06:50:41.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 1 -- start start 2024-09-16T06:50:41.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.941+0000 7fa1a1e5d700 10 monclient: init 2024-09-16T06:50:41.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-a con 0x7fa19c0973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-c con 0x7fa19c097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a0e5b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fa19c0973c0 legacy=0x7fa19c13c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient(hunting): picked mon.noname-b con 0x7fa19c1400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c072660 con 0x7fa19c0973c0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c06c4e0 con 0x7fa19c1400d0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19c0a3100 con 0x7fa19c097eb0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient(hunting): authenticate will time out at 1589.080838s 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa19bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa19c097eb0 0x7fa19c13ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41656/0 (socket says 172.21.15.76:41656) 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa19bfff700 1 -- 172.21.15.76:0/1282505108 learned_addr learned my addr 172.21.15.76:0/1282505108 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520986366 0 0) 0x7fa19c0a3100 con 0x7fa19c097eb0 2024-09-16T06:50:41.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): my global_id is 244181 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1880035d0 con 0x7fa19c097eb0 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453684593 0 0) 0x7fa19c06c4e0 con 0x7fa19c1400d0 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): my global_id is 224347 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c0a3100 con 0x7fa19c1400d0 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015457011 0 0) 0x7fa1880035d0 con 0x7fa19c097eb0 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa19c06c4e0 con 0x7fa19c097eb0 2024-09-16T06:50:41.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa18c004580 con 0x7fa19c097eb0 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): dump: 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2829405610 0 0) 0x7fa19c06c4e0 con 0x7fa19c097eb0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 >> v1:172.21.15.189:6789/0 conn(0x7fa19c1400d0 legacy=0x7fa19c1d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 >> v1:172.21.15.76:6789/0 conn(0x7fa19c0973c0 legacy=0x7fa19c13c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: _finish_hunting 0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 monclient: found mon.c 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c09a540 con 0x7fa19c097eb0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: _finish_auth 0 2024-09-16T06:50:41.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: _check_auth_tickets 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 5 monclient: authenticate success, global_id 244181 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _renew_subs 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19c13fd70 con 0x7fa19c097eb0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _renew_subs 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19c13e9d0 con 0x7fa19c097eb0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa18c004860 con 0x7fa19c097eb0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa19affd700 4 set_mon_vals no callback set 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa18c004bf0 con 0x7fa19c097eb0 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:41.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 10 monclient: dump: 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:41.960 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.945+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa18c004dd0 con 0x7fa19c097eb0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1a1e5d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1a1e5d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa18c044220 con 0x7fa19c097eb0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1a1e5d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c005100 con 0x7fa19c097eb0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.949+0000 7fa1a0e5b700 10 monclient: get_auth_request con 0x7fa188036180 auth_method 0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.953+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa18c014c20 con 0x7fa19c097eb0 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.953+0000 7fa1997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:41.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:41.953+0000 7fa1997fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:42.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1a1e5d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:42.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1a1e5d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:42.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1a1e5d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:42.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa17c005930 con 0x7fa19c097eb0 2024-09-16T06:50:42.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1997fa700 1 -- 172.21.15.76:0/1282505108 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1274460392) 0x7fa18c01d210 con 0x7fa19c097eb0 2024-09-16T06:50:42.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:42.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.189+0000 7fa1997fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:42.191 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa188036180 legacy=0x7fa188038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 10 monclient: shutdown 2024-09-16T06:50:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 >> v1:172.21.15.76:6790/0 conn(0x7fa19c097eb0 legacy=0x7fa19c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 shutdown_connections 2024-09-16T06:50:42.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 >> 172.21.15.76:0/1282505108 conn(0x7fa19c0a0030 msgr2=0x7fa19c0963f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:42.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 shutdown_connections 2024-09-16T06:50:42.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:42.193+0000 7fa1a1e5d700 1 -- 172.21.15.76:0/1282505108 wait complete. 2024-09-16T06:50:45.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.325+0000 7f3ef92ad700 10 monclient: get_monmap_and_config 2024-09-16T06:50:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.325+0000 7f3ef92ad700 10 monclient: build_initial_monmap 2024-09-16T06:50:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: monmap: 2024-09-16T06:50:45.327 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:45.331953+0000 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:45.331953+0000 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 Processor -- start 2024-09-16T06:50:45.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- start start 2024-09-16T06:50:45.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: init 2024-09-16T06:50:45.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:45.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:50:45.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-c con 0x7f3ef4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:45.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-a con 0x7f3ef4099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-b con 0x7f3ef4139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef4098990 con 0x7f3ef4099090 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef4097f40 con 0x7f3ef4097b60 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef409d610 con 0x7f3ef4139500 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient(hunting): authenticate will time out at 1592.465240s 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef37fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3ef4099090 legacy=0x7f3ef409f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef8aac700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3ef4139500 0x7f3ef413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58844/0 (socket says 172.21.15.76:58844) 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef8aac700 1 -- 172.21.15.76:0/2443657223 learned_addr learned my addr 172.21.15.76:0/2443657223 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971683272 0 0) 0x7f3ef409d610 con 0x7f3ef4139500 2024-09-16T06:50:45.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient(hunting): my global_id is 244184 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3edc0035d0 con 0x7f3ef4139500 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218187558 0 0) 0x7f3edc0035d0 con 0x7f3ef4139500 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 >> v1:172.21.15.189:6789/0 conn(0x7f3ef4097b60 legacy=0x7f3ef409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 >> v1:172.21.15.76:6789/0 conn(0x7f3ef4099090 legacy=0x7f3ef409f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 monclient: found mon.noname-b 2024-09-16T06:50:45.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef40986b0 con 0x7f3ef4139500 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 5 monclient: authenticate success, global_id 244184 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3eec001990 con 0x7f3ef4139500 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: dump: 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:45.339 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3eec001cb0 con 0x7f3ef4139500 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:45.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 1 -- 172.21.15.76:0/2443657223 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3eec002080 con 0x7f3ef4139500 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 4 set_mon_vals no callback set 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef2ffd700 10 monclient: dump: 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:45.341 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 10 monclient: shutdown 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2443657223 >> v1:172.21.15.76:6790/0 conn(0x7f3ef4139500 legacy=0x7f3ef413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2443657223 shutdown_connections 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2443657223 >> 172.21.15.76:0/2443657223 conn(0x7f3ef40a0030 msgr2=0x7f3ef40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2443657223 shutdown_connections 2024-09-16T06:50:45.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.329+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2443657223 wait complete. 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: build_initial_monmap 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: monmap: 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 Processor -- start 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- start start 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: init 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-b con 0x7f3ef4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-c con 0x7f3ef4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient(hunting): picked mon.noname-a con 0x7f3ef4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef409c390 con 0x7f3ef4139500 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef4071c50 con 0x7f3ef4097b60 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef409a540 con 0x7f3ef4099090 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient(hunting): authenticate will time out at 1592.469831s 2024-09-16T06:50:45.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef8aac700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3ef4139500 legacy=0x7f3ef41d2120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3ef4099090 0x7f3ef40709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58860/0 (socket says 172.21.15.76:58860) 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef37fe700 1 -- 172.21.15.76:0/2259838276 learned_addr learned my addr 172.21.15.76:0/2259838276 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072940225 0 0) 0x7f3ef409a540 con 0x7f3ef4099090 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): my global_id is 244187 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ee00035d0 con 0x7f3ef4099090 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 597959272 0 0) 0x7f3ee00035d0 con 0x7f3ef4099090 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ef409a540 con 0x7f3ef4099090 2024-09-16T06:50:45.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3ee8004200 con 0x7f3ef4099090 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): dump: 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:45.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3589958004 0 0) 0x7f3ef409a540 con 0x7f3ef4099090 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 >> v1:172.21.15.189:6789/0 conn(0x7f3ef4097b60 legacy=0x7f3ef40702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 >> v1:172.21.15.76:6789/0 conn(0x7f3ef4139500 legacy=0x7f3ef41d2120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 monclient: found mon.c 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef409bdd0 con 0x7f3ef4099090 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:50:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 5 monclient: authenticate success, global_id 244187 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _renew_subs 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef409c0b0 con 0x7f3ef4099090 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _renew_subs 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef40716d0 con 0x7f3ef4099090 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ee8004520 con 0x7f3ef4099090 2024-09-16T06:50:45.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3ee80048b0 con 0x7f3ef4099090 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef27fc700 4 set_mon_vals no callback set 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.333+0000 7f3ef0ff9700 10 monclient: dump: 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:45.349 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3ee8004ad0 con 0x7f3ef4099090 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef92ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef92ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef92ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed4005100 con 0x7f3ef4099090 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3ee8043f80 con 0x7f3ef4099090 2024-09-16T06:50:45.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.337+0000 7f3ef3fff700 10 monclient: get_auth_request con 0x7f3ee00360a0 auth_method 0 2024-09-16T06:50:45.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.341+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3ee80148c0 con 0x7f3ef4099090 2024-09-16T06:50:45.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.341+0000 7f3ef0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:45.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.341+0000 7f3ef0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:45.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef92ad700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:45.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef92ad700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:45.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef92ad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:45.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3ed4005930 con 0x7f3ef4099090 2024-09-16T06:50:45.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef0ff9700 1 -- 172.21.15.76:0/2259838276 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2265526104) 0x7f3ee8014690 con 0x7f3ef4099090 2024-09-16T06:50:45.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:45.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.577+0000 7f3ef0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:45.577 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3ee00360a0 legacy=0x7f3ee0038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 10 monclient: shutdown 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 >> v1:172.21.15.76:6790/0 conn(0x7f3ef4099090 legacy=0x7f3ef40709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 shutdown_connections 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 >> 172.21.15.76:0/2259838276 conn(0x7f3ef40a0030 msgr2=0x7f3ef40a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 shutdown_connections 2024-09-16T06:50:45.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:45.581+0000 7f3ef92ad700 1 -- 172.21.15.76:0/2259838276 wait complete. 2024-09-16T06:50:48.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: get_monmap_and_config 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: build_initial_monmap 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: monmap: 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:48.711 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:48.716285+0000 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:48.716285+0000 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:48.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 1 Processor -- start 2024-09-16T06:50:48.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 1 -- start start 2024-09-16T06:50:48.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: init 2024-09-16T06:50:48.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:48.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:48.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-a con 0x7fc1480b8530 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-b con 0x7fc1480b8910 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc14f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc1480241b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-c con 0x7fc1480c1dd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148022f30 con 0x7fc1480b8530 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148021c80 con 0x7fc1480c1dd0 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc14effd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1480b8910 0x7fc1480c06b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:58870/0 (socket says 172.21.15.76:58870) 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc14effd700 1 -- 172.21.15.76:0/3790168848 learned_addr learned my addr 172.21.15.76:0/3790168848 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148023210 con 0x7fc1480b8910 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:48.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.713+0000 7fc154d8b700 10 monclient(hunting): authenticate will time out at 1595.851809s 2024-09-16T06:50:48.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:48.917+0000 7fc14f7fe700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc1480241b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:49.321+0000 7fc14f7fe700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc1480241b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:50.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:50.125+0000 7fc14f7fe700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc1480241b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): tick 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6790/0 conn(0x7fc1480b8910 legacy=0x7fc1480c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.189:6789/0 conn(0x7fc1480c1dd0 legacy=0x7fc1480c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc1480241b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-c con 0x7fc134003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-b con 0x7fc1340073d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-a con 0x7fc1480b8530 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14ffff700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148026140 con 0x7fc1480b8530 2024-09-16T06:50:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148022f30 con 0x7fc134003740 2024-09-16T06:50:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.713+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc134010920 con 0x7fc1340073d0 2024-09-16T06:50:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.717+0000 7fc14d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:51.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:51.917+0000 7fc14ffff700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:52.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:52.321+0000 7fc14ffff700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:53.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:53.125+0000 7fc14ffff700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): tick 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6790/0 conn(0x7fc1340073d0 legacy=0x7fc134009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.189:6789/0 conn(0x7fc134003740 legacy=0x7fc134005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-b con 0x7fc134010c30 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-a con 0x7fc1480b8530 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): picked mon.noname-c con 0x7fc134012e30 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc134011b30 con 0x7fc1480b8530 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc148026140 con 0x7fc134012e30 2024-09-16T06:50:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc134005b90 con 0x7fc134010c30 2024-09-16T06:50:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14d7fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.717+0000 7fc14effd700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:54.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:54.921+0000 7fc14effd700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:55.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:55.325+0000 7fc14effd700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.129+0000 7fc14effd700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893552422 0 0) 0x7fc148026140 con 0x7fc134012e30 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient(hunting): my global_id is 224350 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1400035d0 con 0x7fc134012e30 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4115321821 0 0) 0x7fc1400035d0 con 0x7fc134012e30 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6790/0 conn(0x7fc134010c30 legacy=0x7fc134011070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.76:6789/0 conn(0x7fc1480b8530 legacy=0x7fc13400d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:56.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 monclient: found mon.noname-c 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc134005d90 con 0x7fc134012e30 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc15004ec20 con 0x7fc134012e30 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 5 monclient: authenticate success, global_id 224350 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: dump: 2024-09-16T06:50:56.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc15004ef50 con 0x7fc134012e30 2024-09-16T06:50:56.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 1 -- 172.21.15.76:0/3790168848 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc15006a310 con 0x7fc134012e30 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 4 set_mon_vals no callback set 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc14e7fc700 10 monclient: dump: 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.755 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:56.756 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 10 monclient: shutdown 2024-09-16T06:50:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:56.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 >> v1:172.21.15.189:6789/0 conn(0x7fc134012e30 legacy=0x7fc134006360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 shutdown_connections 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 >> 172.21.15.76:0/3790168848 conn(0x7fc148026f70 msgr2=0x7fc148027380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 shutdown_connections 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 1 -- 172.21.15.76:0/3790168848 wait complete. 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.753+0000 7fc154d8b700 10 monclient: build_initial_monmap 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: monmap: 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:56.761 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 Processor -- start 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- start start 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: init 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-c con 0x7fc134010c30 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-a con 0x7fc134012e30 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient(hunting): picked mon.noname-b con 0x7fc1480b8530 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1480bfca0 con 0x7fc134012e30 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc14effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc134012e30 legacy=0x7fc1481554f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1480ba8e0 con 0x7fc1480b8530 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1480bf270 con 0x7fc134010c30 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient(hunting): authenticate will time out at 1603.894119s 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc14f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc134010c30 0x7fc1480b7440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41450/0 (socket says 172.21.15.76:41450) 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc14f7fe700 1 -- 172.21.15.76:0/643819802 learned_addr learned my addr 172.21.15.76:0/643819802 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417533720 0 0) 0x7fc134005d90 con 0x7fc1480b8530 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): my global_id is 224353 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc134011840 con 0x7fc1480b8530 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2817093763 0 0) 0x7fc134011840 con 0x7fc1480b8530 2024-09-16T06:50:56.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc134005d90 con 0x7fc1480b8530 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc15004f5c0 con 0x7fc1480b8530 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): dump: 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:56.764 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2664740096 0 0) 0x7fc134005d90 con 0x7fc1480b8530 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 >> v1:172.21.15.76:6790/0 conn(0x7fc134010c30 legacy=0x7fc1480b7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 >> v1:172.21.15.76:6789/0 conn(0x7fc134012e30 legacy=0x7fc1481554f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 monclient: found mon.b 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1480bcec0 con 0x7fc1480b8530 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient: _finish_auth 0 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:50:56.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc13bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 5 monclient: authenticate success, global_id 224353 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _renew_subs 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1480c1950 con 0x7fc1480b8530 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _renew_subs 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.757+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1480bef90 con 0x7fc1480b8530 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc150069400 con 0x7fc1480b8530 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc150066de0 con 0x7fc1480b8530 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc14dffb700 4 set_mon_vals no callback set 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 10 monclient: dump: 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:56.768 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc150066f40 con 0x7fc1480b8530 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc154d8b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc154d8b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc154d8b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:50:56.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc130005100 con 0x7fc1480b8530 2024-09-16T06:50:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc1500bd8c0 con 0x7fc1480b8530 2024-09-16T06:50:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.761+0000 7fc14f7fe700 10 monclient: get_auth_request con 0x7fc134037b30 auth_method 0 2024-09-16T06:50:56.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.769+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc150069620 con 0x7fc1480b8530 2024-09-16T06:50:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.769+0000 7fc13bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:56.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:56.769+0000 7fc13bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:57.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.001+0000 7fc154d8b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:50:57.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.001+0000 7fc154d8b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:57.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.001+0000 7fc154d8b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:50:57.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.001+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc130005930 con 0x7fc1480b8530 2024-09-16T06:50:57.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.001+0000 7fc13bfff700 1 -- 172.21.15.76:0/643819802 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3084341871) 0x7fc150066b40 con 0x7fc1480b8530 2024-09-16T06:50:57.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc13bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:50:57.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc13bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:57.003 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc134037b30 legacy=0x7fc134039ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 10 monclient: shutdown 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 >> v1:172.21.15.189:6789/0 conn(0x7fc1480b8530 legacy=0x7fc148158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 shutdown_connections 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 >> 172.21.15.76:0/643819802 conn(0x7fc148026f70 msgr2=0x7fc148029290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 shutdown_connections 2024-09-16T06:50:57.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.005+0000 7fc154d8b700 1 -- 172.21.15.76:0/643819802 wait complete. 2024-09-16T06:50:57.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:50:57.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:50:57.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:50:57.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: get_monmap_and_config 2024-09-16T06:50:57.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: build_initial_monmap 2024-09-16T06:50:57.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: monmap: 2024-09-16T06:50:57.132 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:57.137641+0000 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:57.137641+0000 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:57.133 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 1 Processor -- start 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 1 -- start start 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: init 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient(hunting): picked mon.noname-c con 0x7fd92009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient(hunting): picked mon.noname-b con 0x7fd92009c460 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient(hunting): picked mon.noname-a con 0x7fd92009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92009a040 con 0x7fd92009fd20 2024-09-16T06:50:57.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92009ac80 con 0x7fd92009f230 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92009def0 con 0x7fd92009c460 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 10 monclient(hunting): authenticate will time out at 1604.270866s 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd92009fd20 legacy=0x7fd920142160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd917fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd92009c460 0x7fd92013ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41460/0 (socket says 172.21.15.76:41460) 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd917fff700 1 -- 172.21.15.76:0/132335081 learned_addr learned my addr 172.21.15.76:0/132335081 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186469418 0 0) 0x7fd92009def0 con 0x7fd92009c460 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient(hunting): my global_id is 244190 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c0035d0 con 0x7fd92009c460 2024-09-16T06:50:57.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813882533 0 0) 0x7fd90c0035d0 con 0x7fd92009c460 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 >> v1:172.21.15.189:6789/0 conn(0x7fd92009f230 legacy=0x7fd92009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 >> v1:172.21.15.76:6789/0 conn(0x7fd92009fd20 legacy=0x7fd920142160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92009a410 con 0x7fd92009c460 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd925630700 5 monclient: authenticate success, global_id 244190 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd910001610 con 0x7fd92009c460 2024-09-16T06:50:57.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.133+0000 7fd91e7fc700 10 monclient: dump: 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd910001930 con 0x7fd92009c460 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 1 -- 172.21.15.76:0/132335081 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd910001cc0 con 0x7fd92009c460 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 4 set_mon_vals no callback set 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91e7fc700 10 monclient: dump: 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.140 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: shutdown 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- 172.21.15.76:0/132335081 >> v1:172.21.15.76:6790/0 conn(0x7fd92009c460 legacy=0x7fd92013ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- 172.21.15.76:0/132335081 shutdown_connections 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- 172.21.15.76:0/132335081 >> 172.21.15.76:0/132335081 conn(0x7fd9200a3e50 msgr2=0x7fd9200a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- 172.21.15.76:0/132335081 shutdown_connections 2024-09-16T06:50:57.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- 172.21.15.76:0/132335081 wait complete. 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: build_initial_monmap 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: monmap: 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:57.146 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 Processor -- start 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- start start 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: init 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient(hunting): picked mon.noname-b con 0x7fd92009c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient(hunting): picked mon.noname-c con 0x7fd92009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient(hunting): picked mon.noname-a con 0x7fd92009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920072450 con 0x7fd92009fd20 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920138e90 con 0x7fd92009c460 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd91f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd92009fd20 legacy=0x7fd9201d66d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200a2760 con 0x7fd92009f230 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd925630700 10 monclient(hunting): authenticate will time out at 1604.275807s 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd917fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd92009f230 0x7fd9201377a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41464/0 (socket says 172.21.15.76:41464) 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.137+0000 7fd917fff700 1 -- 172.21.15.76:0/2062996488 learned_addr learned my addr 172.21.15.76:0/2062996488 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844436497 0 0) 0x7fd9200a2760 con 0x7fd92009f230 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): my global_id is 244193 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9000035d0 con 0x7fd92009f230 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467988182 0 0) 0x7fd920138e90 con 0x7fd92009c460 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): my global_id is 224359 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9200a2760 con 0x7fd92009c460 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994279242 0 0) 0x7fd9000035d0 con 0x7fd92009f230 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd920138e90 con 0x7fd92009f230 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd9100015d0 con 0x7fd92009f230 2024-09-16T06:50:57.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): dump: 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 850289517 0 0) 0x7fd920138e90 con 0x7fd92009f230 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.189:6789/0 conn(0x7fd92009c460 legacy=0x7fd920137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6789/0 conn(0x7fd92009fd20 legacy=0x7fd9201d66d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 monclient: found mon.c 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920138980 con 0x7fd92009f230 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:57.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 5 monclient: authenticate success, global_id 244193 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 10 monclient: _renew_subs 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd920139990 con 0x7fd92009f230 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 10 monclient: _renew_subs 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd920142820 con 0x7fd92009f230 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd910004cf0 con 0x7fd92009f230 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd910005000 con 0x7fd92009f230 2024-09-16T06:50:57.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd91dffb700 4 set_mon_vals no callback set 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: dump: 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd91000fa70 con 0x7fd92009f230 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd904006590 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd910044090 con 0x7fd92009f230 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd91f7fe700 10 monclient: get_auth_request con 0x7fd9000361d0 auth_method 0 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883358009 0 0) 0x7fd904006590 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient(hunting): my global_id is 244196 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920138e90 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1275875871 0 0) 0x7fd920138e90 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd904006590 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd9080016b0 con 0x7fd904002830 2024-09-16T06:50:57.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 10 monclient: dump: 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4042072397 0 0) 0x7fd904006590 con 0x7fd904002830 2024-09-16T06:50:57.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.141+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd90400a190 con 0x7fd904002830 2024-09-16T06:50:57.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.145+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd908001c90 con 0x7fd904002830 2024-09-16T06:50:57.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.145+0000 7fd9177fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:57.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.145+0000 7fd9177fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:57.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.145+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6790/0 conn(0x7fd904002830 legacy=0x7fd904004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.145+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6790/0 conn(0x7fd904002830 legacy=0x7fd904004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:57.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd925630700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:57.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd904009e30 con 0x7fd904009150 2024-09-16T06:50:57.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033809577 0 0) 0x7fd904009e30 con 0x7fd904009150 2024-09-16T06:50:57.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient(hunting): my global_id is 244199 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904006590 con 0x7fd904009150 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1561622836 0 0) 0x7fd904006590 con 0x7fd904009150 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd904009e30 con 0x7fd904009150 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd908003a50 con 0x7fd904009150 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient: dump: 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.158 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1664169849 0 0) 0x7fd904009e30 con 0x7fd904009150 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd90400c340 con 0x7fd904009150 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4037 (unknown 0 0 2821698898) 0x7fd908004fd0 con 0x7fd904009150 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6790/0 conn(0x7fd904009150 legacy=0x7fd9040095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.153+0000 7fd9177fe700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6790/0 conn(0x7fd904009150 legacy=0x7fd9040095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 112, 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:57.160 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.161 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.162 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:57.163 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.164 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.165 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.167 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:57.168 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:57.169 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.170 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:57.171 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:57.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd9000361d0 legacy=0x7fd900038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 10 monclient: shutdown 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 >> v1:172.21.15.76:6790/0 conn(0x7fd92009f230 legacy=0x7fd9201377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd91f7fe700 1 -- 172.21.15.76:0/2062996488 reap_dead start 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 shutdown_connections 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 >> 172.21.15.76:0/2062996488 conn(0x7fd9200a3e50 msgr2=0x7fd92009c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 shutdown_connections 2024-09-16T06:50:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.157+0000 7fd925630700 1 -- 172.21.15.76:0/2062996488 wait complete. 2024-09-16T06:50:57.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:50:57.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:50:57.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:50:57.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 10 monclient: get_monmap_and_config 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 10 monclient: build_initial_monmap 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 10 monclient: monmap: 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:50:57.298893+0000 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:50:57.298893+0000 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:50:57.294 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 1 Processor -- start 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 1 -- start start 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.293+0000 7fc87ae6b700 10 monclient: init 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:50:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-a con 0x7fc87409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-b con 0x7fc87409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-c con 0x7fc874139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87409cd80 con 0x7fc87409f230 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc874099fd0 con 0x7fc874139b70 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87409a410 con 0x7fc87409fd20 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient(hunting): authenticate will time out at 1604.432449s 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc878c07700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc87409f230 legacy=0x7fc87409f610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc87409fd20 0x7fc874138450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41474/0 (socket says 172.21.15.76:41474) 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86bfff700 1 -- 172.21.15.76:0/4275301531 learned_addr learned my addr 172.21.15.76:0/4275301531 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009688020 0 0) 0x7fc87409a410 con 0x7fc87409fd20 2024-09-16T06:50:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): my global_id is 244202 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c0035d0 con 0x7fc87409fd20 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005696227 0 0) 0x7fc874099fd0 con 0x7fc874139b70 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient(hunting): my global_id is 224362 2024-09-16T06:50:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc87409a410 con 0x7fc874139b70 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1355719812 0 0) 0x7fc85c0035d0 con 0x7fc87409fd20 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 >> v1:172.21.15.189:6789/0 conn(0x7fc874139b70 legacy=0x7fc87413bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 >> v1:172.21.15.76:6789/0 conn(0x7fc87409f230 legacy=0x7fc87409f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:50:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87409de90 con 0x7fc87409fd20 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 5 monclient: authenticate success, global_id 244202 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc864001610 con 0x7fc87409fd20 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: dump: 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.302 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc864001830 con 0x7fc87409fd20 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient: get_monmap_and_config success 2024-09-16T06:50:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 4 set_mon_vals no callback set 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 1 -- 172.21.15.76:0/4275301531 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc864001b40 con 0x7fc87409fd20 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc86b7fe700 10 monclient: dump: 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.304 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 10 monclient: shutdown 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- 172.21.15.76:0/4275301531 >> v1:172.21.15.76:6790/0 conn(0x7fc87409fd20 legacy=0x7fc874138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- 172.21.15.76:0/4275301531 shutdown_connections 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- 172.21.15.76:0/4275301531 >> 172.21.15.76:0/4275301531 conn(0x7fc8740a3e50 msgr2=0x7fc8740a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- 172.21.15.76:0/4275301531 shutdown_connections 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.297+0000 7fc87ae6b700 1 -- 172.21.15.76:0/4275301531 wait complete. 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient: build_initial_monmap 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient: monmap: 2024-09-16T06:50:57.305 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:50:57.306 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 1 Processor -- start 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 1 -- start start 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient: init 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient: _reopen_session rank -1 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-b con 0x7fc87409f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-c con 0x7fc87409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient(hunting): picked mon.noname-a con 0x7fc874139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87411cfc0 con 0x7fc874139b70 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc874134030 con 0x7fc87409f230 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc879408700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc874139b70 legacy=0x7fc8741d25f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:50:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc874072450 con 0x7fc87409fd20 2024-09-16T06:50:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient(hunting): _renew_subs 2024-09-16T06:50:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc87ae6b700 10 monclient(hunting): authenticate will time out at 1604.438476s 2024-09-16T06:50:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc86bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc87409fd20 0x7fc8741ceec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41482/0 (socket says 172.21.15.76:41482) 2024-09-16T06:50:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc86bfff700 1 -- 172.21.15.76:0/3318630759 learned_addr learned my addr 172.21.15.76:0/3318630759 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694889957 0 0) 0x7fc874072450 con 0x7fc87409fd20 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): my global_id is 244205 2024-09-16T06:50:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8600035d0 con 0x7fc87409fd20 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53593975 0 0) 0x7fc874134030 con 0x7fc87409f230 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): my global_id is 224365 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874072450 con 0x7fc87409f230 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997976152 0 0) 0x7fc8600035d0 con 0x7fc87409fd20 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc874134030 con 0x7fc87409fd20 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8640015d0 con 0x7fc87409fd20 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): dump: 2024-09-16T06:50:57.312 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3055855540 0 0) 0x7fc874072450 con 0x7fc87409f230 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8600035d0 con 0x7fc87409f230 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc870004170 con 0x7fc87409f230 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 10 monclient(hunting): dump: 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.301+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2832280196 0 0) 0x7fc874134030 con 0x7fc87409fd20 2024-09-16T06:50:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6789/0 conn(0x7fc87409f230 legacy=0x7fc874133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.76:6789/0 conn(0x7fc874139b70 legacy=0x7fc8741d25f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: _finish_hunting 0 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 monclient: found mon.c 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87411ca00 con 0x7fc87409fd20 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: _finish_auth 0 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: _check_auth_tickets 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 5 monclient: authenticate success, global_id 244205 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 10 monclient: _renew_subs 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87411cce0 con 0x7fc87409fd20 2024-09-16T06:50:57.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 10 monclient: _renew_subs 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87411bc60 con 0x7fc87409fd20 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc864004b40 con 0x7fc87409fd20 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc864004ed0 con 0x7fc87409fd20 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: dump: 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.316 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc86a7fc700 4 set_mon_vals no callback set 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc854006590 con 0x7fc854002830 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc8640050b0 con 0x7fc87409fd20 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc8640440e0 con 0x7fc87409fd20 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc879408700 10 monclient: get_auth_request con 0x7fc8600362c0 auth_method 0 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525068864 0 0) 0x7fc854006590 con 0x7fc854002830 2024-09-16T06:50:57.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient(hunting): my global_id is 224368 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874134030 con 0x7fc854002830 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3006227757 0 0) 0x7fc874134030 con 0x7fc854002830 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc854006590 con 0x7fc854002830 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8700047f0 con 0x7fc854002830 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 10 monclient: dump: 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.318 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1616593958 0 0) 0x7fc854006590 con 0x7fc854002830 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.305+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc85400a190 con 0x7fc854002830 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.309+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc870006ba0 con 0x7fc854002830 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.309+0000 7fc8697fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:50:57.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.309+0000 7fc8697fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:50:57.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.309+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6789/0 conn(0x7fc854002830 legacy=0x7fc854004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.309+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6789/0 conn(0x7fc854002830 legacy=0x7fc854004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:57.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc87ae6b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc854009e30 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220166713 0 0) 0x7fc854009e30 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient(hunting): my global_id is 224371 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc854006590 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3749759026 0 0) 0x7fc854006590 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc854009e30 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc870005040 con 0x7fc854009150 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:50:57.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.317+0000 7fc8697fa700 10 monclient: dump: 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3007123860 0 0) 0x7fc854009e30 con 0x7fc854009150 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc85400c340 con 0x7fc854009150 2024-09-16T06:50:57.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 2567082832) 0x7fc870006440 con 0x7fc854009150 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6789/0 conn(0x7fc854009150 legacy=0x7fc8540095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc8697fa700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6789/0 conn(0x7fc854009150 legacy=0x7fc8540095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:57.324 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 112, 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.325 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.326 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:50:57.327 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:50:57.328 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:50:57.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:50:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:57.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.335 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:50:57.336 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc8600362c0 legacy=0x7fc860038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 10 monclient: shutdown 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 >> v1:172.21.15.76:6790/0 conn(0x7fc87409fd20 legacy=0x7fc8741ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc879408700 1 -- 172.21.15.76:0/3318630759 reap_dead start 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc879408700 1 -- 172.21.15.76:0/3318630759 reap_dead start 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 shutdown_connections 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 >> 172.21.15.76:0/3318630759 conn(0x7fc8740a3e50 msgr2=0x7fc87409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 shutdown_connections 2024-09-16T06:50:57.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:50:57.321+0000 7fc87ae6b700 1 -- 172.21.15.76:0/3318630759 wait complete. 2024-09-16T06:50:57.339 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:50:57.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:51:17.340 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:51:17.340 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:51:17.340 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:51:17.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:51:17.344 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:51:17.344 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:51:17.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:17.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.452+0000 7fc41d7e3700 10 monclient: get_monmap_and_config 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.452+0000 7fc41d7e3700 10 monclient: build_initial_monmap 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.452+0000 7fc41d7e3700 10 monclient: monmap: 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:17.459275+0000 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:17.459275+0000 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:17.456 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.452+0000 7fc41d7e3700 1 Processor -- start 2024-09-16T06:51:17.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- start start 2024-09-16T06:51:17.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient: init 2024-09-16T06:51:17.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-a con 0x7fc41809f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-c con 0x7fc418071a70 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-b con 0x7fc41809fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418071400 con 0x7fc41809f230 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418099d60 con 0x7fc418071a70 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41809a040 con 0x7fc41809fd20 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient(hunting): authenticate will time out at 1624.593001s 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc416ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc41809f230 0x7fc41809f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33970/0 (socket says 172.21.15.76:33970) 2024-09-16T06:51:17.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc416ffd700 1 -- 172.21.15.76:0/581944119 learned_addr learned my addr 172.21.15.76:0/581944119 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514556435 0 0) 0x7fc41809a040 con 0x7fc41809fd20 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient(hunting): my global_id is 244208 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4040035d0 con 0x7fc41809fd20 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982559272 0 0) 0x7fc4040035d0 con 0x7fc41809fd20 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 >> v1:172.21.15.189:6789/0 conn(0x7fc418071a70 legacy=0x7fc41813a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 >> v1:172.21.15.76:6789/0 conn(0x7fc41809f230 legacy=0x7fc41809f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 monclient: found mon.noname-b 2024-09-16T06:51:17.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41809a320 con 0x7fc41809fd20 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc410001990 con 0x7fc41809fd20 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: dump: 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.462 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 5 monclient: authenticate success, global_id 244208 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc410001cb0 con 0x7fc41809fd20 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:17.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 1 -- 172.21.15.76:0/581944119 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc410002040 con 0x7fc41809fd20 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 4 set_mon_vals no callback set 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc4167fc700 10 monclient: dump: 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.464 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 10 monclient: shutdown 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- 172.21.15.76:0/581944119 >> v1:172.21.15.76:6790/0 conn(0x7fc41809fd20 legacy=0x7fc41813dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- 172.21.15.76:0/581944119 shutdown_connections 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- 172.21.15.76:0/581944119 >> 172.21.15.76:0/581944119 conn(0x7fc4180a3e50 msgr2=0x7fc4180a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- 172.21.15.76:0/581944119 shutdown_connections 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.456+0000 7fc41d7e3700 1 -- 172.21.15.76:0/581944119 wait complete. 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: build_initial_monmap 2024-09-16T06:51:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: monmap: 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 Processor -- start 2024-09-16T06:51:17.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- start start 2024-09-16T06:51:17.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: init 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-a con 0x7fc418071a70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-b con 0x7fc41809f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient(hunting): picked mon.noname-c con 0x7fc41809fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41809b430 con 0x7fc418071a70 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc418072450 con 0x7fc41809f230 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4181385a0 con 0x7fc41809fd20 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient(hunting): authenticate will time out at 1624.598166s 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc4177fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc41809fd20 0x7fc4181d2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48234/0 (socket says 172.21.15.76:48234) 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc4177fe700 1 -- 172.21.15.76:0/2749243543 learned_addr learned my addr 172.21.15.76:0/2749243543 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40ffff700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6789/0 conn(0x7fc41809f230 legacy=0x7fc4181cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40ffff700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6789/0 conn(0x7fc41809f230 legacy=0x7fc4181cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156260502 0 0) 0x7fc4181385a0 con 0x7fc41809fd20 2024-09-16T06:51:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40ffff700 1 --1- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6789/0 conn(0x7fc41809f230 0x7fc4181cf460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40ffff700 1 --1- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6789/0 conn(0x7fc41809f230 0x7fc4181cf460 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): my global_id is 244211 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f80035d0 con 0x7fc41809fd20 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318300722 0 0) 0x7fc3f80035d0 con 0x7fc41809fd20 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4181385a0 con 0x7fc41809fd20 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc410001990 con 0x7fc41809fd20 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): dump: 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1912627837 0 0) 0x7fc4181385a0 con 0x7fc41809fd20 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6789/0 conn(0x7fc41809f230 legacy=0x7fc4181cf460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.76:6789/0 conn(0x7fc418071a70 legacy=0x7fc4181ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 monclient: found mon.c 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4181382c0 con 0x7fc41809fd20 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 5 monclient: authenticate success, global_id 244211 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _renew_subs 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc418136790 con 0x7fc41809fd20 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _renew_subs 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc418137580 con 0x7fc41809fd20 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc410003d90 con 0x7fc41809fd20 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc410004120 con 0x7fc41809fd20 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc415ffb700 4 set_mon_vals no callback set 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.460+0000 7fc40f7fe700 10 monclient: dump: 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc410004300 con 0x7fc41809fd20 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc41d7e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc41d7e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc41d7e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc005100 con 0x7fc41809fd20 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc410044400 con 0x7fc41809fd20 2024-09-16T06:51:17.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.464+0000 7fc416ffd700 10 monclient: get_auth_request con 0x7fc3f80360f0 auth_method 0 2024-09-16T06:51:17.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.472+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc410014f40 con 0x7fc41809fd20 2024-09-16T06:51:17.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.472+0000 7fc40f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:17.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.472+0000 7fc40f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:17.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc41d7e3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:17.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc41d7e3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:17.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc41d7e3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3fc005930 con 0x7fc41809fd20 2024-09-16T06:51:17.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc40f7fe700 1 -- 172.21.15.76:0/2749243543 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3941069071) 0x7fc4100133b0 con 0x7fc41809fd20 2024-09-16T06:51:17.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc40f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:17.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.700+0000 7fc40f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:17.702 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc3f80360f0 legacy=0x7fc3f80385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 10 monclient: shutdown 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 >> v1:172.21.15.76:6790/0 conn(0x7fc41809fd20 legacy=0x7fc4181d2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 shutdown_connections 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 >> 172.21.15.76:0/2749243543 conn(0x7fc4180a3e50 msgr2=0x7fc41813e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:17.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 shutdown_connections 2024-09-16T06:51:17.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.704+0000 7fc41d7e3700 1 -- 172.21.15.76:0/2749243543 wait complete. 2024-09-16T06:51:17.716 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:51:17.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:51:17.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:51:17.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: get_monmap_and_config 2024-09-16T06:51:17.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: build_initial_monmap 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: monmap: 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:17.829716+0000 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:17.829716+0000 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:17.825 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 1 Processor -- start 2024-09-16T06:51:17.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 1 -- start start 2024-09-16T06:51:17.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: init 2024-09-16T06:51:17.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:17.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient(hunting): picked mon.noname-a con 0x7f016009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient(hunting): picked mon.noname-c con 0x7f0160098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient(hunting): picked mon.noname-b con 0x7f0160099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016009d560 con 0x7f016009ed60 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016009f140 con 0x7f0160098be0 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016009d790 con 0x7f0160099030 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0167578700 10 monclient(hunting): authenticate will time out at 1624.963067s 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0166576700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f016009ed60 0x7f01600984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33982/0 (socket says 172.21.15.76:33982) 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0166576700 1 -- 172.21.15.76:0/3227574659 learned_addr learned my addr 172.21.15.76:0/3227574659 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875022857 0 0) 0x7f016009d790 con 0x7f0160099030 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): my global_id is 244214 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480035d0 con 0x7f0160099030 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685436024 0 0) 0x7f016009f140 con 0x7f0160098be0 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 10 monclient(hunting): my global_id is 224377 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.824+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f016009d790 con 0x7f0160098be0 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252125401 0 0) 0x7f016009d790 con 0x7f0160098be0 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 >> v1:172.21.15.76:6790/0 conn(0x7f0160099030 legacy=0x7f016013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 >> v1:172.21.15.76:6789/0 conn(0x7f016009ed60 legacy=0x7f01600984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: _finish_hunting 0 2024-09-16T06:51:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 monclient: found mon.noname-c 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016009f620 con 0x7f0160098be0 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: _finish_auth 0 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: _check_auth_tickets 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f01540015f0 con 0x7f0160098be0 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: dump: 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 5 monclient: authenticate success, global_id 224377 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01540018a0 con 0x7f0160098be0 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 1 -- 172.21.15.76:0/3227574659 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0154001c70 con 0x7f0160098be0 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 4 set_mon_vals no callback set 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165574700 10 monclient: dump: 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: shutdown 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- 172.21.15.76:0/3227574659 >> v1:172.21.15.189:6789/0 conn(0x7f0160098be0 legacy=0x7f01601374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- 172.21.15.76:0/3227574659 shutdown_connections 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- 172.21.15.76:0/3227574659 >> 172.21.15.76:0/3227574659 conn(0x7f01600a0030 msgr2=0x7f01600a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- 172.21.15.76:0/3227574659 shutdown_connections 2024-09-16T06:51:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- 172.21.15.76:0/3227574659 wait complete. 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: build_initial_monmap 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: monmap: 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 Processor -- start 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- start start 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: init 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient(hunting): picked mon.noname-c con 0x7f0160098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient(hunting): picked mon.noname-a con 0x7f0160099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient(hunting): picked mon.noname-b con 0x7f016009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0160095280 con 0x7f0160099030 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600a1250 con 0x7f016009ed60 2024-09-16T06:51:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0160093670 con 0x7f0160098be0 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0167578700 10 monclient(hunting): authenticate will time out at 1624.967367s 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0166576700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0160098be0 0x7f01600912d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48252/0 (socket says 172.21.15.76:48252) 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0165d75700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0160099030 0x7f01600919e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33998/0 (socket says 172.21.15.76:33998) 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.828+0000 7f0166576700 1 -- 172.21.15.76:0/1836965102 learned_addr learned my addr 172.21.15.76:0/1836965102 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923000783 0 0) 0x7f0160093670 con 0x7f0160098be0 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): my global_id is 244217 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c0035d0 con 0x7f0160098be0 2024-09-16T06:51:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475938917 0 0) 0x7f014c0035d0 con 0x7f0160098be0 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0160093670 con 0x7f0160098be0 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0158001b20 con 0x7f0160098be0 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): dump: 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536828138 0 0) 0x7f01600a1250 con 0x7f016009ed60 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient(hunting): my global_id is 224380 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c0035d0 con 0x7f016009ed60 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488683601 0 0) 0x7f0160093670 con 0x7f0160098be0 2024-09-16T06:51:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.189:6789/0 conn(0x7f016009ed60 legacy=0x7f01601d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6789/0 conn(0x7f0160099030 legacy=0x7f01600919e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: _finish_hunting 0 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 monclient: found mon.c 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600985c0 con 0x7f0160098be0 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: _finish_auth 0 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: _check_auth_tickets 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 5 monclient: authenticate success, global_id 244217 2024-09-16T06:51:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 10 monclient: _renew_subs 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0160092150 con 0x7f0160098be0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 10 monclient: _renew_subs 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0160094fa0 con 0x7f0160098be0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0160097050 con 0x7f0160096c70 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0158004c10 con 0x7f0160098be0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0158004fa0 con 0x7f0160098be0 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:17.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 10 monclient: dump: 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f0164d73700 4 set_mon_vals no callback set 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0158005180 con 0x7f0160098be0 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.832+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0158044380 con 0x7f0160098be0 2024-09-16T06:51:17.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851897864 0 0) 0x7f0160097050 con 0x7f0160096c70 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient(hunting): my global_id is 254097 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160093670 con 0x7f0160096c70 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f0166d77700 10 monclient: get_auth_request con 0x7f014c036160 auth_method 0 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3184075318 0 0) 0x7f0160093670 con 0x7f0160096c70 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0160097050 con 0x7f0160096c70 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f01540013b0 con 0x7f0160096c70 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient: dump: 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.847 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2546429346 0 0) 0x7f0160097050 con 0x7f0160096c70 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f016009aaa0 con 0x7f0160096c70 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0154005830 con 0x7f0160096c70 2024-09-16T06:51:17.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6789/0 conn(0x7f0160096c70 legacy=0x7f0160061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.836+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6789/0 conn(0x7f0160096c70 legacy=0x7f0160061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f016008e720 con 0x7f016002c820 2024-09-16T06:51:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373880719 0 0) 0x7f016008e720 con 0x7f016002c820 2024-09-16T06:51:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient(hunting): my global_id is 254100 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160097050 con 0x7f016002c820 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 763210851 0 0) 0x7f0160097050 con 0x7f016002c820 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f016008e720 con 0x7f016002c820 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0154003b30 con 0x7f016002c820 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient: dump: 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2182447372 0 0) 0x7f016008e720 con 0x7f016002c820 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f016009b0f0 con 0x7f016002c820 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1274061780) 0x7f01540059a0 con 0x7f016002c820 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:17.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6789/0 conn(0x7f016002c820 legacy=0x7f016008e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f015effd700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6789/0 conn(0x7f016002c820 legacy=0x7f016008e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:17.865 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.189:6812/3098810674 conn(0x7f014c036160 legacy=0x7f014c038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 10 monclient: shutdown 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 >> v1:172.21.15.76:6790/0 conn(0x7f0160098be0 legacy=0x7f01600912d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0166d77700 1 -- 172.21.15.76:0/1836965102 reap_dead start 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0166d77700 1 -- 172.21.15.76:0/1836965102 reap_dead start 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 shutdown_connections 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 >> 172.21.15.76:0/1836965102 conn(0x7f01600a0030 msgr2=0x7f01600702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 shutdown_connections 2024-09-16T06:51:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.848+0000 7f0167578700 1 -- 172.21.15.76:0/1836965102 wait complete. 2024-09-16T06:51:17.867 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:17.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:51:17.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: get_monmap_and_config 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: build_initial_monmap 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: monmap: 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:17.990108+0000 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:17.990108+0000 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:17.998 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 1 Processor -- start 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 1 -- start start 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: init 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-c con 0x7f46dc097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-a con 0x7f46dc099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-b con 0x7f46dc139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:17.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc098990 con 0x7f46dc099090 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc097f40 con 0x7f46dc097b60 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc09d610 con 0x7f46dc139500 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46db59e700 10 monclient(hunting): authenticate will time out at 1625.123420s 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46d1d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f46dc099090 0x7f46dc09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34036/0 (socket says 172.21.15.76:34036) 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.984+0000 7f46d1d9b700 1 -- 172.21.15.76:0/616607124 learned_addr learned my addr 172.21.15.76:0/616607124 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272693685 0 0) 0x7f46dc09d610 con 0x7f46dc139500 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): my global_id is 244220 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c80035d0 con 0x7f46dc139500 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008456005 0 0) 0x7f46dc098990 con 0x7f46dc099090 2024-09-16T06:51:18.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): my global_id is 254103 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc09d610 con 0x7f46dc099090 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257515428 0 0) 0x7f46dc097f40 con 0x7f46dc097b60 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient(hunting): my global_id is 224383 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc098990 con 0x7f46dc097b60 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472804849 0 0) 0x7f46dc09d610 con 0x7f46dc099090 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 >> v1:172.21.15.76:6790/0 conn(0x7f46dc139500 legacy=0x7f46dc13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 >> v1:172.21.15.189:6789/0 conn(0x7f46dc097b60 legacy=0x7f46dc09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 monclient: found mon.noname-a 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0986b0 con 0x7f46dc099090 2024-09-16T06:51:18.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46c40015f0 con 0x7f46dc099090 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 5 monclient: authenticate success, global_id 254103 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: dump: 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.003 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c4001810 con 0x7f46dc099090 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 1 -- 172.21.15.76:0/616607124 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46c4001be0 con 0x7f46dc099090 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 4 set_mon_vals no callback set 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:18.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d9d9b700 10 monclient: dump: 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: shutdown 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- 172.21.15.76:0/616607124 >> v1:172.21.15.76:6789/0 conn(0x7f46dc099090 legacy=0x7f46dc09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- 172.21.15.76:0/616607124 shutdown_connections 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- 172.21.15.76:0/616607124 >> 172.21.15.76:0/616607124 conn(0x7f46dc0a0030 msgr2=0x7f46dc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- 172.21.15.76:0/616607124 shutdown_connections 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- 172.21.15.76:0/616607124 wait complete. 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: build_initial_monmap 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: monmap: 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:18.006 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 Processor -- start 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- start start 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: init 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-b con 0x7f46dc097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-a con 0x7f46dc099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient(hunting): picked mon.noname-c con 0x7f46dc139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc128820 con 0x7f46dc099090 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc0a2b80 con 0x7f46dc097b60 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc09b670 con 0x7f46dc139500 2024-09-16T06:51:18.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46db59e700 10 monclient(hunting): authenticate will time out at 1625.127228s 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46dad9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f46dc139500 0x7f46dc1d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48264/0 (socket says 172.21.15.76:48264) 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46dad9d700 1 -- 172.21.15.76:0/1760391550 learned_addr learned my addr 172.21.15.76:0/1760391550 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814688654 0 0) 0x7f46dc128820 con 0x7f46dc099090 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): my global_id is 254106 2024-09-16T06:51:18.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc0035d0 con 0x7f46dc099090 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648177081 0 0) 0x7f46dc09b670 con 0x7f46dc139500 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 10 monclient(hunting): my global_id is 244223 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.988+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc128820 con 0x7f46dc139500 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683663332 0 0) 0x7f46dc0a2b80 con 0x7f46dc097b60 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): my global_id is 224386 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc09b670 con 0x7f46dc097b60 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 37102394 0 0) 0x7f46bc0035d0 con 0x7f46dc099090 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46dc0a2b80 con 0x7f46dc099090 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46c4001b20 con 0x7f46dc099090 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): dump: 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.010 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2817930982 0 0) 0x7f46dc128820 con 0x7f46dc139500 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46bc0035d0 con 0x7f46dc139500 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d4001c50 con 0x7f46dc139500 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): dump: 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1071170741 0 0) 0x7f46dc09b670 con 0x7f46dc097b60 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46dc128820 con 0x7f46dc097b60 2024-09-16T06:51:18.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4173568276 0 0) 0x7f46dc0a2b80 con 0x7f46dc099090 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6790/0 conn(0x7f46dc139500 legacy=0x7f46dc1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.189:6789/0 conn(0x7f46dc097b60 legacy=0x7f46dc09a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 monclient: found mon.a 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0712f0 con 0x7f46dc099090 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 5 monclient: authenticate success, global_id 254106 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 10 monclient: _renew_subs 2024-09-16T06:51:18.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc128540 con 0x7f46dc099090 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 10 monclient: _renew_subs 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc09c5a0 con 0x7f46dc099090 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c4004bc0 con 0x7f46dc099090 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46c4004ed0 con 0x7f46dc099090 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d959a700 4 set_mon_vals no callback set 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: dump: 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.014 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46c0006590 con 0x7f46c0002830 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133785425 0 0) 0x7f46c0006590 con 0x7f46c0002830 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient(hunting): my global_id is 244226 2024-09-16T06:51:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc0a2b80 con 0x7f46c0002830 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f46c40050b0 con 0x7f46dc099090 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f46c4044090 con 0x7f46dc099090 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1770226952 0 0) 0x7f46dc0a2b80 con 0x7f46c0002830 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c0006590 con 0x7f46c0002830 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46cc0036a0 con 0x7f46c0002830 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 10 monclient: dump: 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.016 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46dad9d700 10 monclient: get_auth_request con 0x7f46bc0364e0 auth_method 0 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 689434333 0 0) 0x7f46c0006590 con 0x7f46c0002830 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.992+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46c000a190 con 0x7f46c0002830 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.996+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f46cc005840 con 0x7f46c0002830 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.996+0000 7f46d37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.996+0000 7f46d37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.996+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6790/0 conn(0x7f46c0002830 legacy=0x7f46c0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:17.996+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6790/0 conn(0x7f46c0002830 legacy=0x7f46c0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46db59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46c0009e30 con 0x7f46c0009150 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397006405 0 0) 0x7f46c0009e30 con 0x7f46c0009150 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient(hunting): my global_id is 244229 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0006590 con 0x7f46c0009150 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3300680965 0 0) 0x7f46c0006590 con 0x7f46c0009150 2024-09-16T06:51:18.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c0009e30 con 0x7f46c0009150 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46cc003dd0 con 0x7f46c0009150 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 10 monclient: dump: 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.020 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1477531746 0 0) 0x7f46c0009e30 con 0x7f46c0009150 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.004+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46c000c340 con 0x7f46c0009150 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 1559507459) 0x7f46cc0059b0 con 0x7f46c0009150 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46d37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46d37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6790/0 conn(0x7f46c0009150 legacy=0x7f46c00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46d37fe700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6790/0 conn(0x7f46c0009150 legacy=0x7f46c00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:18.021 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:18.035 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:18.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.189:6812/3098810674 conn(0x7f46bc0364e0 legacy=0x7f46bc0389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 10 monclient: shutdown 2024-09-16T06:51:18.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 >> v1:172.21.15.76:6789/0 conn(0x7f46dc099090 legacy=0x7f46dc09af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46dad9d700 1 -- 172.21.15.76:0/1760391550 reap_dead start 2024-09-16T06:51:18.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46dad9d700 1 -- 172.21.15.76:0/1760391550 reap_dead start 2024-09-16T06:51:18.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 shutdown_connections 2024-09-16T06:51:18.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 >> 172.21.15.76:0/1760391550 conn(0x7f46dc0a0030 msgr2=0x7f46dc0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 shutdown_connections 2024-09-16T06:51:18.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.008+0000 7f46db59e700 1 -- 172.21.15.76:0/1760391550 wait complete. 2024-09-16T06:51:18.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:18.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:51:18.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:51:18.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.156+0000 7fa7d4fe5700 10 monclient: get_monmap_and_config 2024-09-16T06:51:18.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.156+0000 7fa7d4fe5700 10 monclient: build_initial_monmap 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.156+0000 7fa7d4fe5700 10 monclient: monmap: 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:18.163893+0000 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:18.163893+0000 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:18.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 1 Processor -- start 2024-09-16T06:51:18.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 1 -- start start 2024-09-16T06:51:18.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient: init 2024-09-16T06:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-c con 0x7fa7d0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-b con 0x7fa7d006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:18.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-a con 0x7fa7d006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d006c280 con 0x7fa7d006cf50 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d006bfa0 con 0x7fa7d0132330 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d009ace0 con 0x7fa7d006d480 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient(hunting): authenticate will time out at 1625.298134s 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7cffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa7d006cf50 0x7fa7d013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34060/0 (socket says 172.21.15.76:34060) 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ceffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa7d006d480 0x7fa7d013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48302/0 (socket says 172.21.15.76:48302) 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7cffff700 1 -- 172.21.15.76:0/4036132982 learned_addr learned my addr 172.21.15.76:0/4036132982 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644126555 0 0) 0x7fa7d006c280 con 0x7fa7d006cf50 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): my global_id is 254109 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc0035d0 con 0x7fa7d006cf50 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268249897 0 0) 0x7fa7d009ace0 con 0x7fa7d006d480 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): my global_id is 244232 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d006c280 con 0x7fa7d006d480 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020774084 0 0) 0x7fa7d006bfa0 con 0x7fa7d0132330 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient(hunting): my global_id is 224389 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d009ace0 con 0x7fa7d0132330 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177706247 0 0) 0x7fa7d006c280 con 0x7fa7d006d480 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 >> v1:172.21.15.189:6789/0 conn(0x7fa7d0132330 legacy=0x7fa7d0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 >> v1:172.21.15.76:6789/0 conn(0x7fa7d006cf50 legacy=0x7fa7d013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 monclient: found mon.noname-b 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d009ae70 con 0x7fa7d006d480 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7b8001520 con 0x7fa7d006d480 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 5 monclient: authenticate success, global_id 244232 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: dump: 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.174 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b80017c0 con 0x7fa7d006d480 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 1 -- 172.21.15.76:0/4036132982 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7b80019f0 con 0x7fa7d006d480 2024-09-16T06:51:18.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7d4fe5700 4 set_mon_vals no callback set 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.160+0000 7fa7ce7fc700 10 monclient: dump: 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.176 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.177 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: shutdown 2024-09-16T06:51:18.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/4036132982 >> v1:172.21.15.76:6790/0 conn(0x7fa7d006d480 legacy=0x7fa7d013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/4036132982 shutdown_connections 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/4036132982 >> 172.21.15.76:0/4036132982 conn(0x7fa7d00a41c0 msgr2=0x7fa7d00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/4036132982 shutdown_connections 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/4036132982 wait complete. 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: build_initial_monmap 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: monmap: 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:18.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 Processor -- start 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- start start 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: init 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-b con 0x7fa7d006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-c con 0x7fa7d006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:18.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d00a1a00 con 0x7fa7d0132330 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d01ca670 con 0x7fa7d006cf50 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d012dcf0 con 0x7fa7d006d480 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient(hunting): authenticate will time out at 1625.301853s 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7ceffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa7d006d480 0x7fa7d01c9f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48304/0 (socket says 172.21.15.76:48304) 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7ceffd700 1 -- 172.21.15.76:0/2092619103 learned_addr learned my addr 172.21.15.76:0/2092619103 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395331969 0 0) 0x7fa7d012dcf0 con 0x7fa7d006d480 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): my global_id is 244235 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b00035d0 con 0x7fa7d006d480 2024-09-16T06:51:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037950883 0 0) 0x7fa7d00a1a00 con 0x7fa7d0132330 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): my global_id is 254112 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d012dcf0 con 0x7fa7d0132330 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097014592 0 0) 0x7fa7b00035d0 con 0x7fa7d006d480 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d00a1a00 con 0x7fa7d006d480 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7b8004890 con 0x7fa7d006d480 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): dump: 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.183 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400495334 0 0) 0x7fa7d01ca670 con 0x7fa7d006cf50 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): my global_id is 224392 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b00035d0 con 0x7fa7d006cf50 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524407948 0 0) 0x7fa7d012dcf0 con 0x7fa7d0132330 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d01ca670 con 0x7fa7d0132330 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7c8001490 con 0x7fa7d0132330 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient(hunting): dump: 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994929084 0 0) 0x7fa7d00a1a00 con 0x7fa7d006d480 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6789/0 conn(0x7fa7d006cf50 legacy=0x7fa7d01c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.76:6789/0 conn(0x7fa7d0132330 legacy=0x7fa7d01d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 monclient: found mon.c 2024-09-16T06:51:18.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d012e2b0 con 0x7fa7d006d480 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 5 monclient: authenticate success, global_id 244235 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _renew_subs 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d012dfd0 con 0x7fa7d006d480 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _renew_subs 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.164+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d00a0f70 con 0x7fa7d006d480 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7d4fe5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa7d00008d0 con 0x7fa7d009c510 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b8004ca0 con 0x7fa7d006d480 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7b8001120 con 0x7fa7d006d480 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7cd7fa700 4 set_mon_vals no callback set 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: dump: 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.189 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477048777 0 0) 0x7fa7d00008d0 con 0x7fa7d009c510 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient(hunting): my global_id is 224395 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d00a1a00 con 0x7fa7d009c510 2024-09-16T06:51:18.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa7b800f8c0 con 0x7fa7d006d480 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa7b8044050 con 0x7fa7d006d480 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2538641417 0 0) 0x7fa7d00a1a00 con 0x7fa7d009c510 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d00008d0 con 0x7fa7d009c510 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7cffff700 10 monclient: get_auth_request con 0x7fa7b0036430 auth_method 0 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7c0001740 con 0x7fa7d009c510 2024-09-16T06:51:18.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: dump: 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.193 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 577494988 0 0) 0x7fa7d00008d0 con 0x7fa7d009c510 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7d009ae70 con 0x7fa7d009c510 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa7c0001a00 con 0x7fa7d009c510 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6789/0 conn(0x7fa7d009c510 legacy=0x7fa7d009c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.168+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6789/0 conn(0x7fa7d009c510 legacy=0x7fa7d009c8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7d4fe5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:18.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa7d009d3e0 con 0x7fa7d009d000 2024-09-16T06:51:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324266961 0 0) 0x7fa7d009d3e0 con 0x7fa7d009d000 2024-09-16T06:51:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient(hunting): my global_id is 224398 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d00008d0 con 0x7fa7d009d000 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2243477021 0 0) 0x7fa7d00008d0 con 0x7fa7d009d000 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d009d3e0 con 0x7fa7d009d000 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa7c0003a30 con 0x7fa7d009d000 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.176+0000 7fa7c7fff700 10 monclient: dump: 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.201 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2010762359 0 0) 0x7fa7d009d3e0 con 0x7fa7d009d000 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7d00a3450 con 0x7fa7d009d000 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2237316479) 0x7fa7c0003c40 con 0x7fa7d009d000 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6789/0 conn(0x7fa7d009d000 legacy=0x7fa7d002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7c7fff700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6789/0 conn(0x7fa7d009d000 legacy=0x7fa7d002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:18.204 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:18.204 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:18.205 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:18.206 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.207 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.208 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.209 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.210 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:18.211 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:18.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:18.213 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:18.214 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:18.215 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:18.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:18.217 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.218 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:18.220 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:18.221 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:18.222 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.223 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.224 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:18.225 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:18.226 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:18.226 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:18.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa7b0036430 legacy=0x7fa7b00388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 10 monclient: shutdown 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 >> v1:172.21.15.76:6790/0 conn(0x7fa7d006d480 legacy=0x7fa7d01c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7cffff700 1 -- 172.21.15.76:0/2092619103 reap_dead start 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 shutdown_connections 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 >> 172.21.15.76:0/2092619103 conn(0x7fa7d00a41c0 msgr2=0x7fa7d0129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 shutdown_connections 2024-09-16T06:51:18.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.180+0000 7fa7d4fe5700 1 -- 172.21.15.76:0/2092619103 wait complete. 2024-09-16T06:51:18.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:18.229 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:51:18.229 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:51:18.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.344+0000 7fdffe5e5700 10 monclient: get_monmap_and_config 2024-09-16T06:51:18.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.344+0000 7fdffe5e5700 10 monclient: build_initial_monmap 2024-09-16T06:51:18.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.344+0000 7fdffe5e5700 10 monclient: monmap: 2024-09-16T06:51:18.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:18.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:18.351576+0000 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:18.351576+0000 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:18.347 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 Processor -- start 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- start start 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: init 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-b con 0x7fdff8097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-c con 0x7fdff8099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-a con 0x7fdff8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8098990 con 0x7fdff8139500 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8097f40 con 0x7fdff8099090 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff809d610 con 0x7fdff8097b60 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient(hunting): authenticate will time out at 1625.484924s 2024-09-16T06:51:18.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffdde4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdff8139500 0x7fdff813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34080/0 (socket says 172.21.15.76:34080) 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffd5e3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdff8097b60 0x7fdff809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48316/0 (socket says 172.21.15.76:48316) 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffdde4700 1 -- 172.21.15.76:0/1025682395 learned_addr learned my addr 172.21.15.76:0/1025682395 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169804675 0 0) 0x7fdff8098990 con 0x7fdff8139500 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): my global_id is 254115 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe40035d0 con 0x7fdff8139500 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763880091 0 0) 0x7fdff809d610 con 0x7fdff8097b60 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient(hunting): my global_id is 244238 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8098990 con 0x7fdff8097b60 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434206409 0 0) 0x7fdfe40035d0 con 0x7fdff8139500 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 >> v1:172.21.15.76:6790/0 conn(0x7fdff8097b60 legacy=0x7fdff809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 >> v1:172.21.15.189:6789/0 conn(0x7fdff8099090 legacy=0x7fdff809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 monclient: found mon.noname-a 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:18.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff80986b0 con 0x7fdff8139500 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdff0001990 con 0x7fdff8139500 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 5 monclient: authenticate success, global_id 254115 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: dump: 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff0001c30 con 0x7fdff8139500 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 1 -- 172.21.15.76:0/1025682395 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdff0001fc0 con 0x7fdff8139500 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 4 set_mon_vals no callback set 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdfeffff700 10 monclient: dump: 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: shutdown 2024-09-16T06:51:18.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- 172.21.15.76:0/1025682395 >> v1:172.21.15.76:6789/0 conn(0x7fdff8139500 legacy=0x7fdff813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- 172.21.15.76:0/1025682395 shutdown_connections 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- 172.21.15.76:0/1025682395 >> 172.21.15.76:0/1025682395 conn(0x7fdff80a0030 msgr2=0x7fdff80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- 172.21.15.76:0/1025682395 shutdown_connections 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 1 -- 172.21.15.76:0/1025682395 wait complete. 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: build_initial_monmap 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.348+0000 7fdffe5e5700 10 monclient: monmap: 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:18.357 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 Processor -- start 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- start start 2024-09-16T06:51:18.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: init 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-b con 0x7fdff8097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-a con 0x7fdff8099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient(hunting): picked mon.noname-c con 0x7fdff8139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff813c2a0 con 0x7fdff8099090 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff806ccb0 con 0x7fdff8097b60 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff80a2d60 con 0x7fdff8139500 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient(hunting): authenticate will time out at 1625.489318s 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffcde2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdff8099090 0x7fdff806c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34090/0 (socket says 172.21.15.76:34090) 2024-09-16T06:51:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffcde2700 1 -- 172.21.15.76:0/2794773128 learned_addr learned my addr 172.21.15.76:0/2794773128 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166806266 0 0) 0x7fdff80a2d60 con 0x7fdff8139500 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): my global_id is 244241 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfdc0035d0 con 0x7fdff8139500 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717797089 0 0) 0x7fdff8097f40 con 0x7fdff8099090 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): my global_id is 254118 2024-09-16T06:51:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff80a2d60 con 0x7fdff8099090 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360191497 0 0) 0x7fdff806ccb0 con 0x7fdff8097b60 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): my global_id is 224404 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8097f40 con 0x7fdff8097b60 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279676665 0 0) 0x7fdfdc0035d0 con 0x7fdff8139500 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff806ccb0 con 0x7fdff8139500 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdff0001da0 con 0x7fdff8139500 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): dump: 2024-09-16T06:51:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.366 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.366 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.366 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024808609 0 0) 0x7fdff8097f40 con 0x7fdff8097b60 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfdc0035d0 con 0x7fdff8097b60 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdff40018d0 con 0x7fdff8097b60 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): dump: 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534806333 0 0) 0x7fdff80a2d60 con 0x7fdff8099090 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff8097f40 con 0x7fdff8099090 2024-09-16T06:51:18.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdfe8001690 con 0x7fdff8099090 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient(hunting): dump: 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 695317869 0 0) 0x7fdff806ccb0 con 0x7fdff8139500 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 >> v1:172.21.15.189:6789/0 conn(0x7fdff8097b60 legacy=0x7fdff806be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 >> v1:172.21.15.76:6789/0 conn(0x7fdff8099090 legacy=0x7fdff806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 monclient: found mon.c 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff809a540 con 0x7fdff8139500 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 5 monclient: authenticate success, global_id 244241 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _renew_subs 2024-09-16T06:51:18.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff806f460 con 0x7fdff8139500 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _renew_subs 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff806def0 con 0x7fdff8139500 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff0005020 con 0x7fdff8139500 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdff0005330 con 0x7fdff8139500 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfef7fe700 4 set_mon_vals no callback set 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.352+0000 7fdfedffb700 10 monclient: dump: 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:18.372 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdff000fda0 con 0x7fdff8139500 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fdff00441f0 con 0x7fdff8139500 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdffe5e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdffe5e5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdffe5e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8005100 con 0x7fdff8139500 2024-09-16T06:51:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.356+0000 7fdffd5e3700 10 monclient: get_auth_request con 0x7fdfdc036500 auth_method 0 2024-09-16T06:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.364+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdff0014d50 con 0x7fdff8139500 2024-09-16T06:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.364+0000 7fdfedffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:18.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.364+0000 7fdfedffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:18.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.596+0000 7fdffe5e5700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:51:18.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.596+0000 7fdffe5e5700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:51:18.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.596+0000 7fdffe5e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:18.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.596+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fdfd8005bc0 con 0x7fdff8139500 2024-09-16T06:51:18.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdfedffb700 1 -- 172.21.15.76:0/2794773128 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fdff00131c0 con 0x7fdff8139500 2024-09-16T06:51:18.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdfedffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:51:18.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdfedffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:18.599 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=13,logm=50} crc {auth=2059436809,config=1097430493,health=2177675957,kv=1001081567,logm=4096554459}) 2024-09-16T06:51:18.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=13,logm=49} crc {auth=3192584295,config=1097430493,health=3379137625,kv=2283260164,logm=4250821756}) 2024-09-16T06:51:18.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=13,logm=50} crc {auth=2059436809,config=1097430493,health=2177675957,kv=1001081567,logm=4096554459}) 2024-09-16T06:51:18.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=8,kv=12,logm=53} crc {auth=2601693707,config=1097430493,health=1598409518,kv=1500079442,logm=1777407988}) 2024-09-16T06:51:18.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdfdc036500 legacy=0x7fdfdc0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 10 monclient: shutdown 2024-09-16T06:51:18.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:18.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 >> v1:172.21.15.76:6790/0 conn(0x7fdff8139500 legacy=0x7fdff81cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:18.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 shutdown_connections 2024-09-16T06:51:18.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 >> 172.21.15.76:0/2794773128 conn(0x7fdff80a0030 msgr2=0x7fdff813aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:18.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 shutdown_connections 2024-09-16T06:51:18.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:18.600+0000 7fdffe5e5700 1 -- 172.21.15.76:0/2794773128 wait complete. 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668447795}) 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768709226}) 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668447795}) 2024-09-16T06:51:18.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616309876}) 2024-09-16T06:51:18.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112508696}) 2024-09-16T06:51:18.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28404579}) 2024-09-16T06:51:18.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112508696}) 2024-09-16T06:51:18.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558016656}) 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288212030}) 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175806468}) 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288212030}) 2024-09-16T06:51:18.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310958273}) 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947751793}) 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648770462}) 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947751793}) 2024-09-16T06:51:18.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517519193}) 2024-09-16T06:51:18.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845785749}) 2024-09-16T06:51:18.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436288289}) 2024-09-16T06:51:18.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845785749}) 2024-09-16T06:51:18.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708491984}) 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2682817231}) 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2517129442}) 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2682817231}) 2024-09-16T06:51:18.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638810815}) 2024-09-16T06:51:18.615 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:51:18.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4268190541}) 2024-09-16T06:51:18.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910911052}) 2024-09-16T06:51:18.616 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.616 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4268190541}) 2024-09-16T06:51:18.616 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959877695}) 2024-09-16T06:51:18.617 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.617 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594733176}) 2024-09-16T06:51:18.617 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977413357}) 2024-09-16T06:51:18.617 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594733176}) 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441867786}) 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2424488229}) 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488255737}) 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2424488229}) 2024-09-16T06:51:18.618 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4193944286}) 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358809836}) 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551902141}) 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358809836}) 2024-09-16T06:51:18.619 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=150545486}) 2024-09-16T06:51:18.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2086935351}) 2024-09-16T06:51:18.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063080138}) 2024-09-16T06:51:18.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2086935351}) 2024-09-16T06:51:18.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060933436}) 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703763655}) 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116015831}) 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703763655}) 2024-09-16T06:51:18.622 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=633053090}) 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310422553}) 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212570617}) 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310422553}) 2024-09-16T06:51:18.623 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=441813776}) 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423384879}) 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516877002}) 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423384879}) 2024-09-16T06:51:18.624 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162641174}) 2024-09-16T06:51:18.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805483889}) 2024-09-16T06:51:18.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326629207}) 2024-09-16T06:51:18.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.625 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805483889}) 2024-09-16T06:51:18.626 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790609406}) 2024-09-16T06:51:18.626 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.626 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991780325}) 2024-09-16T06:51:18.626 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039721099}) 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991780325}) 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.624+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194431089}) 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987284024}) 2024-09-16T06:51:18.627 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003889549}) 2024-09-16T06:51:18.628 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.628 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987284024}) 2024-09-16T06:51:18.628 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084906676}) 2024-09-16T06:51:18.628 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.629 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717602022}) 2024-09-16T06:51:18.629 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186806389}) 2024-09-16T06:51:18.629 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.629 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717602022}) 2024-09-16T06:51:18.629 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220145704}) 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985683015}) 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566034710}) 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985683015}) 2024-09-16T06:51:18.630 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=17236690}) 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387211503}) 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736189461}) 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387211503}) 2024-09-16T06:51:18.631 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.628+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557715494}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723971780}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276091486}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723971780}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3039983915}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332672549}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3518639223}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332672549}) 2024-09-16T06:51:18.633 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974630326}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371065359}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243756674}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371065359}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.632+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1433920023}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254251216}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281146619}) 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.635 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254251216}) 2024-09-16T06:51:18.636 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717031018}) 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3728763309}) 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281303312}) 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3728763309}) 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255025004}) 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.637 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173045809}) 2024-09-16T06:51:18.638 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036780505}) 2024-09-16T06:51:18.638 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.638 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173045809}) 2024-09-16T06:51:18.638 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=116397917}) 2024-09-16T06:51:18.639 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.639 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155651240}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138597320}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155651240}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.636+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880011282}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172856550}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224871976}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172856550}) 2024-09-16T06:51:18.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646445851}) 2024-09-16T06:51:18.641 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951893077}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028141226}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951893077}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363091677}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821394903}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=222723234}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821394903}) 2024-09-16T06:51:18.642 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.640+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056008561}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281988276}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401856384}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281988276}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258807898}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32894348}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912296298}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32894348}) 2024-09-16T06:51:18.644 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268807807}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005072996}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880701354}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005072996}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751872777}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096360834}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652423102}) 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096360834}) 2024-09-16T06:51:18.647 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.644+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434349120}) 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124318115}) 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=261221217}) 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124318115}) 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082073809}) 2024-09-16T06:51:18.648 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2044374174}) 2024-09-16T06:51:18.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473287944}) 2024-09-16T06:51:18.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2044374174}) 2024-09-16T06:51:18.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=511740197}) 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420831668}) 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632693605}) 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420831668}) 2024-09-16T06:51:18.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.648+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706424633}) 2024-09-16T06:51:18.651 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210966565}) 2024-09-16T06:51:18.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688917656}) 2024-09-16T06:51:18.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210966565}) 2024-09-16T06:51:18.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953335586}) 2024-09-16T06:51:18.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257918095}) 2024-09-16T06:51:18.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844659968}) 2024-09-16T06:51:18.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257918095}) 2024-09-16T06:51:18.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.652+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3847236082}) 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1991283610}) 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642628497}) 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1991283610}) 2024-09-16T06:51:18.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181184861}) 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268165107}) 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393114077}) 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268165107}) 2024-09-16T06:51:18.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.656+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2792367971}) 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355594778}) 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305202812}) 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355594778}) 2024-09-16T06:51:18.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1314937479}) 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881778547}) 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240400279}) 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881778547}) 2024-09-16T06:51:18.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674526620}) 2024-09-16T06:51:18.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552649462}) 2024-09-16T06:51:18.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570919311}) 2024-09-16T06:51:18.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552649462}) 2024-09-16T06:51:18.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.660+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154926128}) 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116775025}) 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427082816}) 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116775025}) 2024-09-16T06:51:18.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257676272}) 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010760941}) 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236417617}) 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010760941}) 2024-09-16T06:51:18.664 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263601561}) 2024-09-16T06:51:18.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240835500}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120834637}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240835500}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627108013}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662821474}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490815459}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662821474}) 2024-09-16T06:51:18.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.664+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391897900}) 2024-09-16T06:51:18.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404739558}) 2024-09-16T06:51:18.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519647018}) 2024-09-16T06:51:18.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404739558}) 2024-09-16T06:51:18.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567676929}) 2024-09-16T06:51:18.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052408136}) 2024-09-16T06:51:18.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040827832}) 2024-09-16T06:51:18.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052408136}) 2024-09-16T06:51:18.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568586205}) 2024-09-16T06:51:18.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895918527}) 2024-09-16T06:51:18.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347127103}) 2024-09-16T06:51:18.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895918527}) 2024-09-16T06:51:18.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3419092840}) 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378760270}) 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86506056}) 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378760270}) 2024-09-16T06:51:18.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.668+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=791382202}) 2024-09-16T06:51:18.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586138749}) 2024-09-16T06:51:18.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245657889}) 2024-09-16T06:51:18.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586138749}) 2024-09-16T06:51:18.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490280792}) 2024-09-16T06:51:18.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949659772}) 2024-09-16T06:51:18.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557646711}) 2024-09-16T06:51:18.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949659772}) 2024-09-16T06:51:18.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.672+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566173409}) 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561877069}) 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171665246}) 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561877069}) 2024-09-16T06:51:18.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075052160}) 2024-09-16T06:51:18.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550842900}) 2024-09-16T06:51:18.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458851368}) 2024-09-16T06:51:18.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550842900}) 2024-09-16T06:51:18.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346124560}) 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082084690}) 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891194370}) 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082084690}) 2024-09-16T06:51:18.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746685943}) 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493849012}) 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309024276}) 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.676+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493849012}) 2024-09-16T06:51:18.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1594818264}) 2024-09-16T06:51:18.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368011697}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502384488}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368011697}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2131849060}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1593304079}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998981072}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1593304079}) 2024-09-16T06:51:18.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2057330246}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1688413841}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820443518}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1688413841}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.680+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342657854}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472659074}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273432741}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472659074}) 2024-09-16T06:51:18.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317313455}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004555526}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377245340}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004555526}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2772547285}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496069228}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695422394}) 2024-09-16T06:51:18.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496069228}) 2024-09-16T06:51:18.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388592031}) 2024-09-16T06:51:18.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329354597}) 2024-09-16T06:51:18.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807033049}) 2024-09-16T06:51:18.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329354597}) 2024-09-16T06:51:18.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.684+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750841315}) 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=3288759273,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1929318895,mon_config_key=1714515979,monmap=1887442722,osd_metadata=304085106,osd_pg_creating=3462100867,osd_snap=3823785925}) 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=3,osd_metadata=1} crc {logm=503750014,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2886307118,mon_config_key=2727055987,monmap=1887442722,osd_metadata=1949918646}) 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=3288759273,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1929318895,mon_config_key=1714515979,monmap=1887442722,osd_metadata=304085106,osd_pg_creating=3462100867,osd_snap=3823785925}) 2024-09-16T06:51:18.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.688+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=20,monmap=2,osd_metadata=2} crc {logm=632485857,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2886307118,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2293246139}) 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=1,osdmap=54} crc {osd_snap=4249038086,osdmap=461567408}) 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=51} crc {osd_snap=4249038086,osdmap=3468259517}) 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=1,osdmap=54} crc {osd_snap=4249038086,osdmap=461567408}) 2024-09-16T06:51:18.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:18.696+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=1,osdmap=53} crc {osd_snap=4249038086,osdmap=3635649866}) 2024-09-16T06:51:19.616 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:51:19.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: get_monmap_and_config 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: build_initial_monmap 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: monmap: 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:19.737963+0000 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:19.737963+0000 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:19.733 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 1 Processor -- start 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 1 -- start start 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: init 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.732+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-a con 0x7f33440964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-b con 0x7f334409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-c con 0x7f3344137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344098560 con 0x7f33440964f0 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344099ec0 con 0x7f3344137e70 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334409b5f0 con 0x7f334409b1c0 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient(hunting): authenticate will time out at 1626.872256s 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3348ad6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f33440964f0 0x7f33440968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34100/0 (socket says 172.21.15.76:34100) 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3343fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f334409b1c0 0x7f3344136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48344/0 (socket says 172.21.15.76:48344) 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3348ad6700 1 -- 172.21.15.76:0/400788433 learned_addr learned my addr 172.21.15.76:0/400788433 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475635739 0 0) 0x7f3344098560 con 0x7f33440964f0 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): my global_id is 254121 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33340035d0 con 0x7f33440964f0 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108365485 0 0) 0x7f334409b5f0 con 0x7f334409b1c0 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient(hunting): my global_id is 244244 2024-09-16T06:51:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3344098560 con 0x7f334409b1c0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390290776 0 0) 0x7f33340035d0 con 0x7f33440964f0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 >> v1:172.21.15.76:6790/0 conn(0x7f334409b1c0 legacy=0x7f3344136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 >> v1:172.21.15.189:6789/0 conn(0x7f3344137e70 legacy=0x7f334413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 monclient: found mon.noname-a 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344099be0 con 0x7f33440964f0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3330001610 con 0x7f33440964f0 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: dump: 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 5 monclient: authenticate success, global_id 254121 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3330001830 con 0x7f33440964f0 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 1 -- 172.21.15.76:0/400788433 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3330001bc0 con 0x7f33440964f0 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f33437fe700 10 monclient: dump: 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 4 set_mon_vals no callback set 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 10 monclient: shutdown 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- 172.21.15.76:0/400788433 >> v1:172.21.15.76:6789/0 conn(0x7f33440964f0 legacy=0x7f33440968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- 172.21.15.76:0/400788433 shutdown_connections 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- 172.21.15.76:0/400788433 >> 172.21.15.76:0/400788433 conn(0x7f33440a0030 msgr2=0x7f33440a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- 172.21.15.76:0/400788433 shutdown_connections 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.736+0000 7f3349ad8700 1 -- 172.21.15.76:0/400788433 wait complete. 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient: build_initial_monmap 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient: monmap: 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:19.744 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:19.744 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 1 Processor -- start 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 1 -- start start 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient: init 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-a con 0x7f33440964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-b con 0x7f334409b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient(hunting): picked mon.noname-c con 0x7f3344137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344091430 con 0x7f33440964f0 2024-09-16T06:51:19.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344091710 con 0x7f334409b1c0 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334408f070 con 0x7f3344137e70 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33492d7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3344137e70 0x7f33441d0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48356/0 (socket says 172.21.15.76:48356) 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33492d7700 1 -- 172.21.15.76:0/84951994 learned_addr learned my addr 172.21.15.76:0/84951994 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f3349ad8700 10 monclient(hunting): authenticate will time out at 1626.878640s 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155996619 0 0) 0x7f334408f070 con 0x7f3344137e70 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): my global_id is 244247 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33280035d0 con 0x7f3344137e70 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210802718 0 0) 0x7f3344091430 con 0x7f33440964f0 2024-09-16T06:51:19.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): my global_id is 254124 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334408f070 con 0x7f33440964f0 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104612520 0 0) 0x7f3344091710 con 0x7f334409b1c0 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): my global_id is 224410 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3344091430 con 0x7f334409b1c0 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885650203 0 0) 0x7f33280035d0 con 0x7f3344137e70 2024-09-16T06:51:19.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3344091710 con 0x7f3344137e70 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f333c001c70 con 0x7f3344137e70 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): dump: 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.752 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018998573 0 0) 0x7f334408f070 con 0x7f33440964f0 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33280035d0 con 0x7f33440964f0 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3330001510 con 0x7f33440964f0 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.740+0000 7f33417fa700 10 monclient(hunting): dump: 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.753 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1983560874 0 0) 0x7f3344091430 con 0x7f334409b1c0 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f334408f070 con 0x7f334409b1c0 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33380018b0 con 0x7f334409b1c0 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient(hunting): dump: 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.754 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304362258 0 0) 0x7f3344091710 con 0x7f3344137e70 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 >> v1:172.21.15.189:6789/0 conn(0x7f334409b1c0 legacy=0x7f334408e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 >> v1:172.21.15.76:6789/0 conn(0x7f33440964f0 legacy=0x7f334408e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: _finish_hunting 0 2024-09-16T06:51:19.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 monclient: found mon.c 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334408f3f0 con 0x7f3344137e70 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: _finish_auth 0 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: _check_auth_tickets 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 5 monclient: authenticate success, global_id 244247 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _renew_subs 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3344097e50 con 0x7f3344137e70 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _renew_subs 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33440a2db0 con 0x7f3344137e70 2024-09-16T06:51:19.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f333c002000 con 0x7f3344137e70 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f333c003e90 con 0x7f3344137e70 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3342ffd700 4 set_mon_vals no callback set 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 10 monclient: dump: 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f333c003ff0 con 0x7f3344137e70 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f333c044180 con 0x7f3344137e70 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3324005100 con 0x7f3344137e70 2024-09-16T06:51:19.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.744+0000 7f3348ad6700 10 monclient: get_auth_request con 0x7f33280365e0 auth_method 0 2024-09-16T06:51:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.752+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f333c014aa0 con 0x7f3344137e70 2024-09-16T06:51:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.752+0000 7f33417fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:19.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.752+0000 7f33417fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f3349ad8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f3349ad8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f3349ad8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:19.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3324005930 con 0x7f3344137e70 2024-09-16T06:51:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f33417fa700 1 -- 172.21.15.76:0/84951994 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2666084736) 0x7f333c012f10 con 0x7f3344137e70 2024-09-16T06:51:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f33417fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:19.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.984+0000 7f33417fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:19.985 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 >> v1:172.21.15.189:6812/3098810674 conn(0x7f33280365e0 legacy=0x7f3328038aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 10 monclient: shutdown 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 >> v1:172.21.15.76:6790/0 conn(0x7f3344137e70 legacy=0x7f33441d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 shutdown_connections 2024-09-16T06:51:19.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 >> 172.21.15.76:0/84951994 conn(0x7f33440a0030 msgr2=0x7f334409bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:19.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 shutdown_connections 2024-09-16T06:51:19.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:19.988+0000 7f3349ad8700 1 -- 172.21.15.76:0/84951994 wait complete. 2024-09-16T06:51:20.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:51:20.004 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:51:20.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:51:20.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:51:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 10 monclient: get_monmap_and_config 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 10 monclient: monmap: 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:20.123030+0000 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:20.123030+0000 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:20.118 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 1 Processor -- start 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 1 -- start start 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.116+0000 7fd676697700 10 monclient: init 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient(hunting): picked mon.noname-c con 0x7fd670097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient(hunting): picked mon.noname-b con 0x7fd670099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient(hunting): picked mon.noname-a con 0x7fd670139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd670098990 con 0x7fd670139500 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd670097f40 con 0x7fd670097b60 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67009d610 con 0x7fd670099090 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient(hunting): authenticate will time out at 1627.256321s 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd674e94700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd670099090 0x7fd67009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48368/0 (socket says 172.21.15.76:48368) 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd674e94700 1 -- 172.21.15.76:0/758419669 learned_addr learned my addr 172.21.15.76:0/758419669 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236288143 0 0) 0x7fd67009d610 con 0x7fd670099090 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): my global_id is 244250 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd65c0035d0 con 0x7fd670099090 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195555371 0 0) 0x7fd670098990 con 0x7fd670139500 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient(hunting): my global_id is 254127 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd67009d610 con 0x7fd670139500 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628732903 0 0) 0x7fd65c0035d0 con 0x7fd670099090 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 >> v1:172.21.15.189:6789/0 conn(0x7fd670097b60 legacy=0x7fd67009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 >> v1:172.21.15.76:6789/0 conn(0x7fd670139500 legacy=0x7fd67013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 monclient: found mon.noname-b 2024-09-16T06:51:20.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700986b0 con 0x7fd670099090 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 5 monclient: authenticate success, global_id 244250 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c001610 con 0x7fd670099090 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: dump: 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.130 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd66c001830 con 0x7fd670099090 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 4 set_mon_vals no callback set 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 1 -- 172.21.15.76:0/758419669 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c001bc0 con 0x7fd670099090 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd663fff700 10 monclient: dump: 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: shutdown 2024-09-16T06:51:20.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- 172.21.15.76:0/758419669 >> v1:172.21.15.76:6790/0 conn(0x7fd670099090 legacy=0x7fd67009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- 172.21.15.76:0/758419669 shutdown_connections 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- 172.21.15.76:0/758419669 >> 172.21.15.76:0/758419669 conn(0x7fd6700a0030 msgr2=0x7fd6700a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- 172.21.15.76:0/758419669 shutdown_connections 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- 172.21.15.76:0/758419669 wait complete. 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: monmap: 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.133 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 Processor -- start 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 1 -- start start 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.120+0000 7fd676697700 10 monclient: init 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient(hunting): picked mon.noname-b con 0x7fd670097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient(hunting): picked mon.noname-a con 0x7fd670099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient(hunting): picked mon.noname-c con 0x7fd670139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67013c390 con 0x7fd670099090 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67008f7b0 con 0x7fd670097b60 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6700a2bf0 con 0x7fd670139500 2024-09-16T06:51:20.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient(hunting): authenticate will time out at 1627.260933s 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd675e96700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd670139500 0x7fd6701d0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48384/0 (socket says 172.21.15.76:48384) 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd675e96700 1 -- 172.21.15.76:0/3287639131 learned_addr learned my addr 172.21.15.76:0/3287639131 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334731561 0 0) 0x7fd6700986b0 con 0x7fd670099090 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): my global_id is 254130 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6540035d0 con 0x7fd670099090 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806977147 0 0) 0x7fd6700a2bf0 con 0x7fd670139500 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): my global_id is 244253 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6700986b0 con 0x7fd670139500 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2478152746 0 0) 0x7fd6540035d0 con 0x7fd670099090 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6700a2bf0 con 0x7fd670099090 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c0019a0 con 0x7fd670099090 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.136 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572424470 0 0) 0x7fd67008f7b0 con 0x7fd670097b60 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): my global_id is 224416 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6540035d0 con 0x7fd670097b60 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009339937 0 0) 0x7fd6700986b0 con 0x7fd670139500 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd67008f7b0 con 0x7fd670139500 2024-09-16T06:51:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd668001c30 con 0x7fd670139500 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.138 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2363671393 0 0) 0x7fd6700a2bf0 con 0x7fd670099090 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6790/0 conn(0x7fd670139500 legacy=0x7fd6701d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.189:6789/0 conn(0x7fd670097b60 legacy=0x7fd670138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 monclient: found mon.a 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67009a540 con 0x7fd670099090 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd66c001c60 con 0x7fd670099090 2024-09-16T06:51:20.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd66c004ce0 con 0x7fd670099090 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 5 monclient: authenticate success, global_id 254130 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _renew_subs 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd6637fe700 4 set_mon_vals no callback set 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67008f4d0 con 0x7fd670099090 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient: dump: 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.140 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _renew_subs 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67008c960 con 0x7fd670099090 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd66c00f7e0 con 0x7fd670099090 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd670139b90 con 0x7fd670138e90 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd66c043c10 con 0x7fd670099090 2024-09-16T06:51:20.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556523657 0 0) 0x7fd670139b90 con 0x7fd670138e90 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 10 monclient(hunting): my global_id is 254133 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6700a2bf0 con 0x7fd670138e90 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd675e96700 10 monclient: get_auth_request con 0x7fd654036490 auth_method 0 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.124+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 141694240 0 0) 0x7fd6700a2bf0 con 0x7fd670138e90 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd670139b90 con 0x7fd670138e90 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd664001790 con 0x7fd670138e90 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 10 monclient: dump: 2024-09-16T06:51:20.142 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2133074236 0 0) 0x7fd670139b90 con 0x7fd670138e90 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd67009fb80 con 0x7fd670138e90 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd664001ca0 con 0x7fd670138e90 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:20.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6789/0 conn(0x7fd670138e90 legacy=0x7fd67006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.128+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6789/0 conn(0x7fd670138e90 legacy=0x7fd67006dea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:20.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd670139270 con 0x7fd67002c6b0 2024-09-16T06:51:20.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802187707 0 0) 0x7fd670139270 con 0x7fd67002c6b0 2024-09-16T06:51:20.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient(hunting): my global_id is 254136 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd670139b90 con 0x7fd67002c6b0 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2459049748 0 0) 0x7fd670139b90 con 0x7fd67002c6b0 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd670139270 con 0x7fd67002c6b0 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd664003a30 con 0x7fd67002c6b0 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient: dump: 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.146 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2099326796 0 0) 0x7fd670139270 con 0x7fd67002c6b0 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd670099540 con 0x7fd67002c6b0 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2355249224) 0x7fd664004f30 con 0x7fd67002c6b0 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6789/0 conn(0x7fd67002c6b0 legacy=0x7fd67006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd661ffb700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6789/0 conn(0x7fd67002c6b0 legacy=0x7fd67006e5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.149 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.150 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.151 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.152 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:20.153 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.154 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:20.156 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:20.157 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.159 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:20.159 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd654036490 legacy=0x7fd654038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 10 monclient: shutdown 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 >> v1:172.21.15.76:6789/0 conn(0x7fd670099090 legacy=0x7fd6701cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd675e96700 1 -- 172.21.15.76:0/3287639131 reap_dead start 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd675e96700 1 -- 172.21.15.76:0/3287639131 reap_dead start 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 shutdown_connections 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 >> 172.21.15.76:0/3287639131 conn(0x7fd6700a0030 msgr2=0x7fd67013c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 shutdown_connections 2024-09-16T06:51:20.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.140+0000 7fd676697700 1 -- 172.21.15.76:0/3287639131 wait complete. 2024-09-16T06:51:20.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:20.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:51:20.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:51:20.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 10 monclient: get_monmap_and_config 2024-09-16T06:51:20.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 10 monclient: monmap: 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:20.282858+0000 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:20.282858+0000 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:20.278 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 1 Processor -- start 2024-09-16T06:51:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 1 -- start start 2024-09-16T06:51:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.276+0000 7fc1f73d3700 10 monclient: init 2024-09-16T06:51:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-c con 0x7fc1f009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-b con 0x7fc1f0098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-a con 0x7fc1f0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f009d560 con 0x7fc1f0099030 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f009f140 con 0x7fc1f009ed60 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f009d790 con 0x7fc1f0098be0 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient(hunting): authenticate will time out at 1627.416860s 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f5bd0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1f0098be0 0x7fc1f01374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48390/0 (socket says 172.21.15.76:48390) 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f5bd0700 1 -- 172.21.15.76:0/3801002103 learned_addr learned my addr 172.21.15.76:0/3801002103 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f6bd2700 1 --1- 172.21.15.76:0/3801002103 >> v1:172.21.15.76:6789/0 conn(0x7fc1f0099030 0x7fc1f013ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34160/0 (socket says 172.21.15.76:34160) 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233230653 0 0) 0x7fc1f009d790 con 0x7fc1f0098be0 2024-09-16T06:51:20.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): my global_id is 244256 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc0035d0 con 0x7fc1f0098be0 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35460967 0 0) 0x7fc1f009d560 con 0x7fc1f0099030 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): my global_id is 254139 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f009d790 con 0x7fc1f0099030 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289009231 0 0) 0x7fc1f009f140 con 0x7fc1f009ed60 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient(hunting): my global_id is 224419 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f009d560 con 0x7fc1f009ed60 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30468588 0 0) 0x7fc1f009d790 con 0x7fc1f0099030 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 >> v1:172.21.15.76:6790/0 conn(0x7fc1f0098be0 legacy=0x7fc1f01374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 >> v1:172.21.15.189:6789/0 conn(0x7fc1f009ed60 legacy=0x7fc1f00984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 monclient: found mon.noname-a 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f009f620 con 0x7fc1f0099030 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 5 monclient: authenticate success, global_id 254139 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1e8001970 con 0x7fc1f0099030 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: dump: 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.285 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1e8001c90 con 0x7fc1f0099030 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 1 -- 172.21.15.76:0/3801002103 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1e8002060 con 0x7fc1f0099030 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 4 set_mon_vals no callback set 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f53cf700 10 monclient: dump: 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.287 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 10 monclient: shutdown 2024-09-16T06:51:20.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3801002103 >> v1:172.21.15.76:6789/0 conn(0x7fc1f0099030 legacy=0x7fc1f013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3801002103 shutdown_connections 2024-09-16T06:51:20.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3801002103 >> 172.21.15.76:0/3801002103 conn(0x7fc1f00a0030 msgr2=0x7fc1f00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3801002103 shutdown_connections 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.280+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3801002103 wait complete. 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: monmap: 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:20.294 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 Processor -- start 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 -- start start 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: init 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-a con 0x7fc1f0098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-c con 0x7fc1f0099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient(hunting): picked mon.noname-b con 0x7fc1f009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f0098540 con 0x7fc1f0098be0 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f0092bc0 con 0x7fc1f009ed60 2024-09-16T06:51:20.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00a2d50 con 0x7fc1f0099030 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient(hunting): authenticate will time out at 1627.422293s 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f63d1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc1f0098be0 0x7fc1f00916d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34166/0 (socket says 172.21.15.76:34166) 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f5bd0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1f0099030 0x7fc1f0091de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48406/0 (socket says 172.21.15.76:48406) 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f63d1700 1 -- 172.21.15.76:0/3557245024 learned_addr learned my addr 172.21.15.76:0/3557245024 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790631482 0 0) 0x7fc1f0098540 con 0x7fc1f0098be0 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): my global_id is 254142 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d40035d0 con 0x7fc1f0098be0 2024-09-16T06:51:20.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890798162 0 0) 0x7fc1f00a2d50 con 0x7fc1f0099030 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): my global_id is 244259 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f0098540 con 0x7fc1f0099030 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1997287745 0 0) 0x7fc1d40035d0 con 0x7fc1f0098be0 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f00a2d50 con 0x7fc1f0098be0 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1ec001990 con 0x7fc1f0098be0 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): dump: 2024-09-16T06:51:20.297 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601307776 0 0) 0x7fc1f0092bc0 con 0x7fc1f009ed60 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): my global_id is 224422 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d40035d0 con 0x7fc1f009ed60 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109283670 0 0) 0x7fc1f0098540 con 0x7fc1f0099030 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f0092bc0 con 0x7fc1f0099030 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1e4001740 con 0x7fc1f0099030 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient(hunting): dump: 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.299 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2580494846 0 0) 0x7fc1f00a2d50 con 0x7fc1f0098be0 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6790/0 conn(0x7fc1f0099030 legacy=0x7fc1f0091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.189:6789/0 conn(0x7fc1f009ed60 legacy=0x7fc1f01d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 monclient: found mon.a 2024-09-16T06:51:20.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00953a0 con 0x7fc1f0098be0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 5 monclient: authenticate success, global_id 254142 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: _renew_subs 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f00924f0 con 0x7fc1f0098be0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.284+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1ec0036a0 con 0x7fc1f0098be0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1ec0039b0 con 0x7fc1f0098be0 2024-09-16T06:51:20.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: dump: 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f73d3700 10 monclient: _renew_subs 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f73d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f4bce700 4 set_mon_vals no callback set 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f0093c70 con 0x7fc1f0098be0 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc1ec003bd0 con 0x7fc1f0098be0 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f73d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc1ec043e30 con 0x7fc1f0098be0 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc1f00a2510 con 0x7fc1f002c820 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1f5bd0700 10 monclient: get_auth_request con 0x7fc1d40364e0 auth_method 0 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892407455 0 0) 0x7fc1f00a2510 con 0x7fc1f002c820 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient(hunting): my global_id is 244262 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f00a2d50 con 0x7fc1f002c820 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3841429736 0 0) 0x7fc1f00a2d50 con 0x7fc1f002c820 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f00a2510 con 0x7fc1f002c820 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1e80019f0 con 0x7fc1f002c820 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: dump: 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.304 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1371178458 0 0) 0x7fc1f00a2510 con 0x7fc1f002c820 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1f00a31a0 con 0x7fc1f002c820 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc1e8001fa0 con 0x7fc1f002c820 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6790/0 conn(0x7fc1f002c820 legacy=0x7fc1f0061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.288+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6790/0 conn(0x7fc1f002c820 legacy=0x7fc1f0061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1f73d3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc1f006a9a0 con 0x7fc1f008e010 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671088223 0 0) 0x7fc1f006a9a0 con 0x7fc1f008e010 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient(hunting): my global_id is 244265 2024-09-16T06:51:20.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f00a2510 con 0x7fc1f008e010 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 901824615 0 0) 0x7fc1f00a2510 con 0x7fc1f008e010 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f006a9a0 con 0x7fc1f008e010 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1e8005390 con 0x7fc1f008e010 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient: dump: 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.308 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 105416028 0 0) 0x7fc1f006a9a0 con 0x7fc1f008e010 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc1f009aaa0 con 0x7fc1f008e010 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 3385796396) 0x7fc1e8006fe0 con 0x7fc1f008e010 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.296+0000 7fc1e2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6790/0 conn(0x7fc1f008e010 legacy=0x7fc1f01d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1e2ffd700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6790/0 conn(0x7fc1f008e010 legacy=0x7fc1f01d7a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.309 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:20.310 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.311 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.312 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:20.313 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.314 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.315 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:20.316 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.317 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:20.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:20.321 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc1d40364e0 legacy=0x7fc1d40389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 10 monclient: shutdown 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 >> v1:172.21.15.76:6789/0 conn(0x7fc1f0098be0 legacy=0x7fc1f00916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f6bd2700 1 -- 172.21.15.76:0/3557245024 reap_dead start 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f6bd2700 1 -- 172.21.15.76:0/3557245024 reap_dead start 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 shutdown_connections 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 >> 172.21.15.76:0/3557245024 conn(0x7fc1f00a0030 msgr2=0x7fc1f00702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 shutdown_connections 2024-09-16T06:51:20.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.300+0000 7fc1f73d3700 1 -- 172.21.15.76:0/3557245024 wait complete. 2024-09-16T06:51:20.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:20.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:51:20.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:51:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: get_monmap_and_config 2024-09-16T06:51:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: monmap: 2024-09-16T06:51:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.447 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:20.452586+0000 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:20.452586+0000 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:20.448 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 1 Processor -- start 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 1 -- start start 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: init 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-a con 0x7f1e28097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-c con 0x7f1e28099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-b con 0x7f1e28139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e28098990 con 0x7f1e28097b60 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e28097f40 con 0x7f1e28099090 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2809d610 con 0x7f1e28139500 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient(hunting): authenticate will time out at 1627.585857s 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2da89700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1e28097b60 0x7f1e2809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34180/0 (socket says 172.21.15.76:34180) 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2da89700 1 -- 172.21.15.76:0/1113075284 learned_addr learned my addr 172.21.15.76:0/1113075284 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516651466 0 0) 0x7f1e28098990 con 0x7f1e28097b60 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): my global_id is 254145 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e100035d0 con 0x7f1e28097b60 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481307645 0 0) 0x7f1e2809d610 con 0x7f1e28139500 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): my global_id is 244268 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28098990 con 0x7f1e28139500 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479160616 0 0) 0x7f1e28097f40 con 0x7f1e28099090 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient(hunting): my global_id is 224425 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e2809d610 con 0x7f1e28099090 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124202604 0 0) 0x7f1e100035d0 con 0x7f1e28097b60 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 >> v1:172.21.15.76:6790/0 conn(0x7f1e28139500 legacy=0x7f1e2813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 >> v1:172.21.15.189:6789/0 conn(0x7f1e28099090 legacy=0x7f1e2809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 monclient: found mon.noname-a 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280986b0 con 0x7f1e28097b60 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 5 monclient: authenticate success, global_id 254145 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e1c001610 con 0x7f1e28097b60 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: dump: 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e1c0018f0 con 0x7f1e28097b60 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 1 -- 172.21.15.76:0/1113075284 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e1c001c80 con 0x7f1e28097b60 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ca87700 10 monclient: dump: 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.458 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.448+0000 7f1e2ea8b700 4 set_mon_vals no callback set 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: shutdown 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/1113075284 >> v1:172.21.15.76:6789/0 conn(0x7f1e28097b60 legacy=0x7f1e2809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/1113075284 shutdown_connections 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/1113075284 >> 172.21.15.76:0/1113075284 conn(0x7f1e280a0030 msgr2=0x7f1e280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/1113075284 shutdown_connections 2024-09-16T06:51:20.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/1113075284 wait complete. 2024-09-16T06:51:20.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: monmap: 2024-09-16T06:51:20.465 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.465 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 Processor -- start 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 1 -- start start 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: init 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-b con 0x7f1e28097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.452+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-a con 0x7f1e28099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient(hunting): picked mon.noname-c con 0x7f1e28139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e28139330 con 0x7f1e28099090 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2808f430 con 0x7f1e28097b60 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2813c270 con 0x7f1e28139500 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient(hunting): authenticate will time out at 1627.592115s 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2d288700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1e28099090 0x7f1e28138c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34196/0 (socket says 172.21.15.76:34196) 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2d288700 1 -- 172.21.15.76:0/2229291805 learned_addr learned my addr 172.21.15.76:0/2229291805 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2e28a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1e28139500 0x7f1e281cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48440/0 (socket says 172.21.15.76:48440) 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773910120 0 0) 0x7f1e2813c270 con 0x7f1e28139500 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): my global_id is 244271 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e140035d0 con 0x7f1e28139500 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550694416 0 0) 0x7f1e2808f430 con 0x7f1e28097b60 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): my global_id is 224428 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e2813c270 con 0x7f1e28097b60 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909205738 0 0) 0x7f1e28139330 con 0x7f1e28099090 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): my global_id is 254148 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e2808f430 con 0x7f1e28099090 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 873273030 0 0) 0x7f1e140035d0 con 0x7f1e28139500 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e28139330 con 0x7f1e28139500 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e200018b0 con 0x7f1e28139500 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): dump: 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.469 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1898627867 0 0) 0x7f1e2808f430 con 0x7f1e28099090 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e140035d0 con 0x7f1e28099090 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e18001c70 con 0x7f1e28099090 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): dump: 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172412583 0 0) 0x7f1e2813c270 con 0x7f1e28097b60 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e2808f430 con 0x7f1e28097b60 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e1c001430 con 0x7f1e28097b60 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): dump: 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2080725794 0 0) 0x7f1e28139330 con 0x7f1e28139500 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6789/0 conn(0x7f1e28097b60 legacy=0x7f1e28138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.76:6789/0 conn(0x7f1e28099090 legacy=0x7f1e28138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 monclient: found mon.c 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280a2ca0 con 0x7f1e28139500 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 5 monclient: authenticate success, global_id 244271 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient: _renew_subs 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2808f150 con 0x7f1e28139500 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient: _renew_subs 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2808c5a0 con 0x7f1e28139500 2024-09-16T06:51:20.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e200036a0 con 0x7f1e28139500 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e200039b0 con 0x7f1e28139500 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e27fff700 4 set_mon_vals no callback set 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient: dump: 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.475 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e08006590 con 0x7f1e08002830 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1e20003b10 con 0x7f1e28139500 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1e20043d10 con 0x7f1e28139500 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718986928 0 0) 0x7f1e08006590 con 0x7f1e08002830 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 10 monclient(hunting): my global_id is 224431 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28139330 con 0x7f1e08002830 2024-09-16T06:51:20.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.456+0000 7f1e2d288700 10 monclient: get_auth_request con 0x7f1e14036560 auth_method 0 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 231126536 0 0) 0x7f1e28139330 con 0x7f1e08002830 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e08006590 con 0x7f1e08002830 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e1c004b90 con 0x7f1e08002830 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 10 monclient: dump: 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.477 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1690483316 0 0) 0x7f1e08006590 con 0x7f1e08002830 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e0800a190 con 0x7f1e08002830 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1e1c004e80 con 0x7f1e08002830 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6789/0 conn(0x7f1e08002830 legacy=0x7f1e08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.460+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6789/0 conn(0x7f1e08002830 legacy=0x7f1e08004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.468+0000 7f1e2ea8b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.468+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1e08009e50 con 0x7f1e08009170 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901368485 0 0) 0x7f1e08009e50 con 0x7f1e08009170 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient(hunting): my global_id is 224434 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e08006590 con 0x7f1e08009170 2024-09-16T06:51:20.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2254631333 0 0) 0x7f1e08006590 con 0x7f1e08009170 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e08009e50 con 0x7f1e08009170 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e1c0054e0 con 0x7f1e08009170 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient: dump: 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2270053911 0 0) 0x7f1e08009e50 con 0x7f1e08009170 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1e0800c340 con 0x7f1e08009170 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 3504571085) 0x7f1e1c006850 con 0x7f1e08009170 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6789/0 conn(0x7f1e08009170 legacy=0x7f1e08009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e267fc700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6789/0 conn(0x7f1e08009170 legacy=0x7f1e08009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:20.483 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 114, 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:20.484 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:20.485 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:20.486 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:20.487 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.488 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.490 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:20.491 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:20.492 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:20.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:51:20.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:20.495 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:20.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1e14036560 legacy=0x7f1e14038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 10 monclient: shutdown 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 >> v1:172.21.15.76:6790/0 conn(0x7f1e28139500 legacy=0x7f1e281cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2e28a700 1 -- 172.21.15.76:0/2229291805 reap_dead start 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2e28a700 1 -- 172.21.15.76:0/2229291805 reap_dead start 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 shutdown_connections 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 >> 172.21.15.76:0/2229291805 conn(0x7f1e280a0030 msgr2=0x7f1e28139c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 shutdown_connections 2024-09-16T06:51:20.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.472+0000 7f1e2ea8b700 1 -- 172.21.15.76:0/2229291805 wait complete. 2024-09-16T06:51:20.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:20.498 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:51:20.498 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:51:20.498 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:51:20.498 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:51:20.498 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:51:20.498 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:51:20.531 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:51:20.532 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:51:20.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:20.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.648+0000 7f32368e6700 10 monclient: get_monmap_and_config 2024-09-16T06:51:20.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.648+0000 7f32368e6700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.648+0000 7f32368e6700 10 monclient: monmap: 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:20.655774+0000 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:20.655774+0000 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:20.651 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:20.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 Processor -- start 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- start start 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: init 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-c con 0x7f323009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-b con 0x7f3230098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-a con 0x7f3230099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323009d560 con 0x7f3230099030 2024-09-16T06:51:20.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323009f140 con 0x7f323009ed60 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323009d790 con 0x7f3230098be0 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient(hunting): authenticate will time out at 1627.789025s 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32360e5700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3230099030 legacy=0x7f323013ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32350e3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3230098be0 0x7f32301374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48456/0 (socket says 172.21.15.76:48456) 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32350e3700 1 -- 172.21.15.76:0/2250052253 learned_addr learned my addr 172.21.15.76:0/2250052253 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859202823 0 0) 0x7f323009d790 con 0x7f3230098be0 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): my global_id is 244274 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c0057e0 con 0x7f3230098be0 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811236550 0 0) 0x7f323009f140 con 0x7f323009ed60 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient(hunting): my global_id is 224437 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f323009d790 con 0x7f323009ed60 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3274124207 0 0) 0x7f321c0057e0 con 0x7f3230098be0 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 >> v1:172.21.15.189:6789/0 conn(0x7f323009ed60 legacy=0x7f32300984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 >> v1:172.21.15.76:6789/0 conn(0x7f3230099030 legacy=0x7f323013ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 monclient: found mon.noname-b 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323009f620 con 0x7f3230098be0 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3220001610 con 0x7f3230098be0 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 5 monclient: authenticate success, global_id 244274 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:20.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: dump: 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:20.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32200018f0 con 0x7f3230098be0 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 4 set_mon_vals no callback set 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 1 -- 172.21.15.76:0/2250052253 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3220001c80 con 0x7f3230098be0 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32348e2700 10 monclient: dump: 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: shutdown 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- 172.21.15.76:0/2250052253 >> v1:172.21.15.76:6790/0 conn(0x7f3230098be0 legacy=0x7f32301374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- 172.21.15.76:0/2250052253 shutdown_connections 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- 172.21.15.76:0/2250052253 >> 172.21.15.76:0/2250052253 conn(0x7f32300a0030 msgr2=0x7f32300a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- 172.21.15.76:0/2250052253 shutdown_connections 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 1 -- 172.21.15.76:0/2250052253 wait complete. 2024-09-16T06:51:20.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: build_initial_monmap 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.652+0000 7f32368e6700 10 monclient: monmap: 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:20.662 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 Processor -- start 2024-09-16T06:51:20.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- start start 2024-09-16T06:51:20.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: init 2024-09-16T06:51:20.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:20.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-a con 0x7f3230098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-c con 0x7f3230099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient(hunting): picked mon.noname-b con 0x7f323009ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32301b0240 con 0x7f3230098be0 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32358e4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3230098be0 legacy=0x7f32301b75d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32301b2060 con 0x7f323009ed60 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32300a14e0 con 0x7f3230099030 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient(hunting): authenticate will time out at 1627.793664s 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32350e3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3230099030 0x7f32301ca7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:48466/0 (socket says 172.21.15.76:48466) 2024-09-16T06:51:20.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32350e3700 1 -- 172.21.15.76:0/2828041527 learned_addr learned my addr 172.21.15.76:0/2828041527 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78395266 0 0) 0x7f32300a14e0 con 0x7f3230099030 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): my global_id is 244277 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32140035d0 con 0x7f3230099030 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652142255 0 0) 0x7f32301b2060 con 0x7f323009ed60 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:20.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): my global_id is 224440 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32300a14e0 con 0x7f323009ed60 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846845244 0 0) 0x7f32140035d0 con 0x7f3230099030 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32301b2060 con 0x7f3230099030 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f32200015d0 con 0x7f3230099030 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): dump: 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938200218 0 0) 0x7f32301b2060 con 0x7f3230099030 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 >> v1:172.21.15.189:6789/0 conn(0x7f323009ed60 legacy=0x7f32301cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 >> v1:172.21.15.76:6789/0 conn(0x7f3230098be0 legacy=0x7f32301b75d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 monclient: found mon.c 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32300988d0 con 0x7f3230099030 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 5 monclient: authenticate success, global_id 244277 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _renew_subs 2024-09-16T06:51:20.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323009a540 con 0x7f3230099030 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _renew_subs 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32301afd20 con 0x7f3230099030 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3220004b90 con 0x7f3230099030 2024-09-16T06:51:20.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:20.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3220004f20 con 0x7f3230099030 2024-09-16T06:51:20.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:20.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:20.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32267fc700 10 monclient: dump: 2024-09-16T06:51:20.674 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:20.675 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.656+0000 7f32277fe700 4 set_mon_vals no callback set 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32368e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3220005080 con 0x7f3230099030 2024-09-16T06:51:20.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32368e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3220043ed0 con 0x7f3230099030 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32368e6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3218005100 con 0x7f3230099030 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.660+0000 7f32358e4700 10 monclient: get_auth_request con 0x7f3214035fa0 auth_method 0 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.668+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3220006670 con 0x7f3230099030 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.668+0000 7f32267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:20.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.668+0000 7f32267fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:20.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-16T06:51:20.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.900+0000 7f32368e6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.900+0000 7f32368e6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.900+0000 7f32368e6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.900+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3218005930 con 0x7f3230099030 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32267fc700 1 -- 172.21.15.76:0/2828041527 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 639133375) 0x7f3220012ea0 con 0x7f3230099030 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:20.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32267fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:20.904 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:20.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3214035fa0 legacy=0x7f3214038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 10 monclient: shutdown 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 >> v1:172.21.15.76:6790/0 conn(0x7f3230099030 legacy=0x7f32301ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 shutdown_connections 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 >> 172.21.15.76:0/2828041527 conn(0x7f32300a0030 msgr2=0x7f32300972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 shutdown_connections 2024-09-16T06:51:20.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:20.904+0000 7f32368e6700 1 -- 172.21.15.76:0/2828041527 wait complete. 2024-09-16T06:51:23.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 10 monclient: get_monmap_and_config 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 10 monclient: build_initial_monmap 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 10 monclient: monmap: 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:24.038392+0000 2024-09-16T06:51:24.033 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:24.038392+0000 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:24.034 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 1 Processor -- start 2024-09-16T06:51:24.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 1 -- start start 2024-09-16T06:51:24.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.032+0000 7ff1459be700 10 monclient: init 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-c con 0x7ff14009f800 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-b con 0x7ff140071ce0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-a con 0x7ff14009cca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400a0340 con 0x7ff14009cca0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14009fbe0 con 0x7ff14009f800 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400a0610 con 0x7ff140071ce0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient(hunting): authenticate will time out at 1631.172555s 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13f7fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff14009cca0 legacy=0x7ff14013a510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1367fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff140071ce0 0x7ff14009a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53180/0 (socket says 172.21.15.76:53180) 2024-09-16T06:51:24.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1367fc700 1 -- 172.21.15.76:0/351330639 learned_addr learned my addr 172.21.15.76:0/351330639 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968508881 0 0) 0x7ff1400a0610 con 0x7ff140071ce0 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient(hunting): my global_id is 244280 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c0035d0 con 0x7ff140071ce0 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 918144450 0 0) 0x7ff12c0035d0 con 0x7ff140071ce0 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 >> v1:172.21.15.189:6789/0 conn(0x7ff14009f800 legacy=0x7ff140099c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 >> v1:172.21.15.76:6789/0 conn(0x7ff14009cca0 legacy=0x7ff14013a510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:24.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14009bfe0 con 0x7ff140071ce0 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff128001630 con 0x7ff140071ce0 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: dump: 2024-09-16T06:51:24.040 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1280018e0 con 0x7ff140071ce0 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 5 monclient: authenticate success, global_id 244280 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:24.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 1 -- 172.21.15.76:0/351330639 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff128001c70 con 0x7ff140071ce0 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 4 set_mon_vals no callback set 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff13e7fc700 10 monclient: dump: 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:24.042 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: shutdown 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- 172.21.15.76:0/351330639 >> v1:172.21.15.76:6790/0 conn(0x7ff140071ce0 legacy=0x7ff14009a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- 172.21.15.76:0/351330639 shutdown_connections 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- 172.21.15.76:0/351330639 >> 172.21.15.76:0/351330639 conn(0x7ff1400a3ed0 msgr2=0x7ff1400a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- 172.21.15.76:0/351330639 shutdown_connections 2024-09-16T06:51:24.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 1 -- 172.21.15.76:0/351330639 wait complete. 2024-09-16T06:51:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: build_initial_monmap 2024-09-16T06:51:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.036+0000 7ff1459be700 10 monclient: monmap: 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 Processor -- start 2024-09-16T06:51:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- start start 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: init 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-b con 0x7ff140071ce0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-a con 0x7ff14009cca0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient(hunting): picked mon.noname-c con 0x7ff14009f800 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff140125b70 con 0x7ff14009cca0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14006b6f0 con 0x7ff140071ce0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1367fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7ff14009cca0 legacy=0x7ff14006d130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400a2660 con 0x7ff14009f800 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient(hunting): authenticate will time out at 1631.177380s 2024-09-16T06:51:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff14009f800 0x7ff1401d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53184/0 (socket says 172.21.15.76:53184) 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13f7fe700 1 -- 172.21.15.76:0/2357598842 learned_addr learned my addr 172.21.15.76:0/2357598842 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474133887 0 0) 0x7ff1400a2660 con 0x7ff14009f800 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): my global_id is 244283 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1200035d0 con 0x7ff14009f800 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1254386420 0 0) 0x7ff1200035d0 con 0x7ff14009f800 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1400a2660 con 0x7ff14009f800 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff138004430 con 0x7ff14009f800 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): dump: 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1341163925 0 0) 0x7ff1400a2660 con 0x7ff14009f800 2024-09-16T06:51:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 >> v1:172.21.15.189:6789/0 conn(0x7ff140071ce0 legacy=0x7ff14006ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 >> v1:172.21.15.76:6789/0 conn(0x7ff14009cca0 legacy=0x7ff14006d130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 monclient: found mon.c 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400a22c0 con 0x7ff14009f800 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient: _finish_auth 0 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:51:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff13cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 5 monclient: authenticate success, global_id 244283 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _renew_subs 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14006e390 con 0x7ff14009f800 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _renew_subs 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.040+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14006d840 con 0x7ff14009f800 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff1459be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff1459be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff1459be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14006daf0 con 0x7ff14009f800 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1380046e0 con 0x7ff14009f800 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff138004a70 con 0x7ff14009f800 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13d7fa700 4 set_mon_vals no callback set 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 10 monclient: dump: 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:24.054 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff138004bd0 con 0x7ff14009f800 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff138043ff0 con 0x7ff14009f800 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.044+0000 7ff13effd700 10 monclient: get_auth_request con 0x7ff120036190 auth_method 0 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.052+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff1380146a0 con 0x7ff14009f800 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.052+0000 7ff13cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.052+0000 7ff13cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff1459be700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff1459be700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff1459be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:24.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff1400697f0 con 0x7ff14009f800 2024-09-16T06:51:24.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff13cff9700 1 -- 172.21.15.76:0/2357598842 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1995745438) 0x7ff138012d40 con 0x7ff14009f800 2024-09-16T06:51:24.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff13cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:24.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.292+0000 7ff13cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:24.293 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff120036190 legacy=0x7ff120038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 10 monclient: shutdown 2024-09-16T06:51:24.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 >> v1:172.21.15.76:6790/0 conn(0x7ff14009f800 legacy=0x7ff1401d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 shutdown_connections 2024-09-16T06:51:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 >> 172.21.15.76:0/2357598842 conn(0x7ff1400a3ed0 msgr2=0x7ff1400708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 shutdown_connections 2024-09-16T06:51:24.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:24.296+0000 7ff1459be700 1 -- 172.21.15.76:0/2357598842 wait complete. 2024-09-16T06:51:26.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-16T06:51:27.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:27.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: get_monmap_and_config 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: build_initial_monmap 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: monmap: 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:27.432548+0000 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:27.432548+0000 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:27.428 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 1 Processor -- start 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 1 -- start start 2024-09-16T06:51:27.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: init 2024-09-16T06:51:27.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:27.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:27.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient(hunting): picked mon.noname-b con 0x7f249409f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:27.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient(hunting): picked mon.noname-c con 0x7f249409fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:27.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient(hunting): picked mon.noname-a con 0x7f2494139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:27.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249409cd80 con 0x7f2494139b70 2024-09-16T06:51:27.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2494099fd0 con 0x7f249409fd20 2024-09-16T06:51:27.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249409a410 con 0x7f249409f230 2024-09-16T06:51:27.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:27.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24937fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f249413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:27.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f24996da700 10 monclient(hunting): authenticate will time out at 1634.565919s 2024-09-16T06:51:27.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f2492ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f249409f230 0x7f249409f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53196/0 (socket says 172.21.15.76:53196) 2024-09-16T06:51:27.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.428+0000 7f2492ffd700 1 -- 172.21.15.76:0/3421738078 learned_addr learned my addr 172.21.15.76:0/3421738078 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:27.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:27.632+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f249413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:28.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:28.036+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f249413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:28.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:28.840+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f249413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:30.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): tick 2024-09-16T06:51:30.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:51:30.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:51:30.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6790/0 conn(0x7f249409f230 legacy=0x7f249409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.189:6789/0 conn(0x7f249409fd20 legacy=0x7f2494138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f249413bf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-b con 0x7f2474003740 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-c con 0x7f24740073d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-a con 0x7f2494139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249409de90 con 0x7f2494139b70 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249409cd80 con 0x7f24740073d0 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2474010920 con 0x7f2474003740 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:30.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.428+0000 7f24917fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:30.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:30.632+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:31.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:31.036+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:31.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:31.840+0000 7f24937fe700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:33.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): tick 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 monclient(hunting): continuing hunt 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6790/0 conn(0x7f2474003740 legacy=0x7f2474005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.189:6789/0 conn(0x7f24740073d0 legacy=0x7f2474009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-c con 0x7f247400f9a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-a con 0x7f2494139b70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:33.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): picked mon.noname-b con 0x7f2474003740 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2474011b90 con 0x7f2494139b70 2024-09-16T06:51:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249409de90 con 0x7f247400f9a0 2024-09-16T06:51:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740106b0 con 0x7f2474003740 2024-09-16T06:51:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f24917fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.428+0000 7f248bfff700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:33.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:33.632+0000 7f248bfff700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:34.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.036+0000 7f248bfff700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:34.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.840+0000 7f248bfff700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:34.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.976+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632952531 0 0) 0x7f249409de90 con 0x7f247400f9a0 2024-09-16T06:51:34.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.976+0000 7f24927fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:34.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.976+0000 7f24927fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:34.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.976+0000 7f24927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:34.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.976+0000 7f24927fc700 10 monclient(hunting): my global_id is 224443 2024-09-16T06:51:34.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24800035d0 con 0x7f247400f9a0 2024-09-16T06:51:34.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2544931562 0 0) 0x7f24800035d0 con 0x7f247400f9a0 2024-09-16T06:51:34.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6790/0 conn(0x7f2474003740 legacy=0x7f2474005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:34.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.76:6789/0 conn(0x7f2494139b70 legacy=0x7f247400d150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:34.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 monclient: found mon.noname-c 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2474008860 con 0x7f247400f9a0 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 5 monclient: authenticate success, global_id 224443 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2484001590 con 0x7f247400f9a0 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: dump: 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:34.981 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24840017b0 con 0x7f247400f9a0 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 1 -- 172.21.15.76:0/3421738078 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2484001bc0 con 0x7f247400f9a0 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:34.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 4 set_mon_vals no callback set 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24927fc700 10 monclient: dump: 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:34.983 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:34.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 10 monclient: shutdown 2024-09-16T06:51:34.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:34.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 1 -- 172.21.15.76:0/3421738078 >> v1:172.21.15.189:6789/0 conn(0x7f247400f9a0 legacy=0x7f247400fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:34.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 1 -- 172.21.15.76:0/3421738078 shutdown_connections 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 1 -- 172.21.15.76:0/3421738078 >> 172.21.15.76:0/3421738078 conn(0x7f24940a3e50 msgr2=0x7f24940a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 1 -- 172.21.15.76:0/3421738078 shutdown_connections 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.980+0000 7f24996da700 1 -- 172.21.15.76:0/3421738078 wait complete. 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient: build_initial_monmap 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient: monmap: 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 1 Processor -- start 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 1 -- start start 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient: init 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient(hunting): picked mon.noname-b con 0x7f2474003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient(hunting): picked mon.noname-a con 0x7f247400f9a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient(hunting): picked mon.noname-c con 0x7f2494139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2494072450 con 0x7f247400f9a0 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24941398e0 con 0x7f2474003740 2024-09-16T06:51:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249411e560 con 0x7f2494139b70 2024-09-16T06:51:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248bfff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f247400f9a0 legacy=0x7f24941ccf30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24996da700 10 monclient(hunting): authenticate will time out at 1642.122658s 2024-09-16T06:51:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24937fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2494139b70 0x7f24941d0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55786/0 (socket says 172.21.15.76:55786) 2024-09-16T06:51:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f24937fe700 1 -- 172.21.15.76:0/4019622208 learned_addr learned my addr 172.21.15.76:0/4019622208 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609099393 0 0) 0x7f24941398e0 con 0x7f2474003740 2024-09-16T06:51:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 10 monclient(hunting): my global_id is 224446 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.984+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2474010010 con 0x7f2474003740 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 935303854 0 0) 0x7f2474010010 con 0x7f2474003740 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24941398e0 con 0x7f2474003740 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2484001ac0 con 0x7f2474003740 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient(hunting): dump: 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:34.991 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3749966608 0 0) 0x7f24941398e0 con 0x7f2474003740 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 >> v1:172.21.15.76:6790/0 conn(0x7f2494139b70 legacy=0x7f24941d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 >> v1:172.21.15.76:6789/0 conn(0x7f247400f9a0 legacy=0x7f24941ccf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 monclient: found mon.b 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249406ca50 con 0x7f2474003740 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:34.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 5 monclient: authenticate success, global_id 224446 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _renew_subs 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249411e280 con 0x7f2474003740 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _renew_subs 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f249411ce30 con 0x7f2474003740 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2484004b40 con 0x7f2474003740 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2484004e50 con 0x7f2474003740 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:34.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 10 monclient: dump: 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f2491ffb700 4 set_mon_vals no callback set 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f248400f8c0 con 0x7f2474003740 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:34.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:34.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:51:34.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24940678b0 con 0x7f2474003740 2024-09-16T06:51:34.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2484043e30 con 0x7f2474003740 2024-09-16T06:51:34.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.988+0000 7f248bfff700 10 monclient: get_auth_request con 0x7f2474037720 auth_method 0 2024-09-16T06:51:34.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.996+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2484014600 con 0x7f2474003740 2024-09-16T06:51:34.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.996+0000 7f248b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:34.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:34.996+0000 7f248b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:35.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f24996da700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:35.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f24996da700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f24996da700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f249406fcc0 con 0x7f2474003740 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f248b7fe700 1 -- 172.21.15.76:0/4019622208 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 347742014) 0x7f249406fcc0 con 0x7f2474003740 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f248b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.232+0000 7f248b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:35.234 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:35.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2474037720 legacy=0x7f2474039be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 10 monclient: shutdown 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 >> v1:172.21.15.189:6789/0 conn(0x7f2474003740 legacy=0x7f24941c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 shutdown_connections 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 >> 172.21.15.76:0/4019622208 conn(0x7f24940a3e50 msgr2=0x7f249406cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 shutdown_connections 2024-09-16T06:51:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.236+0000 7f24996da700 1 -- 172.21.15.76:0/4019622208 wait complete. 2024-09-16T06:51:35.248 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:51:35.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:51:35.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.364+0000 7fe20b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.364+0000 7fe20b59e700 10 monclient: build_initial_monmap 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.364+0000 7fe20b59e700 10 monclient: monmap: 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:35.371243+0000 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:35.371243+0000 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:35.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 Processor -- start 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- start start 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: init 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fe20c09b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fe20c137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c098560 con 0x7fe20c137e70 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c099ec0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c09b5f0 con 0x7fe20c0964f0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient(hunting): authenticate will time out at 1642.504771s 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20ad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe20c137e70 legacy=0x7fe20c13a260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20a59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe20c0964f0 0x7fe20c0968d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55800/0 (socket says 172.21.15.76:55800) 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20a59c700 1 -- 172.21.15.76:0/1045623394 learned_addr learned my addr 172.21.15.76:0/1045623394 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101471867 0 0) 0x7fe20c09b5f0 con 0x7fe20c0964f0 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): my global_id is 244286 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80035d0 con 0x7fe20c0964f0 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918078775 0 0) 0x7fe20c099ec0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient(hunting): my global_id is 224449 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c09b5f0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260348280 0 0) 0x7fe1f80035d0 con 0x7fe20c0964f0 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 >> v1:172.21.15.189:6789/0 conn(0x7fe20c09b1c0 legacy=0x7fe20c136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 >> v1:172.21.15.76:6789/0 conn(0x7fe20c137e70 legacy=0x7fe20c13a260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:51:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 monclient: found mon.noname-b 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c099be0 con 0x7fe20c0964f0 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: _finish_auth 0 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 5 monclient: authenticate success, global_id 244286 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1fc001610 con 0x7fe20c0964f0 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: dump: 2024-09-16T06:51:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1fc0018f0 con 0x7fe20c0964f0 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 4 set_mon_vals no callback set 2024-09-16T06:51:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 1 -- 172.21.15.76:0/1045623394 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1fc001c80 con 0x7fe20c0964f0 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe209d9b700 10 monclient: dump: 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: shutdown 2024-09-16T06:51:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- 172.21.15.76:0/1045623394 >> v1:172.21.15.76:6790/0 conn(0x7fe20c0964f0 legacy=0x7fe20c0968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- 172.21.15.76:0/1045623394 shutdown_connections 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- 172.21.15.76:0/1045623394 >> 172.21.15.76:0/1045623394 conn(0x7fe20c0a0030 msgr2=0x7fe20c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- 172.21.15.76:0/1045623394 shutdown_connections 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 1 -- 172.21.15.76:0/1045623394 wait complete. 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: build_initial_monmap 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.368+0000 7fe20b59e700 10 monclient: monmap: 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:35.376 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:35.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:35.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:35.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 Processor -- start 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- start start 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: init 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fe20c09b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:35.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fe20c137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c091250 con 0x7fe20c137e70 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0901a0 con 0x7fe20c0964f0 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c091f80 con 0x7fe20c09b1c0 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient(hunting): authenticate will time out at 1642.509439s 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20ad9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe20c137e70 legacy=0x7fe20c1d0060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe201d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe20c09b1c0 0x7fe20c08e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55814/0 (socket says 172.21.15.76:55814) 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe201d9b700 1 -- 172.21.15.76:0/4165172664 learned_addr learned my addr 172.21.15.76:0/4165172664 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058610921 0 0) 0x7fe20c091f80 con 0x7fe20c09b1c0 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): my global_id is 244289 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec0035d0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616105652 0 0) 0x7fe20c0901a0 con 0x7fe20c0964f0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): my global_id is 224452 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c091f80 con 0x7fe20c0964f0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141611683 0 0) 0x7fe1ec0035d0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe20c0901a0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1f40019d0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): dump: 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:35.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2537447618 0 0) 0x7fe20c091f80 con 0x7fe20c0964f0 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1ec0035d0 con 0x7fe20c0964f0 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1fc0015d0 con 0x7fe20c0964f0 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient(hunting): dump: 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3840834280 0 0) 0x7fe20c0901a0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.189:6789/0 conn(0x7fe20c0964f0 legacy=0x7fe20c08e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6789/0 conn(0x7fe20c137e70 legacy=0x7fe20c1d0060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 monclient: found mon.c 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c09be70 con 0x7fe20c09b1c0 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:35.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 5 monclient: authenticate success, global_id 244289 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _renew_subs 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c09a540 con 0x7fe20c09b1c0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _renew_subs 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c091ca0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f4001cc0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20959a700 4 set_mon_vals no callback set 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1f4003af0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 10 monclient: dump: 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:35.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe1f4003cd0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.372+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe1f0006590 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fe1f4043ee0 con 0x7fe20c09b1c0 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299916910 0 0) 0x7fe1f0006590 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient(hunting): my global_id is 244292 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c0901a0 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe20ad9d700 10 monclient: get_auth_request con 0x7fe1ec0362a0 auth_method 0 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2336633849 0 0) 0x7fe20c0901a0 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f0006590 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1fc004b90 con 0x7fe1f0002830 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient: dump: 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1613693405 0 0) 0x7fe1f0006590 con 0x7fe1f0002830 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1f000a190 con 0x7fe1f0002830 2024-09-16T06:51:35.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe1fc004f00 con 0x7fe1f0002830 2024-09-16T06:51:35.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:35.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:35.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6790/0 conn(0x7fe1f0002830 legacy=0x7fe1f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.376+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6790/0 conn(0x7fe1f0002830 legacy=0x7fe1f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.384+0000 7fe20b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.384+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe1f0009e50 con 0x7fe1f0009170 2024-09-16T06:51:35.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337654016 0 0) 0x7fe1f0009e50 con 0x7fe1f0009170 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient(hunting): my global_id is 244295 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f0006590 con 0x7fe1f0009170 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4180313168 0 0) 0x7fe1f0006590 con 0x7fe1f0009170 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f0009e50 con 0x7fe1f0009170 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe1fc005500 con 0x7fe1f0009170 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient: dump: 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3976454932 0 0) 0x7fe1f0009e50 con 0x7fe1f0009170 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe1f000c340 con 0x7fe1f0009170 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4037 (unknown 0 0 1064249635) 0x7fe1fc0068d0 con 0x7fe1f0009170 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6790/0 conn(0x7fe1f0009170 legacy=0x7fe1f0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe2037fe700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6790/0 conn(0x7fe1f0009170 legacy=0x7fe1f0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 118, 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:35.395 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.396 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:35.398 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.399 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:35.402 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.403 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.404 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.405 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:35.406 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe1ec0362a0 legacy=0x7fe1ec038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 10 monclient: shutdown 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 >> v1:172.21.15.76:6790/0 conn(0x7fe20c09b1c0 legacy=0x7fe20c08e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20ad9d700 1 -- 172.21.15.76:0/4165172664 reap_dead start 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20ad9d700 1 -- 172.21.15.76:0/4165172664 reap_dead start 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 shutdown_connections 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 >> 172.21.15.76:0/4165172664 conn(0x7fe20c0a0030 msgr2=0x7fe20c0a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 shutdown_connections 2024-09-16T06:51:35.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.388+0000 7fe20b59e700 1 -- 172.21.15.76:0/4165172664 wait complete. 2024-09-16T06:51:35.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:51:35.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:51:35.408 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 10 monclient: get_monmap_and_config 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 10 monclient: build_initial_monmap 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 10 monclient: monmap: 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:35.534515+0000 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:35.534515+0000 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:35.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 1 Processor -- start 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 1 -- start start 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.528+0000 7f015a6ca700 10 monclient: init 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-b con 0x7f0154097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-c con 0x7f0154099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-a con 0x7f0154139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154098990 con 0x7f0154139500 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154097f40 con 0x7f0154099090 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015409d610 con 0x7f0154097b60 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient(hunting): authenticate will time out at 1642.668301s 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f01596c8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0154097b60 0x7f015409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55838/0 (socket says 172.21.15.76:55838) 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f0159ec9700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0154139500 legacy=0x7f015413b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f01596c8700 1 -- 172.21.15.76:0/3548496317 learned_addr learned my addr 172.21.15.76:0/3548496317 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494581239 0 0) 0x7f015409d610 con 0x7f0154097b60 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient(hunting): my global_id is 244298 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01400035d0 con 0x7f0154097b60 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416015022 0 0) 0x7f01400035d0 con 0x7f0154097b60 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 >> v1:172.21.15.189:6789/0 conn(0x7f0154099090 legacy=0x7f015409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 >> v1:172.21.15.76:6789/0 conn(0x7f0154139500 legacy=0x7f015413b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 monclient: found mon.noname-b 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540986b0 con 0x7f0154097b60 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: _finish_auth 0 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0150001610 con 0x7f0154097b60 2024-09-16T06:51:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 5 monclient: authenticate success, global_id 244298 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: dump: 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0150001930 con 0x7f0154097b60 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 1 -- 172.21.15.76:0/3548496317 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0150001cc0 con 0x7f0154097b60 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 4 set_mon_vals no callback set 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f014bfff700 10 monclient: dump: 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: shutdown 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:35.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- 172.21.15.76:0/3548496317 >> v1:172.21.15.76:6790/0 conn(0x7f0154097b60 legacy=0x7f015409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- 172.21.15.76:0/3548496317 shutdown_connections 2024-09-16T06:51:35.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- 172.21.15.76:0/3548496317 >> 172.21.15.76:0/3548496317 conn(0x7f01540a0030 msgr2=0x7f01540a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:35.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- 172.21.15.76:0/3548496317 shutdown_connections 2024-09-16T06:51:35.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 -- 172.21.15.76:0/3548496317 wait complete. 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: build_initial_monmap 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 10 monclient: monmap: 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:35.544 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.532+0000 7f015a6ca700 1 Processor -- start 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- start start 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: init 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-a con 0x7f0154097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-b con 0x7f0154099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient(hunting): picked mon.noname-c con 0x7f0154139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154071b10 con 0x7f0154097b60 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154136af0 con 0x7f0154099090 2024-09-16T06:51:35.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015409a540 con 0x7f0154139500 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient(hunting): authenticate will time out at 1642.672910s 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f01596c8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f0154097b60 legacy=0x7f0154135cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0159ec9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0154139500 0x7f01541d2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55842/0 (socket says 172.21.15.76:55842) 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0159ec9700 1 -- 172.21.15.76:0/1246815980 learned_addr learned my addr 172.21.15.76:0/1246815980 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362059577 0 0) 0x7f015409a540 con 0x7f0154139500 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): my global_id is 244301 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01380035d0 con 0x7f0154139500 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179191919 0 0) 0x7f0154136af0 con 0x7f0154099090 2024-09-16T06:51:35.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): my global_id is 224458 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f015409a540 con 0x7f0154099090 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052595513 0 0) 0x7f01380035d0 con 0x7f0154139500 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0154136af0 con 0x7f0154139500 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f014c004460 con 0x7f0154139500 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190100933 0 0) 0x7f0154136af0 con 0x7f0154139500 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6789/0 conn(0x7f0154099090 legacy=0x7f01541363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.76:6789/0 conn(0x7f0154097b60 legacy=0x7f0154135cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 monclient: found mon.c 2024-09-16T06:51:35.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154131fa0 con 0x7f0154139500 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 5 monclient: authenticate success, global_id 244301 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _renew_subs 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154138970 con 0x7f0154139500 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _renew_subs 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f015413c8c0 con 0x7f0154139500 2024-09-16T06:51:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f014c004680 con 0x7f0154139500 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f014c004a10 con 0x7f0154139500 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f0149ffb700 10 monclient: dump: 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.536+0000 7f014b7fe700 4 set_mon_vals no callback set 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f015a6ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0134006590 con 0x7f0134002830 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f014c004bf0 con 0x7f0154139500 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f014c044090 con 0x7f0154139500 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0158ec7700 10 monclient: get_auth_request con 0x7f0138036060 auth_method 0 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516573539 0 0) 0x7f0134006590 con 0x7f0134002830 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient(hunting): my global_id is 224461 2024-09-16T06:51:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0154136af0 con 0x7f0134002830 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 386001687 0 0) 0x7f0154136af0 con 0x7f0134002830 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0134006590 con 0x7f0134002830 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0150001610 con 0x7f0134002830 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient: dump: 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3226145749 0 0) 0x7f0134006590 con 0x7f0134002830 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f013400a190 con 0x7f0134002830 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0150001b80 con 0x7f0134002830 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6789/0 conn(0x7f0134002830 legacy=0x7f0134004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.540+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6789/0 conn(0x7f0134002830 legacy=0x7f0134004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.552+0000 7f015a6ca700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:35.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.552+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0134009e30 con 0x7f0134009150 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159738987 0 0) 0x7f0134009e30 con 0x7f0134009150 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient(hunting): my global_id is 224464 2024-09-16T06:51:35.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134006590 con 0x7f0134009150 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2043689957 0 0) 0x7f0134006590 con 0x7f0134009150 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0134009e30 con 0x7f0134009150 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0150004d80 con 0x7f0134009150 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:35.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient: dump: 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3577733666 0 0) 0x7f0134009e30 con 0x7f0134009150 2024-09-16T06:51:35.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f013400c340 con 0x7f0134009150 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 4288662153) 0x7f0150004f90 con 0x7f0134009150 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6789/0 conn(0x7f0134009150 legacy=0x7f01340095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0149ffb700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6789/0 conn(0x7f0134009150 legacy=0x7f01340095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:35.558 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 118, 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:35.559 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:35.560 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.561 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.561 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.561 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.561 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.561 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.563 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:35.564 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:35.565 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:35.566 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:35.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:35.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.571 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:35.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:35.573 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:35.573 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0138036060 legacy=0x7f0138038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 10 monclient: shutdown 2024-09-16T06:51:35.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f0159ec9700 1 -- 172.21.15.76:0/1246815980 reap_dead start 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 >> v1:172.21.15.76:6790/0 conn(0x7f0154139500 legacy=0x7f01541d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 shutdown_connections 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 >> 172.21.15.76:0/1246815980 conn(0x7f01540a0030 msgr2=0x7f01540702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 shutdown_connections 2024-09-16T06:51:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:35.556+0000 7f015a6ca700 1 -- 172.21.15.76:0/1246815980 wait complete. 2024-09-16T06:51:35.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:51:35.575 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:51:55.577 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:51:55.577 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:51:55.577 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:51:55.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:51:55.581 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:51:55.581 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:51:55.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:55.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.699+0000 7f1bc49b2700 10 monclient: get_monmap_and_config 2024-09-16T06:51:55.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.699+0000 7f1bc49b2700 10 monclient: build_initial_monmap 2024-09-16T06:51:55.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.699+0000 7f1bc49b2700 10 monclient: monmap: 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:55.703709+0000 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:55.703709+0000 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:55.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 Processor -- start 2024-09-16T06:51:55.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- start start 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: init 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-c con 0x7f1bc00964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-a con 0x7f1bc009b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-b con 0x7f1bc0137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc0098560 con 0x7f1bc009b1c0 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc0099ec0 con 0x7f1bc00964f0 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc009b5f0 con 0x7f1bc0137e70 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:55.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient(hunting): authenticate will time out at 1662.837102s 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1bc0137e70 0x7f1bc013a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60190/0 (socket says 172.21.15.76:60190) 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbffff700 1 -- 172.21.15.76:0/1204851233 learned_addr learned my addr 172.21.15.76:0/1204851233 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905284446 0 0) 0x7f1bc009b5f0 con 0x7f1bc0137e70 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): my global_id is 244304 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba80035d0 con 0x7f1bc0137e70 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121040013 0 0) 0x7f1bc0099ec0 con 0x7f1bc00964f0 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:55.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient(hunting): my global_id is 224467 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc009b5f0 con 0x7f1bc00964f0 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802045726 0 0) 0x7f1ba80035d0 con 0x7f1bc0137e70 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 >> v1:172.21.15.189:6789/0 conn(0x7f1bc00964f0 legacy=0x7f1bc00968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 >> v1:172.21.15.76:6789/0 conn(0x7f1bc009b1c0 legacy=0x7f1bc0136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 monclient: found mon.noname-b 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc0099be0 con 0x7f1bc0137e70 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: _finish_auth 0 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:51:55.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 5 monclient: authenticate success, global_id 244304 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1bb0001610 con 0x7f1bc0137e70 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: dump: 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:55.709 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb00018f0 con 0x7f1bc0137e70 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:55.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 1 -- 172.21.15.76:0/1204851233 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1bb0001c80 con 0x7f1bc0137e70 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 4 set_mon_vals no callback set 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bbe7fc700 10 monclient: dump: 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:55.711 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: shutdown 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1204851233 >> v1:172.21.15.76:6790/0 conn(0x7f1bc0137e70 legacy=0x7f1bc013a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1204851233 shutdown_connections 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1204851233 >> 172.21.15.76:0/1204851233 conn(0x7f1bc00a0030 msgr2=0x7f1bc00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1204851233 shutdown_connections 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1204851233 wait complete. 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: build_initial_monmap 2024-09-16T06:51:55.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.703+0000 7f1bc49b2700 10 monclient: monmap: 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 Processor -- start 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- start start 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: init 2024-09-16T06:51:55.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-c con 0x7f1bc00964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-b con 0x7f1bc009b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient(hunting): picked mon.noname-a con 0x7f1bc0137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc0095380 con 0x7f1bc0137e70 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc008be30 con 0x7f1bc009b1c0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc008c6d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient(hunting): authenticate will time out at 1662.840695s 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bbf7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1bc00964f0 0x7f1bc0094c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60200/0 (socket says 172.21.15.76:60200) 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bbf7fe700 1 -- 172.21.15.76:0/1281698583 learned_addr learned my addr 172.21.15.76:0/1281698583 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255171324 0 0) 0x7f1bc008c6d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): my global_id is 244307 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bac0035d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2420094728 0 0) 0x7f1bac0035d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc008c6d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195062600 0 0) 0x7f1bc008be30 con 0x7f1bc009b1c0 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): my global_id is 224470 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bac0035d0 con 0x7f1bc009b1c0 2024-09-16T06:51:55.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1bb8001cf0 con 0x7f1bc00964f0 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): dump: 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:55.716 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 322073729 0 0) 0x7f1bc008c6d0 con 0x7f1bc00964f0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 >> v1:172.21.15.189:6789/0 conn(0x7f1bc009b1c0 legacy=0x7f1bc01cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 >> v1:172.21.15.76:6789/0 conn(0x7f1bc0137e70 legacy=0x7f1bc01d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 monclient: found mon.c 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc013b310 con 0x7f1bc00964f0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: _finish_auth 0 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:55.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 5 monclient: authenticate success, global_id 244307 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _renew_subs 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc00a2df0 con 0x7f1bc00964f0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _renew_subs 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc008c110 con 0x7f1bc00964f0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb80020a0 con 0x7f1bc00964f0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1bb8001f20 con 0x7f1bc00964f0 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 10 monclient: dump: 2024-09-16T06:51:55.718 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:55.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bbdffb700 4 set_mon_vals no callback set 2024-09-16T06:51:55.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1bb8003f50 con 0x7f1bc00964f0 2024-09-16T06:51:55.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:55.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.707+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba0005100 con 0x7f1bc00964f0 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.711+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1bb8044270 con 0x7f1bc00964f0 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.711+0000 7f1bbeffd700 10 monclient: get_auth_request con 0x7f1bac036200 auth_method 0 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.719+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1bb8014c30 con 0x7f1bc00964f0 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.719+0000 7f1ba7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.719+0000 7f1ba7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:55.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1ba0005930 con 0x7f1bc00964f0 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1ba7fff700 1 -- 172.21.15.76:0/1281698583 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3301497674) 0x7f1bb801d200 con 0x7f1bc00964f0 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1ba7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1ba7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:55.948 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1bac036200 legacy=0x7f1bac0386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 10 monclient: shutdown 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 >> v1:172.21.15.76:6790/0 conn(0x7f1bc00964f0 legacy=0x7f1bc0094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 shutdown_connections 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 >> 172.21.15.76:0/1281698583 conn(0x7f1bc00a0030 msgr2=0x7f1bc009bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 shutdown_connections 2024-09-16T06:51:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:55.951+0000 7f1bc49b2700 1 -- 172.21.15.76:0/1281698583 wait complete. 2024-09-16T06:51:55.967 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:51:55.967 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:51:55.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:51:56.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: get_monmap_and_config 2024-09-16T06:51:56.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: monmap: 2024-09-16T06:51:56.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:56.084333+0000 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:56.084333+0000 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 Processor -- start 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- start start 2024-09-16T06:51:56.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: init 2024-09-16T06:51:56.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:56.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-a con 0x7f1ea0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-c con 0x7f1ea0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-b con 0x7f1ea0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0098990 con 0x7f1ea0097b60 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0097f40 con 0x7f1ea0099090 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea009d610 con 0x7f1ea0139500 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient(hunting): authenticate will time out at 1663.217485s 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea565d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1ea0139500 0x7f1ea013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60206/0 (socket says 172.21.15.76:60206) 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea4e5c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1ea0097b60 0x7f1ea009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55956/0 (socket says 172.21.15.76:55956) 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea565d700 1 -- 172.21.15.76:0/3512538826 learned_addr learned my addr 172.21.15.76:0/3512538826 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192195567 0 0) 0x7f1ea009d610 con 0x7f1ea0139500 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): my global_id is 244310 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e900035d0 con 0x7f1ea0139500 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929422873 0 0) 0x7f1ea0097f40 con 0x7f1ea0099090 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient(hunting): my global_id is 224473 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea009d610 con 0x7f1ea0099090 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 665600933 0 0) 0x7f1ea009d610 con 0x7f1ea0099090 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 >> v1:172.21.15.76:6790/0 conn(0x7f1ea0139500 legacy=0x7f1ea013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 >> v1:172.21.15.76:6789/0 conn(0x7f1ea0097b60 legacy=0x7f1ea009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 monclient: found mon.noname-c 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00986b0 con 0x7f1ea0099090 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e8c0015f0 con 0x7f1ea0099090 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: dump: 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.088 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 5 monclient: authenticate success, global_id 224473 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e8c001910 con 0x7f1ea0099090 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 1 -- 172.21.15.76:0/3512538826 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e8c001ca0 con 0x7f1ea0099090 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:56.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 4 set_mon_vals no callback set 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1e9f7fe700 10 monclient: dump: 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 10 monclient: shutdown 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/3512538826 >> v1:172.21.15.189:6789/0 conn(0x7f1ea0099090 legacy=0x7f1ea009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/3512538826 shutdown_connections 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/3512538826 >> 172.21.15.76:0/3512538826 conn(0x7f1ea00a0030 msgr2=0x7f1ea00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/3512538826 shutdown_connections 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.083+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/3512538826 wait complete. 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: monmap: 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.091 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 Processor -- start 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- start start 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: init 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-b con 0x7f1ea0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-c con 0x7f1ea0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient(hunting): picked mon.noname-a con 0x7f1ea0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0128b00 con 0x7f1ea0139500 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0128890 con 0x7f1ea0097b60 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea009b670 con 0x7f1ea0099090 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient(hunting): authenticate will time out at 1663.221049s 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1ea0099090 0x7f1ea009af60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60208/0 (socket says 172.21.15.76:60208) 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9ffff700 1 -- 172.21.15.76:0/542472840 learned_addr learned my addr 172.21.15.76:0/542472840 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850096804 0 0) 0x7f1ea009b670 con 0x7f1ea0099090 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): my global_id is 244313 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e840035d0 con 0x7f1ea0099090 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99816719 0 0) 0x7f1e840035d0 con 0x7f1ea0099090 2024-09-16T06:51:56.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea009b670 con 0x7f1ea0099090 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e8c003760 con 0x7f1ea0099090 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): dump: 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4095400873 0 0) 0x7f1ea009b670 con 0x7f1ea0099090 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.189:6789/0 conn(0x7f1ea0097b60 legacy=0x7f1ea009a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6789/0 conn(0x7f1ea0139500 legacy=0x7f1ea01d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 monclient: found mon.c 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea013c0b0 con 0x7f1ea0099090 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1e9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 5 monclient: authenticate success, global_id 244313 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _renew_subs 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea00a2c10 con 0x7f1ea0099090 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _renew_subs 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea009c850 con 0x7f1ea0099090 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.087+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ea00994e0 con 0x7f1ea0135800 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e8c003cc0 con 0x7f1ea0099090 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e8c004050 con 0x7f1ea0099090 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9effd700 4 set_mon_vals no callback set 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient: dump: 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1e8c004230 con 0x7f1ea0099090 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1e8c0440c0 con 0x7f1ea0099090 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1ea565d700 10 monclient: get_auth_request con 0x7f1e840360d0 auth_method 0 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148737684 0 0) 0x7f1ea0128890 con 0x7f1ea0135800 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 10 monclient(hunting): my global_id is 264099 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.091+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea009b670 con 0x7f1ea0135800 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2933302790 0 0) 0x7f1ea009b670 con 0x7f1ea0135800 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea0128890 con 0x7f1ea0135800 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e94001490 con 0x7f1ea0135800 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 10 monclient: dump: 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.100 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4123627156 0 0) 0x7f1ea0128890 con 0x7f1ea0135800 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ea009fb80 con 0x7f1ea0135800 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1e94001ad0 con 0x7f1ea0135800 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6789/0 conn(0x7f1ea0135800 legacy=0x7f1ea0135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.095+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6789/0 conn(0x7f1ea0135800 legacy=0x7f1ea0135be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1ea00994e0 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146358585 0 0) 0x7f1ea0098990 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient(hunting): my global_id is 264102 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0128890 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3576050934 0 0) 0x7f1ea0128890 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea0098990 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e94004890 con 0x7f1ea01362f0 2024-09-16T06:51:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient: dump: 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2121203113 0 0) 0x7f1ea0098990 con 0x7f1ea01362f0 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ea00973c0 con 0x7f1ea01362f0 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1277679814) 0x7f1e940058c0 con 0x7f1ea01362f0 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6789/0 conn(0x7f1ea01362f0 legacy=0x7f1ea002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1e9d7fa700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6789/0 conn(0x7f1ea01362f0 legacy=0x7f1ea002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:56.105 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.106 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.107 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:56.108 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.109 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:56.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.114 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.115 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:56.116 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:56.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1e840360d0 legacy=0x7f1e84038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 10 monclient: shutdown 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 >> v1:172.21.15.76:6790/0 conn(0x7f1ea0099090 legacy=0x7f1ea009af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.103+0000 7f1ea565d700 1 -- 172.21.15.76:0/542472840 reap_dead start 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.107+0000 7f1ea565d700 1 -- 172.21.15.76:0/542472840 reap_dead start 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.107+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 shutdown_connections 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.107+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 >> 172.21.15.76:0/542472840 conn(0x7f1ea00a0030 msgr2=0x7f1ea013ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.107+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 shutdown_connections 2024-09-16T06:51:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.107+0000 7f1ea5e5e700 1 -- 172.21.15.76:0/542472840 wait complete. 2024-09-16T06:51:56.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:56.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:51:56.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 10 monclient: get_monmap_and_config 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 10 monclient: monmap: 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:56.238237+0000 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:56.238237+0000 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.233 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.234 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.234 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:56.234 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:56.234 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 1 Processor -- start 2024-09-16T06:51:56.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 1 -- start start 2024-09-16T06:51:56.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.235+0000 7f5461a22700 10 monclient: init 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-b con 0x7f545c135500 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-a con 0x7f545c09a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-c con 0x7f545c13b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c0a67b0 con 0x7f545c09a9d0 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c0a7320 con 0x7f545c13b980 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c09ade0 con 0x7f545c135500 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): authenticate will time out at 1663.372510s 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f545c09a9d0 0x7f545c13a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56002/0 (socket says 172.21.15.76:56002) 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545a7fc700 1 -- 172.21.15.76:0/460990966 learned_addr learned my addr 172.21.15.76:0/460990966 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545affd700 1 --1- 172.21.15.76:0/460990966 >> v1:172.21.15.76:6790/0 conn(0x7f545c135500 0x7f545c1358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60220/0 (socket says 172.21.15.76:60220) 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208007609 0 0) 0x7f545c0a67b0 con 0x7f545c09a9d0 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): my global_id is 264105 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54440035d0 con 0x7f545c09a9d0 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592282721 0 0) 0x7f545c09ade0 con 0x7f545c135500 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient(hunting): my global_id is 244316 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c0a67b0 con 0x7f545c135500 2024-09-16T06:51:56.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224357833 0 0) 0x7f54440035d0 con 0x7f545c09a9d0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 >> v1:172.21.15.76:6790/0 conn(0x7f545c135500 legacy=0x7f545c1358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 >> v1:172.21.15.189:6789/0 conn(0x7f545c13b980 legacy=0x7f545c13dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 monclient: found mon.noname-a 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c099d10 con 0x7f545c09a9d0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f544c001610 con 0x7f545c09a9d0 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:56.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: dump: 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 5 monclient: authenticate success, global_id 264105 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f544c0018f0 con 0x7f545c09a9d0 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 4 set_mon_vals no callback set 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 1 -- 172.21.15.76:0/460990966 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f544c001cc0 con 0x7f545c09a9d0 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5459ffb700 10 monclient: dump: 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: shutdown 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- 172.21.15.76:0/460990966 >> v1:172.21.15.76:6789/0 conn(0x7f545c09a9d0 legacy=0x7f545c13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- 172.21.15.76:0/460990966 shutdown_connections 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- 172.21.15.76:0/460990966 >> 172.21.15.76:0/460990966 conn(0x7f545c0a42f0 msgr2=0x7f545c0a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- 172.21.15.76:0/460990966 shutdown_connections 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- 172.21.15.76:0/460990966 wait complete. 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: monmap: 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:56.248 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:56.249 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:56.249 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.249 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 Processor -- start 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- start start 2024-09-16T06:51:56.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: init 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-a con 0x7f545c09a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-b con 0x7f545c135500 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): picked mon.noname-c con 0x7f545c13b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c06d430 con 0x7f545c09a9d0 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c12fc10 con 0x7f545c135500 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545c132470 con 0x7f545c13b980 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f5461a22700 10 monclient(hunting): authenticate will time out at 1663.375738s 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f545c09a9d0 0x7f545c132890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56004/0 (socket says 172.21.15.76:56004) 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545affd700 1 -- 172.21.15.76:0/2204973097 learned_addr learned my addr 172.21.15.76:0/2204973097 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.239+0000 7f545b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f545c13b980 0x7f545c1d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60228/0 (socket says 172.21.15.76:60228) 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616431361 0 0) 0x7f545c06d430 con 0x7f545c09a9d0 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): my global_id is 264108 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54480035d0 con 0x7f545c09a9d0 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647954986 0 0) 0x7f545c132470 con 0x7f545c13b980 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): my global_id is 244319 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c06d430 con 0x7f545c13b980 2024-09-16T06:51:56.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404558084 0 0) 0x7f54480035d0 con 0x7f545c09a9d0 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f545c132470 con 0x7f545c09a9d0 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f54500018d0 con 0x7f545c09a9d0 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): dump: 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.255 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032663853 0 0) 0x7f545c12fc10 con 0x7f545c135500 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): my global_id is 224479 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54480035d0 con 0x7f545c135500 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025629882 0 0) 0x7f545c06d430 con 0x7f545c13b980 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f545c12fc10 con 0x7f545c13b980 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5454001bb0 con 0x7f545c13b980 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): dump: 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574823269 0 0) 0x7f545c132470 con 0x7f545c09a9d0 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6790/0 conn(0x7f545c13b980 legacy=0x7f545c1d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.189:6789/0 conn(0x7f545c135500 legacy=0x7f545c1ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 monclient: found mon.a 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c134f00 con 0x7f545c09a9d0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 5 monclient: authenticate success, global_id 264108 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 10 monclient: _renew_subs 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545c131bd0 con 0x7f545c09a9d0 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 10 monclient: _renew_subs 2024-09-16T06:51:56.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f545c1300c0 con 0x7f545c09a9d0 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5450001ca0 con 0x7f545c09a9d0 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5450004ce0 con 0x7f545c09a9d0 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f54597fa700 4 set_mon_vals no callback set 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: dump: 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f545000f850 con 0x7f545c09a9d0 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f543c006590 con 0x7f543c002830 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5450043ed0 con 0x7f545c09a9d0 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177077627 0 0) 0x7f543c006590 con 0x7f543c002830 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient(hunting): my global_id is 244322 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c132470 con 0x7f543c002830 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f545b7fe700 10 monclient: get_auth_request con 0x7f54480363b0 auth_method 0 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2329947207 0 0) 0x7f545c132470 con 0x7f543c002830 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f543c006590 con 0x7f543c002830 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f544c001690 con 0x7f543c002830 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 10 monclient: dump: 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.261 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3942386489 0 0) 0x7f543c006590 con 0x7f543c002830 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.243+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f543c00a190 con 0x7f543c002830 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.247+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f544c001cb0 con 0x7f543c002830 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.247+0000 7f5443fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.247+0000 7f5443fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.247+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6790/0 conn(0x7f543c002830 legacy=0x7f543c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.247+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6790/0 conn(0x7f543c002830 legacy=0x7f543c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f543c009e50 con 0x7f543c009170 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933429989 0 0) 0x7f543c009e50 con 0x7f543c009170 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient(hunting): my global_id is 244325 2024-09-16T06:51:56.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c006590 con 0x7f543c009170 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 593431750 0 0) 0x7f543c006590 con 0x7f543c009170 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f543c009e50 con 0x7f543c009170 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f544c004e00 con 0x7f543c009170 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient: dump: 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.265 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3805928019 0 0) 0x7f543c009e50 con 0x7f543c009170 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f543c00c340 con 0x7f543c009170 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 4133317221) 0x7f544c005010 con 0x7f543c009170 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6790/0 conn(0x7f543c009170 legacy=0x7f543c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5443fff700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6790/0 conn(0x7f543c009170 legacy=0x7f543c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.267 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.268 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:56.270 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.271 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:56.273 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.276 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.277 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:56.278 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.189:6812/3098810674 conn(0x7f54480363b0 legacy=0x7f5448038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 10 monclient: shutdown 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 >> v1:172.21.15.76:6789/0 conn(0x7f545c09a9d0 legacy=0x7f545c132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f545b7fe700 1 -- 172.21.15.76:0/2204973097 reap_dead start 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f545b7fe700 1 -- 172.21.15.76:0/2204973097 reap_dead start 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 shutdown_connections 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 >> 172.21.15.76:0/2204973097 conn(0x7f545c0a42f0 msgr2=0x7f545c12c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 shutdown_connections 2024-09-16T06:51:56.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.255+0000 7f5461a22700 1 -- 172.21.15.76:0/2204973097 wait complete. 2024-09-16T06:51:56.281 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:56.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:51:56.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: get_monmap_and_config 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: monmap: 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:56.402229+0000 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:56.402229+0000 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.397 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:56.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:56.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 1 Processor -- start 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 1 -- start start 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: init 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-b con 0x7ff4c009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-c con 0x7ff4c009d520 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-a con 0x7ff4c009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c009b520 con 0x7ff4c009fd20 2024-09-16T06:51:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c0071f30 con 0x7ff4c009d520 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c009b0e0 con 0x7ff4c009f230 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c67fc700 10 monclient(hunting): authenticate will time out at 1663.535486s 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4bffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff4c009f230 0x7ff4c009f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60268/0 (socket says 172.21.15.76:60268) 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4bffff700 1 -- 172.21.15.76:0/2631435648 learned_addr learned my addr 172.21.15.76:0/2631435648 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.399+0000 7ff4c4d99700 1 --1- 172.21.15.76:0/2631435648 >> v1:172.21.15.76:6789/0 conn(0x7ff4c009fd20 0x7ff4c013ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56010/0 (socket says 172.21.15.76:56010) 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990025009 0 0) 0x7ff4c009b520 con 0x7ff4c009fd20 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): my global_id is 264111 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a80035d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939484386 0 0) 0x7ff4c009b0e0 con 0x7ff4c009f230 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): my global_id is 244328 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c009b520 con 0x7ff4c009f230 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058099900 0 0) 0x7ff4c0071f30 con 0x7ff4c009d520 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient(hunting): my global_id is 224482 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c009b0e0 con 0x7ff4c009d520 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2983410113 0 0) 0x7ff4a80035d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 >> v1:172.21.15.76:6790/0 conn(0x7ff4c009f230 legacy=0x7ff4c009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 >> v1:172.21.15.189:6789/0 conn(0x7ff4c009d520 legacy=0x7ff4c0137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 monclient: found mon.noname-a 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c009c7d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b8001990 con 0x7ff4c009fd20 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 5 monclient: authenticate success, global_id 264111 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: dump: 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b8001bb0 con 0x7ff4c009fd20 2024-09-16T06:51:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 1 -- 172.21.15.76:0/2631435648 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b8001f00 con 0x7ff4c009fd20 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 4 set_mon_vals no callback set 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4beffd700 10 monclient: dump: 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 10 monclient: shutdown 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- 172.21.15.76:0/2631435648 >> v1:172.21.15.76:6789/0 conn(0x7ff4c009fd20 legacy=0x7ff4c013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- 172.21.15.76:0/2631435648 shutdown_connections 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- 172.21.15.76:0/2631435648 >> 172.21.15.76:0/2631435648 conn(0x7ff4c00a3e50 msgr2=0x7ff4c00a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- 172.21.15.76:0/2631435648 shutdown_connections 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- 172.21.15.76:0/2631435648 wait complete. 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 10 monclient: monmap: 2024-09-16T06:51:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 Processor -- start 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 1 -- start start 2024-09-16T06:51:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.403+0000 7ff4c67fc700 10 monclient: init 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-a con 0x7ff4c009f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-c con 0x7ff4c009fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient(hunting): picked mon.noname-b con 0x7ff4c01cb950 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c012ff30 con 0x7ff4c009f230 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00a2b30 con 0x7ff4c01cb950 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c0072450 con 0x7ff4c009fd20 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bf7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff4c009fd20 0x7ff4c01ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60272/0 (socket says 172.21.15.76:60272) 2024-09-16T06:51:56.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bf7fe700 1 -- 172.21.15.76:0/233869804 learned_addr learned my addr 172.21.15.76:0/233869804 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient(hunting): authenticate will time out at 1663.540777s 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194664174 0 0) 0x7ff4c0072450 con 0x7ff4c009fd20 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): my global_id is 244331 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ac0035d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969891318 0 0) 0x7ff4c012ff30 con 0x7ff4c009f230 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): my global_id is 264114 2024-09-16T06:51:56.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c0072450 con 0x7ff4c009f230 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957871729 0 0) 0x7ff4ac0035d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4c012ff30 con 0x7ff4c009fd20 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b40018d0 con 0x7ff4c009fd20 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): dump: 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413617010 0 0) 0x7ff4c00a2b30 con 0x7ff4c01cb950 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): my global_id is 224485 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ac0035d0 con 0x7ff4c01cb950 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384185496 0 0) 0x7ff4c0072450 con 0x7ff4c009f230 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4c00a2b30 con 0x7ff4c009f230 2024-09-16T06:51:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b00019d0 con 0x7ff4c009f230 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient(hunting): dump: 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 621877524 0 0) 0x7ff4c012ff30 con 0x7ff4c009fd20 2024-09-16T06:51:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6789/0 conn(0x7ff4c01cb950 legacy=0x7ff4c01cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.76:6789/0 conn(0x7ff4c009f230 legacy=0x7ff4c01309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 monclient: found mon.c 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c011dd90 con 0x7ff4c009fd20 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 5 monclient: authenticate success, global_id 244331 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _renew_subs 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c011e480 con 0x7ff4c009fd20 2024-09-16T06:51:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _renew_subs 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c011f8c0 con 0x7ff4c009fd20 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b40036c0 con 0x7ff4c009fd20 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b4003a50 con 0x7ff4c009fd20 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4be7fc700 4 set_mon_vals no callback set 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 10 monclient: dump: 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff4a0006590 con 0x7ff4a0002830 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff4b4003c30 con 0x7ff4c009fd20 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.407+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff4b4043df0 con 0x7ff4c009fd20 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4c4d99700 10 monclient: get_auth_request con 0x7ff4ac0364e0 auth_method 0 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259845886 0 0) 0x7ff4a0006590 con 0x7ff4a0002830 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient(hunting): my global_id is 224488 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c012ff30 con 0x7ff4a0002830 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2019425808 0 0) 0x7ff4c012ff30 con 0x7ff4a0002830 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4a0006590 con 0x7ff4a0002830 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b0004a30 con 0x7ff4a0002830 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient: dump: 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 378024698 0 0) 0x7ff4a0006590 con 0x7ff4a0002830 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4a000a190 con 0x7ff4a0002830 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff4b0004cd0 con 0x7ff4a0002830 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6789/0 conn(0x7ff4a0002830 legacy=0x7ff4a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.411+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6789/0 conn(0x7ff4a0002830 legacy=0x7ff4a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4c67fc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:56.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff4a0009e30 con 0x7ff4a0009150 2024-09-16T06:51:56.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150962259 0 0) 0x7ff4a0009e30 con 0x7ff4a0009150 2024-09-16T06:51:56.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient(hunting): my global_id is 224491 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a0006590 con 0x7ff4a0009150 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 836973458 0 0) 0x7ff4a0006590 con 0x7ff4a0009150 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4a0009e30 con 0x7ff4a0009150 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff4b00052a0 con 0x7ff4a0009150 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 10 monclient: dump: 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.426 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3034168246 0 0) 0x7ff4a0009e30 con 0x7ff4a0009150 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.423+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff4a000c340 con 0x7ff4a0009150 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 1973275363) 0x7ff4b0006630 con 0x7ff4a0009150 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4bcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4bcff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6789/0 conn(0x7ff4a0009150 legacy=0x7ff4a00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4bcff9700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6789/0 conn(0x7ff4a0009150 legacy=0x7ff4a00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:56.428 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.429 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.430 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:56.431 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:56.432 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:56.434 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.435 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.436 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.437 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.438 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:56.439 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff4ac0364e0 legacy=0x7ff4ac0389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 10 monclient: shutdown 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 >> v1:172.21.15.76:6790/0 conn(0x7ff4c009fd20 legacy=0x7ff4c01ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c4d99700 1 -- 172.21.15.76:0/233869804 reap_dead start 2024-09-16T06:51:56.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c4d99700 1 -- 172.21.15.76:0/233869804 reap_dead start 2024-09-16T06:51:56.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 shutdown_connections 2024-09-16T06:51:56.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 >> 172.21.15.76:0/233869804 conn(0x7ff4c00a3e50 msgr2=0x7ff4c0099af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 shutdown_connections 2024-09-16T06:51:56.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.427+0000 7ff4c67fc700 1 -- 172.21.15.76:0/233869804 wait complete. 2024-09-16T06:51:56.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:56.442 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:51:56.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:51:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: get_monmap_and_config 2024-09-16T06:51:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: monmap: 2024-09-16T06:51:56.552 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:56.557598+0000 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:56.557598+0000 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:56.553 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 1 Processor -- start 2024-09-16T06:51:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 1 -- start start 2024-09-16T06:51:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: init 2024-09-16T06:51:56.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-c con 0x7f39a8097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f39a8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-a con 0x7f39a8139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a8098990 con 0x7f39a8139500 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a8097f40 con 0x7f39a8097b60 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a809d610 con 0x7f39a8099090 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a759e700 10 monclient(hunting): authenticate will time out at 1663.690958s 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a5d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f39a8099090 0x7f39a809f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60286/0 (socket says 172.21.15.76:60286) 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a5d9b700 1 -- 172.21.15.76:0/96833351 learned_addr learned my addr 172.21.15.76:0/96833351 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360714452 0 0) 0x7f39a809d610 con 0x7f39a8099090 2024-09-16T06:51:56.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): my global_id is 244334 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900035d0 con 0x7f39a8099090 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358816663 0 0) 0x7f39a8098990 con 0x7f39a8139500 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient(hunting): my global_id is 264117 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a809d610 con 0x7f39a8139500 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823143278 0 0) 0x7f39900035d0 con 0x7f39a8099090 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 >> v1:172.21.15.189:6789/0 conn(0x7f39a8097b60 legacy=0x7f39a809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 >> v1:172.21.15.76:6789/0 conn(0x7f39a8139500 legacy=0x7f39a813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 monclient: found mon.noname-b 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80986b0 con 0x7f39a8099090 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f399c001610 con 0x7f39a8099090 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.555+0000 7f39a559a700 10 monclient: dump: 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.560 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 5 monclient: authenticate success, global_id 244334 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f399c001870 con 0x7f39a8099090 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 1 -- 172.21.15.76:0/96833351 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f399c001c00 con 0x7f39a8099090 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:56.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 4 set_mon_vals no callback set 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a559a700 10 monclient: dump: 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: shutdown 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/96833351 >> v1:172.21.15.76:6790/0 conn(0x7f39a8099090 legacy=0x7f39a809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/96833351 shutdown_connections 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/96833351 >> 172.21.15.76:0/96833351 conn(0x7f39a80a0030 msgr2=0x7f39a80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/96833351 shutdown_connections 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/96833351 wait complete. 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: build_initial_monmap 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: monmap: 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:56.563 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 Processor -- start 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- start start 2024-09-16T06:51:56.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: init 2024-09-16T06:51:56.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:56.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:56.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-a con 0x7f39a8097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-c con 0x7f39a8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f39a8139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f39a8097b60 0x7f39a80702a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56046/0 (socket says 172.21.15.76:56046) 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a5d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f39a8099090 0x7f39a80709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60298/0 (socket says 172.21.15.76:60298) 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a659c700 1 -- 172.21.15.76:0/2939616666 learned_addr learned my addr 172.21.15.76:0/2939616666 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80710c0 con 0x7f39a8097b60 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.559+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a809a540 con 0x7f39a8139500 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a809b940 con 0x7f39a8099090 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient(hunting): authenticate will time out at 1663.696082s 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776578640 0 0) 0x7f39a80710c0 con 0x7f39a8097b60 2024-09-16T06:51:56.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): my global_id is 264120 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39880035d0 con 0x7f39a8097b60 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689923915 0 0) 0x7f39a809b940 con 0x7f39a8099090 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): my global_id is 244337 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a80710c0 con 0x7f39a8099090 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1901191538 0 0) 0x7f39880035d0 con 0x7f39a8097b60 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a809b940 con 0x7f39a8097b60 2024-09-16T06:51:56.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39980019e0 con 0x7f39a8097b60 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): dump: 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.570 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682439051 0 0) 0x7f39a809a540 con 0x7f39a8139500 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): my global_id is 224497 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39880035d0 con 0x7f39a8139500 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733066869 0 0) 0x7f39a80710c0 con 0x7f39a8099090 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a809a540 con 0x7f39a8099090 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f399c001510 con 0x7f39a8099090 2024-09-16T06:51:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient(hunting): dump: 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2775253529 0 0) 0x7f39a809b940 con 0x7f39a8097b60 2024-09-16T06:51:56.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 >> v1:172.21.15.76:6790/0 conn(0x7f39a8099090 legacy=0x7f39a80709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 >> v1:172.21.15.189:6789/0 conn(0x7f39a8139500 legacy=0x7f39a81d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 monclient: found mon.a 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a809bc20 con 0x7f39a8097b60 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: _finish_auth 0 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 5 monclient: authenticate success, global_id 264120 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _renew_subs 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a809b600 con 0x7f39a8097b60 2024-09-16T06:51:56.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3998003970 con 0x7f39a8097b60 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3998003d00 con 0x7f39a8097b60 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 10 monclient: dump: 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:56.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _renew_subs 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a4d99700 4 set_mon_vals no callback set 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a809c600 con 0x7f39a8097b60 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a809bfe0 con 0x7f39a8097b60 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3998003f70 con 0x7f39a8097b60 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3998043ee0 con 0x7f39a8097b60 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.563+0000 7f39a5d9b700 10 monclient: get_auth_request con 0x7f39880364b0 auth_method 0 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.571+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3998014730 con 0x7f39a8097b60 2024-09-16T06:51:56.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.571+0000 7f3996ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:56.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.571+0000 7f3996ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:56.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.803+0000 7f39a759e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:51:56.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.803+0000 7f39a759e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:51:56.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.803+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:56.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.803+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f39a80672a0 con 0x7f39a8097b60 2024-09-16T06:51:56.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=11,kv=10,logm=53} crc {auth=2339909065,config=3009771555,health=407870978,kv=2014955901,logm=3208594676}) 2024-09-16T06:51:56.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=13,logm=48} crc {auth=72144797,config=1097430493,health=4281930021,kv=4150243238,logm=294943082}) 2024-09-16T06:51:56.805 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f3996ffd700 1 -- 172.21.15.76:0/2939616666 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3998014550 con 0x7f39a8097b60 2024-09-16T06:51:56.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f3996ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:51:56.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f3996ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:56.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=11,kv=10,logm=53} crc {auth=2339909065,config=3009771555,health=407870978,kv=2014955901,logm=3208594676}) 2024-09-16T06:51:56.806 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.807+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=13,logm=47} crc {auth=3645436769,config=1097430493,health=4281930021,kv=1001081567,logm=3001828537}) 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 >> v1:172.21.15.189:6812/3098810674 conn(0x7f39880364b0 legacy=0x7f3988038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 10 monclient: shutdown 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 >> v1:172.21.15.76:6789/0 conn(0x7f39a8097b60 legacy=0x7f39a80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 shutdown_connections 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 >> 172.21.15.76:0/2939616666 conn(0x7f39a80a0030 msgr2=0x7f39a813ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 shutdown_connections 2024-09-16T06:51:56.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f39a759e700 1 -- 172.21.15.76:0/2939616666 wait complete. 2024-09-16T06:51:56.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4261644562}) 2024-09-16T06:51:56.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301022394}) 2024-09-16T06:51:56.808 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4261644562}) 2024-09-16T06:51:56.809 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.811+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985855714}) 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3956186965}) 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042658941}) 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3956186965}) 2024-09-16T06:51:56.811 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942587450}) 2024-09-16T06:51:56.813 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518515771}) 2024-09-16T06:51:56.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970911135}) 2024-09-16T06:51:56.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518515771}) 2024-09-16T06:51:56.814 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.815+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188109214}) 2024-09-16T06:51:56.815 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2631340532}) 2024-09-16T06:51:56.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817646590}) 2024-09-16T06:51:56.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2631340532}) 2024-09-16T06:51:56.816 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2599347800}) 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476886523}) 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4112104204}) 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476886523}) 2024-09-16T06:51:56.817 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922046644}) 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2134290870}) 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747390181}) 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2134290870}) 2024-09-16T06:51:56.818 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.819+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976094800}) 2024-09-16T06:51:56.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:51:56.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.821 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2453862299}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904464640}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2453862299}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493598489}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2364208050}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039247617}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2364208050}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046890948}) 2024-09-16T06:51:56.822 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538032628}) 2024-09-16T06:51:56.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133619733}) 2024-09-16T06:51:56.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538032628}) 2024-09-16T06:51:56.823 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.823+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505641298}) 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317866920}) 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744091916}) 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317866920}) 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556846579}) 2024-09-16T06:51:56.824 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269498895}) 2024-09-16T06:51:56.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102971049}) 2024-09-16T06:51:56.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.825 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269498895}) 2024-09-16T06:51:56.826 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229378996}) 2024-09-16T06:51:56.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446454831}) 2024-09-16T06:51:56.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923736462}) 2024-09-16T06:51:56.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.827 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446454831}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=897327457}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440094945}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405466814}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440094945}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.827+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353480913}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074738827}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106579215}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074738827}) 2024-09-16T06:51:56.828 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=687190175}) 2024-09-16T06:51:56.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592742147}) 2024-09-16T06:51:56.829 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390295158}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592742147}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3867012137}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681161237}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927286741}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681161237}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364281488}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449832938}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305033483}) 2024-09-16T06:51:56.830 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449832938}) 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532616819}) 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842660732}) 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177762001}) 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.831 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842660732}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.831+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588269253}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175590564}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201501864}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175590564}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527132913}) 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.832 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=668642240}) 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432756275}) 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=668642240}) 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305285363}) 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.833 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913376786}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016944785}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913376786}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266226518}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759638417}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637171311}) 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.834 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759638417}) 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.835+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742798400}) 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2232466304}) 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239842356}) 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2232466304}) 2024-09-16T06:51:56.835 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3252275492}) 2024-09-16T06:51:56.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076060707}) 2024-09-16T06:51:56.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216920261}) 2024-09-16T06:51:56.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.836 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076060707}) 2024-09-16T06:51:56.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2580267550}) 2024-09-16T06:51:56.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862341829}) 2024-09-16T06:51:56.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135125453}) 2024-09-16T06:51:56.837 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862341829}) 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668856499}) 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106556453}) 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422760227}) 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106556453}) 2024-09-16T06:51:56.838 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.839+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056628861}) 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259262003}) 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168337137}) 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259262003}) 2024-09-16T06:51:56.839 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017627389}) 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484632483}) 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000431568}) 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484632483}) 2024-09-16T06:51:56.840 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1711650633}) 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861155248}) 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027049235}) 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861155248}) 2024-09-16T06:51:56.841 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56683492}) 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3659783808}) 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921647856}) 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3659783808}) 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312140093}) 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.842 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594807678}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858456414}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594807678}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.843+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504623567}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728967436}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595134352}) 2024-09-16T06:51:56.843 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728967436}) 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048301666}) 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064202559}) 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737243791}) 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064202559}) 2024-09-16T06:51:56.844 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583597583}) 2024-09-16T06:51:56.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163755775}) 2024-09-16T06:51:56.845 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032652951}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163755775}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=953563368}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950731717}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769293715}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950731717}) 2024-09-16T06:51:56.846 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.847+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925051138}) 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2263309722}) 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227711458}) 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2263309722}) 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364307144}) 2024-09-16T06:51:56.847 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192686055}) 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616190713}) 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192686055}) 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146702942}) 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.848 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228117547}) 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746713142}) 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228117547}) 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2130677255}) 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874850316}) 2024-09-16T06:51:56.849 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494465043}) 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874850316}) 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502195585}) 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179415481}) 2024-09-16T06:51:56.850 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355109390}) 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179415481}) 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.851+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4112700518}) 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642529907}) 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764926345}) 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.851 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642529907}) 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567515192}) 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624943381}) 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893914423}) 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.852 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624943381}) 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277149083}) 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242510654}) 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029192074}) 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242510654}) 2024-09-16T06:51:56.853 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782619265}) 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460609881}) 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210894316}) 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460609881}) 2024-09-16T06:51:56.854 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.855+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327090678}) 2024-09-16T06:51:56.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798712020}) 2024-09-16T06:51:56.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669396200}) 2024-09-16T06:51:56.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.855 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798712020}) 2024-09-16T06:51:56.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555419274}) 2024-09-16T06:51:56.856 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.857 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273126723}) 2024-09-16T06:51:56.857 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860731436}) 2024-09-16T06:51:56.857 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.857 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273126723}) 2024-09-16T06:51:56.857 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488758399}) 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804505040}) 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4135200714}) 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804505040}) 2024-09-16T06:51:56.858 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.859+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3341426095}) 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3080988538}) 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3116150557}) 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3080988538}) 2024-09-16T06:51:56.859 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4135706658}) 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847311114}) 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274609210}) 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847311114}) 2024-09-16T06:51:56.860 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037121630}) 2024-09-16T06:51:56.861 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.861 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317030111}) 2024-09-16T06:51:56.861 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674762069}) 2024-09-16T06:51:56.861 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.861 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317030111}) 2024-09-16T06:51:56.862 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614893161}) 2024-09-16T06:51:56.862 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.862 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199278249}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099136272}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199278249}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.863+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989277644}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949158385}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392064660}) 2024-09-16T06:51:56.863 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.864 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949158385}) 2024-09-16T06:51:56.864 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556689888}) 2024-09-16T06:51:56.864 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.864 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591723824}) 2024-09-16T06:51:56.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211369028}) 2024-09-16T06:51:56.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591723824}) 2024-09-16T06:51:56.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131405014}) 2024-09-16T06:51:56.865 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.866 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1194106585}) 2024-09-16T06:51:56.866 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494279119}) 2024-09-16T06:51:56.866 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.866 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1194106585}) 2024-09-16T06:51:56.866 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.867+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502658597}) 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100050728}) 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664217689}) 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100050728}) 2024-09-16T06:51:56.867 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1814182745}) 2024-09-16T06:51:56.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.868 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973855307}) 2024-09-16T06:51:56.869 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912707733}) 2024-09-16T06:51:56.869 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.869 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973855307}) 2024-09-16T06:51:56.869 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079436311}) 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976086463}) 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577824910}) 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976086463}) 2024-09-16T06:51:56.870 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.871+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177247778}) 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887287365}) 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527805526}) 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887287365}) 2024-09-16T06:51:56.871 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252310402}) 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202995346}) 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327229929}) 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202995346}) 2024-09-16T06:51:56.873 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3100833069}) 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1129261569}) 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613657287}) 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1129261569}) 2024-09-16T06:51:56.874 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.875+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974334060}) 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950648570}) 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564615448}) 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950648570}) 2024-09-16T06:51:56.875 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808740297}) 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633677065}) 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593671628}) 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633677065}) 2024-09-16T06:51:56.876 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=569841066}) 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191483616}) 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050363875}) 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191483616}) 2024-09-16T06:51:56.877 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737512517}) 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345405368}) 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337012968}) 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345405368}) 2024-09-16T06:51:56.878 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.879+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680215591}) 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874074925}) 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706414952}) 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874074925}) 2024-09-16T06:51:56.879 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682129099}) 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573280316}) 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152010237}) 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573280316}) 2024-09-16T06:51:56.880 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077961188}) 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467327638}) 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233221114}) 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467327638}) 2024-09-16T06:51:56.881 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771582517}) 2024-09-16T06:51:56.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134861035}) 2024-09-16T06:51:56.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.883+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211994406}) 2024-09-16T06:51:56.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.882 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134861035}) 2024-09-16T06:51:56.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505626308}) 2024-09-16T06:51:56.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=735112835}) 2024-09-16T06:51:56.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085948480}) 2024-09-16T06:51:56.883 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=735112835}) 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015919072}) 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=165935082}) 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251931708}) 2024-09-16T06:51:56.884 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=165935082}) 2024-09-16T06:51:56.885 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457877370}) 2024-09-16T06:51:56.886 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.887+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.891+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=920456406,mdsmap=1531151339,mgr=278123252,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3281115673,mon_config_key=1174667612,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=409308773}) 2024-09-16T06:51:56.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.891+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=42} crc {logm=1393015320,mdsmap=2147943554,mgr=2759932744,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3462562250,mon_config_key=1026285949,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=1984348070}) 2024-09-16T06:51:56.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.891+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.891+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=920456406,mdsmap=1531151339,mgr=278123252,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3281115673,mon_config_key=1174667612,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=409308773}) 2024-09-16T06:51:56.887 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.891+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=6,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=45} crc {logm=1489998038,mdsmap=1228700967,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3501463026,mon_config_key=2387200789,monmap=1887442722,osd_metadata=3822138956,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4290509389}) 2024-09-16T06:51:56.893 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=3974555831}) 2024-09-16T06:51:56.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=12} crc {osdmap=18489797}) 2024-09-16T06:51:56.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:51:56.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=3974555831}) 2024-09-16T06:51:56.894 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:51:56.895+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=2514999911}) 2024-09-16T06:51:57.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:51:57.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: get_monmap_and_config 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: build_initial_monmap 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: monmap: 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:57.945323+0000 2024-09-16T06:51:57.940 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:57.945323+0000 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:57.941 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 1 Processor -- start 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 1 -- start start 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: init 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:51:57.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-c con 0x7f37d4097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-b con 0x7f37d4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-a con 0x7f37d4139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4098990 con 0x7f37d4139500 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4097f40 con 0x7f37d4097b60 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d409d610 con 0x7f37d4099090 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 10 monclient(hunting): authenticate will time out at 1665.078564s 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d96e8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f37d4139500 0x7f37d413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56050/0 (socket says 172.21.15.76:56050) 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d96e8700 1 -- 172.21.15.76:0/2989307430 learned_addr learned my addr 172.21.15.76:0/2989307430 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606033396 0 0) 0x7f37d4098990 con 0x7f37d4139500 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): my global_id is 264123 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37bc0035d0 con 0x7f37d4139500 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704515537 0 0) 0x7f37d409d610 con 0x7f37d4099090 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): my global_id is 244340 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4098990 con 0x7f37d4099090 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729218457 0 0) 0x7f37d4097f40 con 0x7f37d4097b60 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient(hunting): my global_id is 224500 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d409d610 con 0x7f37d4097b60 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641846521 0 0) 0x7f37bc0035d0 con 0x7f37d4139500 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 >> v1:172.21.15.76:6790/0 conn(0x7f37d4099090 legacy=0x7f37d409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 >> v1:172.21.15.189:6789/0 conn(0x7f37d4097b60 legacy=0x7f37d409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 monclient: found mon.noname-a 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40986b0 con 0x7f37d4139500 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:57.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 5 monclient: authenticate success, global_id 264123 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37cc001990 con 0x7f37d4139500 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37cb7fe700 10 monclient: dump: 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.943+0000 7f37d9ee9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37cc001bb0 con 0x7f37d4139500 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 4 set_mon_vals no callback set 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 1 -- 172.21.15.76:0/2989307430 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37cc001f40 con 0x7f37d4139500 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cb7fe700 10 monclient: dump: 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.956 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: shutdown 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- 172.21.15.76:0/2989307430 >> v1:172.21.15.76:6789/0 conn(0x7f37d4139500 legacy=0x7f37d413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- 172.21.15.76:0/2989307430 shutdown_connections 2024-09-16T06:51:57.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- 172.21.15.76:0/2989307430 >> 172.21.15.76:0/2989307430 conn(0x7f37d40a0030 msgr2=0x7f37d40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- 172.21.15.76:0/2989307430 shutdown_connections 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- 172.21.15.76:0/2989307430 wait complete. 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: build_initial_monmap 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: monmap: 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:57.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 Processor -- start 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- start start 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: init 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-a con 0x7f37d4097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-c con 0x7f37d4099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient(hunting): picked mon.noname-b con 0x7f37d4139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d413c270 con 0x7f37d4097b60 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4091980 con 0x7f37d4139500 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d40a2cd0 con 0x7f37d4099090 2024-09-16T06:51:57.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d9ee9700 10 monclient(hunting): authenticate will time out at 1665.082543s 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f37d4099090 0x7f37d41cef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60312/0 (socket says 172.21.15.76:60312) 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37cbfff700 1 -- 172.21.15.76:0/851116091 learned_addr learned my addr 172.21.15.76:0/851116091 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37d8ee7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f37d4097b60 0x7f37d4138a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56056/0 (socket says 172.21.15.76:56056) 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536349511 0 0) 0x7f37d40a2cd0 con 0x7f37d4099090 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): my global_id is 244343 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c00035d0 con 0x7f37d4099090 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991171156 0 0) 0x7f37d413c270 con 0x7f37d4097b60 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): my global_id is 264126 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d40a2cd0 con 0x7f37d4097b60 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522910022 0 0) 0x7f37d4091980 con 0x7f37d4139500 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): my global_id is 224503 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d413c270 con 0x7f37d4139500 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 742867534 0 0) 0x7f37d40a2cd0 con 0x7f37d4097b60 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d4091980 con 0x7f37d4097b60 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37d00018b0 con 0x7f37d4097b60 2024-09-16T06:51:57.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): dump: 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:57.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444007029 0 0) 0x7f37c00035d0 con 0x7f37d4099090 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d40a2cd0 con 0x7f37d4099090 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37c40018f0 con 0x7f37d4099090 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 10 monclient(hunting): dump: 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.963 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221456298 0 0) 0x7f37d413c270 con 0x7f37d4139500 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.947+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37c00035d0 con 0x7f37d4139500 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37cc001ec0 con 0x7f37d4139500 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient(hunting): dump: 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.965 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2297527832 0 0) 0x7f37d40a2cd0 con 0x7f37d4099090 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 >> v1:172.21.15.189:6789/0 conn(0x7f37d4139500 legacy=0x7f37d41d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 >> v1:172.21.15.76:6789/0 conn(0x7f37d4097b60 legacy=0x7f37d4138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 monclient: found mon.c 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d408ff70 con 0x7f37d4099090 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: _finish_auth 0 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:51:57.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 5 monclient: authenticate success, global_id 244343 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _renew_subs 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d40916a0 con 0x7f37d4099090 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _renew_subs 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d4139150 con 0x7f37d4099090 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37c40036c0 con 0x7f37d4099090 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f37c40039d0 con 0x7f37d4099090 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 10 monclient: dump: 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:57.967 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37caffd700 4 set_mon_vals no callback set 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f37c4003bf0 con 0x7f37d4099090 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f37c4043d50 con 0x7f37d4099090 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d8ee7700 10 monclient: get_auth_request con 0x7f37c0036540 auth_method 0 2024-09-16T06:51:57.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.951+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b4005100 con 0x7f37d4099090 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.959+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f37c4004440 con 0x7f37d4099090 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.959+0000 7f37c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:57.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:57.959+0000 7f37c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:58.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37d9ee9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37d9ee9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37d9ee9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f37b4005930 con 0x7f37d4099090 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37c97fa700 1 -- 172.21.15.76:0/851116091 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2965771205) 0x7f37c4012c40 con 0x7f37d4099090 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.187+0000 7f37c97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:58.186 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 >> v1:172.21.15.189:6812/3098810674 conn(0x7f37c0036540 legacy=0x7f37c0038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 10 monclient: shutdown 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 >> v1:172.21.15.76:6790/0 conn(0x7f37d4099090 legacy=0x7f37d41cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 shutdown_connections 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 >> 172.21.15.76:0/851116091 conn(0x7f37d40a0030 msgr2=0x7f37d4139c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 shutdown_connections 2024-09-16T06:51:58.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.191+0000 7f37d9ee9700 1 -- 172.21.15.76:0/851116091 wait complete. 2024-09-16T06:51:58.199 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:51:58.199 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:51:58.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:51:58.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:51:58.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: get_monmap_and_config 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: monmap: 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:58.320216+0000 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:58.320216+0000 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:58.315 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:58.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 1 Processor -- start 2024-09-16T06:51:58.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 1 -- start start 2024-09-16T06:51:58.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: init 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-b con 0x7f6bcc0291d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-c con 0x7f6bcc0244f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-a con 0x7f6bcc0c1e50 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc026450 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0273a0 con 0x7f6bcc0244f0 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc027680 con 0x7f6bcc0291d0 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient(hunting): authenticate will time out at 1665.453627s 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bd9e10700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6bcc0c1e50 0x7f6bcc0c4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56070/0 (socket says 172.21.15.76:56070) 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bd9e10700 1 -- 172.21.15.76:0/452462512 learned_addr learned my addr 172.21.15.76:0/452462512 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bd960f700 1 --1- 172.21.15.76:0/452462512 >> v1:172.21.15.76:6790/0 conn(0x7f6bcc0291d0 0x7f6bcc023af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60322/0 (socket says 172.21.15.76:60322) 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249431914 0 0) 0x7f6bcc026450 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): my global_id is 264129 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc00035d0 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907390427 0 0) 0x7f6bcc027680 con 0x7f6bcc0291d0 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): my global_id is 244346 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc026450 con 0x7f6bcc0291d0 2024-09-16T06:51:58.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032589413 0 0) 0x7f6bcc0273a0 con 0x7f6bcc0244f0 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient(hunting): my global_id is 224506 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc027680 con 0x7f6bcc0244f0 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186591886 0 0) 0x7f6bc00035d0 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 >> v1:172.21.15.76:6790/0 conn(0x7f6bcc0291d0 legacy=0x7f6bcc023af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 >> v1:172.21.15.189:6789/0 conn(0x7f6bcc0244f0 legacy=0x7f6bcc0c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 monclient: found mon.noname-a 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0bc2a0 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bd4069400 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 5 monclient: authenticate success, global_id 264129 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: dump: 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd404ec20 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 1 -- 172.21.15.76:0/452462512 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bd40008d0 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bda611700 4 set_mon_vals no callback set 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.319+0000 7f6bcbfff700 10 monclient: dump: 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: shutdown 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- 172.21.15.76:0/452462512 >> v1:172.21.15.76:6789/0 conn(0x7f6bcc0c1e50 legacy=0x7f6bcc0c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- 172.21.15.76:0/452462512 shutdown_connections 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- 172.21.15.76:0/452462512 >> 172.21.15.76:0/452462512 conn(0x7f6bcc005e70 msgr2=0x7f6bcc006280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- 172.21.15.76:0/452462512 shutdown_connections 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- 172.21.15.76:0/452462512 wait complete. 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: monmap: 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:58.327 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:58.327 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 Processor -- start 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- start start 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: init 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-c con 0x7f6bcc0291d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-b con 0x7f6bcc0c1e50 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient(hunting): picked mon.noname-a con 0x7f6bcc01e1a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc01ec00 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc020c60 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc02d060 con 0x7f6bcc0291d0 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bda611700 10 monclient(hunting): authenticate will time out at 1665.458723s 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bd9e10700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6bcc01e1a0 0x7f6bcc158b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56076/0 (socket says 172.21.15.76:56076) 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bd9e10700 1 -- 172.21.15.76:0/1347121126 learned_addr learned my addr 172.21.15.76:0/1347121126 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056526180 0 0) 0x7f6bcc01ec00 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): my global_id is 264132 2024-09-16T06:51:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb80035d0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94778975 0 0) 0x7f6bcc02d060 con 0x7f6bcc0291d0 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): my global_id is 244349 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc01ec00 con 0x7f6bcc0291d0 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217985649 0 0) 0x7f6bcc020c60 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): my global_id is 224509 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc02d060 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 399822368 0 0) 0x7f6bb80035d0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bcc020c60 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bd40008d0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2482843528 0 0) 0x7f6bcc01ec00 con 0x7f6bcc0291d0 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bb80035d0 con 0x7f6bcc0291d0 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bc4001120 con 0x7f6bcc0291d0 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.323+0000 7f6bc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.336 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357346504 0 0) 0x7f6bcc02d060 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bcc01ec00 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bd0001790 con 0x7f6bcc0c1e50 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.337 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802132595 0 0) 0x7f6bcc020c60 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6790/0 conn(0x7f6bcc0291d0 legacy=0x7f6bcc0b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.189:6789/0 conn(0x7f6bcc0c1e50 legacy=0x7f6bcc155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 monclient: found mon.a 2024-09-16T06:51:58.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc01f7b0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 5 monclient: authenticate success, global_id 264132 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 10 monclient: _renew_subs 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc020f40 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 10 monclient: _renew_subs 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc01f530 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd4069400 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bd404f5c0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bcb7fe700 4 set_mon_vals no callback set 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: dump: 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6bb4006590 con 0x7f6bb4002830 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6bd4067aa0 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6bd40bdb80 con 0x7f6bcc01e1a0 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691022285 0 0) 0x7f6bb4006590 con 0x7f6bb4002830 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient(hunting): my global_id is 264135 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc020c60 con 0x7f6bb4002830 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bd8e0e700 10 monclient: get_auth_request con 0x7f6bb80366a0 auth_method 0 2024-09-16T06:51:58.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2447884732 0 0) 0x7f6bcc020c60 con 0x7f6bb4002830 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bb4006590 con 0x7f6bb4002830 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bc4004c20 con 0x7f6bb4002830 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: dump: 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.342 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4051210516 0 0) 0x7f6bb4006590 con 0x7f6bb4002830 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bb400a190 con 0x7f6bb4002830 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6bc4004ec0 con 0x7f6bb4002830 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6789/0 conn(0x7f6bb4002830 legacy=0x7f6bb4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.327+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6789/0 conn(0x7f6bb4002830 legacy=0x7f6bb4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.335+0000 7f6bda611700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.335+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6bb4009e50 con 0x7f6bb4009170 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256846460 0 0) 0x7f6bb4009e50 con 0x7f6bb4009170 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient(hunting): my global_id is 264138 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb4006590 con 0x7f6bb4009170 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2218010259 0 0) 0x7f6bb4006590 con 0x7f6bb4009170 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bb4009e50 con 0x7f6bb4009170 2024-09-16T06:51:58.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6bc40054c0 con 0x7f6bb4009170 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient: dump: 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 96394005 0 0) 0x7f6bb4009e50 con 0x7f6bb4009170 2024-09-16T06:51:58.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6bb400c340 con 0x7f6bb4009170 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2343910234) 0x7f6bc4005730 con 0x7f6bb4009170 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6789/0 conn(0x7f6bb4009170 legacy=0x7f6bb4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bc9ffb700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6789/0 conn(0x7f6bb4009170 legacy=0x7f6bb4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:58.358 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6bb80366a0 legacy=0x7f6bb8038b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 10 monclient: shutdown 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bd9e10700 1 -- 172.21.15.76:0/1347121126 reap_dead start 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 >> v1:172.21.15.76:6789/0 conn(0x7f6bcc01e1a0 legacy=0x7f6bcc158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 shutdown_connections 2024-09-16T06:51:58.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 >> 172.21.15.76:0/1347121126 conn(0x7f6bcc005e70 msgr2=0x7f6bcc02beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 shutdown_connections 2024-09-16T06:51:58.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.339+0000 7f6bda611700 1 -- 172.21.15.76:0/1347121126 wait complete. 2024-09-16T06:51:58.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:58.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:51:58.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:51:58.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.483+0000 7f87b9a61700 10 monclient: get_monmap_and_config 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.483+0000 7f87b9a61700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.483+0000 7f87b9a61700 10 monclient: monmap: 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:58.487010+0000 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:58.487010+0000 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:58.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.483+0000 7f87b9a61700 1 Processor -- start 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.483+0000 7f87b9a61700 1 -- start start 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: init 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-a con 0x7f87b40a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-b con 0x7f87b40a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-c con 0x7f87b413b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b409b3e0 con 0x7f87b40a20c0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40a3450 con 0x7f87b413b6b0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b409b1a0 con 0x7f87b40a24a0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient(hunting): authenticate will time out at 1665.620770s 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b8a5f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f87b40a20c0 0x7f87b406cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56100/0 (socket says 172.21.15.76:56100) 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b8a5f700 1 -- 172.21.15.76:0/2181800718 learned_addr learned my addr 172.21.15.76:0/2181800718 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f87b40a24a0 0x7f87b40a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60340/0 (socket says 172.21.15.76:60340) 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41159508 0 0) 0x7f87b409b1a0 con 0x7f87b40a24a0 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): my global_id is 244352 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c0035d0 con 0x7f87b40a24a0 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131683894 0 0) 0x7f87b409b3e0 con 0x7f87b40a20c0 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): my global_id is 264141 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b409b1a0 con 0x7f87b40a20c0 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011476369 0 0) 0x7f87b40a3450 con 0x7f87b413b6b0 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient(hunting): my global_id is 224512 2024-09-16T06:51:58.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b409b3e0 con 0x7f87b413b6b0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2078151312 0 0) 0x7f879c0035d0 con 0x7f87b40a24a0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 >> v1:172.21.15.189:6789/0 conn(0x7f87b413b6b0 legacy=0x7f87b413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 >> v1:172.21.15.76:6789/0 conn(0x7f87b40a20c0 legacy=0x7f87b406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 monclient: found mon.noname-b 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b406bd30 con 0x7f87b40a24a0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87a8001610 con 0x7f87b40a24a0 2024-09-16T06:51:58.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 5 monclient: authenticate success, global_id 244352 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: dump: 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87a8001880 con 0x7f87b40a24a0 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 4 set_mon_vals no callback set 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 1 -- 172.21.15.76:0/2181800718 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87a8001c10 con 0x7f87b40a24a0 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b37fe700 10 monclient: dump: 2024-09-16T06:51:58.491 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: shutdown 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- 172.21.15.76:0/2181800718 >> v1:172.21.15.76:6790/0 conn(0x7f87b40a24a0 legacy=0x7f87b40a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- 172.21.15.76:0/2181800718 shutdown_connections 2024-09-16T06:51:58.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- 172.21.15.76:0/2181800718 >> 172.21.15.76:0/2181800718 conn(0x7f87b40a41c0 msgr2=0x7f87b40a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- 172.21.15.76:0/2181800718 shutdown_connections 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 -- 172.21.15.76:0/2181800718 wait complete. 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 10 monclient: monmap: 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:58.493 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.487+0000 7f87b9a61700 1 Processor -- start 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- start start 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: init 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-a con 0x7f87b40a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-b con 0x7f87b40a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient(hunting): picked mon.noname-c con 0x7f87b413b6b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b413a8e0 con 0x7f87b40a20c0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40a6f50 con 0x7f87b40a24a0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40a1500 con 0x7f87b413b6b0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient(hunting): authenticate will time out at 1665.624482s 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b8a5f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f87b40a20c0 0x7f87b4137230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56114/0 (socket says 172.21.15.76:56114) 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b8a5f700 1 -- 172.21.15.76:0/2101386179 learned_addr learned my addr 172.21.15.76:0/2101386179 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687547920 0 0) 0x7f87b40a1500 con 0x7f87b413b6b0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): my global_id is 244355 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a00035d0 con 0x7f87b413b6b0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829690207 0 0) 0x7f87b413a8e0 con 0x7f87b40a20c0 2024-09-16T06:51:58.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): my global_id is 264144 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40a1500 con 0x7f87b40a20c0 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913716974 0 0) 0x7f87b40a6f50 con 0x7f87b40a24a0 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): my global_id is 224515 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b413a8e0 con 0x7f87b40a24a0 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717254777 0 0) 0x7f87b40a1500 con 0x7f87b40a20c0 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b40a6f50 con 0x7f87b40a20c0 2024-09-16T06:51:58.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87a40018b0 con 0x7f87b40a20c0 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): dump: 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1818937003 0 0) 0x7f87a00035d0 con 0x7f87b413b6b0 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b40a1500 con 0x7f87b413b6b0 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87ac001c30 con 0x7f87b413b6b0 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): dump: 2024-09-16T06:51:58.502 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.503 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 69195975 0 0) 0x7f87b40a6f50 con 0x7f87b40a20c0 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6790/0 conn(0x7f87b413b6b0 legacy=0x7f87b41d23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.189:6789/0 conn(0x7f87b40a24a0 legacy=0x7f87b4137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 monclient: found mon.a 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b413aea0 con 0x7f87b40a20c0 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 5 monclient: authenticate success, global_id 264144 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _renew_subs 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b413a600 con 0x7f87b40a20c0 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _renew_subs 2024-09-16T06:51:58.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b413b2c0 con 0x7f87b40a20c0 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87a4004890 con 0x7f87b40a20c0 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87a4001d80 con 0x7f87b40a20c0 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b2ffd700 4 set_mon_vals no callback set 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient: dump: 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f87a4004c20 con 0x7f87b40a20c0 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8794006590 con 0x7f8794002830 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f87a4043da0 con 0x7f87b40a20c0 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b3fff700 10 monclient: get_auth_request con 0x7f87a0036450 auth_method 0 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429944475 0 0) 0x7f8794006590 con 0x7f8794002830 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 10 monclient(hunting): my global_id is 244358 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.491+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b40a6f50 con 0x7f8794002830 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1015107486 0 0) 0x7f87b40a6f50 con 0x7f8794002830 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8794006590 con 0x7f8794002830 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87ac003a40 con 0x7f8794002830 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 10 monclient: dump: 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3586628876 0 0) 0x7f8794006590 con 0x7f8794002830 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f879400a190 con 0x7f8794002830 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f87ac003d60 con 0x7f8794002830 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6790/0 conn(0x7f8794002830 legacy=0x7f8794004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.495+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6790/0 conn(0x7f8794002830 legacy=0x7f8794004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b9a61700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8794009e30 con 0x7f8794009150 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674339959 0 0) 0x7f8794009e30 con 0x7f8794009150 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient(hunting): my global_id is 244361 2024-09-16T06:51:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8794006590 con 0x7f8794009150 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 538501699 0 0) 0x7f8794006590 con 0x7f8794009150 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8794009e30 con 0x7f8794009150 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87ac004370 con 0x7f8794009150 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.503+0000 7f87b17fa700 10 monclient: dump: 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1105802224 0 0) 0x7f8794009e30 con 0x7f8794009150 2024-09-16T06:51:58.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f879400c340 con 0x7f8794009150 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 1667661642) 0x7f87ac005770 con 0x7f8794009150 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6790/0 conn(0x7f8794009150 legacy=0x7f87940095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b17fa700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6790/0 conn(0x7f8794009150 legacy=0x7f87940095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:58.527 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.189:6812/3098810674 conn(0x7f87a0036450 legacy=0x7f87a0038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 10 monclient: shutdown 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 >> v1:172.21.15.76:6789/0 conn(0x7f87b40a20c0 legacy=0x7f87b4137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9260700 1 -- 172.21.15.76:0/2101386179 reap_dead start 2024-09-16T06:51:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9260700 1 -- 172.21.15.76:0/2101386179 reap_dead start 2024-09-16T06:51:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 shutdown_connections 2024-09-16T06:51:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 >> 172.21.15.76:0/2101386179 conn(0x7f87b40a41c0 msgr2=0x7f87b40a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 shutdown_connections 2024-09-16T06:51:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.507+0000 7f87b9a61700 1 -- 172.21.15.76:0/2101386179 wait complete. 2024-09-16T06:51:58.530 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:58.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:51:58.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: get_monmap_and_config 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: monmap: 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:58.652373+0000 2024-09-16T06:51:58.647 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:58.652373+0000 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:58.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 1 Processor -- start 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 1 -- start start 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: init 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-b con 0x7f0a20132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-a con 0x7f0a2006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-c con 0x7f0a2006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2006c280 con 0x7f0a2006d480 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2006bfa0 con 0x7f0a2006cf50 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2009ace0 con 0x7f0a20132330 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient(hunting): authenticate will time out at 1665.786303s 2024-09-16T06:51:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0a20132330 0x7f0a20132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60374/0 (socket says 172.21.15.76:60374) 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1ffff700 1 -- 172.21.15.76:0/590587875 learned_addr learned my addr 172.21.15.76:0/590587875 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648162685 0 0) 0x7f0a2009ace0 con 0x7f0a20132330 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): my global_id is 244364 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c0035d0 con 0x7f0a20132330 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976155221 0 0) 0x7f0a2006c280 con 0x7f0a2006d480 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): my global_id is 264147 2024-09-16T06:51:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2009ace0 con 0x7f0a2006d480 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068715642 0 0) 0x7f0a2006bfa0 con 0x7f0a2006cf50 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient(hunting): my global_id is 224518 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2006c280 con 0x7f0a2006cf50 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621168249 0 0) 0x7f0a0c0035d0 con 0x7f0a20132330 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 >> v1:172.21.15.189:6789/0 conn(0x7f0a2006cf50 legacy=0x7f0a2013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 >> v1:172.21.15.76:6789/0 conn(0x7f0a2006d480 legacy=0x7f0a2013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:51:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2009ae70 con 0x7f0a20132330 2024-09-16T06:51:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 5 monclient: authenticate success, global_id 244364 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a10001610 con 0x7f0a20132330 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: dump: 2024-09-16T06:51:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a10001830 con 0x7f0a20132330 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 1 -- 172.21.15.76:0/590587875 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a10001a20 con 0x7f0a20132330 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a2537b700 4 set_mon_vals no callback set 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.651+0000 7f0a1f7fe700 10 monclient: dump: 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.657 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: shutdown 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/590587875 >> v1:172.21.15.76:6790/0 conn(0x7f0a20132330 legacy=0x7f0a20132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/590587875 shutdown_connections 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/590587875 >> 172.21.15.76:0/590587875 conn(0x7f0a200a41c0 msgr2=0x7f0a200a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/590587875 shutdown_connections 2024-09-16T06:51:58.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/590587875 wait complete. 2024-09-16T06:51:58.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: monmap: 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 Processor -- start 2024-09-16T06:51:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- start start 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: init 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-c con 0x7f0a2006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-b con 0x7f0a2006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient(hunting): picked mon.noname-a con 0x7f0a20132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a201392d0 con 0x7f0a20132330 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a20136190 con 0x7f0a2006d480 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2009e8b0 con 0x7f0a2006cf50 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient(hunting): authenticate will time out at 1665.789752s 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0a2006cf50 0x7f0a20135370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60382/0 (socket says 172.21.15.76:60382) 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1ffff700 1 -- 172.21.15.76:0/2233862052 learned_addr learned my addr 172.21.15.76:0/2233862052 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439609840 0 0) 0x7f0a201392d0 con 0x7f0a20132330 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): my global_id is 264150 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000035d0 con 0x7f0a20132330 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735781955 0 0) 0x7f0a2009e8b0 con 0x7f0a2006cf50 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): my global_id is 244367 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a201392d0 con 0x7f0a2006cf50 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884595873 0 0) 0x7f0a20136190 con 0x7f0a2006d480 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): my global_id is 224521 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a2009e8b0 con 0x7f0a2006d480 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272635291 0 0) 0x7f0a000035d0 con 0x7f0a20132330 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a20136190 con 0x7f0a20132330 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a18001c30 con 0x7f0a20132330 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:51:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): dump: 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.669 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261929387 0 0) 0x7f0a201392d0 con 0x7f0a2006cf50 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a000035d0 con 0x7f0a2006cf50 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a100019a0 con 0x7f0a2006cf50 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): dump: 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269620454 0 0) 0x7f0a2009e8b0 con 0x7f0a2006d480 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a201392d0 con 0x7f0a2006d480 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a080019d0 con 0x7f0a2006d480 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient(hunting): dump: 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3941110904 0 0) 0x7f0a20136190 con 0x7f0a20132330 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.76:6790/0 conn(0x7f0a2006cf50 legacy=0x7f0a20135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6789/0 conn(0x7f0a2006d480 legacy=0x7f0a20135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 monclient: found mon.a 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20131700 con 0x7f0a20132330 2024-09-16T06:51:58.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 5 monclient: authenticate success, global_id 264150 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _renew_subs 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a201378c0 con 0x7f0a20132330 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _renew_subs 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201310d0 con 0x7f0a20132330 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a18003a40 con 0x7f0a20132330 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a18003d50 con 0x7f0a20132330 2024-09-16T06:51:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1effd700 4 set_mon_vals no callback set 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.655+0000 7f0a1d7fa700 10 monclient: dump: 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a2537b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a04006590 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0a1800e7c0 con 0x7f0a20132330 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0a18042ee0 con 0x7f0a20132330 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299513758 0 0) 0x7f0a04006590 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient(hunting): my global_id is 224524 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20136190 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a177fe700 10 monclient: get_auth_request con 0x7f0a00036460 auth_method 0 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 724859175 0 0) 0x7f0a20136190 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a04006590 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a10004d30 con 0x7f0a04002830 2024-09-16T06:51:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient: dump: 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 661729202 0 0) 0x7f0a04006590 con 0x7f0a04002830 2024-09-16T06:51:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a0400a190 con 0x7f0a04002830 2024-09-16T06:51:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0a100070b0 con 0x7f0a04002830 2024-09-16T06:51:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6789/0 conn(0x7f0a04002830 legacy=0x7f0a04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.659+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6789/0 conn(0x7f0a04002830 legacy=0x7f0a04004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a2537b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:51:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0a04009e50 con 0x7f0a04009170 2024-09-16T06:51:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379167627 0 0) 0x7f0a04009e50 con 0x7f0a04009170 2024-09-16T06:51:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient(hunting): my global_id is 224527 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a04006590 con 0x7f0a04009170 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3138578046 0 0) 0x7f0a04006590 con 0x7f0a04009170 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a04009e50 con 0x7f0a04009170 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0a100055c0 con 0x7f0a04009170 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.671+0000 7f0a1d7fa700 10 monclient: dump: 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.679 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1793233376 0 0) 0x7f0a04009e50 con 0x7f0a04009170 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a0400c340 con 0x7f0a04009170 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 539293009) 0x7f0a100069c0 con 0x7f0a04009170 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6789/0 conn(0x7f0a04009170 legacy=0x7f0a04009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a1d7fa700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6789/0 conn(0x7f0a04009170 legacy=0x7f0a04009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 120, 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:51:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:51:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:51:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:51:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:51:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:51:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:51:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:51:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:51:58.696 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0a00036460 legacy=0x7f0a00038920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 10 monclient: shutdown 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 >> v1:172.21.15.76:6789/0 conn(0x7f0a20132330 legacy=0x7f0a201d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a24b7a700 1 -- 172.21.15.76:0/2233862052 reap_dead start 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a24b7a700 1 -- 172.21.15.76:0/2233862052 reap_dead start 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 shutdown_connections 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 >> 172.21.15.76:0/2233862052 conn(0x7f0a200a41c0 msgr2=0x7f0a200a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 shutdown_connections 2024-09-16T06:51:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.675+0000 7f0a2537b700 1 -- 172.21.15.76:0/2233862052 wait complete. 2024-09-16T06:51:58.698 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:51:58.698 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:51:58.698 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:51:58.698 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:51:58.698 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:51:58.699 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:51:58.699 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:51:58.759 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:51:58.759 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:51:58.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 10 monclient: get_monmap_and_config 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 10 monclient: monmap: 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:51:58.883161+0000 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:51:58.883161+0000 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.878 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.879 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:51:58.879 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:51:58.879 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 1 Processor -- start 2024-09-16T06:51:58.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 1 -- start start 2024-09-16T06:51:58.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.879+0000 7fec27fff700 10 monclient: init 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-a con 0x7fec2809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-b con 0x7fec28098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-c con 0x7fec28099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2809d560 con 0x7fec2809ed60 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2809f140 con 0x7fec28099030 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2809d790 con 0x7fec28098be0 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): authenticate will time out at 1666.016433s 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec26ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fec2809ed60 0x7fec280984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56152/0 (socket says 172.21.15.76:56152) 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec26ffd700 1 -- 172.21.15.76:0/680770326 learned_addr learned my addr 172.21.15.76:0/680770326 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec277fe700 1 -- 172.21.15.76:0/680770326 >> v1:172.21.15.189:6789/0 conn(0x7fec28099030 legacy=0x7fec2813ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824752783 0 0) 0x7fec2809d790 con 0x7fec28098be0 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient(hunting): my global_id is 244370 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec140057e0 con 0x7fec28098be0 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340475591 0 0) 0x7fec140057e0 con 0x7fec28098be0 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 >> v1:172.21.15.189:6789/0 conn(0x7fec28099030 legacy=0x7fec2813ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 >> v1:172.21.15.76:6789/0 conn(0x7fec2809ed60 legacy=0x7fec280984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 monclient: found mon.noname-b 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2809f620 con 0x7fec28098be0 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 5 monclient: authenticate success, global_id 244370 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fec10001610 con 0x7fec28098be0 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: dump: 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.885 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec10001930 con 0x7fec28098be0 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 4 set_mon_vals no callback set 2024-09-16T06:51:58.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 1 -- 172.21.15.76:0/680770326 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fec10001cc0 con 0x7fec28098be0 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec25ffb700 10 monclient: dump: 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: shutdown 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- 172.21.15.76:0/680770326 >> v1:172.21.15.76:6790/0 conn(0x7fec28098be0 legacy=0x7fec281374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- 172.21.15.76:0/680770326 shutdown_connections 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- 172.21.15.76:0/680770326 >> 172.21.15.76:0/680770326 conn(0x7fec280a0030 msgr2=0x7fec280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- 172.21.15.76:0/680770326 shutdown_connections 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- 172.21.15.76:0/680770326 wait complete. 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: build_initial_monmap 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: monmap: 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:51:58.888 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 Processor -- start 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 1 -- start start 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: init 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.883+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-c con 0x7fec28098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-b con 0x7fec28099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient(hunting): picked mon.noname-a con 0x7fec2809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:51:58.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec28090810 con 0x7fec2809ed60 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec28090d40 con 0x7fec28099030 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec28091020 con 0x7fec28098be0 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient(hunting): authenticate will time out at 1666.020054s 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec26ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fec28098be0 0x7fec28096590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60406/0 (socket says 172.21.15.76:60406) 2024-09-16T06:51:58.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec26ffd700 1 -- 172.21.15.76:0/395068219 learned_addr learned my addr 172.21.15.76:0/395068219 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:51:58.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec267fc700 1 -- 172.21.15.76:0/395068219 >> v1:172.21.15.189:6789/0 conn(0x7fec28099030 legacy=0x7fec281ca9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:51:58.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487387007 0 0) 0x7fec28091020 con 0x7fec28098be0 2024-09-16T06:51:58.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:51:58.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): my global_id is 244373 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec080035d0 con 0x7fec28098be0 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922640461 0 0) 0x7fec080035d0 con 0x7fec28098be0 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec28091020 con 0x7fec28098be0 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fec20001b20 con 0x7fec28098be0 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): dump: 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.893 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:51:58.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 359231514 0 0) 0x7fec28091020 con 0x7fec28098be0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 >> v1:172.21.15.189:6789/0 conn(0x7fec28099030 legacy=0x7fec281ca9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 >> v1:172.21.15.76:6789/0 conn(0x7fec2809ed60 legacy=0x7fec281ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 monclient: found mon.c 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec280a1350 con 0x7fec28098be0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 5 monclient: authenticate success, global_id 244373 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _renew_subs 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec28098780 con 0x7fec28098be0 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _renew_subs 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2808ec90 con 0x7fec28098be0 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec20004c10 con 0x7fec28098be0 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec24ff9700 4 set_mon_vals no callback set 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fec20004fa0 con 0x7fec28098be0 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 10 monclient: dump: 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:51:58.896 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fec20005200 con 0x7fec28098be0 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c005100 con 0x7fec28098be0 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fec200442c0 con 0x7fec28098be0 2024-09-16T06:51:58.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.887+0000 7fec267fc700 10 monclient: get_auth_request con 0x7fec08035ff0 auth_method 0 2024-09-16T06:51:58.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.895+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fec20014d20 con 0x7fec28098be0 2024-09-16T06:51:58.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.895+0000 7fec1f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:51:58.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:58.895+0000 7fec1f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:51:59.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec27fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec27fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec27fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fec0c005930 con 0x7fec28098be0 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec1f7fe700 1 -- 172.21.15.76:0/395068219 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 137841914) 0x7fec20014910 con 0x7fec28098be0 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec1f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:51:59.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.127+0000 7fec1f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:51:59.127 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:51:59.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 >> v1:172.21.15.189:6812/3098810674 conn(0x7fec08035ff0 legacy=0x7fec080384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 10 monclient: shutdown 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 >> v1:172.21.15.76:6790/0 conn(0x7fec28098be0 legacy=0x7fec28096590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 shutdown_connections 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 >> 172.21.15.76:0/395068219 conn(0x7fec280a0030 msgr2=0x7fec2806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 shutdown_connections 2024-09-16T06:51:59.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:51:59.131+0000 7fec27fff700 1 -- 172.21.15.76:0/395068219 wait complete. 2024-09-16T06:52:02.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:02.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: get_monmap_and_config 2024-09-16T06:52:02.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: build_initial_monmap 2024-09-16T06:52:02.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: monmap: 2024-09-16T06:52:02.259 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:02.264480+0000 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:02.264480+0000 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:02.260 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 Processor -- start 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- start start 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: init 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient(hunting): picked mon.noname-b con 0x7f04180207b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient(hunting): picked mon.noname-a con 0x7f04180229f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:02.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient(hunting): picked mon.noname-c con 0x7f0418021e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418021bf0 con 0x7f04180229f0 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041801c340 con 0x7f0418021e70 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041801d500 con 0x7f04180207b0 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient(hunting): authenticate will time out at 1669.397840s 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f04180207b0 0x7f0418020b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60412/0 (socket says 172.21.15.76:60412) 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041f7fe700 1 -- 172.21.15.76:0/1838195185 learned_addr learned my addr 172.21.15.76:0/1838195185 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041ffff700 1 -- 172.21.15.76:0/1838195185 >> v1:172.21.15.189:6789/0 conn(0x7f0418021e70 legacy=0x7f04180c0510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937805167 0 0) 0x7f041801d500 con 0x7f04180207b0 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:02.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient(hunting): my global_id is 244376 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04080046e0 con 0x7f04180207b0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852076889 0 0) 0x7f04080046e0 con 0x7f04180207b0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 >> v1:172.21.15.189:6789/0 conn(0x7f0418021e70 legacy=0x7f04180c0510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 >> v1:172.21.15.76:6789/0 conn(0x7f04180229f0 legacy=0x7f04180bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041801d920 con 0x7f04180207b0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:02.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 5 monclient: authenticate success, global_id 244376 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0410001610 con 0x7f04180207b0 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: dump: 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:02.266 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04100018a0 con 0x7f04180207b0 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 4 set_mon_vals no callback set 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 1 -- 172.21.15.76:0/1838195185 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0410001c70 con 0x7f04180207b0 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f041e7fc700 10 monclient: dump: 2024-09-16T06:52:02.267 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 10 monclient: shutdown 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:02.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- 172.21.15.76:0/1838195185 >> v1:172.21.15.76:6790/0 conn(0x7f04180207b0 legacy=0x7f0418020b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- 172.21.15.76:0/1838195185 shutdown_connections 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.263+0000 7f042487b700 1 -- 172.21.15.76:0/1838195185 >> 172.21.15.76:0/1838195185 conn(0x7f0418005b40 msgr2=0x7f0418005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- 172.21.15.76:0/1838195185 shutdown_connections 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- 172.21.15.76:0/1838195185 wait complete. 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: build_initial_monmap 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: monmap: 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:02.269 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:02.270 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:02.270 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:02.270 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 Processor -- start 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- start start 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: init 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient(hunting): picked mon.noname-a con 0x7f04180207b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:02.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient(hunting): picked mon.noname-c con 0x7f0418021e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient(hunting): picked mon.noname-b con 0x7f04180229f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041801f780 con 0x7f04180207b0 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041814a4d0 con 0x7f04180229f0 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180204a0 con 0x7f0418021e70 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient(hunting): authenticate will time out at 1669.402230s 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f041f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f04180207b0 0x7f0418149dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:56166/0 (socket says 172.21.15.76:56166) 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f041f7fe700 1 -- 172.21.15.76:0/3159521501 learned_addr learned my addr 172.21.15.76:0/3159521501 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:02.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f041ffff700 1 -- 172.21.15.76:0/3159521501 >> v1:172.21.15.189:6789/0 conn(0x7f04180229f0 legacy=0x7f0418158820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643743110 0 0) 0x7f04180204a0 con 0x7f0418021e70 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): my global_id is 244379 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f040c0035d0 con 0x7f0418021e70 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644100791 0 0) 0x7f040c0035d0 con 0x7f0418021e70 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04180204a0 con 0x7f0418021e70 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f04140017e0 con 0x7f0418021e70 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): dump: 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:02.276 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 721956248 0 0) 0x7f04180204a0 con 0x7f0418021e70 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 >> v1:172.21.15.189:6789/0 conn(0x7f04180229f0 legacy=0x7f0418158820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 >> v1:172.21.15.76:6789/0 conn(0x7f04180207b0 legacy=0x7f0418149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:02.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: _finish_hunting 0 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 monclient: found mon.c 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041812ec90 con 0x7f0418021e70 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: _finish_auth 0 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: _check_auth_tickets 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 5 monclient: authenticate success, global_id 244379 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _renew_subs 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0418131720 con 0x7f0418021e70 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _renew_subs 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181301d0 con 0x7f0418021e70 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04140036c0 con 0x7f0418021e70 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0414003a50 con 0x7f0418021e70 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f0407fff700 10 monclient: dump: 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:02.279 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.267+0000 7f041d7fa700 4 set_mon_vals no callback set 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0414003bb0 con 0x7f0418021e70 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f042487b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f042487b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f042487b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0418131070 con 0x7f0418021e70 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0414042c70 con 0x7f0418021e70 2024-09-16T06:52:02.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.271+0000 7f041f7fe700 10 monclient: get_auth_request con 0x7f040c036100 auth_method 0 2024-09-16T06:52:02.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.279+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f04140135b0 con 0x7f0418021e70 2024-09-16T06:52:02.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.279+0000 7f0407fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:02.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.279+0000 7f0407fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0418008040 con 0x7f0418021e70 2024-09-16T06:52:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f0407fff700 1 -- 172.21.15.76:0/3159521501 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1490780891) 0x7f0414011a20 con 0x7f0418021e70 2024-09-16T06:52:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f0407fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:02.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f0407fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:02.508 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 >> v1:172.21.15.189:6812/3098810674 conn(0x7f040c036100 legacy=0x7f040c0385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 10 monclient: shutdown 2024-09-16T06:52:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 >> v1:172.21.15.76:6790/0 conn(0x7f0418021e70 legacy=0x7f04181550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 shutdown_connections 2024-09-16T06:52:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 >> 172.21.15.76:0/3159521501 conn(0x7f0418005b40 msgr2=0x7f041801dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 shutdown_connections 2024-09-16T06:52:02.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:02.511+0000 7f042487b700 1 -- 172.21.15.76:0/3159521501 wait complete. 2024-09-16T06:52:05.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:05.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: get_monmap_and_config 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: build_initial_monmap 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: monmap: 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:05.644718+0000 2024-09-16T06:52:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:05.644718+0000 2024-09-16T06:52:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:05.644 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 1 Processor -- start 2024-09-16T06:52:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 1 -- start start 2024-09-16T06:52:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: init 2024-09-16T06:52:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:05.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:05.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-a con 0x7f7ce409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:05.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-b con 0x7f7ce409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:05.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-c con 0x7f7ce4139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:05.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce409cd80 con 0x7f7ce409f230 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4099fd0 con 0x7f7ce4139b70 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce409a410 con 0x7f7ce409fd20 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce9fde700 10 monclient(hunting): authenticate will time out at 1672.778011s 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce3fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7ce413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:05.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce37fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7ce409f230 0x7f7ce409f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50530/0 (socket says 172.21.15.76:50530) 2024-09-16T06:52:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.643+0000 7f7ce37fe700 1 -- 172.21.15.76:0/2085341991 learned_addr learned my addr 172.21.15.76:0/2085341991 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:05.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:05.847+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7ce413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:06.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:06.251+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7ce413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:07.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:07.055+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7ce413bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): tick 2024-09-16T06:52:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:52:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:52:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:52:08.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6790/0 conn(0x7f7ce409fd20 legacy=0x7f7ce4138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7ce413bf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6789/0 conn(0x7f7ce409f230 legacy=0x7f7ce409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7cc4003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f7cc40073d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f7ce4139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce409de90 con 0x7f7cc4003740 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4099fd0 con 0x7f7ce4139b70 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc4010940 con 0x7f7cc40073d0 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce1ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:08.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.643+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:08.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:08.847+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:09.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:09.251+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:10.055+0000 7f7ce3fff700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): tick 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6790/0 conn(0x7f7cc40073d0 legacy=0x7f7cc4009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7ce4139b70 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6789/0 conn(0x7f7cc4003740 legacy=0x7f7cc4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f7cc4010bd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:11.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7cc40110b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f7cc40073d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc4011b90 con 0x7f7cc40110b0 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4099fd0 con 0x7f7cc4010bd0 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc40116c0 con 0x7f7cc40073d0 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce1ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:11.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.643+0000 7f7ce37fe700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7cc4010bd0 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:11.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:11.847+0000 7f7ce37fe700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7cc4010bd0 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:12.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.251+0000 7f7ce37fe700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7cc4010bd0 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674126434 0 0) 0x7f7cc4011b90 con 0x7f7cc40110b0 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 10 monclient(hunting): my global_id is 264153 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.783+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd00035d0 con 0x7f7cc40110b0 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276495011 0 0) 0x7f7cd00035d0 con 0x7f7cc40110b0 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6790/0 conn(0x7f7cc40073d0 legacy=0x7f7cc4009860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:12.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.189:6789/0 conn(0x7f7cc4010bd0 legacy=0x7f7cc400d190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40118a0 con 0x7f7cc40110b0 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ccc001550 con 0x7f7cc40110b0 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 5 monclient: authenticate success, global_id 264153 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: dump: 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ccc0017f0 con 0x7f7cc40110b0 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 1 -- 172.21.15.76:0/2085341991 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7ccc001bc0 con 0x7f7cc40110b0 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 4 set_mon_vals no callback set 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce2ffd700 10 monclient: dump: 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:12.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 10 monclient: shutdown 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- 172.21.15.76:0/2085341991 >> v1:172.21.15.76:6789/0 conn(0x7f7cc40110b0 legacy=0x7f7cc4011e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- 172.21.15.76:0/2085341991 shutdown_connections 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- 172.21.15.76:0/2085341991 >> 172.21.15.76:0/2085341991 conn(0x7f7ce40a3e50 msgr2=0x7f7ce40a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- 172.21.15.76:0/2085341991 shutdown_connections 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- 172.21.15.76:0/2085341991 wait complete. 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 10 monclient: build_initial_monmap 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 10 monclient: monmap: 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:12.790 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 Processor -- start 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 1 -- start start 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.787+0000 7f7ce9fde700 10 monclient: init 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-b con 0x7f7cc40073d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:12.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-c con 0x7f7cc4010bd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient(hunting): picked mon.noname-a con 0x7f7ce41326e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce411f6b0 con 0x7f7ce41326e0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce4072450 con 0x7f7cc40073d0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce40a29a0 con 0x7f7cc4010bd0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient(hunting): authenticate will time out at 1679.924156s 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce37fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7cc40073d0 legacy=0x7f7ce4131fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7ce41326e0 0x7f7ce41d2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50550/0 (socket says 172.21.15.76:50550) 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce3fff700 1 -- 172.21.15.76:0/313363664 learned_addr learned my addr 172.21.15.76:0/313363664 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860751474 0 0) 0x7f7ce411f6b0 con 0x7f7ce41326e0 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:12.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): my global_id is 264156 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4011b30 con 0x7f7ce41326e0 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782372291 0 0) 0x7f7cc4011b30 con 0x7f7ce41326e0 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ce411f6b0 con 0x7f7ce41326e0 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7cdc004610 con 0x7f7ce41326e0 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:12.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1949754333 0 0) 0x7f7ce411f6b0 con 0x7f7ce41326e0 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 >> v1:172.21.15.76:6790/0 conn(0x7f7cc4010bd0 legacy=0x7f7ce41cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 >> v1:172.21.15.189:6789/0 conn(0x7f7cc40073d0 legacy=0x7f7ce4131fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 monclient: found mon.a 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce409ca80 con 0x7f7ce41326e0 2024-09-16T06:52:12.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 5 monclient: authenticate success, global_id 264156 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _renew_subs 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce411f990 con 0x7f7ce41326e0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _renew_subs 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce411ca80 con 0x7f7ce41326e0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cdc004a20 con 0x7f7ce41326e0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7cdc004db0 con 0x7f7ce41326e0 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce0ff9700 10 monclient: dump: 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce27fc700 4 set_mon_vals no callback set 2024-09-16T06:52:12.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.791+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce40678b0 con 0x7f7ce41326e0 2024-09-16T06:52:12.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.795+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7cdc004f90 con 0x7f7ce41326e0 2024-09-16T06:52:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.795+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7cdc044000 con 0x7f7ce41326e0 2024-09-16T06:52:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.795+0000 7f7ce37fe700 10 monclient: get_auth_request con 0x7f7cc4037c50 auth_method 0 2024-09-16T06:52:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.799+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7cdc0149a0 con 0x7f7ce41326e0 2024-09-16T06:52:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.799+0000 7f7ce0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:12.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:12.799+0000 7f7ce0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce9fde700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce9fde700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce9fde700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7ce4069620 con 0x7f7ce41326e0 2024-09-16T06:52:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce0ff9700 1 -- 172.21.15.76:0/313363664 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1294782229) 0x7f7cdc012e10 con 0x7f7ce41326e0 2024-09-16T06:52:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.035+0000 7f7ce0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:13.034 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":124,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7cc4037c50 legacy=0x7f7cc403a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 10 monclient: shutdown 2024-09-16T06:52:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 >> v1:172.21.15.76:6789/0 conn(0x7f7ce41326e0 legacy=0x7f7ce41d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 shutdown_connections 2024-09-16T06:52:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 >> 172.21.15.76:0/313363664 conn(0x7f7ce40a3e50 msgr2=0x7f7ce4133960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 shutdown_connections 2024-09-16T06:52:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.039+0000 7f7ce9fde700 1 -- 172.21.15.76:0/313363664 wait complete. 2024-09-16T06:52:13.048 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:52:13.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:52:13.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: get_monmap_and_config 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: build_initial_monmap 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: monmap: 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:13.162515+0000 2024-09-16T06:52:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:13.162515+0000 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:13.158 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 1 Processor -- start 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 1 -- start start 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: init 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-b con 0x7f92a009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-c con 0x7f92a0098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.159+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-a con 0x7f92a0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a009d560 con 0x7f92a0099030 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a009f140 con 0x7f92a0098be0 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a009d790 con 0x7f92a009ed60 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient(hunting): authenticate will time out at 1680.296064s 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a5cfb700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f92a0098be0 legacy=0x7f92a01374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a6cfd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f92a0099030 0x7f92a013ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50566/0 (socket says 172.21.15.76:50566) 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a64fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f92a009ed60 0x7f92a00984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:32918/0 (socket says 172.21.15.76:32918) 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a6cfd700 1 -- 172.21.15.76:0/3820364978 learned_addr learned my addr 172.21.15.76:0/3820364978 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474440710 0 0) 0x7f92a009d560 con 0x7f92a0099030 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): my global_id is 264159 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92880035d0 con 0x7f92a0099030 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868760853 0 0) 0x7f92a009d790 con 0x7f92a009ed60 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient(hunting): my global_id is 244382 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a009d560 con 0x7f92a009ed60 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163245176 0 0) 0x7f92880035d0 con 0x7f92a0099030 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 >> v1:172.21.15.76:6790/0 conn(0x7f92a009ed60 legacy=0x7f92a00984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 >> v1:172.21.15.189:6789/0 conn(0x7f92a0098be0 legacy=0x7f92a01374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: _finish_hunting 0 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 monclient: found mon.noname-a 2024-09-16T06:52:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a009f620 con 0x7f92a0099030 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: _finish_auth 0 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: _check_auth_tickets 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 5 monclient: authenticate success, global_id 264159 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f929c001610 con 0x7f92a0099030 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: dump: 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f929c0018f0 con 0x7f92a0099030 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 1 -- 172.21.15.76:0/3820364978 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f929c001cc0 con 0x7f92a0099030 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 4 set_mon_vals no callback set 2024-09-16T06:52:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a54fa700 10 monclient: dump: 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient: shutdown 2024-09-16T06:52:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- 172.21.15.76:0/3820364978 >> v1:172.21.15.76:6789/0 conn(0x7f92a0099030 legacy=0x7f92a013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- 172.21.15.76:0/3820364978 shutdown_connections 2024-09-16T06:52:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- 172.21.15.76:0/3820364978 >> 172.21.15.76:0/3820364978 conn(0x7f92a00a0030 msgr2=0x7f92a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- 172.21.15.76:0/3820364978 shutdown_connections 2024-09-16T06:52:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- 172.21.15.76:0/3820364978 wait complete. 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient: build_initial_monmap 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient: monmap: 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 Processor -- start 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 1 -- start start 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.163+0000 7f92a74fe700 10 monclient: init 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-b con 0x7f92a0098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-a con 0x7f92a0099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient(hunting): picked mon.noname-c con 0x7f92a009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a00931a0 con 0x7f92a0099030 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a009e750 con 0x7f92a0098be0 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a0092900 con 0x7f92a009ed60 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a64fc700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f92a0098be0 legacy=0x7f92a00966f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient(hunting): authenticate will time out at 1680.301257s 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a6cfd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f92a009ed60 0x7f92a01d0250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:32924/0 (socket says 172.21.15.76:32924) 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a5cfb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f92a0099030 0x7f92a01ccb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50568/0 (socket says 172.21.15.76:50568) 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a6cfd700 1 -- 172.21.15.76:0/175856875 learned_addr learned my addr 172.21.15.76:0/175856875 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104343205 0 0) 0x7f92a0092900 con 0x7f92a009ed60 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): my global_id is 244385 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f928c0035d0 con 0x7f92a009ed60 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136751910 0 0) 0x7f92a00931a0 con 0x7f92a0099030 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): my global_id is 264162 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0092900 con 0x7f92a0099030 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725086567 0 0) 0x7f928c0035d0 con 0x7f92a009ed60 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a00931a0 con 0x7f92a009ed60 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f929c001c00 con 0x7f92a009ed60 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:13.175 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302026346 0 0) 0x7f92a0092900 con 0x7f92a0099030 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f928c0035d0 con 0x7f92a0099030 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9290004520 con 0x7f92a0099030 2024-09-16T06:52:13.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667817236 0 0) 0x7f92a00931a0 con 0x7f92a009ed60 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.189:6789/0 conn(0x7f92a0098be0 legacy=0x7f92a00966f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6789/0 conn(0x7f92a0099030 legacy=0x7f92a01ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 monclient: found mon.c 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0092be0 con 0x7f92a009ed60 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 5 monclient: authenticate success, global_id 244385 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _renew_subs 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a01c7ce0 con 0x7f92a009ed60 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _renew_subs 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a01c8660 con 0x7f92a009ed60 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f929c004b40 con 0x7f92a009ed60 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f929c001d60 con 0x7f92a009ed60 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9297fff700 4 set_mon_vals no callback set 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: dump: 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.179 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9280006590 con 0x7f9280002830 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092038592 0 0) 0x7f9280006590 con 0x7f9280002830 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient(hunting): my global_id is 264165 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a00931a0 con 0x7f9280002830 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3202458892 0 0) 0x7f92a00931a0 con 0x7f9280002830 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9280006590 con 0x7f9280002830 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9298001950 con 0x7f9280002830 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.167+0000 7f9296ffd700 10 monclient: dump: 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.181 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3300428454 0 0) 0x7f9280006590 con 0x7f9280002830 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f928000a190 con 0x7f9280002830 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f929c00f760 con 0x7f92a009ed60 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f929c043e30 con 0x7f92a009ed60 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9298001b80 con 0x7f9280002830 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6789/0 conn(0x7f9280002830 legacy=0x7f9280004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6789/0 conn(0x7f9280002830 legacy=0x7f9280004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:13.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.171+0000 7f92a5cfb700 10 monclient: get_auth_request con 0x7f928c0365f0 auth_method 0 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.175+0000 7f92a74fe700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.175+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f928000c4e0 con 0x7f9280009150 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631924435 0 0) 0x7f928000c4e0 con 0x7f9280009150 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient(hunting): my global_id is 264168 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9280006590 con 0x7f9280009150 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4242632506 0 0) 0x7f9280006590 con 0x7f9280009150 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f928000c4e0 con 0x7f9280009150 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9298004c20 con 0x7f9280009150 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient: dump: 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.185 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3385682150 0 0) 0x7f928000c4e0 con 0x7f9280009150 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9280009cf0 con 0x7f9280009150 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 2103653261) 0x7f9298005f30 con 0x7f9280009150 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6789/0 conn(0x7f9280009150 legacy=0x7f92800095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f9296ffd700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6789/0 conn(0x7f9280009150 legacy=0x7f92800095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 124, 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:13.187 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:13.188 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.189 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:13.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.191 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.192 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:13.193 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.194 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.196 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:13.197 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.189:6812/3098810674 conn(0x7f928c0365f0 legacy=0x7f928c038ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 10 monclient: shutdown 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 >> v1:172.21.15.76:6790/0 conn(0x7f92a009ed60 legacy=0x7f92a01d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a6cfd700 1 -- 172.21.15.76:0/175856875 reap_dead start 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a6cfd700 1 -- 172.21.15.76:0/175856875 reap_dead start 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 shutdown_connections 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 >> 172.21.15.76:0/175856875 conn(0x7f92a00a0030 msgr2=0x7f92a006e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:13.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 shutdown_connections 2024-09-16T06:52:13.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.179+0000 7f92a74fe700 1 -- 172.21.15.76:0/175856875 wait complete. 2024-09-16T06:52:13.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:52:13.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:52:13.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:52:13.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: get_monmap_and_config 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: build_initial_monmap 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: monmap: 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:13.320862+0000 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:13.320862+0000 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:13.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 1 Processor -- start 2024-09-16T06:52:13.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 1 -- start start 2024-09-16T06:52:13.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: init 2024-09-16T06:52:13.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-a con 0x7ff6f0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-b con 0x7ff6f006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-c con 0x7ff6f006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f006c280 con 0x7ff6f0132330 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f006bfa0 con 0x7ff6f006cf50 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f009ace0 con 0x7ff6f006d480 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 10 monclient(hunting): authenticate will time out at 1680.454618s 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6effff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7ff6f006cf50 legacy=0x7ff6f013da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6eeffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff6f006d480 0x7ff6f013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:32926/0 (socket says 172.21.15.76:32926) 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ef7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff6f0132330 0x7ff6f0132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50608/0 (socket says 172.21.15.76:50608) 2024-09-16T06:52:13.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6eeffd700 1 -- 172.21.15.76:0/1726493343 learned_addr learned my addr 172.21.15.76:0/1726493343 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604177739 0 0) 0x7ff6f009ace0 con 0x7ff6f006d480 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): my global_id is 244388 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc0035d0 con 0x7ff6f006d480 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840884929 0 0) 0x7ff6f006c280 con 0x7ff6f0132330 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient(hunting): my global_id is 264171 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f009ace0 con 0x7ff6f0132330 2024-09-16T06:52:13.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297904127 0 0) 0x7ff6f009ace0 con 0x7ff6f0132330 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 >> v1:172.21.15.76:6790/0 conn(0x7ff6f006d480 legacy=0x7ff6f013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 >> v1:172.21.15.189:6789/0 conn(0x7ff6f006cf50 legacy=0x7ff6f013da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 monclient: found mon.noname-a 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f009ae70 con 0x7ff6f0132330 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: _finish_auth 0 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e00015f0 con 0x7ff6f0132330 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 5 monclient: authenticate success, global_id 264171 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:13.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6ee7fc700 10 monclient: dump: 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.319+0000 7ff6f4c31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:13.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e0001910 con 0x7ff6f0132330 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 1 -- 172.21.15.76:0/1726493343 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e0001b40 con 0x7ff6f0132330 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 4 set_mon_vals no callback set 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ee7fc700 10 monclient: dump: 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: shutdown 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- 172.21.15.76:0/1726493343 >> v1:172.21.15.76:6789/0 conn(0x7ff6f0132330 legacy=0x7ff6f0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- 172.21.15.76:0/1726493343 shutdown_connections 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- 172.21.15.76:0/1726493343 >> 172.21.15.76:0/1726493343 conn(0x7ff6f00a41c0 msgr2=0x7ff6f00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- 172.21.15.76:0/1726493343 shutdown_connections 2024-09-16T06:52:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- 172.21.15.76:0/1726493343 wait complete. 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: build_initial_monmap 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: monmap: 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 Processor -- start 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- start start 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: init 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-a con 0x7ff6f006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:13.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-b con 0x7ff6f006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient(hunting): picked mon.noname-c con 0x7ff6f0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f009e8b0 con 0x7ff6f006cf50 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f01304b0 con 0x7ff6f006d480 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0130790 con 0x7ff6f0132330 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ef7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff6f006cf50 0x7ff6f013e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50614/0 (socket says 172.21.15.76:50614) 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6f4c31700 10 monclient(hunting): authenticate will time out at 1680.459675s 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6ef7fe700 1 -- 172.21.15.76:0/489290138 learned_addr learned my addr 172.21.15.76:0/489290138 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.323+0000 7ff6eeffd700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.189:6789/0 conn(0x7ff6f006d480 legacy=0x7ff6f01ce990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805629895 0 0) 0x7ff6f0130790 con 0x7ff6f0132330 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): my global_id is 244391 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d00035d0 con 0x7ff6f0132330 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211222304 0 0) 0x7ff6f009e8b0 con 0x7ff6f006cf50 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): my global_id is 264174 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0130790 con 0x7ff6f006cf50 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743211516 0 0) 0x7ff6f0130790 con 0x7ff6f006cf50 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f009e8b0 con 0x7ff6f006cf50 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e0004890 con 0x7ff6f006cf50 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:13.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): dump: 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3736444043 0 0) 0x7ff6d00035d0 con 0x7ff6f0132330 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f0130790 con 0x7ff6f0132330 2024-09-16T06:52:13.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e80043b0 con 0x7ff6f0132330 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): dump: 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 636078648 0 0) 0x7ff6f0130790 con 0x7ff6f0132330 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.189:6789/0 conn(0x7ff6f006d480 legacy=0x7ff6f01ce990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6789/0 conn(0x7ff6f006cf50 legacy=0x7ff6f013e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 monclient: found mon.c 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0130a70 con 0x7ff6f0132330 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: _finish_auth 0 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 5 monclient: authenticate success, global_id 244391 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 10 monclient: _renew_subs 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f0130dc0 con 0x7ff6f0132330 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e80045d0 con 0x7ff6f0132330 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 10 monclient: _renew_subs 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f01c9ac0 con 0x7ff6f0132330 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e80048e0 con 0x7ff6f0132330 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: dump: 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.337 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6edffb700 4 set_mon_vals no callback set 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6f00a3430 con 0x7ff6f013e900 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441859213 0 0) 0x7ff6e0004ca0 con 0x7ff6f013e900 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient(hunting): my global_id is 244394 2024-09-16T06:52:13.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0130790 con 0x7ff6f013e900 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff6e8004a40 con 0x7ff6f0132330 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff6e8043ea0 con 0x7ff6f0132330 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 684894037 0 0) 0x7ff6f0130790 con 0x7ff6f013e900 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e0004ca0 con 0x7ff6f013e900 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e0005020 con 0x7ff6f013e900 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 10 monclient: dump: 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.339 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6eeffd700 10 monclient: get_auth_request con 0x7ff6d0036330 auth_method 0 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1739936305 0 0) 0x7ff6e0004ca0 con 0x7ff6f013e900 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.327+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6f0130fb0 con 0x7ff6f013e900 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.331+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff6e0007280 con 0x7ff6f013e900 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.331+0000 7ff6e7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.331+0000 7ff6e7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.331+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6790/0 conn(0x7ff6f013e900 legacy=0x7ff6f009fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.331+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6790/0 conn(0x7ff6f013e900 legacy=0x7ff6f009fad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6f4c31700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff6f00a3430 con 0x7ff6f012ef00 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946732559 0 0) 0x7ff6e0005510 con 0x7ff6f012ef00 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient(hunting): my global_id is 244397 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e0004ca0 con 0x7ff6f012ef00 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2431274261 0 0) 0x7ff6e0004ca0 con 0x7ff6f012ef00 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e0005510 con 0x7ff6f012ef00 2024-09-16T06:52:13.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff6e0005950 con 0x7ff6f012ef00 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.335+0000 7ff6e7fff700 10 monclient: dump: 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:13.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4287493409 0 0) 0x7ff6e0005510 con 0x7ff6f012ef00 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6f00672a0 con 0x7ff6f012ef00 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 3053336893) 0x7ff6e0006e50 con 0x7ff6f012ef00 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6790/0 conn(0x7ff6f012ef00 legacy=0x7ff6f002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6e7fff700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6790/0 conn(0x7ff6f012ef00 legacy=0x7ff6f002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 124, 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.345 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.346 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:13.347 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:13.348 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:13.349 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.350 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.351 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:13.352 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:13.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:13.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:13.355 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:13.356 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff6d0036330 legacy=0x7ff6d00387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 10 monclient: shutdown 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 >> v1:172.21.15.76:6790/0 conn(0x7ff6f0132330 legacy=0x7ff6f01d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6effff700 1 -- 172.21.15.76:0/489290138 reap_dead start 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6effff700 1 -- 172.21.15.76:0/489290138 reap_dead start 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 shutdown_connections 2024-09-16T06:52:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 >> 172.21.15.76:0/489290138 conn(0x7ff6f00a41c0 msgr2=0x7ff6f009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 shutdown_connections 2024-09-16T06:52:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:13.339+0000 7ff6f4c31700 1 -- 172.21.15.76:0/489290138 wait complete. 2024-09-16T06:52:13.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:52:13.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:52:33.359 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:52:33.359 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:52:33.359 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:52:33.360 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:52:33.363 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:52:33.363 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:52:33.363 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:33.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 10 monclient: get_monmap_and_config 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 10 monclient: build_initial_monmap 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 10 monclient: monmap: 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:33.490966+0000 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:33.490966+0000 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:33.486 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 1 Processor -- start 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 1 -- start start 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.486+0000 7f5ad6a97700 10 monclient: init 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-c con 0x7f5ac801f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-a con 0x7f5ac8025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-b con 0x7f5ac8025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac801f8a0 con 0x7f5ac8025410 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80233f0 con 0x7f5ac801f4c0 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac8020050 con 0x7f5ac8025860 2024-09-16T06:52:33.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient(hunting): authenticate will time out at 1700.624268s 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad5294700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5ac8025410 0x7f5ac80bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39946/0 (socket says 172.21.15.76:39946) 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad5294700 1 -- 172.21.15.76:0/619470734 learned_addr learned my addr 172.21.15.76:0/619470734 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098390299 0 0) 0x7f5ac801f8a0 con 0x7f5ac8025410 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): my global_id is 264177 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5abc0035d0 con 0x7f5ac8025410 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889092600 0 0) 0x7f5ac8020050 con 0x7f5ac8025860 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient(hunting): my global_id is 244400 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac801f8a0 con 0x7f5ac8025860 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797046798 0 0) 0x7f5abc0035d0 con 0x7f5ac8025410 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 >> v1:172.21.15.76:6790/0 conn(0x7f5ac8025860 legacy=0x7f5ac80c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 >> v1:172.21.15.189:6789/0 conn(0x7f5ac801f4c0 legacy=0x7f5ac8024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: _finish_hunting 0 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 monclient: found mon.noname-a 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac8021a90 con 0x7f5ac8025410 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: _finish_auth 0 2024-09-16T06:52:33.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: _check_auth_tickets 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 5 monclient: authenticate success, global_id 264177 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5acc001610 con 0x7f5ac8025410 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: dump: 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5acc001900 con 0x7f5ac8025410 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 1 -- 172.21.15.76:0/619470734 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5acc001c90 con 0x7f5ac8025410 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad4a93700 10 monclient: dump: 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 4 set_mon_vals no callback set 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 10 monclient: shutdown 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- 172.21.15.76:0/619470734 >> v1:172.21.15.76:6789/0 conn(0x7f5ac8025410 legacy=0x7f5ac80bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- 172.21.15.76:0/619470734 shutdown_connections 2024-09-16T06:52:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- 172.21.15.76:0/619470734 >> 172.21.15.76:0/619470734 conn(0x7f5ac8005b40 msgr2=0x7f5ac8005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- 172.21.15.76:0/619470734 shutdown_connections 2024-09-16T06:52:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.490+0000 7f5ad6a97700 1 -- 172.21.15.76:0/619470734 wait complete. 2024-09-16T06:52:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient: build_initial_monmap 2024-09-16T06:52:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient: monmap: 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 1 Processor -- start 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 1 -- start start 2024-09-16T06:52:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient: init 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-c con 0x7f5ac801f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-a con 0x7f5ac8025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient(hunting): picked mon.noname-b con 0x7f5ac8025860 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 0x7f5ac81508a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42046/0 (socket says 172.21.15.76:42046) 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80192e0 con 0x7f5ac8025410 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 -- 172.21.15.76:0/3899752127 learned_addr learned my addr 172.21.15.76:0/3899752127 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac801be80 con 0x7f5ac8025860 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80204a0 con 0x7f5ac801f4c0 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad6a97700 10 monclient(hunting): authenticate will time out at 1700.631028s 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 legacy=0x7f5ac81508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:52:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 legacy=0x7f5ac81508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:52:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 --1- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 0x7f5ac81508a0 :-1 s=OPENED pgs=1505 cs=1 l=1).handle_message read tag failed 2024-09-16T06:52:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ad5a95700 1 --1- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 0x7f5ac81508a0 :-1 s=OPENED pgs=1505 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:52:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.494+0000 7f5ac27fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896001786 0 0) 0x7f5ac80192e0 con 0x7f5ac8025410 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): my global_id is 264180 2024-09-16T06:52:33.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab40035d0 con 0x7f5ac8025410 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130839665 0 0) 0x7f5ab40035d0 con 0x7f5ac8025410 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ac80192e0 con 0x7f5ac8025410 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5acc0015d0 con 0x7f5ac8025410 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): dump: 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3497655311 0 0) 0x7f5ac80192e0 con 0x7f5ac8025410 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6790/0 conn(0x7f5ac801f4c0 legacy=0x7f5ac81508a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.189:6789/0 conn(0x7f5ac8025860 legacy=0x7f5ac8159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 monclient: found mon.a 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac8131be0 con 0x7f5ac8025410 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: _finish_auth 0 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 5 monclient: authenticate success, global_id 264180 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _renew_subs 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac8131900 con 0x7f5ac8025410 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _renew_subs 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac801e2f0 con 0x7f5ac8025410 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5acc004ca0 con 0x7f5ac8025410 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5acc004fb0 con 0x7f5ac8025410 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac3fff700 4 set_mon_vals no callback set 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 10 monclient: dump: 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.529 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:33.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5acc00fa20 con 0x7f5ac8025410 2024-09-16T06:52:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.526+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab8005100 con 0x7f5ac8025410 2024-09-16T06:52:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.530+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5acc043f40 con 0x7f5ac8025410 2024-09-16T06:52:33.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.530+0000 7f5ad5a95700 10 monclient: get_auth_request con 0x7f5ab40360b0 auth_method 0 2024-09-16T06:52:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.538+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5acc014810 con 0x7f5ac8025410 2024-09-16T06:52:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.538+0000 7f5ac27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:33.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.538+0000 7f5ac27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:33.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ad6a97700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:33.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ad6a97700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ad6a97700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5ab8005930 con 0x7f5ac8025410 2024-09-16T06:52:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ac27fc700 1 -- 172.21.15.76:0/3899752127 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2790923852) 0x7f5acc012eb0 con 0x7f5ac8025410 2024-09-16T06:52:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ac27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.774+0000 7f5ac27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:33.773 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5ab40360b0 legacy=0x7f5ab4038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 10 monclient: shutdown 2024-09-16T06:52:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 >> v1:172.21.15.76:6789/0 conn(0x7f5ac8025410 legacy=0x7f5ac81558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 shutdown_connections 2024-09-16T06:52:33.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 >> 172.21.15.76:0/3899752127 conn(0x7f5ac8005b40 msgr2=0x7f5ac801aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:33.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 shutdown_connections 2024-09-16T06:52:33.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.778+0000 7f5ad6a97700 1 -- 172.21.15.76:0/3899752127 wait complete. 2024-09-16T06:52:33.791 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:52:33.791 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:52:33.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 10 monclient: get_monmap_and_config 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 10 monclient: build_initial_monmap 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 10 monclient: monmap: 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:33.906451+0000 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:33.906451+0000 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:33.901 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.902 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:33.902 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:33.902 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:33.902 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 1 Processor -- start 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 1 -- start start 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.902+0000 7f2d69262700 10 monclient: init 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-a con 0x7f2d64132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:33.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-c con 0x7f2d6406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-b con 0x7f2d6406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6406c280 con 0x7f2d64132330 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6406bfa0 con 0x7f2d6406d480 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6409ace0 con 0x7f2d6406cf50 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient(hunting): authenticate will time out at 1701.040398s 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d63fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2d64132330 0x7f2d64132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55058/0 (socket says 172.21.15.76:55058) 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d63fff700 1 -- 172.21.15.76:0/941155674 learned_addr learned my addr 172.21.15.76:0/941155674 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420071270 0 0) 0x7f2d6406c280 con 0x7f2d64132330 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): my global_id is 264183 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d500035d0 con 0x7f2d64132330 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751833321 0 0) 0x7f2d6409ace0 con 0x7f2d6406cf50 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): my global_id is 244403 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d6406c280 con 0x7f2d6406cf50 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904622216 0 0) 0x7f2d6406bfa0 con 0x7f2d6406d480 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient(hunting): my global_id is 274099 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d6409ace0 con 0x7f2d6406d480 2024-09-16T06:52:33.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635543287 0 0) 0x7f2d6406c280 con 0x7f2d6406cf50 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 >> v1:172.21.15.189:6789/0 conn(0x7f2d6406d480 legacy=0x7f2d6413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 >> v1:172.21.15.76:6789/0 conn(0x7f2d64132330 legacy=0x7f2d64132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 monclient: found mon.noname-b 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6409ae70 con 0x7f2d6406cf50 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d5c001970 con 0x7f2d6406cf50 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: dump: 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 5 monclient: authenticate success, global_id 244403 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:33.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d5c001c90 con 0x7f2d6406cf50 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 4 set_mon_vals no callback set 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 1 -- 172.21.15.76:0/941155674 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d5c001e80 con 0x7f2d6406cf50 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d62ffd700 10 monclient: dump: 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.910 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: shutdown 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- 172.21.15.76:0/941155674 >> v1:172.21.15.76:6790/0 conn(0x7f2d6406cf50 legacy=0x7f2d6413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- 172.21.15.76:0/941155674 shutdown_connections 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- 172.21.15.76:0/941155674 >> 172.21.15.76:0/941155674 conn(0x7f2d640a41c0 msgr2=0x7f2d640a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- 172.21.15.76:0/941155674 shutdown_connections 2024-09-16T06:52:33.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 -- 172.21.15.76:0/941155674 wait complete. 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: build_initial_monmap 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 10 monclient: monmap: 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.906+0000 7f2d69262700 1 Processor -- start 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- start start 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: init 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-a con 0x7f2d6406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-c con 0x7f2d6406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient(hunting): picked mon.noname-b con 0x7f2d64132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d641ca600 con 0x7f2d6406cf50 2024-09-16T06:52:33.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6412f810 con 0x7f2d64132330 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6412faf0 con 0x7f2d6406d480 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient(hunting): authenticate will time out at 1701.045370s 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d637fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2d6406d480 0x7f2d641c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43562/0 (socket says 172.21.15.76:43562) 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d637fe700 1 -- 172.21.15.76:0/4194093162 learned_addr learned my addr 172.21.15.76:0/4194093162 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487915101 0 0) 0x7f2d641ca600 con 0x7f2d6406cf50 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): my global_id is 264186 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d440035d0 con 0x7f2d6406cf50 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807229314 0 0) 0x7f2d6412faf0 con 0x7f2d6406d480 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): my global_id is 244406 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d641ca600 con 0x7f2d6406d480 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1957719103 0 0) 0x7f2d440035d0 con 0x7f2d6406cf50 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d6412faf0 con 0x7f2d6406cf50 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453941543 0 0) 0x7f2d6412f810 con 0x7f2d64132330 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): my global_id is 274102 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d440035d0 con 0x7f2d64132330 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d4c001850 con 0x7f2d6406cf50 2024-09-16T06:52:33.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:33.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500769505 0 0) 0x7f2d641ca600 con 0x7f2d6406d480 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d6412f810 con 0x7f2d6406d480 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d540018b0 con 0x7f2d6406d480 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 444112530 0 0) 0x7f2d6412faf0 con 0x7f2d6406cf50 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6790/0 conn(0x7f2d6406d480 legacy=0x7f2d641c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.189:6789/0 conn(0x7f2d64132330 legacy=0x7f2d641d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 monclient: found mon.a 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6412fdd0 con 0x7f2d6406cf50 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: _finish_auth 0 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:33.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 5 monclient: authenticate success, global_id 264186 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _renew_subs 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d641300b0 con 0x7f2d6406cf50 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _renew_subs 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6412e5e0 con 0x7f2d6406cf50 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d4c004890 con 0x7f2d6406cf50 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d4c004ba0 con 0x7f2d6406cf50 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d627fc700 4 set_mon_vals no callback set 2024-09-16T06:52:33.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 10 monclient: dump: 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.910+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2d4c00f610 con 0x7f2d6406cf50 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2d4c043d50 con 0x7f2d6406cf50 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d69262700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2d48006590 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d637fe700 10 monclient: get_auth_request con 0x7f2d44036510 auth_method 0 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371115241 0 0) 0x7f2d48006590 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient(hunting): my global_id is 264189 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d6412faf0 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3572762534 0 0) 0x7f2d6412faf0 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d48006590 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d5c001580 con 0x7f2d48002830 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient: dump: 2024-09-16T06:52:33.926 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1952994189 0 0) 0x7f2d48006590 con 0x7f2d48002830 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d4800a190 con 0x7f2d48002830 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2d5c001960 con 0x7f2d48002830 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:33.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6789/0 conn(0x7f2d48002830 legacy=0x7f2d48004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.914+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6789/0 conn(0x7f2d48002830 legacy=0x7f2d48004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d69262700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2d48009e50 con 0x7f2d48009170 2024-09-16T06:52:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355429439 0 0) 0x7f2d48009e50 con 0x7f2d48009170 2024-09-16T06:52:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 10 monclient(hunting): my global_id is 264192 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.922+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d48006590 con 0x7f2d48009170 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1345004499 0 0) 0x7f2d48006590 con 0x7f2d48009170 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d48009e50 con 0x7f2d48009170 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d5c003db0 con 0x7f2d48009170 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 10 monclient: dump: 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:33.930 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2660966103 0 0) 0x7f2d48009e50 con 0x7f2d48009170 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d4800c340 con 0x7f2d48009170 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3296128161) 0x7f2d5c004030 con 0x7f2d48009170 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6789/0 conn(0x7f2d48009170 legacy=0x7f2d48009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d60ff9700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6789/0 conn(0x7f2d48009170 legacy=0x7f2d48009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:33.932 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:33.933 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:33.934 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:33.935 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:33.936 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:33.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.938 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:33.939 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:33.940 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:33.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:33.942 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:33.943 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2d44036510 legacy=0x7f2d440389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 10 monclient: shutdown 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 >> v1:172.21.15.76:6789/0 conn(0x7f2d6406cf50 legacy=0x7f2d641c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d68a61700 1 -- 172.21.15.76:0/4194093162 reap_dead start 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d68a61700 1 -- 172.21.15.76:0/4194093162 reap_dead start 2024-09-16T06:52:33.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 shutdown_connections 2024-09-16T06:52:33.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 >> 172.21.15.76:0/4194093162 conn(0x7f2d640a41c0 msgr2=0x7f2d6409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:33.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 shutdown_connections 2024-09-16T06:52:33.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:33.926+0000 7f2d69262700 1 -- 172.21.15.76:0/4194093162 wait complete. 2024-09-16T06:52:33.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:33.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:52:33.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:52:34.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 10 monclient: get_monmap_and_config 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 10 monclient: monmap: 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:34.070967+0000 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:34.070967+0000 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:34.066 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 1 Processor -- start 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 1 -- start start 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.066+0000 7f0ae5efb700 10 monclient: init 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-a con 0x7f0ae0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-c con 0x7f0ae0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-b con 0x7f0ae0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae0098990 con 0x7f0ae0097b60 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae0097f40 con 0x7f0ae0099090 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae009d610 con 0x7f0ae0139500 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient(hunting): authenticate will time out at 1701.204476s 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae4ef9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0ae0097b60 0x7f0ae009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55108/0 (socket says 172.21.15.76:55108) 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae4ef9700 1 -- 172.21.15.76:0/3387431944 learned_addr learned my addr 172.21.15.76:0/3387431944 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770661391 0 0) 0x7f0ae0098990 con 0x7f0ae0097b60 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): my global_id is 264195 2024-09-16T06:52:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac80035d0 con 0x7f0ae0097b60 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107904981 0 0) 0x7f0ae009d610 con 0x7f0ae0139500 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): my global_id is 244409 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae0098990 con 0x7f0ae0139500 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629219776 0 0) 0x7f0ae0097f40 con 0x7f0ae0099090 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient(hunting): my global_id is 274105 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae009d610 con 0x7f0ae0099090 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998564164 0 0) 0x7f0ac80035d0 con 0x7f0ae0097b60 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 >> v1:172.21.15.76:6790/0 conn(0x7f0ae0139500 legacy=0x7f0ae013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 >> v1:172.21.15.189:6789/0 conn(0x7f0ae0099090 legacy=0x7f0ae009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 monclient: found mon.noname-a 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae00986b0 con 0x7f0ae0097b60 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad0001610 con 0x7f0ae0097b60 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 5 monclient: authenticate success, global_id 264195 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: dump: 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ad00018f0 con 0x7f0ae0097b60 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 1 -- 172.21.15.76:0/3387431944 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad0001c80 con 0x7f0ae0097b60 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 4 set_mon_vals no callback set 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0adf7fe700 10 monclient: dump: 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 10 monclient: shutdown 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- 172.21.15.76:0/3387431944 >> v1:172.21.15.76:6789/0 conn(0x7f0ae0097b60 legacy=0x7f0ae009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- 172.21.15.76:0/3387431944 shutdown_connections 2024-09-16T06:52:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- 172.21.15.76:0/3387431944 >> 172.21.15.76:0/3387431944 conn(0x7f0ae00a0030 msgr2=0x7f0ae00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- 172.21.15.76:0/3387431944 shutdown_connections 2024-09-16T06:52:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.070+0000 7f0ae5efb700 1 -- 172.21.15.76:0/3387431944 wait complete. 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: monmap: 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 Processor -- start 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- start start 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: init 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-b con 0x7f0ae0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-c con 0x7f0ae0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient(hunting): picked mon.noname-a con 0x7f0ae0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae006ccb0 con 0x7f0ae0139500 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae006f3d0 con 0x7f0ae0097b60 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae006ee10 con 0x7f0ae0099090 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient(hunting): authenticate will time out at 1701.209821s 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae56fa700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0ae0139500 0x7f0ae01cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55114/0 (socket says 172.21.15.76:55114) 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae56fa700 1 -- 172.21.15.76:0/2562059214 learned_addr learned my addr 172.21.15.76:0/2562059214 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333137597 0 0) 0x7f0ae006ccb0 con 0x7f0ae0139500 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): my global_id is 264198 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0acc0035d0 con 0x7f0ae0139500 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149351798 0 0) 0x7f0ae006ee10 con 0x7f0ae0099090 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): my global_id is 244412 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae006ccb0 con 0x7f0ae0099090 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326120659 0 0) 0x7f0ae006f3d0 con 0x7f0ae0097b60 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): my global_id is 274108 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae006ee10 con 0x7f0ae0097b60 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226430937 0 0) 0x7f0acc0035d0 con 0x7f0ae0139500 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae006f3d0 con 0x7f0ae0139500 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad8001c30 con 0x7f0ae0139500 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): dump: 2024-09-16T06:52:34.085 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2635535325 0 0) 0x7f0ae006ccb0 con 0x7f0ae0099090 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0acc0035d0 con 0x7f0ae0099090 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad40018b0 con 0x7f0ae0099090 2024-09-16T06:52:34.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient(hunting): dump: 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931831578 0 0) 0x7f0ae006f3d0 con 0x7f0ae0139500 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6790/0 conn(0x7f0ae0099090 legacy=0x7f0ae006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.189:6789/0 conn(0x7f0ae0097b60 legacy=0x7f0ae006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 monclient: found mon.a 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae006f0f0 con 0x7f0ae0139500 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 5 monclient: authenticate success, global_id 264198 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _renew_subs 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae01390c0 con 0x7f0ae0139500 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ad8003a40 con 0x7f0ae0139500 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad8003e50 con 0x7f0ae0139500 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:34.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0add7fa700 10 monclient: dump: 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _renew_subs 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae006eba0 con 0x7f0ae0139500 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.074+0000 7f0adeffd700 4 set_mon_vals no callback set 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0ae5efb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ae00a24c0 con 0x7f0ae006d390 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0ad8003fb0 con 0x7f0ae0139500 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0ad8043e00 con 0x7f0ae0139500 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305750950 0 0) 0x7f0ae00a24c0 con 0x7f0ae006d390 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient(hunting): my global_id is 244415 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae006f3d0 con 0x7f0ae006d390 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0adffff700 10 monclient: get_auth_request con 0x7f0acc036400 auth_method 0 2024-09-16T06:52:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2597210929 0 0) 0x7f0ae006f3d0 con 0x7f0ae006d390 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae00a24c0 con 0x7f0ae006d390 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad0001430 con 0x7f0ae006d390 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient: dump: 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1752220458 0 0) 0x7f0ae00a24c0 con 0x7f0ae006d390 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ae009fb80 con 0x7f0ae006d390 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0ad0004890 con 0x7f0ae006d390 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6790/0 conn(0x7f0ae006d390 legacy=0x7f0ae0135900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.078+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6790/0 conn(0x7f0ae006d390 legacy=0x7f0ae0135900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0ae5efb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ae006eba0 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310084553 0 0) 0x7f0ae006ee10 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 10 monclient(hunting): my global_id is 244418 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.086+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae00a24c0 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2749464648 0 0) 0x7f0ae00a24c0 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae006ee10 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0ad0004c80 con 0x7f0ae002c6b0 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 10 monclient: dump: 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3838401816 0 0) 0x7f0ae006ee10 con 0x7f0ae002c6b0 2024-09-16T06:52:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0ae0099500 con 0x7f0ae002c6b0 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 3666800611) 0x7f0ad0005070 con 0x7f0ae002c6b0 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6790/0 conn(0x7f0ae002c6b0 legacy=0x7f0ae0136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0add7fa700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6790/0 conn(0x7f0ae002c6b0 legacy=0x7f0ae0136010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:52:34.097 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.098 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:34.100 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:34.101 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.102 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.103 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.104 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:34.105 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.107 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.108 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:34.109 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:34.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0acc036400 legacy=0x7f0acc0388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 10 monclient: shutdown 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 >> v1:172.21.15.76:6789/0 conn(0x7f0ae0139500 legacy=0x7f0ae01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae56fa700 1 -- 172.21.15.76:0/2562059214 reap_dead start 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae56fa700 1 -- 172.21.15.76:0/2562059214 reap_dead start 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 shutdown_connections 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 >> 172.21.15.76:0/2562059214 conn(0x7f0ae00a0030 msgr2=0x7f0ae013aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 shutdown_connections 2024-09-16T06:52:34.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.090+0000 7f0ae5efb700 1 -- 172.21.15.76:0/2562059214 wait complete. 2024-09-16T06:52:34.111 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:34.111 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:52:34.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.226+0000 7fbe00a0a700 10 monclient: get_monmap_and_config 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.226+0000 7fbe00a0a700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.226+0000 7fbe00a0a700 10 monclient: monmap: 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:34.231338+0000 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:34.231338+0000 2024-09-16T06:52:34.226 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.227 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.227 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.227 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:34.227 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:34.227 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 Processor -- start 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 -- start start 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient: init 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc06c280 con 0x7fbdfc132330 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc06bfa0 con 0x7fbdfc06d480 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09ace0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient(hunting): authenticate will time out at 1701.365397s 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfb7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbdfc132330 0x7fbdfc132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55124/0 (socket says 172.21.15.76:55124) 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfb7fe700 1 -- 172.21.15.76:0/1211452520 learned_addr learned my addr 172.21.15.76:0/1211452520 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373240500 0 0) 0x7fbdfc09ace0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): my global_id is 244421 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde80035d0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993421051 0 0) 0x7fbdfc06c280 con 0x7fbdfc132330 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): my global_id is 264201 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc09ace0 con 0x7fbdfc132330 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193101649 0 0) 0x7fbdfc06bfa0 con 0x7fbdfc06d480 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient(hunting): my global_id is 274111 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc06c280 con 0x7fbdfc06d480 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701950225 0 0) 0x7fbde80035d0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 >> v1:172.21.15.189:6789/0 conn(0x7fbdfc06d480 legacy=0x7fbdfc13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 >> v1:172.21.15.76:6789/0 conn(0x7fbdfc132330 legacy=0x7fbdfc132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 monclient: found mon.noname-b 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc09ae70 con 0x7fbdfc06cf50 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 5 monclient: authenticate success, global_id 244421 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001990 con 0x7fbdfc06cf50 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: dump: 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf4001c70 con 0x7fbdfc06cf50 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 4 set_mon_vals no callback set 2024-09-16T06:52:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 1 -- 172.21.15.76:0/1211452520 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001ea0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbdfaffd700 10 monclient: dump: 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.235 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 10 monclient: shutdown 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.230+0000 7fbe00a0a700 1 -- 172.21.15.76:0/1211452520 >> v1:172.21.15.76:6790/0 conn(0x7fbdfc06cf50 legacy=0x7fbdfc13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- 172.21.15.76:0/1211452520 shutdown_connections 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- 172.21.15.76:0/1211452520 >> 172.21.15.76:0/1211452520 conn(0x7fbdfc0a41c0 msgr2=0x7fbdfc0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- 172.21.15.76:0/1211452520 shutdown_connections 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- 172.21.15.76:0/1211452520 wait complete. 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient: monmap: 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 Processor -- start 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- start start 2024-09-16T06:52:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient: init 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-b con 0x7fbdfc06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-a con 0x7fbdfc06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient(hunting): picked mon.noname-c con 0x7fbdfc132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc12dfc0 con 0x7fbdfc06d480 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc0a1a10 con 0x7fbdfc06cf50 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdfc09e8b0 con 0x7fbdfc132330 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbe00a0a700 10 monclient(hunting): authenticate will time out at 1701.371757s 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbdf2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbdfc06d480 0x7fbdfc1ced60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55140/0 (socket says 172.21.15.76:55140) 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.234+0000 7fbdf2ffd700 1 -- 172.21.15.76:0/2166249021 learned_addr learned my addr 172.21.15.76:0/2166249021 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169407136 0 0) 0x7fbdfc12dfc0 con 0x7fbdfc06d480 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): my global_id is 264204 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc0035d0 con 0x7fbdfc06d480 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509264382 0 0) 0x7fbdfc09e8b0 con 0x7fbdfc132330 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): my global_id is 244424 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc12dfc0 con 0x7fbdfc132330 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440811078 0 0) 0x7fbdfc0a1a10 con 0x7fbdfc06cf50 2024-09-16T06:52:34.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): my global_id is 274114 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc09e8b0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638449560 0 0) 0x7fbddc0035d0 con 0x7fbdfc06d480 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdfc0a1a10 con 0x7fbdfc06d480 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbde4001990 con 0x7fbdfc06d480 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395625180 0 0) 0x7fbdfc12dfc0 con 0x7fbdfc132330 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbddc0035d0 con 0x7fbdfc132330 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdf4001380 con 0x7fbdfc132330 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556793702 0 0) 0x7fbdfc09e8b0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdfc12dfc0 con 0x7fbdfc06cf50 2024-09-16T06:52:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdec001830 con 0x7fbdfc06cf50 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3233034035 0 0) 0x7fbdfc0a1a10 con 0x7fbdfc06d480 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.76:6790/0 conn(0x7fbdfc132330 legacy=0x7fbdfc1d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6789/0 conn(0x7fbdfc06cf50 legacy=0x7fbdfc1307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 monclient: found mon.a 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc12da00 con 0x7fbdfc06d480 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 5 monclient: authenticate success, global_id 264204 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 10 monclient: _renew_subs 2024-09-16T06:52:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc12dd40 con 0x7fbdfc06d480 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 10 monclient: _renew_subs 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc130ef0 con 0x7fbdfc06d480 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde4001d40 con 0x7fbdfc06d480 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbde4003b10 con 0x7fbdfc06d480 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdfa7fc700 4 set_mon_vals no callback set 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient: dump: 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbde0006590 con 0x7fbde0002830 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbde4003cf0 con 0x7fbdfc06d480 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fbde4043e10 con 0x7fbdfc06d480 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdfbfff700 10 monclient: get_auth_request con 0x7fbddc036500 auth_method 0 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865290172 0 0) 0x7fbde0006590 con 0x7fbde0002830 2024-09-16T06:52:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 10 monclient(hunting): my global_id is 274117 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.238+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc0a1a10 con 0x7fbde0002830 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 604838519 0 0) 0x7fbdfc0a1a10 con 0x7fbde0002830 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde0006590 con 0x7fbde0002830 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdec001d50 con 0x7fbde0002830 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 10 monclient: dump: 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.254 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2042373866 0 0) 0x7fbde0006590 con 0x7fbde0002830 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde000a190 con 0x7fbde0002830 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fbdec004c60 con 0x7fbde0002830 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:34.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6789/0 conn(0x7fbde0002830 legacy=0x7fbde0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.242+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6789/0 conn(0x7fbde0002830 legacy=0x7fbde0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.250+0000 7fbe00a0a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.250+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbde0009e30 con 0x7fbde0009150 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394035157 0 0) 0x7fbde0009e30 con 0x7fbde0009150 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient(hunting): my global_id is 274120 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde0006590 con 0x7fbde0009150 2024-09-16T06:52:34.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 442717414 0 0) 0x7fbde0006590 con 0x7fbde0009150 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde0009e30 con 0x7fbde0009150 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbdec0051f0 con 0x7fbde0009150 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient: dump: 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.258 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1482174497 0 0) 0x7fbde0009e30 con 0x7fbde0009150 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbde000c340 con 0x7fbde0009150 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2143464299) 0x7fbdec0065c0 con 0x7fbde0009150 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6789/0 conn(0x7fbde0009150 legacy=0x7fbde00095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdf8ff9700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6789/0 conn(0x7fbde0009150 legacy=0x7fbde00095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:34.259 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:34.260 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.261 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:34.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:34.264 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.267 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:34.268 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:34.269 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:34.270 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:34.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbddc036500 legacy=0x7fbddc0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 10 monclient: shutdown 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 >> v1:172.21.15.76:6789/0 conn(0x7fbdfc06d480 legacy=0x7fbdfc1ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdfbfff700 1 -- 172.21.15.76:0/2166249021 reap_dead start 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbdfbfff700 1 -- 172.21.15.76:0/2166249021 reap_dead start 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 shutdown_connections 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 >> 172.21.15.76:0/2166249021 conn(0x7fbdfc0a41c0 msgr2=0x7fbdfc129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 shutdown_connections 2024-09-16T06:52:34.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.254+0000 7fbe00a0a700 1 -- 172.21.15.76:0/2166249021 wait complete. 2024-09-16T06:52:34.273 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:34.273 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:52:34.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:52:34.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.390+0000 7fc787fff700 10 monclient: get_monmap_and_config 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.390+0000 7fc787fff700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.390+0000 7fc787fff700 10 monclient: monmap: 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:34.395818+0000 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:34.395818+0000 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:34.391 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 Processor -- start 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- start start 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient: init 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc788132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc78806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc78806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78806c280 con 0x7fc788132330 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78806bfa0 con 0x7fc78806cf50 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78809ace0 con 0x7fc78806d480 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient(hunting): authenticate will time out at 1701.529329s 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc786ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc788132330 0x7fc788132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55148/0 (socket says 172.21.15.76:55148) 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc786ffd700 1 -- 172.21.15.76:0/1090815758 learned_addr learned my addr 172.21.15.76:0/1090815758 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc7867fc700 1 --1- 172.21.15.76:0/1090815758 >> v1:172.21.15.76:6790/0 conn(0x7fc78806d480 0x7fc78813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43616/0 (socket says 172.21.15.76:43616) 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193165340 0 0) 0x7fc78806c280 con 0x7fc788132330 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): my global_id is 264207 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7700035d0 con 0x7fc788132330 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335833759 0 0) 0x7fc78809ace0 con 0x7fc78806d480 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient(hunting): my global_id is 244427 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78806c280 con 0x7fc78806d480 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778879167 0 0) 0x7fc78806c280 con 0x7fc78806d480 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 >> v1:172.21.15.189:6789/0 conn(0x7fc78806cf50 legacy=0x7fc78813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 >> v1:172.21.15.76:6789/0 conn(0x7fc788132330 legacy=0x7fc788132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 monclient: found mon.noname-b 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78809ae70 con 0x7fc78806d480 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc780001990 con 0x7fc78806d480 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: dump: 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 5 monclient: authenticate success, global_id 244427 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc780001c30 con 0x7fc78806d480 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 1 -- 172.21.15.76:0/1090815758 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc780001e20 con 0x7fc78806d480 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 4 set_mon_vals no callback set 2024-09-16T06:52:34.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc785ffb700 10 monclient: dump: 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 10 monclient: shutdown 2024-09-16T06:52:34.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- 172.21.15.76:0/1090815758 >> v1:172.21.15.76:6790/0 conn(0x7fc78806d480 legacy=0x7fc78813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- 172.21.15.76:0/1090815758 shutdown_connections 2024-09-16T06:52:34.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- 172.21.15.76:0/1090815758 >> 172.21.15.76:0/1090815758 conn(0x7fc7880a41c0 msgr2=0x7fc7880a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- 172.21.15.76:0/1090815758 shutdown_connections 2024-09-16T06:52:34.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.394+0000 7fc787fff700 1 -- 172.21.15.76:0/1090815758 wait complete. 2024-09-16T06:52:34.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient: build_initial_monmap 2024-09-16T06:52:34.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient: monmap: 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:34.405 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 1 Processor -- start 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 1 -- start start 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient: init 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc78806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc78806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc788132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc788131240 con 0x7fc788132330 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78809e8b0 con 0x7fc78806d480 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78812d870 con 0x7fc78806cf50 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:34.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc786ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc78806cf50 0x7fc78812b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43626/0 (socket says 172.21.15.76:43626) 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc787fff700 10 monclient(hunting): authenticate will time out at 1701.535065s 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc786ffd700 1 -- 172.21.15.76:0/2173668846 learned_addr learned my addr 172.21.15.76:0/2173668846 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548166735 0 0) 0x7fc788131240 con 0x7fc788132330 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): my global_id is 264210 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7740035d0 con 0x7fc788132330 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523618142 0 0) 0x7fc78812d870 con 0x7fc78806cf50 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): my global_id is 244430 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc788131240 con 0x7fc78806cf50 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340026663 0 0) 0x7fc7740035d0 con 0x7fc788132330 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc78812d870 con 0x7fc788132330 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc778001750 con 0x7fc788132330 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.398+0000 7fc76f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): dump: 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.408 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034725671 0 0) 0x7fc78809e8b0 con 0x7fc78806d480 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): my global_id is 274126 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7740035d0 con 0x7fc78806d480 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 240912262 0 0) 0x7fc788131240 con 0x7fc78806cf50 2024-09-16T06:52:34.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc78809e8b0 con 0x7fc78806cf50 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc77c001840 con 0x7fc78806cf50 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient(hunting): dump: 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.410 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2997286854 0 0) 0x7fc78812d870 con 0x7fc788132330 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 >> v1:172.21.15.76:6790/0 conn(0x7fc78806cf50 legacy=0x7fc78812b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 >> v1:172.21.15.189:6789/0 conn(0x7fc78806d480 legacy=0x7fc78812be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 monclient: found mon.a 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78812db50 con 0x7fc788132330 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 5 monclient: authenticate success, global_id 264210 2024-09-16T06:52:34.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _renew_subs 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78812d570 con 0x7fc788132330 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _renew_subs 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78809d660 con 0x7fc788132330 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7780036a0 con 0x7fc788132330 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc778003a30 con 0x7fc788132330 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc7857fa700 4 set_mon_vals no callback set 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 10 monclient: dump: 2024-09-16T06:52:34.412 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc778003c10 con 0x7fc788132330 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc768005100 con 0x7fc788132330 2024-09-16T06:52:34.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc778042a90 con 0x7fc788132330 2024-09-16T06:52:34.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.402+0000 7fc786ffd700 10 monclient: get_auth_request con 0x7fc7740364b0 auth_method 0 2024-09-16T06:52:34.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.410+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc7780133f0 con 0x7fc788132330 2024-09-16T06:52:34.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.410+0000 7fc76f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:34.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.410+0000 7fc76f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.654+0000 7fc787fff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:52:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.654+0000 7fc787fff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:52:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.654+0000 7fc787fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.654+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc768005bc0 con 0x7fc788132330 2024-09-16T06:52:34.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc76f7fe700 1 -- 172.21.15.76:0/2173668846 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fc77801b9e0 con 0x7fc788132330 2024-09-16T06:52:34.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc76f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:52:34.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc76f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=4251684675,config=1097430493,health=1083522342,kv=345781561,logm=2783886737}) 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=3081673291,config=1097430494,health=2949768714,kv=2418047910,logm=3332195046}) 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=4251684675,config=1097430493,health=1083522342,kv=345781561,logm=2783886737}) 2024-09-16T06:52:34.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=10,kv=13,logm=50} crc {auth=2932694181,config=1097430493,health=231875279,kv=1001081567,logm=2877716681}) 2024-09-16T06:52:34.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc7740364b0 legacy=0x7fc774038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 10 monclient: shutdown 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 >> v1:172.21.15.76:6789/0 conn(0x7fc788132330 legacy=0x7fc7881d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 shutdown_connections 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 >> 172.21.15.76:0/2173668846 conn(0x7fc7880a41c0 msgr2=0x7fc788129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 shutdown_connections 2024-09-16T06:52:34.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:34.658+0000 7fc787fff700 1 -- 172.21.15.76:0/2173668846 wait complete. 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899938924}) 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331854712}) 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899938924}) 2024-09-16T06:52:34.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=235955186}) 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60515826}) 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=956687962}) 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60515826}) 2024-09-16T06:52:34.663 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728025858}) 2024-09-16T06:52:34.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737017503}) 2024-09-16T06:52:34.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484903470}) 2024-09-16T06:52:34.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737017503}) 2024-09-16T06:52:34.666 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590231171}) 2024-09-16T06:52:34.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408646087}) 2024-09-16T06:52:34.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067177310}) 2024-09-16T06:52:34.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=408646087}) 2024-09-16T06:52:34.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283222545}) 2024-09-16T06:52:34.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830361453}) 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074334571}) 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830361453}) 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757146712}) 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243723947}) 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625997800}) 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243723947}) 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475941561}) 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416236644}) 2024-09-16T06:52:34.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066954667}) 2024-09-16T06:52:34.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416236644}) 2024-09-16T06:52:34.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094281390}) 2024-09-16T06:52:34.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035955773}) 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153538335}) 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035955773}) 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006389997}) 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915123540}) 2024-09-16T06:52:34.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228957707}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915123540}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2553187048}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187956042}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761637334}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187956042}) 2024-09-16T06:52:34.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998602476}) 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245657825}) 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314605398}) 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245657825}) 2024-09-16T06:52:34.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943570583}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608915834}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034094439}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608915834}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161165866}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192068689}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247881460}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192068689}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2185693}) 2024-09-16T06:52:34.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583405424}) 2024-09-16T06:52:34.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351360885}) 2024-09-16T06:52:34.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583405424}) 2024-09-16T06:52:34.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109091150}) 2024-09-16T06:52:34.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150135771}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013679723}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150135771}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1760313515}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065470275}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=105192159}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065470275}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618112596}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537275294}) 2024-09-16T06:52:34.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769275235}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537275294}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728223937}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926218483}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095988974}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926218483}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3383539037}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799253801}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987846594}) 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799253801}) 2024-09-16T06:52:34.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1671361869}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416738679}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275877218}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416738679}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665817441}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102399137}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432871415}) 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102399137}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3129303183}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824632687}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789504064}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824632687}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3503051133}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020554089}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672005968}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020554089}) 2024-09-16T06:52:34.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075650209}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691271379}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679721571}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691271379}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702554665}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625133449}) 2024-09-16T06:52:34.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455420176}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625133449}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4032323451}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740497320}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478293820}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740497320}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2444990248}) 2024-09-16T06:52:34.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137158258}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170581413}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137158258}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202295488}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841095760}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679241482}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841095760}) 2024-09-16T06:52:34.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407149468}) 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881904403}) 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916142963}) 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881904403}) 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=659885781}) 2024-09-16T06:52:34.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496972478}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253928393}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496972478}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919121818}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124847820}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1883181248}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124847820}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843599407}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141932723}) 2024-09-16T06:52:34.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715320500}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141932723}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831476089}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854481743}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282368701}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854481743}) 2024-09-16T06:52:34.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689883478}) 2024-09-16T06:52:34.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264075055}) 2024-09-16T06:52:34.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350128785}) 2024-09-16T06:52:34.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264075055}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536458895}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555877228}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457398863}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555877228}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786062250}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192548967}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544552658}) 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192548967}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184614748}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882507945}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214295216}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882507945}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969259468}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=405436252}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873517060}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=405436252}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974283871}) 2024-09-16T06:52:34.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143378671}) 2024-09-16T06:52:34.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552656707}) 2024-09-16T06:52:34.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143378671}) 2024-09-16T06:52:34.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529427827}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684251457}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724273584}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684251457}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157923392}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602919488}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090922346}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602919488}) 2024-09-16T06:52:34.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296822638}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155476243}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388611857}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155476243}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443258385}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174746983}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=304934507}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174746983}) 2024-09-16T06:52:34.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1588363423}) 2024-09-16T06:52:34.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111361785}) 2024-09-16T06:52:34.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632438467}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111361785}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059802879}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179396929}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395052605}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179396929}) 2024-09-16T06:52:34.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387851457}) 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403641296}) 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843599805}) 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403641296}) 2024-09-16T06:52:34.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3033178196}) 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1141159881}) 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2814477387}) 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1141159881}) 2024-09-16T06:52:34.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909443559}) 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495500987}) 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879890837}) 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495500987}) 2024-09-16T06:52:34.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708647793}) 2024-09-16T06:52:34.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581774393}) 2024-09-16T06:52:34.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869820266}) 2024-09-16T06:52:34.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581774393}) 2024-09-16T06:52:34.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3458030969}) 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633666617}) 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615945076}) 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633666617}) 2024-09-16T06:52:34.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466475313}) 2024-09-16T06:52:34.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119653020}) 2024-09-16T06:52:34.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670566253}) 2024-09-16T06:52:34.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119653020}) 2024-09-16T06:52:34.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3791762562}) 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212276370}) 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129637230}) 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212276370}) 2024-09-16T06:52:34.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3342757490}) 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208367229}) 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709880436}) 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208367229}) 2024-09-16T06:52:34.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165158765}) 2024-09-16T06:52:34.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482943419}) 2024-09-16T06:52:34.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763971885}) 2024-09-16T06:52:34.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482943419}) 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767295402}) 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220089768}) 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90107554}) 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220089768}) 2024-09-16T06:52:34.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2719828167}) 2024-09-16T06:52:34.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684392129}) 2024-09-16T06:52:34.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903241447}) 2024-09-16T06:52:34.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684392129}) 2024-09-16T06:52:34.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705230591}) 2024-09-16T06:52:34.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3333489201}) 2024-09-16T06:52:34.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209928352}) 2024-09-16T06:52:34.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3333489201}) 2024-09-16T06:52:34.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119169860}) 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408369103}) 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018168304}) 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408369103}) 2024-09-16T06:52:34.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931421907}) 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973223852}) 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580335985}) 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973223852}) 2024-09-16T06:52:34.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167309082}) 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186499885}) 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596980095}) 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186499885}) 2024-09-16T06:52:34.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915544985}) 2024-09-16T06:52:34.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21007293}) 2024-09-16T06:52:34.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706165769}) 2024-09-16T06:52:34.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21007293}) 2024-09-16T06:52:34.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3452795291}) 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277832402}) 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077813584}) 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277832402}) 2024-09-16T06:52:34.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602856097}) 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084699695}) 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455781654}) 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084699695}) 2024-09-16T06:52:34.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211121960}) 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489517875}) 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727101368}) 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489517875}) 2024-09-16T06:52:34.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2933060391}) 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912016335}) 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841892254}) 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912016335}) 2024-09-16T06:52:34.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262640646}) 2024-09-16T06:52:34.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359226544}) 2024-09-16T06:52:34.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757882698}) 2024-09-16T06:52:34.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359226544}) 2024-09-16T06:52:34.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960278368}) 2024-09-16T06:52:34.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942883715}) 2024-09-16T06:52:34.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323440797}) 2024-09-16T06:52:34.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942883715}) 2024-09-16T06:52:34.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118083726}) 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712204803}) 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115742505}) 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712204803}) 2024-09-16T06:52:34.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1876683074}) 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010691614}) 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122267704}) 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010691614}) 2024-09-16T06:52:34.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175540160}) 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053241889}) 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1794055918}) 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053241889}) 2024-09-16T06:52:34.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436721541}) 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827257584}) 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453278212}) 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827257584}) 2024-09-16T06:52:34.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981247385}) 2024-09-16T06:52:34.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442564733}) 2024-09-16T06:52:34.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242058483}) 2024-09-16T06:52:34.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442564733}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4000352532}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555669828}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119133092}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555669828}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875203047}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519339781}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804298018}) 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519339781}) 2024-09-16T06:52:34.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357339094}) 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3403680137,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2487121927}) 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=3667430318,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2223785069}) 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3403680137,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2487121927}) 2024-09-16T06:52:34.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=1030265762,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4192005668}) 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {mgrstat=3576340404,mon_config_key=1752232527,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1777273494}) 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=55} crc {mgrstat=3576340404,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=894461268}) 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {mgrstat=3576340404,mon_config_key=1752232527,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1777273494}) 2024-09-16T06:52:34.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:52:34.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mgrstat=3576340404,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=613831339}) 2024-09-16T06:52:35.673 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:52:35.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: get_monmap_and_config 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: build_initial_monmap 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: monmap: 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:35.796204+0000 2024-09-16T06:52:35.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:35.796204+0000 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:35.792 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 Processor -- start 2024-09-16T06:52:35.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- start start 2024-09-16T06:52:35.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: init 2024-09-16T06:52:35.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-b con 0x7ffa780973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-a con 0x7ffa78097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-c con 0x7ffa781400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa78099350 con 0x7ffa78097eb0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7809f690 con 0x7ffa781400d0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa78099bf0 con 0x7ffa780973c0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient(hunting): authenticate will time out at 1702.929494s 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7e3b7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ffa780973c0 0x7ffa780977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43628/0 (socket says 172.21.15.76:43628) 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7e3b7700 1 -- 172.21.15.76:0/2537858412 learned_addr learned my addr 172.21.15.76:0/2537858412 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747278096 0 0) 0x7ffa78099350 con 0x7ffa78097eb0 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): my global_id is 264213 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa640035d0 con 0x7ffa78097eb0 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505853650 0 0) 0x7ffa78099bf0 con 0x7ffa780973c0 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient(hunting): my global_id is 244433 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78099350 con 0x7ffa780973c0 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882650194 0 0) 0x7ffa640035d0 con 0x7ffa78097eb0 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 >> v1:172.21.15.76:6790/0 conn(0x7ffa780973c0 legacy=0x7ffa780977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 >> v1:172.21.15.189:6789/0 conn(0x7ffa781400d0 legacy=0x7ffa781424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: _finish_hunting 0 2024-09-16T06:52:35.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 monclient: found mon.noname-a 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa78098360 con 0x7ffa78097eb0 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: _finish_auth 0 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: _check_auth_tickets 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa74001610 con 0x7ffa78097eb0 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: dump: 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.803 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 5 monclient: authenticate success, global_id 264213 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa740018b0 con 0x7ffa78097eb0 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 4 set_mon_vals no callback set 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 1 -- 172.21.15.76:0/2537858412 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa74001c40 con 0x7ffa78097eb0 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7d3b5700 10 monclient: dump: 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 10 monclient: shutdown 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/2537858412 >> v1:172.21.15.76:6789/0 conn(0x7ffa78097eb0 legacy=0x7ffa7813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/2537858412 shutdown_connections 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/2537858412 >> 172.21.15.76:0/2537858412 conn(0x7ffa780a0030 msgr2=0x7ffa780a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/2537858412 shutdown_connections 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.794+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/2537858412 wait complete. 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: build_initial_monmap 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: monmap: 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:35.806 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 Processor -- start 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- start start 2024-09-16T06:52:35.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: init 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-b con 0x7ffa780973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-a con 0x7ffa78097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient(hunting): picked mon.noname-c con 0x7ffa781400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa78072660 con 0x7ffa78097eb0 2024-09-16T06:52:35.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7813ebe0 con 0x7ffa780973c0 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7813fa70 con 0x7ffa781400d0 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient(hunting): authenticate will time out at 1702.933892s 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7ebb8700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ffa781400d0 0x7ffa781d6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43632/0 (socket says 172.21.15.76:43632) 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7ebb8700 1 -- 172.21.15.76:0/133603754 learned_addr learned my addr 172.21.15.76:0/133603754 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134412464 0 0) 0x7ffa78072660 con 0x7ffa78097eb0 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): my global_id is 264216 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c0035d0 con 0x7ffa78097eb0 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171441876 0 0) 0x7ffa7813fa70 con 0x7ffa781400d0 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): my global_id is 244436 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78072660 con 0x7ffa781400d0 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781485086 0 0) 0x7ffa7813ebe0 con 0x7ffa780973c0 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): my global_id is 274132 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7813fa70 con 0x7ffa780973c0 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374366298 0 0) 0x7ffa78072660 con 0x7ffa781400d0 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa7813ebe0 con 0x7ffa781400d0 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa70001c00 con 0x7ffa781400d0 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.811 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.812 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2377631801 0 0) 0x7ffa5c0035d0 con 0x7ffa78097eb0 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa78072660 con 0x7ffa78097eb0 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa74001bc0 con 0x7ffa78097eb0 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.814 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1370016833 0 0) 0x7ffa7813fa70 con 0x7ffa780973c0 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c0035d0 con 0x7ffa780973c0 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa680018d0 con 0x7ffa780973c0 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.815 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239110569 0 0) 0x7ffa7813ebe0 con 0x7ffa781400d0 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 >> v1:172.21.15.189:6789/0 conn(0x7ffa780973c0 legacy=0x7ffa781ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 >> v1:172.21.15.76:6789/0 conn(0x7ffa78097eb0 legacy=0x7ffa781d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 monclient: found mon.c 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7809a540 con 0x7ffa781400d0 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: _finish_auth 0 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:35.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 5 monclient: authenticate success, global_id 244436 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _renew_subs 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7806c6b0 con 0x7ffa781400d0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _renew_subs 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7813e860 con 0x7ffa781400d0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa70001fd0 con 0x7ffa781400d0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffa70003e90 con 0x7ffa781400d0 2024-09-16T06:52:35.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa7cbb4700 4 set_mon_vals no callback set 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.798+0000 7ffa6effd700 10 monclient: dump: 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:35.819 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ffa70003ff0 con 0x7ffa781400d0 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa7f3b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa7f3b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa7f3b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa60005100 con 0x7ffa781400d0 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ffa70044020 con 0x7ffa781400d0 2024-09-16T06:52:35.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.802+0000 7ffa7e3b7700 10 monclient: get_auth_request con 0x7ffa5c036510 auth_method 0 2024-09-16T06:52:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.810+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffa70014a60 con 0x7ffa781400d0 2024-09-16T06:52:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.810+0000 7ffa6effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:35.810+0000 7ffa6effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=110 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS= 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS= 2024-09-16T06:52:35.822 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.foo 2024-09-16T06:52:35.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: get_monmap_and_config 2024-09-16T06:52:35.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: build_initial_monmap 2024-09-16T06:52:35.922 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: monmap: 2024-09-16T06:52:35.922 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:52:35.922 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:52:35.922191+0000 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:52:35.922191+0000 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:35.923 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 1 Processor -- start 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 1 -- start start 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: init 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa9ac074af0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa9ac0a1740 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:35.933 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa9ac13b1a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac131e30 con 0x7fa9ac074af0 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac074ed0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac099ed0 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9ab59e700 10 monclient(hunting): authenticate will time out at 1708.434908s 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a9d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9ac0a1740 0x7fa9ac0a1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:45554/0 (socket says 172.21.15.189:45554) 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a9d9b700 1 -- 172.21.15.189:0/1288609703 learned_addr learned my addr 172.21.15.189:0/1288609703 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783716529 0 0) 0x7fa9ac074ed0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): my global_id is 274135 2024-09-16T06:52:35.934 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9980035d0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601424044 0 0) 0x7fa9ac099ed0 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): my global_id is 244439 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac074ed0 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488779387 0 0) 0x7fa9ac131e30 con 0x7fa9ac074af0 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 10 monclient(hunting): my global_id is 264219 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.918+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac099ed0 con 0x7fa9ac074af0 2024-09-16T06:52:35.935 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001004078 0 0) 0x7fa9980035d0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 >> v1:172.21.15.76:6790/0 conn(0x7fa9ac13b1a0 legacy=0x7fa9ac13d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 >> v1:172.21.15.76:6789/0 conn(0x7fa9ac074af0 legacy=0x7fa9ac132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: _finish_hunting 0 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 monclient: found mon.noname-c 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac066dd0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: _finish_auth 0 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: _check_auth_tickets 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:35.936 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 5 monclient: authenticate success, global_id 274135 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa99c001610 con 0x7fa9ac0a1740 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: dump: 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.937 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa99c001880 con 0x7fa9ac0a1740 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 4 set_mon_vals no callback set 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 1 -- 172.21.15.189:0/1288609703 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa99c001c50 con 0x7fa9ac0a1740 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9a959a700 10 monclient: dump: 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.938 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: shutdown 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- 172.21.15.189:0/1288609703 >> v1:172.21.15.189:6789/0 conn(0x7fa9ac0a1740 legacy=0x7fa9ac0a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- 172.21.15.189:0/1288609703 shutdown_connections 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- 172.21.15.189:0/1288609703 >> 172.21.15.189:0/1288609703 conn(0x7fa9ac073f20 msgr2=0x7fa9ac074330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- 172.21.15.189:0/1288609703 shutdown_connections 2024-09-16T06:52:35.939 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- 172.21.15.189:0/1288609703 wait complete. 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: build_initial_monmap 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: monmap: 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.940 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 Processor -- start 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- start start 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: init 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa9ac074af0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa9ac0a1740 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa9ac13b1a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0a0250 con 0x7fa9ac0a1740 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac139970 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac132c30 con 0x7fa9ac074af0 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.922+0000 7fa9ab59e700 10 monclient(hunting): authenticate will time out at 1708.439967s 2024-09-16T06:52:35.941 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9aa59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa9ac074af0 0x7fa9ac09dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:50422/0 (socket says 172.21.15.189:50422) 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9aa59c700 1 -- 172.21.15.189:0/564590325 learned_addr learned my addr 172.21.15.189:0/564590325 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881298156 0 0) 0x7fa9ac139970 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): my global_id is 274138 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa98c0035d0 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123477896 0 0) 0x7fa9ac132c30 con 0x7fa9ac074af0 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): my global_id is 244442 2024-09-16T06:52:35.942 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac139970 con 0x7fa9ac074af0 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237711694 0 0) 0x7fa9ac0a0250 con 0x7fa9ac0a1740 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): my global_id is 264222 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac132c30 con 0x7fa9ac0a1740 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932098558 0 0) 0x7fa9ac139970 con 0x7fa9ac074af0 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9ac0a0250 con 0x7fa9ac074af0 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa994001990 con 0x7fa9ac074af0 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.943 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871329254 0 0) 0x7fa98c0035d0 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9ac139970 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa9a4001410 con 0x7fa9ac13b1a0 2024-09-16T06:52:35.944 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882791183 0 0) 0x7fa9ac132c30 con 0x7fa9ac0a1740 2024-09-16T06:52:35.945 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa98c0035d0 con 0x7fa9ac0a1740 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa99c001b90 con 0x7fa9ac0a1740 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.946 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3271477869 0 0) 0x7fa9ac0a0250 con 0x7fa9ac074af0 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 >> v1:172.21.15.189:6789/0 conn(0x7fa9ac13b1a0 legacy=0x7fa9ac1d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 >> v1:172.21.15.76:6789/0 conn(0x7fa9ac0a1740 legacy=0x7fa9ac09e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 monclient: found mon.c 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac09e8f0 con 0x7fa9ac074af0 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:35.947 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 5 monclient: authenticate success, global_id 244442 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _renew_subs 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac09f610 con 0x7fa9ac074af0 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _renew_subs 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac0a13b0 con 0x7fa9ac074af0 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa994001c70 con 0x7fa9ac074af0 2024-09-16T06:52:35.948 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa994003af0 con 0x7fa9ac074af0 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a3fff700 4 set_mon_vals no callback set 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 10 monclient: dump: 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:35.949 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa990005100 con 0x7fa9ac074af0 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa994003c50 con 0x7fa9ac074af0 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.926+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa994043e70 con 0x7fa9ac074af0 2024-09-16T06:52:35.950 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.930+0000 7fa9a9d9b700 10 monclient: get_auth_request con 0x7fa98c036500 auth_method 0 2024-09-16T06:52:35.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.934+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa9940147b0 con 0x7fa9ac074af0 2024-09-16T06:52:35.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.938+0000 7fa9a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:35.951 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:35.938+0000 7fa9a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:36.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:36.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:36.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffa60005930 con 0x7ffa781400d0 2024-09-16T06:52:36.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa6effd700 1 -- 172.21.15.76:0/133603754 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3528775363) 0x7ffa70012ed0 con 0x7ffa781400d0 2024-09-16T06:52:36.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa6effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:36.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa6effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.036 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 >> v1:172.21.15.189:6812/3098810674 conn(0x7ffa5c036510 legacy=0x7ffa5c0389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 10 monclient: shutdown 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 >> v1:172.21.15.76:6790/0 conn(0x7ffa781400d0 legacy=0x7ffa781d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 shutdown_connections 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 >> 172.21.15.76:0/133603754 conn(0x7ffa780a0030 msgr2=0x7ffa7809c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 shutdown_connections 2024-09-16T06:52:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.038+0000 7ffa7f3b9700 1 -- 172.21.15.76:0/133603754 wait complete. 2024-09-16T06:52:36.050 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:52:36.050 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:52:36.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:52:36.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:52:36.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.158+0000 7f1584e58700 10 monclient: get_monmap_and_config 2024-09-16T06:52:36.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.158+0000 7f1584e58700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.158+0000 7f1584e58700 10 monclient: monmap: 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:36.163808+0000 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:36.163808+0000 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:36.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 Processor -- start 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- start start 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: init 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-c con 0x7f1580132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-a con 0x7f158006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-b con 0x7f158006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158006c280 con 0x7f158006d480 2024-09-16T06:52:36.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158006bfa0 con 0x7f1580132330 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158009ace0 con 0x7f158006cf50 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient(hunting): authenticate will time out at 1703.297165s 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157effd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f158006d480 0x7f158013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55182/0 (socket says 172.21.15.76:55182) 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157effd700 1 -- 172.21.15.76:0/890384294 learned_addr learned my addr 172.21.15.76:0/890384294 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821845456 0 0) 0x7f158006c280 con 0x7f158006d480 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): my global_id is 264225 2024-09-16T06:52:36.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15680035d0 con 0x7f158006d480 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451244193 0 0) 0x7f158009ace0 con 0x7f158006cf50 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): my global_id is 244445 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158006c280 con 0x7f158006cf50 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016845923 0 0) 0x7f158006bfa0 con 0x7f1580132330 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient(hunting): my global_id is 274141 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158009ace0 con 0x7f1580132330 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650412956 0 0) 0x7f15680035d0 con 0x7f158006d480 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 >> v1:172.21.15.76:6790/0 conn(0x7f158006cf50 legacy=0x7f158013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 >> v1:172.21.15.189:6789/0 conn(0x7f1580132330 legacy=0x7f1580132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 monclient: found mon.noname-a 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158009ae70 con 0x7f158006d480 2024-09-16T06:52:36.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 5 monclient: authenticate success, global_id 264225 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1570001610 con 0x7f158006d480 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:36.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: dump: 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.176 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1570001830 con 0x7f158006d480 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 4 set_mon_vals no callback set 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 1 -- 172.21.15.76:0/890384294 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1570001a20 con 0x7f158006d480 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f157e7fc700 10 monclient: dump: 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.178 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.179 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.179 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: shutdown 2024-09-16T06:52:36.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- 172.21.15.76:0/890384294 >> v1:172.21.15.76:6789/0 conn(0x7f158006d480 legacy=0x7f158013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- 172.21.15.76:0/890384294 shutdown_connections 2024-09-16T06:52:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- 172.21.15.76:0/890384294 >> 172.21.15.76:0/890384294 conn(0x7f15800a41c0 msgr2=0x7f15800a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- 172.21.15.76:0/890384294 shutdown_connections 2024-09-16T06:52:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 1 -- 172.21.15.76:0/890384294 wait complete. 2024-09-16T06:52:36.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.162+0000 7f1584e58700 10 monclient: monmap: 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:36.182 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 Processor -- start 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- start start 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: init 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:52:36.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-c con 0x7f158006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-b con 0x7f158006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient(hunting): picked mon.noname-a con 0x7f1580132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158013eb40 con 0x7f1580132330 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580131340 con 0x7f158006d480 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158009e8b0 con 0x7f158006cf50 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient(hunting): authenticate will time out at 1703.300639s 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f157ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1580132330 0x7f15801363b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55198/0 (socket says 172.21.15.76:55198) 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f157ffff700 1 -- 172.21.15.76:0/2219657680 learned_addr learned my addr 172.21.15.76:0/2219657680 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f157f7fe700 1 --1- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6790/0 conn(0x7f158006cf50 0x7f1580135590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43658/0 (socket says 172.21.15.76:43658) 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868143451 0 0) 0x7f158009e8b0 con 0x7f158006cf50 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): my global_id is 244448 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c0035d0 con 0x7f158006cf50 2024-09-16T06:52:36.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566737255 0 0) 0x7f158013eb40 con 0x7f1580132330 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): my global_id is 264228 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158009e8b0 con 0x7f1580132330 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236107568 0 0) 0x7f1580131340 con 0x7f158006d480 2024-09-16T06:52:36.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): my global_id is 274144 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158013eb40 con 0x7f158006d480 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929068109 0 0) 0x7f156c0035d0 con 0x7f158006cf50 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1580131340 con 0x7f158006cf50 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15740018d0 con 0x7f158006cf50 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): dump: 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.188 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1134916114 0 0) 0x7f158013eb40 con 0x7f158006d480 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f156c0035d0 con 0x7f158006d480 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15700019a0 con 0x7f158006d480 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): dump: 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.189 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1585999569 0 0) 0x7f158009e8b0 con 0x7f1580132330 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f158013eb40 con 0x7f1580132330 2024-09-16T06:52:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1578001bf0 con 0x7f1580132330 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): dump: 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 874931586 0 0) 0x7f1580131340 con 0x7f158006cf50 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.189:6789/0 conn(0x7f158006d480 legacy=0x7f1580135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6789/0 conn(0x7f1580132330 legacy=0x7f15801363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 monclient: found mon.c 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158013ed70 con 0x7f158006cf50 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 5 monclient: authenticate success, global_id 244448 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _renew_subs 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1580139510 con 0x7f158006cf50 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _renew_subs 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1580138f50 con 0x7f158006cf50 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15740036c0 con 0x7f158006cf50 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f15740039d0 con 0x7f158006cf50 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f157dffb700 4 set_mon_vals no callback set 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient: dump: 2024-09-16T06:52:36.194 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.195 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.195 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.195 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.195 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.196 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.196 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.196 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.196 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1560006590 con 0x7f1560002830 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975753215 0 0) 0x7f1560006590 con 0x7f1560002830 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 10 monclient(hunting): my global_id is 264231 2024-09-16T06:52:36.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580131340 con 0x7f1560002830 2024-09-16T06:52:36.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.166+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1412715714 0 0) 0x7f1580131340 con 0x7f1560002830 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1560006590 con 0x7f1560002830 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1570001d60 con 0x7f1560002830 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 10 monclient: dump: 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.199 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2701727954 0 0) 0x7f1560006590 con 0x7f1560002830 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f156000a190 con 0x7f1560002830 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1574003b30 con 0x7f158006cf50 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1574043e30 con 0x7f158006cf50 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1570004f40 con 0x7f1560002830 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:36.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6789/0 conn(0x7f1560002830 legacy=0x7f1560004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6789/0 conn(0x7f1560002830 legacy=0x7f1560004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.170+0000 7f157ffff700 10 monclient: get_auth_request con 0x7f156c0365e0 auth_method 0 2024-09-16T06:52:36.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.166+0000 7fa9ab59e700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.166+0000 7fa9ab59e700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.166+0000 7fa9ab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.202 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.166+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7fa9900053b0 con 0x7fa9ac074af0 2024-09-16T06:52:36.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.170+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v35) v1 ==== 80+0+0 (unknown 3476768894 0 0) 0x7fa994012c20 con 0x7fa9ac074af0 2024-09-16T06:52:36.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.170+0000 7fa9a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.203 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.170+0000 7fa9a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.178+0000 7f1584e58700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.178+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1560009e70 con 0x7f1560009130 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645417018 0 0) 0x7f1560009e70 con 0x7f1560009130 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient(hunting): my global_id is 264234 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1560006590 con 0x7f1560009130 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3403722447 0 0) 0x7f1560006590 con 0x7f1560009130 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1560009e70 con 0x7f1560009130 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1570005520 con 0x7f1560009130 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient: dump: 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 370938500 0 0) 0x7f1560009e70 con 0x7f1560009130 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f156000c380 con 0x7f1560009130 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3865999186) 0x7f1570006930 con 0x7f1560009130 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:36.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6789/0 conn(0x7f1560009130 legacy=0x7f15600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1567fff700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6789/0 conn(0x7f1560009130 legacy=0x7f15600095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.220 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:36.220 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:36.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.189:6812/3098810674 conn(0x7f156c0365e0 legacy=0x7f156c038aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 10 monclient: shutdown 2024-09-16T06:52:36.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 >> v1:172.21.15.76:6790/0 conn(0x7f158006cf50 legacy=0x7f1580135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f157ffff700 1 -- 172.21.15.76:0/2219657680 reap_dead start 2024-09-16T06:52:36.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 shutdown_connections 2024-09-16T06:52:36.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 >> 172.21.15.76:0/2219657680 conn(0x7f15800a41c0 msgr2=0x7f15800a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 shutdown_connections 2024-09-16T06:52:36.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.182+0000 7f1584e58700 1 -- 172.21.15.76:0/2219657680 wait complete. 2024-09-16T06:52:36.222 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:36.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:52:36.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:52:36.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: monmap: 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:36.346098+0000 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:36.346098+0000 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:36.341 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9ab59e700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9ab59e700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9ab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7fa990005900 con 0x7fa9ac074af0 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9a2ffd700 1 -- 172.21.15.189:0/564590325 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v35) v1 ==== 80+0+0 (unknown 3476768894 0 0) 0x7fa9940143a0 con 0x7fa9ac074af0 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9a2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T06:52:36.350 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.342+0000 7fa9a2ffd700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 1 Processor -- start 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 1 -- start start 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: init 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f752c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f752c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f752c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c09d560 con 0x7f752c099030 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c09f140 con 0x7f752c09ed60 2024-09-16T06:52:36.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c09d790 con 0x7f752c098be0 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752b59e700 10 monclient(hunting): authenticate will time out at 1703.479387s 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f7523fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f752c098be0 0x7f752c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43674/0 (socket says 172.21.15.76:43674) 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f7523fff700 1 -- 172.21.15.76:0/1432449037 learned_addr learned my addr 172.21.15.76:0/1432449037 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.342+0000 7f752ad9d700 1 --1- 172.21.15.76:0/1432449037 >> v1:172.21.15.76:6789/0 conn(0x7f752c099030 0x7f752c13ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55228/0 (socket says 172.21.15.76:55228) 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952611992 0 0) 0x7f752c09d560 con 0x7f752c099030 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): my global_id is 264237 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75180057e0 con 0x7f752c099030 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652008060 0 0) 0x7f752c09f140 con 0x7f752c09ed60 2024-09-16T06:52:36.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): my global_id is 274147 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c09d560 con 0x7f752c09ed60 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798605688 0 0) 0x7f752c09d790 con 0x7f752c098be0 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient(hunting): my global_id is 244451 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c09f140 con 0x7f752c098be0 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761331090 0 0) 0x7f75180057e0 con 0x7f752c099030 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 >> v1:172.21.15.76:6790/0 conn(0x7f752c098be0 legacy=0x7f752c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 >> v1:172.21.15.189:6789/0 conn(0x7f752c09ed60 legacy=0x7f752c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 monclient: found mon.noname-a 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c09f620 con 0x7f752c099030 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 5 monclient: authenticate success, global_id 264237 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7524001990 con 0x7f752c099030 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:36.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: dump: 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7524001c70 con 0x7f752c099030 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 1 -- 172.21.15.76:0/1432449037 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7524002000 con 0x7f752c099030 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 4 set_mon_vals no callback set 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f7529d9b700 10 monclient: dump: 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: shutdown 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- 172.21.15.76:0/1432449037 >> v1:172.21.15.76:6789/0 conn(0x7f752c099030 legacy=0x7f752c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- 172.21.15.76:0/1432449037 shutdown_connections 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- 172.21.15.76:0/1432449037 >> 172.21.15.76:0/1432449037 conn(0x7f752c0a0030 msgr2=0x7f752c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- 172.21.15.76:0/1432449037 shutdown_connections 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- 172.21.15.76:0/1432449037 wait complete. 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: monmap: 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:36.359 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 Processor -- start 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 1 -- start start 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: init 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.346+0000 7f752b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f752c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f752c099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f752c09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0950f0 con 0x7f752c09ed60 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0a1450 con 0x7f752c099030 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c098880 con 0x7f752c098be0 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient(hunting): authenticate will time out at 1703.484115s 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752ad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f752c09ed60 0x7f752c1d23a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55234/0 (socket says 172.21.15.76:55234) 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752ad9d700 1 -- 172.21.15.76:0/1233293101 learned_addr learned my addr 172.21.15.76:0/1233293101 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409225688 0 0) 0x7f752c0950f0 con 0x7f752c09ed60 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): my global_id is 264240 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f750c0035d0 con 0x7f752c09ed60 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243309590 0 0) 0x7f752c098880 con 0x7f752c098be0 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): my global_id is 244454 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0950f0 con 0x7f752c098be0 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792321134 0 0) 0x7f752c0a1450 con 0x7f752c099030 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): my global_id is 274150 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c098880 con 0x7f752c099030 2024-09-16T06:52:36.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816660784 0 0) 0x7f750c0035d0 con 0x7f752c09ed60 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c0a1450 con 0x7f752c09ed60 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7524001950 con 0x7f752c09ed60 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963794137 0 0) 0x7f752c0950f0 con 0x7f752c098be0 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f750c0035d0 con 0x7f752c098be0 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f75140018b0 con 0x7f752c098be0 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.364 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980823008 0 0) 0x7f752c098880 con 0x7f752c099030 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c0950f0 con 0x7f752c099030 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103717598 0 0) 0x7f752c0a1450 con 0x7f752c09ed60 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6790/0 conn(0x7f752c098be0 legacy=0x7f752c096620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.189:6789/0 conn(0x7f752c099030 legacy=0x7f752c1cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 monclient: found mon.a 2024-09-16T06:52:36.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c09a540 con 0x7f752c09ed60 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 5 monclient: authenticate success, global_id 264240 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient: _renew_subs 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c093980 con 0x7f752c09ed60 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient: _renew_subs 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c096d30 con 0x7f752c09ed60 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7524005000 con 0x7f752c09ed60 2024-09-16T06:52:36.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7524005310 con 0x7f752c09ed60 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752959a700 4 set_mon_vals no callback set 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 10 monclient: dump: 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.367 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7524005470 con 0x7f752c09ed60 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f752c096f80 con 0x7f752c1c9950 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.350+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f75240443d0 con 0x7f752c09ed60 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7523fff700 10 monclient: get_auth_request con 0x7f750c036520 auth_method 0 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316518329 0 0) 0x7f752c09d560 con 0x7f752c1c9950 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient(hunting): my global_id is 244457 2024-09-16T06:52:36.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0a1450 con 0x7f752c1c9950 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1032552603 0 0) 0x7f752c0a1450 con 0x7f752c1c9950 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c09d560 con 0x7f752c1c9950 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7514001d80 con 0x7f752c1c9950 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient: dump: 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1880385936 0 0) 0x7f752c09d560 con 0x7f752c1c9950 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f752c09aaa0 con 0x7f752c1c9950 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f75140059d0 con 0x7f752c1c9950 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6790/0 conn(0x7f752c1c9950 legacy=0x7f752c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.354+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6790/0 conn(0x7f752c1c9950 legacy=0x7f752c08d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa98c036500 legacy=0x7fa98c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 10 monclient: shutdown 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 >> v1:172.21.15.76:6790/0 conn(0x7fa9ac074af0 legacy=0x7fa9ac09dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 shutdown_connections 2024-09-16T06:52:36.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 >> 172.21.15.189:0/564590325 conn(0x7fa9ac073f20 msgr2=0x7fa9ac06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 shutdown_connections 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:52:36.346+0000 7fa9ab59e700 1 -- 172.21.15.189:0/564590325 wait complete. 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:52:36.373 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:52:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f752b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f752c096f80 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851327453 0 0) 0x7f7514003c20 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient(hunting): my global_id is 244460 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c09d560 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2391764422 0 0) 0x7f752c09d560 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7514003c20 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7514004020 con 0x7f752c02c6b0 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient: dump: 2024-09-16T06:52:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4164086734 0 0) 0x7f7514003c20 con 0x7f752c02c6b0 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f752c094850 con 0x7f752c02c6b0 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2402799697) 0x7f7514005b40 con 0x7f752c02c6b0 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6790/0 conn(0x7f752c02c6b0 legacy=0x7f752c08d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.362+0000 7f7522ffd700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6790/0 conn(0x7f752c02c6b0 legacy=0x7f752c08d710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:36.388 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:52:36.388 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.189:6812/3098810674 conn(0x7f750c036520 legacy=0x7f750c0389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 10 monclient: shutdown 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 >> v1:172.21.15.76:6789/0 conn(0x7f752c09ed60 legacy=0x7f752c1d23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752ad9d700 1 -- 172.21.15.76:0/1233293101 reap_dead start 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752ad9d700 1 -- 172.21.15.76:0/1233293101 reap_dead start 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 shutdown_connections 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 >> 172.21.15.76:0/1233293101 conn(0x7f752c0a0030 msgr2=0x7f752c070790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 shutdown_connections 2024-09-16T06:52:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.366+0000 7f752b59e700 1 -- 172.21.15.76:0/1233293101 wait complete. 2024-09-16T06:52:36.391 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:36.391 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:52:36.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-16T06:52:36.462 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:52:36.463 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T06:52:36.463 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T06:52:36.463 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: get_monmap_and_config 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: monmap: 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:36.513542+0000 2024-09-16T06:52:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:36.513542+0000 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:36.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 1 Processor -- start 2024-09-16T06:52:36.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 1 -- start start 2024-09-16T06:52:36.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: init 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-a con 0x7fc268024d00 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-b con 0x7fc2680c2790 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-c con 0x7fc2680c62a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26801fe20 con 0x7fc268024d00 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680250e0 con 0x7fc2680c62a0 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26801f860 con 0x7fc2680c2790 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27712d700 10 monclient(hunting): authenticate will time out at 1703.647136s 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27612b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc268024d00 0x7fc2680227d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55244/0 (socket says 172.21.15.76:55244) 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc27612b700 1 -- 172.21.15.76:0/415220354 learned_addr learned my addr 172.21.15.76:0/415220354 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039328281 0 0) 0x7fc26801f860 con 0x7fc2680c2790 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): my global_id is 244466 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c0035d0 con 0x7fc2680c2790 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411981046 0 0) 0x7fc26801fe20 con 0x7fc268024d00 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 10 monclient(hunting): my global_id is 264246 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.510+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26801f860 con 0x7fc268024d00 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186205000 0 0) 0x7fc2680250e0 con 0x7fc2680c62a0 2024-09-16T06:52:36.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient(hunting): my global_id is 274156 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26801fe20 con 0x7fc2680c62a0 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553463838 0 0) 0x7fc26801f860 con 0x7fc268024d00 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 >> v1:172.21.15.76:6790/0 conn(0x7fc2680c2790 legacy=0x7fc2680c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 >> v1:172.21.15.189:6789/0 conn(0x7fc2680c62a0 legacy=0x7fc2680c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 monclient: found mon.noname-a 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680232d0 con 0x7fc268024d00 2024-09-16T06:52:36.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 5 monclient: authenticate success, global_id 264246 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc260001610 con 0x7fc268024d00 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: dump: 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.517 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2600018f0 con 0x7fc268024d00 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 4 set_mon_vals no callback set 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 1 -- 172.21.15.76:0/415220354 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc260001c80 con 0x7fc268024d00 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc275129700 10 monclient: dump: 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.519 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: shutdown 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- 172.21.15.76:0/415220354 >> v1:172.21.15.76:6789/0 conn(0x7fc268024d00 legacy=0x7fc2680227d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- 172.21.15.76:0/415220354 shutdown_connections 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- 172.21.15.76:0/415220354 >> 172.21.15.76:0/415220354 conn(0x7fc268005b40 msgr2=0x7fc268005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- 172.21.15.76:0/415220354 shutdown_connections 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- 172.21.15.76:0/415220354 wait complete. 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: monmap: 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 Processor -- start 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- start start 2024-09-16T06:52:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: init 2024-09-16T06:52:36.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:36.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-a con 0x7fc268024d00 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-b con 0x7fc2680c2790 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient(hunting): picked mon.noname-c con 0x7fc2680c62a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680acdc0 con 0x7fc268024d00 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680ad0a0 con 0x7fc2680c2790 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2680a2ed0 con 0x7fc2680c62a0 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.514+0000 7fc27712d700 10 monclient(hunting): authenticate will time out at 1703.651941s 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27612b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc268024d00 0x7fc2680c1380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:55246/0 (socket says 172.21.15.76:55246) 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27612b700 1 -- 172.21.15.76:0/1262890352 learned_addr learned my addr 172.21.15.76:0/1262890352 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807316350 0 0) 0x7fc2680a2ed0 con 0x7fc2680c62a0 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): my global_id is 244469 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2540035d0 con 0x7fc2680c62a0 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845482259 0 0) 0x7fc2680acdc0 con 0x7fc268024d00 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): my global_id is 264249 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2680a2ed0 con 0x7fc268024d00 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671799592 0 0) 0x7fc2680ad0a0 con 0x7fc2680c2790 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): my global_id is 274159 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2680acdc0 con 0x7fc2680c2790 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305091130 0 0) 0x7fc2540035d0 con 0x7fc2680c62a0 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2680ad0a0 con 0x7fc2680c62a0 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc270069400 con 0x7fc2680c62a0 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): dump: 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3539163071 0 0) 0x7fc2680ad0a0 con 0x7fc2680c62a0 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6789/0 conn(0x7fc2680c2790 legacy=0x7fc2681598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.76:6789/0 conn(0x7fc268024d00 legacy=0x7fc2680c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 monclient: found mon.c 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680c5ff0 con 0x7fc2680c62a0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 5 monclient: authenticate success, global_id 244469 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 10 monclient: _renew_subs 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2680c5ba0 con 0x7fc2680c62a0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 10 monclient: _renew_subs 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2680a2360 con 0x7fc2680c62a0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc27004ec20 con 0x7fc2680c62a0 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc270066de0 con 0x7fc2680c62a0 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc274928700 4 set_mon_vals no callback set 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient: dump: 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc270066f40 con 0x7fc2680c62a0 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc258006590 con 0x7fc258002830 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc2700bc870 con 0x7fc2680c62a0 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc27592a700 10 monclient: get_auth_request con 0x7fc254036340 auth_method 0 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658260294 0 0) 0x7fc258006590 con 0x7fc258002830 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 10 monclient(hunting): my global_id is 274162 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.518+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2680ad0a0 con 0x7fc258002830 2024-09-16T06:52:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3832871887 0 0) 0x7fc2680ad0a0 con 0x7fc258002830 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc258006590 con 0x7fc258002830 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2600015d0 con 0x7fc258002830 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 10 monclient: dump: 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3653475777 0 0) 0x7fc258006590 con 0x7fc258002830 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc25800a190 con 0x7fc258002830 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc260004890 con 0x7fc258002830 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6789/0 conn(0x7fc258002830 legacy=0x7fc258004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.522+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6789/0 conn(0x7fc258002830 legacy=0x7fc258004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc27712d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc258009e50 con 0x7fc258009170 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646775460 0 0) 0x7fc258009e50 con 0x7fc258009170 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient(hunting): my global_id is 274165 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258006590 con 0x7fc258009170 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2720482861 0 0) 0x7fc258006590 con 0x7fc258009170 2024-09-16T06:52:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc258009e50 con 0x7fc258009170 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc260004e10 con 0x7fc258009170 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.530+0000 7fc266ffd700 10 monclient: dump: 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3229801494 0 0) 0x7fc258009e50 con 0x7fc258009170 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc25800c340 con 0x7fc258009170 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 732577398) 0x7fc2600050a0 con 0x7fc258009170 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6789/0 conn(0x7fc258009170 legacy=0x7fc258009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc266ffd700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6789/0 conn(0x7fc258009170 legacy=0x7fc258009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 126, 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:36.549 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:36.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc254036340 legacy=0x7fc254038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 10 monclient: shutdown 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 >> v1:172.21.15.76:6790/0 conn(0x7fc2680c62a0 legacy=0x7fc26815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27692c700 1 -- 172.21.15.76:0/1262890352 reap_dead start 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27692c700 1 -- 172.21.15.76:0/1262890352 reap_dead start 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 shutdown_connections 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 >> 172.21.15.76:0/1262890352 conn(0x7fc268005b40 msgr2=0x7fc268021770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 shutdown_connections 2024-09-16T06:52:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.534+0000 7fc27712d700 1 -- 172.21.15.76:0/1262890352 wait complete. 2024-09-16T06:52:36.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:52:36.552 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T06:52:36.552 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:52:36.552 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T06:52:36.552 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:52:36.552 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T06:52:36.552 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:52:36.590 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T06:52:36.590 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:52:36.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:36.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: get_monmap_and_config 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: monmap: 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:36.704146+0000 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:36.704146+0000 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:36.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 1 Processor -- start 2024-09-16T06:52:36.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 1 -- start start 2024-09-16T06:52:36.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: init 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-a con 0x7f78d812ec10 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-b con 0x7f78d812eff0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-c con 0x7f78d80a0220 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d8071b30 con 0x7f78d812ec10 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d809f2a0 con 0x7f78d80a0220 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d809f580 con 0x7f78d812eff0 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f78d812eff0 0x7f78d8071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43734/0 (socket says 172.21.15.76:43734) 2024-09-16T06:52:36.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d77fe700 1 -- 172.21.15.76:0/3433127876 learned_addr learned my addr 172.21.15.76:0/3433127876 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dd004700 10 monclient(hunting): authenticate will time out at 1703.839129s 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78dc803700 1 -- 172.21.15.76:0/3433127876 >> v1:172.21.15.189:6789/0 conn(0x7f78d80a0220 legacy=0x7f78d813e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229056218 0 0) 0x7f78d809f580 con 0x7f78d812eff0 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): my global_id is 244478 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c40035d0 con 0x7f78d812eff0 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806756524 0 0) 0x7f78d8071b30 con 0x7f78d812ec10 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 10 monclient(hunting): my global_id is 264258 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d809f580 con 0x7f78d812ec10 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.702+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1238219383 0 0) 0x7f78d809f580 con 0x7f78d812ec10 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 >> v1:172.21.15.76:6790/0 conn(0x7f78d812eff0 legacy=0x7f78d8071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 >> v1:172.21.15.189:6789/0 conn(0x7f78d80a0220 legacy=0x7f78d813e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 monclient: found mon.noname-a 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d809f860 con 0x7f78d812ec10 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78c8001970 con 0x7f78d812ec10 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: dump: 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.706 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 5 monclient: authenticate success, global_id 264258 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78c8001b90 con 0x7f78d812ec10 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 4 set_mon_vals no callback set 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 1 -- 172.21.15.76:0/3433127876 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78c8001fa0 con 0x7f78d812ec10 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78d6ffd700 10 monclient: dump: 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.708 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.708 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.708 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.708 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.708 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 10 monclient: shutdown 2024-09-16T06:52:36.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- 172.21.15.76:0/3433127876 >> v1:172.21.15.76:6789/0 conn(0x7f78d812ec10 legacy=0x7f78d8070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- 172.21.15.76:0/3433127876 shutdown_connections 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- 172.21.15.76:0/3433127876 >> 172.21.15.76:0/3433127876 conn(0x7f78d80a4430 msgr2=0x7f78d80a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- 172.21.15.76:0/3433127876 shutdown_connections 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- 172.21.15.76:0/3433127876 wait complete. 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 10 monclient: build_initial_monmap 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 10 monclient: monmap: 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 Processor -- start 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 1 -- start start 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.706+0000 7f78dd004700 10 monclient: init 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-b con 0x7f78d80a0220 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-c con 0x7f78d812ec10 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient(hunting): picked mon.noname-a con 0x7f78d812eff0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d811cee0 con 0x7f78d812eff0 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d809b0f0 con 0x7f78d80a0220 2024-09-16T06:52:36.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d7fff700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f78d80a0220 legacy=0x7f78d811a450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d811b270 con 0x7f78d812ec10 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient(hunting): authenticate will time out at 1703.845177s 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f78d812ec10 0x7f78d811ab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43748/0 (socket says 172.21.15.76:43748) 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d77fe700 1 -- 172.21.15.76:0/1176069961 learned_addr learned my addr 172.21.15.76:0/1176069961 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604271998 0 0) 0x7f78d811b270 con 0x7f78d812ec10 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): my global_id is 244481 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b80035d0 con 0x7f78d812ec10 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951184013 0 0) 0x7f78d811cee0 con 0x7f78d812eff0 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:36.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): my global_id is 264261 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d811b270 con 0x7f78d812eff0 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3185969604 0 0) 0x7f78b80035d0 con 0x7f78d812ec10 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78d811cee0 con 0x7f78d812ec10 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78c00019f0 con 0x7f78d812ec10 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.716 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 109244780 0 0) 0x7f78d811b270 con 0x7f78d812eff0 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78b80035d0 con 0x7f78d812eff0 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78d0004060 con 0x7f78d812eff0 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.717 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273342139 0 0) 0x7f78d811cee0 con 0x7f78d812ec10 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 >> v1:172.21.15.189:6789/0 conn(0x7f78d80a0220 legacy=0x7f78d811a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 >> v1:172.21.15.76:6789/0 conn(0x7f78d812eff0 legacy=0x7f78d81d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 monclient: found mon.c 2024-09-16T06:52:36.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80a3740 con 0x7f78d812ec10 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 5 monclient: authenticate success, global_id 244481 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _renew_subs 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d812c810 con 0x7f78d812ec10 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78c0001ca0 con 0x7f78d812ec10 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _renew_subs 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d811b750 con 0x7f78d812ec10 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78c0004ce0 con 0x7f78d812ec10 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d4ff9700 10 monclient: dump: 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:36.720 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.710+0000 7f78d67fc700 4 set_mon_vals no callback set 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78dd004700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78dd004700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78dd004700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d80678b0 con 0x7f78d812ec10 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f78c000f750 con 0x7f78d812ec10 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f78c0043ff0 con 0x7f78d812ec10 2024-09-16T06:52:36.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.714+0000 7f78d7fff700 10 monclient: get_auth_request con 0x7f78b80362c0 auth_method 0 2024-09-16T06:52:36.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.718+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f78c00112b0 con 0x7f78d812ec10 2024-09-16T06:52:36.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.718+0000 7f78d4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:36.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.718+0000 7f78d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:36.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:36.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:36.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:36.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78d8131050 con 0x7f78d812ec10 2024-09-16T06:52:36.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78d4ff9700 1 -- 172.21.15.76:0/1176069961 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1789233660) 0x7f78c0012dc0 con 0x7f78d812ec10 2024-09-16T06:52:36.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78d4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:36.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78d4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:36.936 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 >> v1:172.21.15.189:6812/3098810674 conn(0x7f78b80362c0 legacy=0x7f78b8038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 10 monclient: shutdown 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 >> v1:172.21.15.76:6790/0 conn(0x7f78d812ec10 legacy=0x7f78d811ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 shutdown_connections 2024-09-16T06:52:36.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 >> 172.21.15.76:0/1176069961 conn(0x7f78d80a4430 msgr2=0x7f78d809dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:36.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 shutdown_connections 2024-09-16T06:52:36.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:36.938+0000 7f78dd004700 1 -- 172.21.15.76:0/1176069961 wait complete. 2024-09-16T06:52:39.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:40.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: get_monmap_and_config 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: build_initial_monmap 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: monmap: 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:40.072668+0000 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:40.072668+0000 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:40.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:40.069 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 1 Processor -- start 2024-09-16T06:52:40.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 1 -- start start 2024-09-16T06:52:40.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: init 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-a con 0x7f5a5809f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-b con 0x7f5a5809d520 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-c con 0x7f5a5809fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5809b520 con 0x7f5a5809f230 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a58071f30 con 0x7f5a5809fd20 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5809b0e0 con 0x7f5a5809d520 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:40.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient(hunting): authenticate will time out at 1707.205932s 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a577fe700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f5a5809fd20 legacy=0x7f5a5813ab70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a567fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5a5809d520 0x7f5a58137440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43756/0 (socket says 172.21.15.76:43756) 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a567fc700 1 -- 172.21.15.76:0/3478740326 learned_addr learned my addr 172.21.15.76:0/3478740326 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011832624 0 0) 0x7f5a5809b0e0 con 0x7f5a5809d520 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient(hunting): my global_id is 244484 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a380035d0 con 0x7f5a5809d520 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472244733 0 0) 0x7f5a380035d0 con 0x7f5a5809d520 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 >> v1:172.21.15.189:6789/0 conn(0x7f5a5809fd20 legacy=0x7f5a5813ab70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 >> v1:172.21.15.76:6789/0 conn(0x7f5a5809f230 legacy=0x7f5a5809f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:40.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 monclient: found mon.noname-b 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5809c7d0 con 0x7f5a5809d520 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: _finish_auth 0 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5a40001610 con 0x7f5a5809d520 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: dump: 2024-09-16T06:52:40.080 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 5 monclient: authenticate success, global_id 244484 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a400018c0 con 0x7f5a5809d520 2024-09-16T06:52:40.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 1 -- 172.21.15.76:0/3478740326 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5a40001c50 con 0x7f5a5809d520 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a5d99c700 4 set_mon_vals no callback set 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.070+0000 7f5a55ffb700 10 monclient: dump: 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:40.082 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: shutdown 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- 172.21.15.76:0/3478740326 >> v1:172.21.15.76:6790/0 conn(0x7f5a5809d520 legacy=0x7f5a58137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- 172.21.15.76:0/3478740326 shutdown_connections 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- 172.21.15.76:0/3478740326 >> 172.21.15.76:0/3478740326 conn(0x7f5a580a3e50 msgr2=0x7f5a580a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- 172.21.15.76:0/3478740326 shutdown_connections 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- 172.21.15.76:0/3478740326 wait complete. 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: build_initial_monmap 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: monmap: 2024-09-16T06:52:40.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 Processor -- start 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- start start 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: init 2024-09-16T06:52:40.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-c con 0x7f5a5809f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-b con 0x7f5a5809fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient(hunting): picked mon.noname-a con 0x7f5a5813e0e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a58139170 con 0x7f5a5813e0e0 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5813a2a0 con 0x7f5a5809fd20 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a580a2a90 con 0x7f5a5809f230 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a5d99c700 10 monclient(hunting): authenticate will time out at 1707.211558s 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a56ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5a5809f230 0x7f5a58136c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43758/0 (socket says 172.21.15.76:43758) 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a56ffd700 1 -- 172.21.15.76:0/356096577 learned_addr learned my addr 172.21.15.76:0/356096577 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:40.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.074+0000 7f5a567fc700 1 -- 172.21.15.76:0/356096577 >> v1:172.21.15.189:6789/0 conn(0x7f5a5809fd20 legacy=0x7f5a58137350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130207030 0 0) 0x7f5a580a2a90 con 0x7f5a5809f230 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): my global_id is 244487 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0035d0 con 0x7f5a5809f230 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4170305750 0 0) 0x7f5a3c0035d0 con 0x7f5a5809f230 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a580a2a90 con 0x7f5a5809f230 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5a480018b0 con 0x7f5a5809f230 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): dump: 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:40.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2631888747 0 0) 0x7f5a580a2a90 con 0x7f5a5809f230 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 >> v1:172.21.15.189:6789/0 conn(0x7f5a5809fd20 legacy=0x7f5a58137350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 >> v1:172.21.15.76:6789/0 conn(0x7f5a5813e0e0 legacy=0x7f5a581d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 monclient: found mon.c 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a58072450 con 0x7f5a5809f230 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: _finish_auth 0 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 5 monclient: authenticate success, global_id 244487 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _renew_subs 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a58137a60 con 0x7f5a5809f230 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _renew_subs 2024-09-16T06:52:40.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5813ab40 con 0x7f5a5809f230 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a48001ce0 con 0x7f5a5809f230 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5a48003b10 con 0x7f5a5809f230 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a557fa700 4 set_mon_vals no callback set 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 10 monclient: dump: 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:40.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30005100 con 0x7f5a5809f230 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5a48003c70 con 0x7f5a5809f230 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5a48043e90 con 0x7f5a5809f230 2024-09-16T06:52:40.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.078+0000 7f5a567fc700 10 monclient: get_auth_request con 0x7f5a3c0360a0 auth_method 0 2024-09-16T06:52:40.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.086+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5a48014830 con 0x7f5a5809f230 2024-09-16T06:52:40.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.086+0000 7f5a4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:40.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.086+0000 7f5a4ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:40.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.326+0000 7f5a5d99c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:40.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.326+0000 7f5a5d99c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:40.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.326+0000 7f5a5d99c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:40.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.326+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5a30005930 con 0x7f5a5809f230 2024-09-16T06:52:40.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a4ffff700 1 -- 172.21.15.76:0/356096577 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 977881053) 0x7f5a48012ca0 con 0x7f5a5809f230 2024-09-16T06:52:40.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a4ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:40.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a4ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:40.327 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5a3c0360a0 legacy=0x7f5a3c038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 10 monclient: shutdown 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 >> v1:172.21.15.76:6790/0 conn(0x7f5a5809f230 legacy=0x7f5a58136c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 shutdown_connections 2024-09-16T06:52:40.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 >> 172.21.15.76:0/356096577 conn(0x7f5a580a3e50 msgr2=0x7f5a580a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:40.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 shutdown_connections 2024-09-16T06:52:40.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:40.330+0000 7f5a5d99c700 1 -- 172.21.15.76:0/356096577 wait complete. 2024-09-16T06:52:43.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:52:43.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: get_monmap_and_config 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: build_initial_monmap 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: monmap: 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:43.464639+0000 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:43.464639+0000 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:43.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:43.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 1 Processor -- start 2024-09-16T06:52:43.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 1 -- start start 2024-09-16T06:52:43.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: init 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-a con 0x7f7d8409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-c con 0x7f7d8409d520 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-b con 0x7f7d8409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8409b520 con 0x7f7d8409f230 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84071f30 con 0x7f7d8409d520 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8409b0e0 con 0x7f7d8409fd20 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d89b4d700 10 monclient(hunting): authenticate will time out at 1710.598174s 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d7affd700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f7d8409d520 legacy=0x7f7d84137440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d83fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7d8409fd20 0x7f7d8413ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43762/0 (socket says 172.21.15.76:43762) 2024-09-16T06:52:43.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.462+0000 7f7d83fff700 1 -- 172.21.15.76:0/972879827 learned_addr learned my addr 172.21.15.76:0/972879827 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:43.666+0000 7f7d7affd700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d8409d520 legacy=0x7f7d84137440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:44.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:44.070+0000 7f7d7affd700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d8409d520 legacy=0x7f7d84137440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:44.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:44.874+0000 7f7d7affd700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d8409d520 legacy=0x7f7d84137440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:46.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): tick 2024-09-16T06:52:46.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:52:46.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:52:46.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:52:46.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6790/0 conn(0x7f7d8409fd20 legacy=0x7f7d8413ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d8409d520 legacy=0x7f7d84137440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6789/0 conn(0x7f7d8409f230 legacy=0x7f7d8409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f7d64005950 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7d64009540 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f7d6400d230 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8409c7d0 con 0x7f7d64009540 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84071f30 con 0x7f7d6400d230 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64012f00 con 0x7f7d64005950 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d81ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:46.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.462+0000 7f7d83fff700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d6400d230 legacy=0x7f7d6400f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:46.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:46.666+0000 7f7d83fff700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d6400d230 legacy=0x7f7d6400f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:47.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:47.070+0000 7f7d83fff700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d6400d230 legacy=0x7f7d6400f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:47.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:47.874+0000 7f7d83fff700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d6400d230 legacy=0x7f7d6400f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): tick 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6790/0 conn(0x7f7d64005950 legacy=0x7f7d64007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d6400d230 legacy=0x7f7d6400f6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6789/0 conn(0x7f7d64009540 legacy=0x7f7d6400b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f7d64010fa0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f7d64005950 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7d6400f730 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64014170 con 0x7f7d6400f730 2024-09-16T06:52:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84071f30 con 0x7f7d64010fa0 2024-09-16T06:52:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6400fb80 con 0x7f7d64005950 2024-09-16T06:52:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d81ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.462+0000 7f7d837fe700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d6400fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:49.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:49.665+0000 7f7d837fe700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d6400fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:50.069+0000 7f7d837fe700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d6400fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:50.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:50.873+0000 7f7d837fe700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d6400fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891584366 0 0) 0x7f7d64014170 con 0x7f7d6400f730 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient(hunting): my global_id is 264264 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d700035d0 con 0x7f7d6400f730 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3026396344 0 0) 0x7f7d700035d0 con 0x7f7d6400f730 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6790/0 conn(0x7f7d64005950 legacy=0x7f7d64007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d6400fdd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 monclient: found mon.noname-a 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6400b9b0 con 0x7f7d6400f730 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d89b4d700 5 monclient: authenticate success, global_id 264264 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d89b4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d7c0018e0 con 0x7f7d6400f730 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: dump: 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d89b4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d7c001b80 con 0x7f7d6400f730 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 1 -- 172.21.15.76:0/972879827 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d7c001f10 con 0x7f7d6400f730 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d89b4d700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d89b4d700 4 set_mon_vals no callback set 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.189+0000 7f7d82ffd700 10 monclient: dump: 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.193 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.194 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: shutdown 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- 172.21.15.76:0/972879827 >> v1:172.21.15.76:6789/0 conn(0x7f7d6400f730 legacy=0x7f7d64010580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- 172.21.15.76:0/972879827 shutdown_connections 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- 172.21.15.76:0/972879827 >> 172.21.15.76:0/972879827 conn(0x7f7d840a3e50 msgr2=0x7f7d840a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- 172.21.15.76:0/972879827 shutdown_connections 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- 172.21.15.76:0/972879827 wait complete. 2024-09-16T06:52:51.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: build_initial_monmap 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: monmap: 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 Processor -- start 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- start start 2024-09-16T06:52:51.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: init 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-c con 0x7f7d64005950 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-a con 0x7f7d6400f730 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient(hunting): picked mon.noname-b con 0x7f7d64010fa0 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d840a2c00 con 0x7f7d6400f730 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d840a26b0 con 0x7f7d64010fa0 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d84072450 con 0x7f7d64005950 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d89b4d700 10 monclient(hunting): authenticate will time out at 1718.331138s 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d837fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7d64005950 0x7f7d84132670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57100/0 (socket says 172.21.15.76:57100) 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d837fe700 1 -- 172.21.15.76:0/3444029594 learned_addr learned my addr 172.21.15.76:0/3444029594 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:51.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d83fff700 1 -- 172.21.15.76:0/3444029594 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d841ce120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390426709 0 0) 0x7f7d840a2c00 con 0x7f7d6400f730 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 10 monclient(hunting): my global_id is 264267 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.193+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6400be90 con 0x7f7d6400f730 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152698072 0 0) 0x7f7d6400be90 con 0x7f7d6400f730 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d840a2c00 con 0x7f7d6400f730 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d6c004280 con 0x7f7d6400f730 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient(hunting): dump: 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.200 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3330875650 0 0) 0x7f7d840a2c00 con 0x7f7d6400f730 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 >> v1:172.21.15.76:6790/0 conn(0x7f7d64005950 legacy=0x7f7d84132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 >> v1:172.21.15.189:6789/0 conn(0x7f7d64010fa0 legacy=0x7f7d841ce120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:51.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 monclient: found mon.a 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84120d60 con 0x7f7d6400f730 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 5 monclient: authenticate success, global_id 264267 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _renew_subs 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d84121040 con 0x7f7d6400f730 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _renew_subs 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8411ece0 con 0x7f7d6400f730 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d6c004720 con 0x7f7d6400f730 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d6c004ab0 con 0x7f7d6400f730 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d827fc700 4 set_mon_vals no callback set 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 10 monclient: dump: 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.203 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68005100 con 0x7f7d6400f730 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7d6c004d10 con 0x7f7d6400f730 2024-09-16T06:52:51.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7d6c043e20 con 0x7f7d6400f730 2024-09-16T06:52:51.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.197+0000 7f7d837fe700 10 monclient: get_auth_request con 0x7f7d64039f90 auth_method 0 2024-09-16T06:52:51.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.205+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7d6c0147c0 con 0x7f7d6400f730 2024-09-16T06:52:51.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.205+0000 7f7d80ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.205+0000 7f7d80ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=0 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:52:51.396 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:52:51.397 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d89b4d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d89b4d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d89b4d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d68005930 con 0x7f7d6400f730 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d80ff9700 1 -- 172.21.15.76:0/3444029594 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 918471313) 0x7f7d6c01cdb0 con 0x7f7d6400f730 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d80ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:52:51.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.441+0000 7f7d80ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:51.442 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7d64039f90 legacy=0x7f7d6403c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 10 monclient: shutdown 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 >> v1:172.21.15.76:6789/0 conn(0x7f7d6400f730 legacy=0x7f7d841ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 shutdown_connections 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 >> 172.21.15.76:0/3444029594 conn(0x7f7d840a3e50 msgr2=0x7f7d841b8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 shutdown_connections 2024-09-16T06:52:51.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.445+0000 7f7d89b4d700 1 -- 172.21.15.76:0/3444029594 wait complete. 2024-09-16T06:52:51.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:52:51.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:52:51.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:52:51.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.569+0000 7fed40936700 10 monclient: get_monmap_and_config 2024-09-16T06:52:51.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.569+0000 7fed40936700 10 monclient: build_initial_monmap 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.569+0000 7fed40936700 10 monclient: monmap: 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:51.575753+0000 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:51.575753+0000 2024-09-16T06:52:51.571 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:51.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:51.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:51.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:51.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 1 Processor -- start 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 1 -- start start 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient: init 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient(hunting): picked mon.noname-c con 0x7fed3c0a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:51.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient(hunting): picked mon.noname-a con 0x7fed3c130550 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient(hunting): picked mon.noname-b con 0x7fed3c12f890 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c0a16b0 con 0x7fed3c130550 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c0a4c20 con 0x7fed3c0a4840 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c09db90 con 0x7fed3c12f890 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient(hunting): authenticate will time out at 1718.710770s 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed3ad9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fed3c0a4840 legacy=0x7fed3c12fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed3b59e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fed3c12f890 0x7fed3c13b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57104/0 (socket says 172.21.15.76:57104) 2024-09-16T06:52:51.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed3a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed3c130550 0x7fed3c1380b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45842/0 (socket says 172.21.15.76:45842) 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed3b59e700 1 -- 172.21.15.76:0/1340376337 learned_addr learned my addr 172.21.15.76:0/1340376337 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838795620 0 0) 0x7fed3c0a16b0 con 0x7fed3c130550 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): my global_id is 264276 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed240035d0 con 0x7fed3c130550 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816358963 0 0) 0x7fed3c09db90 con 0x7fed3c12f890 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient(hunting): my global_id is 244496 2024-09-16T06:52:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c0a16b0 con 0x7fed3c12f890 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3665557376 0 0) 0x7fed3c0a16b0 con 0x7fed3c12f890 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 >> v1:172.21.15.189:6789/0 conn(0x7fed3c0a4840 legacy=0x7fed3c12fe40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 >> v1:172.21.15.76:6789/0 conn(0x7fed3c130550 legacy=0x7fed3c1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 monclient: found mon.noname-b 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c09e1e0 con 0x7fed3c12f890 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed34001970 con 0x7fed3c12f890 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: dump: 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 5 monclient: authenticate success, global_id 244496 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:51.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed34001c90 con 0x7fed3c12f890 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 1 -- 172.21.15.76:0/1340376337 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed34002060 con 0x7fed3c12f890 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed40936700 4 set_mon_vals no callback set 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.573+0000 7fed39d9b700 10 monclient: dump: 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.593 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: shutdown 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- 172.21.15.76:0/1340376337 >> v1:172.21.15.76:6790/0 conn(0x7fed3c12f890 legacy=0x7fed3c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- 172.21.15.76:0/1340376337 shutdown_connections 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- 172.21.15.76:0/1340376337 >> 172.21.15.76:0/1340376337 conn(0x7fed3c0a4430 msgr2=0x7fed3c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- 172.21.15.76:0/1340376337 shutdown_connections 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- 172.21.15.76:0/1340376337 wait complete. 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: build_initial_monmap 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: monmap: 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:51.594 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 Processor -- start 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- start start 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: init 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient(hunting): picked mon.noname-b con 0x7fed3c0a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:51.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient(hunting): picked mon.noname-a con 0x7fed3c12f890 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 10 monclient(hunting): picked mon.noname-c con 0x7fed3c130550 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.577+0000 7fed40936700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c136460 con 0x7fed3c12f890 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c12e7e0 con 0x7fed3c0a4840 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3c12ed30 con 0x7fed3c130550 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient(hunting): authenticate will time out at 1718.716041s 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3ad9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7fed3c0a4840 legacy=0x7fed3c0703d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed3c12f890 0x7fed3c070ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45854/0 (socket says 172.21.15.76:45854) 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3b59e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fed3c130550 0x7fed3c1cffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57108/0 (socket says 172.21.15.76:57108) 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3a59c700 1 -- 172.21.15.76:0/3634071013 learned_addr learned my addr 172.21.15.76:0/3634071013 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067066502 0 0) 0x7fed3c136460 con 0x7fed3c12f890 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): my global_id is 264279 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed280035d0 con 0x7fed3c12f890 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145538012 0 0) 0x7fed3c12ed30 con 0x7fed3c130550 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): my global_id is 244499 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c136460 con 0x7fed3c130550 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989853576 0 0) 0x7fed280035d0 con 0x7fed3c12f890 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed3c12ed30 con 0x7fed3c12f890 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed300019f0 con 0x7fed3c12f890 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:52:51.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): dump: 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070698767 0 0) 0x7fed3c136460 con 0x7fed3c130550 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed280035d0 con 0x7fed3c130550 2024-09-16T06:52:51.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed34001fa0 con 0x7fed3c130550 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): dump: 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 279557132 0 0) 0x7fed3c12ed30 con 0x7fed3c12f890 2024-09-16T06:52:51.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6790/0 conn(0x7fed3c130550 legacy=0x7fed3c1cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.189:6789/0 conn(0x7fed3c0a4840 legacy=0x7fed3c0703d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 monclient: found mon.a 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c0711f0 con 0x7fed3c12f890 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 5 monclient: authenticate success, global_id 264279 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient: _renew_subs 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c134390 con 0x7fed3c12f890 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient: _renew_subs 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c09b150 con 0x7fed3c12f890 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed300015e0 con 0x7fed3c12f890 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed30004b40 con 0x7fed3c12f890 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3959a700 4 set_mon_vals no callback set 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: dump: 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.601 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fed3000f5b0 con 0x7fed3c12f890 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed1c006590 con 0x7fed1c002830 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fed30043e00 con 0x7fed3c12f890 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884541324 0 0) 0x7fed1c006590 con 0x7fed1c002830 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient(hunting): my global_id is 264282 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c12ed30 con 0x7fed1c002830 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed3b59e700 10 monclient: get_auth_request con 0x7fed280362c0 auth_method 0 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1526592675 0 0) 0x7fed3c12ed30 con 0x7fed1c002830 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed1c006590 con 0x7fed1c002830 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed2c004060 con 0x7fed1c002830 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: dump: 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.603 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 648006706 0 0) 0x7fed1c006590 con 0x7fed1c002830 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed1c00a190 con 0x7fed1c002830 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fed2c006470 con 0x7fed1c002830 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6789/0 conn(0x7fed1c002830 legacy=0x7fed1c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.581+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6789/0 conn(0x7fed1c002830 legacy=0x7fed1c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fed1c009e50 con 0x7fed1c009170 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766985542 0 0) 0x7fed1c009e50 con 0x7fed1c009170 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient(hunting): my global_id is 264285 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c006590 con 0x7fed1c009170 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 660170297 0 0) 0x7fed1c006590 con 0x7fed1c009170 2024-09-16T06:52:51.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed1c009e50 con 0x7fed1c009170 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fed2c0048a0 con 0x7fed1c009170 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient: dump: 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.607 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3845739706 0 0) 0x7fed1c009e50 con 0x7fed1c009170 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed1c00c340 con 0x7fed1c009170 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 2884402822) 0x7fed2c005d70 con 0x7fed1c009170 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6789/0 conn(0x7fed1c009170 legacy=0x7fed1c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed237fe700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6789/0 conn(0x7fed1c009170 legacy=0x7fed1c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 130, 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:51.609 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.610 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.611 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:51.612 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.613 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:51.616 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.618 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:51.619 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.189:6812/3098810674 conn(0x7fed280362c0 legacy=0x7fed28038780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 10 monclient: shutdown 2024-09-16T06:52:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 >> v1:172.21.15.76:6789/0 conn(0x7fed3c12f890 legacy=0x7fed3c070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed3b59e700 1 -- 172.21.15.76:0/3634071013 reap_dead start 2024-09-16T06:52:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed3b59e700 1 -- 172.21.15.76:0/3634071013 reap_dead start 2024-09-16T06:52:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 shutdown_connections 2024-09-16T06:52:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 >> 172.21.15.76:0/3634071013 conn(0x7fed3c0a4430 msgr2=0x7fed3c0a35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 shutdown_connections 2024-09-16T06:52:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.593+0000 7fed40936700 1 -- 172.21.15.76:0/3634071013 wait complete. 2024-09-16T06:52:51.621 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:52:51.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:52:51.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:52:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 10 monclient: get_monmap_and_config 2024-09-16T06:52:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 10 monclient: build_initial_monmap 2024-09-16T06:52:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 10 monclient: monmap: 2024-09-16T06:52:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:52:51.742653+0000 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:52:51.742653+0000 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:52:51.738 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 1 Processor -- start 2024-09-16T06:52:51.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 1 -- start start 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.737+0000 7f2d189a3700 10 monclient: init 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-c con 0x7f2d14135500 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-b con 0x7f2d1409a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-a con 0x7f2d1413b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140a67b0 con 0x7f2d1413b980 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140a7320 con 0x7f2d14135500 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1409ade0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient(hunting): authenticate will time out at 1718.876422s 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1259c700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2d14135500 legacy=0x7f2d141358e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d11d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2d1409a9d0 0x7f2d1413a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57112/0 (socket says 172.21.15.76:57112) 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d11d9b700 1 -- 172.21.15.76:0/3084752196 learned_addr learned my addr 172.21.15.76:0/3084752196 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d12d9d700 1 --1- 172.21.15.76:0/3084752196 >> v1:172.21.15.76:6789/0 conn(0x7f2d1413b980 0x7f2d1413dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:45884/0 (socket says 172.21.15.76:45884) 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847447283 0 0) 0x7f2d1409ade0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): my global_id is 244502 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d000035d0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904616271 0 0) 0x7f2d140a67b0 con 0x7f2d1413b980 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient(hunting): my global_id is 264288 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1409ade0 con 0x7f2d1413b980 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090814890 0 0) 0x7f2d000035d0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 >> v1:172.21.15.189:6789/0 conn(0x7f2d14135500 legacy=0x7f2d141358e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 >> v1:172.21.15.76:6789/0 conn(0x7f2d1413b980 legacy=0x7f2d1413dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 monclient: found mon.noname-b 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14099d10 con 0x7f2d1409a9d0 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d040019b0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 5 monclient: authenticate success, global_id 244502 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: dump: 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d04001c90 con 0x7f2d1409a9d0 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 1 -- 172.21.15.76:0/3084752196 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d04002020 con 0x7f2d1409a9d0 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: get_monmap_and_config success 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 4 set_mon_vals no callback set 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d1159a700 10 monclient: dump: 2024-09-16T06:52:51.756 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.756 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.756 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: shutdown 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- 172.21.15.76:0/3084752196 >> v1:172.21.15.76:6790/0 conn(0x7f2d1409a9d0 legacy=0x7f2d1413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- 172.21.15.76:0/3084752196 shutdown_connections 2024-09-16T06:52:51.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- 172.21.15.76:0/3084752196 >> 172.21.15.76:0/3084752196 conn(0x7f2d140a42f0 msgr2=0x7f2d140a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- 172.21.15.76:0/3084752196 shutdown_connections 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 1 -- 172.21.15.76:0/3084752196 wait complete. 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: build_initial_monmap 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.741+0000 7f2d189a3700 10 monclient: monmap: 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:52:51.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 Processor -- start 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- start start 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: init 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-c con 0x7f2d1409a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-a con 0x7f2d14135500 addr v1:172.21.15.76:6789/0 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient(hunting): picked mon.noname-b con 0x7f2d1413b980 addr v1:172.21.15.189:6789/0 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d14136700 con 0x7f2d14135500 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140a3350 con 0x7f2d1413b980 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1406d430 con 0x7f2d1409a9d0 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:52:51.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient(hunting): authenticate will time out at 1718.880651s 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d12d9d700 1 -- >> v1:172.21.15.189:6789/0 conn(0x7f2d1413b980 legacy=0x7f2d141d2700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.189:6789/0 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d1259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2d1409a9d0 0x7f2d141358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57122/0 (socket says 172.21.15.76:57122) 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d1259c700 1 -- 172.21.15.76:0/943186952 learned_addr learned my addr 172.21.15.76:0/943186952 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455636015 0 0) 0x7f2d1406d430 con 0x7f2d1409a9d0 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): my global_id is 244505 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf40035d0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862523177 0 0) 0x7f2d14136700 con 0x7f2d14135500 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): my global_id is 264291 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1406d430 con 0x7f2d14135500 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439860611 0 0) 0x7f2cf40035d0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d14136700 con 0x7f2d1409a9d0 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cfc004230 con 0x7f2d1409a9d0 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): dump: 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115958779 0 0) 0x7f2d1406d430 con 0x7f2d14135500 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf40035d0 con 0x7f2d14135500 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d040017d0 con 0x7f2d14135500 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient(hunting): dump: 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1763589252 0 0) 0x7f2d14136700 con 0x7f2d1409a9d0 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.189:6789/0 conn(0x7f2d1413b980 legacy=0x7f2d141d2700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6789/0 conn(0x7f2d14135500 legacy=0x7f2d14135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: _finish_hunting 0 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 monclient: found mon.c 2024-09-16T06:52:51.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14134b00 con 0x7f2d1409a9d0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: _finish_auth 0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: _check_auth_tickets 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 5 monclient: authenticate success, global_id 244505 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _renew_subs 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14137ff0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _renew_subs 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d14136b60 con 0x7f2d1409a9d0 2024-09-16T06:52:51.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cfc0044a0 con 0x7f2d1409a9d0 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2cfc004830 con 0x7f2d1409a9d0 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 10 monclient: dump: 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.765 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0bfff700 4 set_mon_vals no callback set 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2cfc004990 con 0x7f2d1409a9d0 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2cf8006590 con 0x7f2cf8002830 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.745+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2cfc043e70 con 0x7f2d1409a9d0 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d11d9b700 10 monclient: get_auth_request con 0x7f2cf4036360 auth_method 0 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060255165 0 0) 0x7f2cf8006590 con 0x7f2cf8002830 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient(hunting): my global_id is 244508 2024-09-16T06:52:51.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14136700 con 0x7f2cf8002830 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3852527510 0 0) 0x7f2d14136700 con 0x7f2cf8002830 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf8006590 con 0x7f2cf8002830 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d0c001740 con 0x7f2cf8002830 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient: dump: 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2747790353 0 0) 0x7f2cf8006590 con 0x7f2cf8002830 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cf800a190 con 0x7f2cf8002830 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2d0c001b30 con 0x7f2cf8002830 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6790/0 conn(0x7f2cf8002830 legacy=0x7f2cf8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.749+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6790/0 conn(0x7f2cf8002830 legacy=0x7f2cf8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.757+0000 7f2d189a3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.757+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2cf8009e30 con 0x7f2cf8009150 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937488721 0 0) 0x7f2cf8009e30 con 0x7f2cf8009150 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient(hunting): _negotiate 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient(hunting): my global_id is 244511 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf8006590 con 0x7f2cf8009150 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1215972457 0 0) 0x7f2cf8006590 con 0x7f2cf8009150 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf8009e30 con 0x7f2cf8009150 2024-09-16T06:52:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2d0c004a80 con 0x7f2cf8009150 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient: dump: 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:52:51.771 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1125198959 0 0) 0x7f2cf8009e30 con 0x7f2cf8009150 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2cf800c340 con 0x7f2cf8009150 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 1680405659) 0x7f2d0c004f00 con 0x7f2cf8009150 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6790/0 conn(0x7f2cf8009150 legacy=0x7f2cf80095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d0affd700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6790/0 conn(0x7f2cf8009150 legacy=0x7f2cf80095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:52:51.772 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 130, 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:52:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:52:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:52:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:52:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:52:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:52:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:52:51.784 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2cf4036360 legacy=0x7f2cf4038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 10 monclient: shutdown 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 >> v1:172.21.15.76:6790/0 conn(0x7f2d1409a9d0 legacy=0x7f2d141358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d12d9d700 1 -- 172.21.15.76:0/943186952 reap_dead start 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d12d9d700 1 -- 172.21.15.76:0/943186952 reap_dead start 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 shutdown_connections 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 >> 172.21.15.76:0/943186952 conn(0x7f2d140a42f0 msgr2=0x7f2d1413e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 shutdown_connections 2024-09-16T06:52:51.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:52:51.761+0000 7f2d189a3700 1 -- 172.21.15.76:0/943186952 wait complete. 2024-09-16T06:52:51.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:52:51.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:52:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=0 2024-09-16T06:52:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T06:52:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-16T06:52:51.900 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-16T06:52:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:52:51.901 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T06:52:51.901 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T06:52:51.902 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T06:52:52.235 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:52:52.235 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T06:52:52.236 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-16T06:52:52.566 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:52:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T06:52:52.567 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T06:52:52.567 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:52:52.915 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T06:52:52.916 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T06:52:52.916 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T06:52:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:52:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:52:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:52:53.206 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=110 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T06:52:53.207 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T06:53:11.788 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T06:53:11.788 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T06:53:11.789 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T06:53:11.789 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T06:53:11.792 INFO:tasks.ceph.mon.b:Started 2024-09-16T06:53:11.792 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:53:11.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:11.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 10 monclient: get_monmap_and_config 2024-09-16T06:53:11.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 10 monclient: build_initial_monmap 2024-09-16T06:53:11.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 10 monclient: monmap: 2024-09-16T06:53:11.908 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:11.908 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:11.909 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:11.913514+0000 2024-09-16T06:53:11.909 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:11.913514+0000 2024-09-16T06:53:11.909 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:11.909 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:11.909 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:11.910 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:11.910 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:11.910 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 1 Processor -- start 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 1 -- start start 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.909+0000 7fcef9ac2700 10 monclient: init 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-b con 0x7fcef40a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-a con 0x7fcef41334a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-c con 0x7fcef4133910 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef40a1620 con 0x7fcef41334a0 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef40a4c20 con 0x7fcef4133910 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef4071f10 con 0x7fcef40a4840 2024-09-16T06:53:11.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef8ac0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fcef40a4840 0x7fcef4132d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55690/0 (socket says 172.21.15.76:55690) 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef8ac0700 1 -- 172.21.15.76:0/2653146224 learned_addr learned my addr 172.21.15.76:0/2653146224 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient(hunting): authenticate will time out at 1739.048542s 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904599761 0 0) 0x7fcef4071f10 con 0x7fcef40a4840 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): my global_id is 244544 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc0035d0 con 0x7fcef40a4840 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758057902 0 0) 0x7fcef40a1620 con 0x7fcef41334a0 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:11.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:11.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:11.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient(hunting): my global_id is 264324 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4071f10 con 0x7fcef41334a0 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447688517 0 0) 0x7fcedc0035d0 con 0x7fcef40a4840 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 >> v1:172.21.15.189:6789/0 conn(0x7fcef4133910 legacy=0x7fcef413b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 >> v1:172.21.15.76:6789/0 conn(0x7fcef41334a0 legacy=0x7fcef41380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 monclient: found mon.noname-b 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:11.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef409e1e0 con 0x7fcef40a4840 2024-09-16T06:53:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:11.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcee4001990 con 0x7fcef40a4840 2024-09-16T06:53:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: dump: 2024-09-16T06:53:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:11.920 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:11.921 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcee4001c70 con 0x7fcef40a4840 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 1 -- 172.21.15.76:0/2653146224 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcee4002000 con 0x7fcef40a4840 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:11.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef37fe700 10 monclient: dump: 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:11.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:11.924 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 5 monclient: authenticate success, global_id 244544 2024-09-16T06:53:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:11.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 4 set_mon_vals no callback set 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 10 monclient: shutdown 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2653146224 >> v1:172.21.15.76:6790/0 conn(0x7fcef40a4840 legacy=0x7fcef4132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2653146224 shutdown_connections 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2653146224 >> 172.21.15.76:0/2653146224 conn(0x7fcef40a4430 msgr2=0x7fcef40a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:11.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2653146224 shutdown_connections 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.913+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2653146224 wait complete. 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient: build_initial_monmap 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient: monmap: 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:11.926 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:11.927 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 1 Processor -- start 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 1 -- start start 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient: init 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-c con 0x7fcef40a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-a con 0x7fcef41334a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient(hunting): picked mon.noname-b con 0x7fcef4133910 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef40716e0 con 0x7fcef41334a0 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef41326d0 con 0x7fcef4133910 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef412bb60 con 0x7fcef40a4840 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcef41334a0 0x7fcef412b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57034/0 (socket says 172.21.15.76:57034) 2024-09-16T06:53:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef3fff700 1 -- 172.21.15.76:0/2467973753 learned_addr learned my addr 172.21.15.76:0/2467973753 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:11.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.917+0000 7fcef9ac2700 10 monclient(hunting): authenticate will time out at 1739.054164s 2024-09-16T06:53:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828981333 0 0) 0x7fcef40716e0 con 0x7fcef41334a0 2024-09-16T06:53:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:11.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): my global_id is 264327 2024-09-16T06:53:11.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee00035d0 con 0x7fcef41334a0 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118350793 0 0) 0x7fcee00035d0 con 0x7fcef41334a0 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcef40716e0 con 0x7fcef41334a0 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcee8001910 con 0x7fcef41334a0 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): dump: 2024-09-16T06:53:11.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3795137539 0 0) 0x7fcef40716e0 con 0x7fcef41334a0 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 >> v1:172.21.15.76:6790/0 conn(0x7fcef40a4840 legacy=0x7fcef412ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 >> v1:172.21.15.189:6789/0 conn(0x7fcef4133910 legacy=0x7fcef41d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 monclient: found mon.a 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef409b0f0 con 0x7fcef41334a0 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: _finish_auth 0 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 5 monclient: authenticate success, global_id 264327 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 10 monclient: _renew_subs 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:11.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef40a3770 con 0x7fcef41334a0 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 10 monclient: _renew_subs 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef412dac0 con 0x7fcef41334a0 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcee8001ce0 con 0x7fcef41334a0 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcee8003b10 con 0x7fcef41334a0 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef2ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 10 monclient: dump: 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:11.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.945+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcee8003cf0 con 0x7fcef41334a0 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fcee8043f50 con 0x7fcef41334a0 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef9ac2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef9ac2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef9ac2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4005100 con 0x7fcef41334a0 2024-09-16T06:53:11.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.949+0000 7fcef8ac0700 10 monclient: get_auth_request con 0x7fcee0036140 auth_method 0 2024-09-16T06:53:11.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.957+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fcee8014670 con 0x7fcef41334a0 2024-09-16T06:53:11.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.957+0000 7fcef17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:11.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:11.957+0000 7fcef17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef9ac2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef9ac2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef9ac2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fced4005930 con 0x7fcef41334a0 2024-09-16T06:53:12.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef17fa700 1 -- 172.21.15.76:0/2467973753 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2513168146) 0x7fcee8012d10 con 0x7fcef41334a0 2024-09-16T06:53:12.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:12.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.193+0000 7fcef17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:12.194 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcee0036140 legacy=0x7fcee0038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 10 monclient: shutdown 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 >> v1:172.21.15.76:6789/0 conn(0x7fcef41334a0 legacy=0x7fcef412b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 shutdown_connections 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 >> 172.21.15.76:0/2467973753 conn(0x7fcef40a4430 msgr2=0x7fcef40a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 shutdown_connections 2024-09-16T06:53:12.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.197+0000 7fcef9ac2700 1 -- 172.21.15.76:0/2467973753 wait complete. 2024-09-16T06:53:12.208 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:53:12.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:53:12.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:53:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.321+0000 7f3e6359e700 10 monclient: get_monmap_and_config 2024-09-16T06:53:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.321+0000 7f3e6359e700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.321+0000 7f3e6359e700 10 monclient: monmap: 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:12.327602+0000 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:12.327602+0000 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 Processor -- start 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- start start 2024-09-16T06:53:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient: init 2024-09-16T06:53:12.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3e6409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-a con 0x7f3e640964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-c con 0x7f3e64138870 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e64098560 con 0x7f3e640964f0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e64099ec0 con 0x7f3e64138870 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e64097030 con 0x7f3e6409b1c0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient(hunting): authenticate will time out at 1739.461033s 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e61d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3e640964f0 0x7f3e64096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57048/0 (socket says 172.21.15.76:57048) 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e61d9b700 1 -- 172.21.15.76:0/1013748003 learned_addr learned my addr 172.21.15.76:0/1013748003 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566533528 0 0) 0x7f3e64097030 con 0x7f3e6409b1c0 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): my global_id is 244547 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c0035d0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403817228 0 0) 0x7f3e64098560 con 0x7f3e640964f0 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient(hunting): my global_id is 264330 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e64097030 con 0x7f3e640964f0 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80864820 0 0) 0x7f3e4c0035d0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 >> v1:172.21.15.189:6789/0 conn(0x7f3e64138870 legacy=0x7f3e6413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 >> v1:172.21.15.76:6789/0 conn(0x7f3e640964f0 legacy=0x7f3e64096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 monclient: found mon.noname-b 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6409b5a0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 5 monclient: authenticate success, global_id 244547 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e54001610 con 0x7f3e6409b1c0 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: dump: 2024-09-16T06:53:12.329 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e54001870 con 0x7f3e6409b1c0 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:12.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 4 set_mon_vals no callback set 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 1 -- 172.21.15.76:0/1013748003 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e54001c00 con 0x7f3e6409b1c0 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6159a700 10 monclient: dump: 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 10 monclient: shutdown 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- 172.21.15.76:0/1013748003 >> v1:172.21.15.76:6790/0 conn(0x7f3e6409b1c0 legacy=0x7f3e6409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- 172.21.15.76:0/1013748003 shutdown_connections 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- 172.21.15.76:0/1013748003 >> 172.21.15.76:0/1013748003 conn(0x7f3e640a0030 msgr2=0x7f3e640a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- 172.21.15.76:0/1013748003 shutdown_connections 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.325+0000 7f3e6359e700 1 -- 172.21.15.76:0/1013748003 wait complete. 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: monmap: 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:12.333 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:12.333 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 Processor -- start 2024-09-16T06:53:12.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- start start 2024-09-16T06:53:12.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: init 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-b con 0x7f3e640964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-c con 0x7f3e6409b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient(hunting): picked mon.noname-a con 0x7f3e64138870 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e64070550 con 0x7f3e64138870 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6409bd10 con 0x7f3e640964f0 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e640a2c90 con 0x7f3e6409b1c0 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient(hunting): authenticate will time out at 1739.465698s 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e61d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3e6409b1c0 0x7f3e641cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55722/0 (socket says 172.21.15.76:55722) 2024-09-16T06:53:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e62d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3e64138870 0x7f3e641d2180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57058/0 (socket says 172.21.15.76:57058) 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e61d9b700 1 -- 172.21.15.76:0/4123546286 learned_addr learned my addr 172.21.15.76:0/4123546286 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828572399 0 0) 0x7f3e640a2c90 con 0x7f3e6409b1c0 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): my global_id is 244550 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e440035d0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672356333 0 0) 0x7f3e64070550 con 0x7f3e64138870 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): my global_id is 264333 2024-09-16T06:53:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e640a2c90 con 0x7f3e64138870 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166891290 0 0) 0x7f3e6409bd10 con 0x7f3e640964f0 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): my global_id is 284101 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e64070550 con 0x7f3e640964f0 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079802967 0 0) 0x7f3e440035d0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e6409bd10 con 0x7f3e6409b1c0 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e580019d0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:12.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4114510604 0 0) 0x7f3e640a2c90 con 0x7f3e64138870 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e440035d0 con 0x7f3e64138870 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e5c001b90 con 0x7f3e64138870 2024-09-16T06:53:12.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1461861218 0 0) 0x7f3e6409bd10 con 0x7f3e6409b1c0 2024-09-16T06:53:12.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.189:6789/0 conn(0x7f3e640964f0 legacy=0x7f3e64131870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6789/0 conn(0x7f3e64138870 legacy=0x7f3e641d2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 monclient: found mon.c 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6409a540 con 0x7f3e6409b1c0 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 5 monclient: authenticate success, global_id 244550 2024-09-16T06:53:12.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _renew_subs 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6406e6c0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _renew_subs 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e64133280 con 0x7f3e6409b1c0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e58001c90 con 0x7f3e6409b1c0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e58004ce0 con 0x7f3e6409b1c0 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e60d99700 4 set_mon_vals no callback set 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.329+0000 7f3e52ffd700 10 monclient: dump: 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3e5800f750 con 0x7f3e6409b1c0 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e6359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3e48006590 con 0x7f3e48002830 2024-09-16T06:53:12.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3e58043e80 con 0x7f3e6409b1c0 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065241336 0 0) 0x7f3e48006590 con 0x7f3e48002830 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient(hunting): my global_id is 264336 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e6409bd10 con 0x7f3e48002830 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e62d9d700 10 monclient: get_auth_request con 0x7f3e44036410 auth_method 0 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2461707698 0 0) 0x7f3e6409bd10 con 0x7f3e48002830 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e48006590 con 0x7f3e48002830 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e540036a0 con 0x7f3e48002830 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient: dump: 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3099072106 0 0) 0x7f3e48006590 con 0x7f3e48002830 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e4800a190 con 0x7f3e48002830 2024-09-16T06:53:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3e540058e0 con 0x7f3e48002830 2024-09-16T06:53:12.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:12.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6789/0 conn(0x7f3e48002830 legacy=0x7f3e48004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.333+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6789/0 conn(0x7f3e48002830 legacy=0x7f3e48004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e6359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:12.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3e48009e30 con 0x7f3e48009150 2024-09-16T06:53:12.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742357859 0 0) 0x7f3e48009e30 con 0x7f3e48009150 2024-09-16T06:53:12.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 10 monclient(hunting): my global_id is 264339 2024-09-16T06:53:12.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.341+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e48006590 con 0x7f3e48009150 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 482309686 0 0) 0x7f3e48006590 con 0x7f3e48009150 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e48009e30 con 0x7f3e48009150 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3e54003d90 con 0x7f3e48009150 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 10 monclient: dump: 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.352 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.353 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.353 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.353 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.353 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 735556072 0 0) 0x7f3e48009e30 con 0x7f3e48009150 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3e4800c340 con 0x7f3e48009150 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1992022556) 0x7f3e54005a50 con 0x7f3e48009150 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:12.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6789/0 conn(0x7f3e48009150 legacy=0x7f3e480095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e52ffd700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6789/0 conn(0x7f3e48009150 legacy=0x7f3e480095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:12.356 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.357 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.358 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:12.359 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.360 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:12.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.362 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.364 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.365 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.366 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:12.367 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:12.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3e44036410 legacy=0x7f3e440388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 10 monclient: shutdown 2024-09-16T06:53:12.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 >> v1:172.21.15.76:6790/0 conn(0x7f3e6409b1c0 legacy=0x7f3e641cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e62d9d700 1 -- 172.21.15.76:0/4123546286 reap_dead start 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e62d9d700 1 -- 172.21.15.76:0/4123546286 reap_dead start 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 shutdown_connections 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 >> 172.21.15.76:0/4123546286 conn(0x7f3e640a0030 msgr2=0x7f3e64138070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 shutdown_connections 2024-09-16T06:53:12.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.345+0000 7f3e6359e700 1 -- 172.21.15.76:0/4123546286 wait complete. 2024-09-16T06:53:12.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:12.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:53:12.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:53:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.485+0000 7fef59982700 10 monclient: get_monmap_and_config 2024-09-16T06:53:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: monmap: 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:12.492006+0000 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:12.492006+0000 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:12.487 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 1 Processor -- start 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 1 -- start start 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: init 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient(hunting): picked mon.noname-b con 0x7fef54132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient(hunting): picked mon.noname-c con 0x7fef5406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient(hunting): picked mon.noname-a con 0x7fef5406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5406c280 con 0x7fef5406cf50 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5406bfa0 con 0x7fef5406d480 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5409ace0 con 0x7fef54132330 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient(hunting): authenticate will time out at 1739.625955s 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59181700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fef5406cf50 0x7fef5413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57096/0 (socket says 172.21.15.76:57096) 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef58980700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef54132330 0x7fef54132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55736/0 (socket says 172.21.15.76:55736) 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59181700 1 -- 172.21.15.76:0/3355006374 learned_addr learned my addr 172.21.15.76:0/3355006374 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016604607 0 0) 0x7fef5406c280 con 0x7fef5406cf50 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): my global_id is 264342 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef3c0035d0 con 0x7fef5406cf50 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228054071 0 0) 0x7fef5409ace0 con 0x7fef54132330 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): my global_id is 244553 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5406c280 con 0x7fef54132330 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287664285 0 0) 0x7fef5406bfa0 con 0x7fef5406d480 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient(hunting): my global_id is 284104 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5409ace0 con 0x7fef5406d480 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618934887 0 0) 0x7fef3c0035d0 con 0x7fef5406cf50 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 >> v1:172.21.15.76:6790/0 conn(0x7fef54132330 legacy=0x7fef54132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 >> v1:172.21.15.189:6789/0 conn(0x7fef5406d480 legacy=0x7fef5413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 monclient: found mon.noname-a 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5409ae70 con 0x7fef5406cf50 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c001990 con 0x7fef5406cf50 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 5 monclient: authenticate success, global_id 264342 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: dump: 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef4c001bb0 con 0x7fef5406cf50 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 1 -- 172.21.15.76:0/3355006374 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c001da0 con 0x7fef5406cf50 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef59982700 4 set_mon_vals no callback set 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.489+0000 7fef537fe700 10 monclient: dump: 2024-09-16T06:53:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: shutdown 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- 172.21.15.76:0/3355006374 >> v1:172.21.15.76:6789/0 conn(0x7fef5406cf50 legacy=0x7fef5413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- 172.21.15.76:0/3355006374 shutdown_connections 2024-09-16T06:53:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- 172.21.15.76:0/3355006374 >> 172.21.15.76:0/3355006374 conn(0x7fef540a41c0 msgr2=0x7fef540a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- 172.21.15.76:0/3355006374 shutdown_connections 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- 172.21.15.76:0/3355006374 wait complete. 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: monmap: 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 Processor -- start 2024-09-16T06:53:12.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- start start 2024-09-16T06:53:12.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: init 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient(hunting): picked mon.noname-c con 0x7fef5406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient(hunting): picked mon.noname-b con 0x7fef5406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient(hunting): picked mon.noname-a con 0x7fef54132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef54131800 con 0x7fef54132330 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef541332e0 con 0x7fef5406d480 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef541312d0 con 0x7fef5406cf50 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef59982700 10 monclient(hunting): authenticate will time out at 1739.630863s 2024-09-16T06:53:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef58980700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef5406cf50 0x7fef5412d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55750/0 (socket says 172.21.15.76:55750) 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef58980700 1 -- 172.21.15.76:0/2416770114 learned_addr learned my addr 172.21.15.76:0/2416770114 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65577142 0 0) 0x7fef54131800 con 0x7fef54132330 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): my global_id is 264345 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef400035d0 con 0x7fef54132330 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557559782 0 0) 0x7fef541312d0 con 0x7fef5406cf50 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): my global_id is 244556 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54131800 con 0x7fef5406cf50 2024-09-16T06:53:12.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115476503 0 0) 0x7fef541332e0 con 0x7fef5406d480 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): my global_id is 284107 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef541312d0 con 0x7fef5406d480 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3809994113 0 0) 0x7fef400035d0 con 0x7fef54132330 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef541332e0 con 0x7fef54132330 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef4c0012a0 con 0x7fef54132330 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): dump: 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.507 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654066834 0 0) 0x7fef54131800 con 0x7fef5406cf50 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef400035d0 con 0x7fef5406cf50 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef440018f0 con 0x7fef5406cf50 2024-09-16T06:53:12.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.493+0000 7fef517fa700 10 monclient(hunting): dump: 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941326851 0 0) 0x7fef541312d0 con 0x7fef5406d480 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef54131800 con 0x7fef5406d480 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1787859865 0 0) 0x7fef400035d0 con 0x7fef5406cf50 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.189:6789/0 conn(0x7fef5406d480 legacy=0x7fef5412dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6789/0 conn(0x7fef54132330 legacy=0x7fef5412e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 monclient: found mon.c 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5409e8b0 con 0x7fef5406cf50 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 5 monclient: authenticate success, global_id 244556 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 10 monclient: _renew_subs 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5413e060 con 0x7fef5406cf50 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 10 monclient: _renew_subs 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5413ea00 con 0x7fef5406cf50 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef44004890 con 0x7fef5406cf50 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef44004ba0 con 0x7fef5406cf50 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: dump: 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.511 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.512 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef52ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef34006590 con 0x7fef34002830 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828053341 0 0) 0x7fef34006590 con 0x7fef34002830 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient(hunting): my global_id is 244559 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef400035d0 con 0x7fef34002830 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fef4400f610 con 0x7fef5406cf50 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fef44043e80 con 0x7fef5406cf50 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1745503863 0 0) 0x7fef400035d0 con 0x7fef34002830 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef34006590 con 0x7fef34002830 2024-09-16T06:53:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef480036c0 con 0x7fef34002830 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: dump: 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1370512740 0 0) 0x7fef34006590 con 0x7fef34002830 2024-09-16T06:53:12.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef59181700 10 monclient: get_auth_request con 0x7fef40036410 auth_method 0 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef3400a190 con 0x7fef34002830 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fef48005980 con 0x7fef34002830 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6790/0 conn(0x7fef34002830 legacy=0x7fef34004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.497+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6790/0 conn(0x7fef34002830 legacy=0x7fef34004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.505+0000 7fef59982700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:12.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef34009e30 con 0x7fef34009150 2024-09-16T06:53:12.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906412592 0 0) 0x7fef34009e30 con 0x7fef34009150 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient(hunting): my global_id is 244562 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef34006590 con 0x7fef34009150 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 971242342 0 0) 0x7fef34006590 con 0x7fef34009150 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef34009e30 con 0x7fef34009150 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef48003f40 con 0x7fef34009150 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient: dump: 2024-09-16T06:53:12.517 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1298898440 0 0) 0x7fef34009e30 con 0x7fef34009150 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef3400c340 con 0x7fef34009150 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 1523304312) 0x7fef48005af0 con 0x7fef34009150 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:12.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6790/0 conn(0x7fef34009150 legacy=0x7fef340095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef517fa700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6790/0 conn(0x7fef34009150 legacy=0x7fef340095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:12.519 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.520 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.521 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.522 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.523 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.524 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.525 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.526 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:12.527 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.528 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:53:12.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:53:12.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:12.531 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:12.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.189:6812/3098810674 conn(0x7fef40036410 legacy=0x7fef400388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 10 monclient: shutdown 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 >> v1:172.21.15.76:6790/0 conn(0x7fef5406cf50 legacy=0x7fef5412d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59181700 1 -- 172.21.15.76:0/2416770114 reap_dead start 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59181700 1 -- 172.21.15.76:0/2416770114 reap_dead start 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 shutdown_connections 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 >> 172.21.15.76:0/2416770114 conn(0x7fef540a41c0 msgr2=0x7fef5409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 shutdown_connections 2024-09-16T06:53:12.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.509+0000 7fef59982700 1 -- 172.21.15.76:0/2416770114 wait complete. 2024-09-16T06:53:12.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:12.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:53:12.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:53:12.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: get_monmap_and_config 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: monmap: 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:12.658085+0000 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:12.658085+0000 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:12.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:12.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 1 Processor -- start 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 1 -- start start 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: init 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-b con 0x7f5c8009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-c con 0x7f5c8009d520 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-a con 0x7f5c8009fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8009b520 con 0x7f5c8009fd20 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c80071f30 con 0x7f5c8009d520 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8009b0e0 con 0x7f5c8009f230 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c875d3700 10 monclient(hunting): authenticate will time out at 1739.791585s 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c85b70700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5c8009fd20 0x7f5c8013ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57108/0 (socket says 172.21.15.76:57108) 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.653+0000 7f5c85b70700 1 -- 172.21.15.76:0/3026987024 learned_addr learned my addr 172.21.15.76:0/3026987024 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600255905 0 0) 0x7f5c8009b520 con 0x7f5c8009fd20 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): my global_id is 264348 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c0035d0 con 0x7f5c8009fd20 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205092119 0 0) 0x7f5c8009b0e0 con 0x7f5c8009f230 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): my global_id is 244565 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8009b520 con 0x7f5c8009f230 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500742472 0 0) 0x7f5c80071f30 con 0x7f5c8009d520 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient(hunting): my global_id is 284110 2024-09-16T06:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8009b0e0 con 0x7f5c8009d520 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2690123065 0 0) 0x7f5c6c0035d0 con 0x7f5c8009fd20 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 >> v1:172.21.15.76:6790/0 conn(0x7f5c8009f230 legacy=0x7f5c8009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 >> v1:172.21.15.189:6789/0 conn(0x7f5c8009d520 legacy=0x7f5c80137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 monclient: found mon.noname-a 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8009c7d0 con 0x7f5c8009fd20 2024-09-16T06:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c78001990 con 0x7f5c8009fd20 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: dump: 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 5 monclient: authenticate success, global_id 264348 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c78001bb0 con 0x7f5c8009fd20 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 4 set_mon_vals no callback set 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 1 -- 172.21.15.76:0/3026987024 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c78001f40 con 0x7f5c8009fd20 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c77fff700 10 monclient: dump: 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 10 monclient: shutdown 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- 172.21.15.76:0/3026987024 >> v1:172.21.15.76:6789/0 conn(0x7f5c8009fd20 legacy=0x7f5c8013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- 172.21.15.76:0/3026987024 shutdown_connections 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- 172.21.15.76:0/3026987024 >> 172.21.15.76:0/3026987024 conn(0x7f5c800a3e50 msgr2=0x7f5c800a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- 172.21.15.76:0/3026987024 shutdown_connections 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- 172.21.15.76:0/3026987024 wait complete. 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 10 monclient: monmap: 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.665 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:12.666 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:12.666 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 Processor -- start 2024-09-16T06:53:12.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 1 -- start start 2024-09-16T06:53:12.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.657+0000 7f5c875d3700 10 monclient: init 2024-09-16T06:53:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-b con 0x7f5c8009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-c con 0x7f5c8009fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient(hunting): picked mon.noname-a con 0x7f5c80139ce0 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8011a570 con 0x7f5c80139ce0 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c80072450 con 0x7f5c8009f230 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8006c910 con 0x7f5c8009fd20 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient(hunting): authenticate will time out at 1739.796375s 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c85b70700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5c80139ce0 0x7f5c801d0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57110/0 (socket says 172.21.15.76:57110) 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c84b6e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5c8009fd20 0x7f5c801395d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:55794/0 (socket says 172.21.15.76:55794) 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c85b70700 1 -- 172.21.15.76:0/2706833403 learned_addr learned my addr 172.21.15.76:0/2706833403 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795065362 0 0) 0x7f5c8011a570 con 0x7f5c80139ce0 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): my global_id is 264351 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c640035d0 con 0x7f5c80139ce0 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653874841 0 0) 0x7f5c8006c910 con 0x7f5c8009fd20 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): my global_id is 244568 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8011a570 con 0x7f5c8009fd20 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153448473 0 0) 0x7f5c80072450 con 0x7f5c8009f230 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): my global_id is 284113 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8006c910 con 0x7f5c8009f230 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1471388971 0 0) 0x7f5c640035d0 con 0x7f5c80139ce0 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c80072450 con 0x7f5c80139ce0 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c78001890 con 0x7f5c80139ce0 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.673 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333351225 0 0) 0x7f5c8011a570 con 0x7f5c8009fd20 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c640035d0 con 0x7f5c8009fd20 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c700018d0 con 0x7f5c8009fd20 2024-09-16T06:53:12.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4019313996 0 0) 0x7f5c80072450 con 0x7f5c80139ce0 2024-09-16T06:53:12.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.76:6790/0 conn(0x7f5c8009fd20 legacy=0x7f5c801395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6789/0 conn(0x7f5c8009f230 legacy=0x7f5c80138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 monclient: found mon.a 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8011b4e0 con 0x7f5c80139ce0 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 5 monclient: authenticate success, global_id 264351 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _renew_subs 2024-09-16T06:53:12.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8011ac40 con 0x7f5c80139ce0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _renew_subs 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c80119e70 con 0x7f5c80139ce0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c78001570 con 0x7f5c80139ce0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c78005060 con 0x7f5c80139ce0 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c777fe700 4 set_mon_vals no callback set 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 10 monclient: dump: 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c60006590 con 0x7f5c60002830 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.661+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5c7800fad0 con 0x7f5c80139ce0 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5c78044220 con 0x7f5c80139ce0 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180579343 0 0) 0x7f5c60006590 con 0x7f5c60002830 2024-09-16T06:53:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient(hunting): my global_id is 284116 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80072450 con 0x7f5c60002830 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c84b6e700 10 monclient: get_auth_request con 0x7f5c64036430 auth_method 0 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 464756 0 0) 0x7f5c80072450 con 0x7f5c60002830 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c60006590 con 0x7f5c60002830 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c7c0017d0 con 0x7f5c60002830 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient: dump: 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 752626757 0 0) 0x7f5c60006590 con 0x7f5c60002830 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c6000a190 con 0x7f5c60002830 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5c7c001b50 con 0x7f5c60002830 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6789/0 conn(0x7f5c60002830 legacy=0x7f5c60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.665+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6789/0 conn(0x7f5c60002830 legacy=0x7f5c60004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c875d3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c60009e50 con 0x7f5c60009170 2024-09-16T06:53:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16690303 0 0) 0x7f5c60009e50 con 0x7f5c60009170 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient(hunting): my global_id is 284119 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c60006590 con 0x7f5c60009170 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3517704290 0 0) 0x7f5c60006590 con 0x7f5c60009170 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c60009e50 con 0x7f5c60009170 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c7c0038c0 con 0x7f5c60009170 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient: dump: 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 656162412 0 0) 0x7f5c60009e50 con 0x7f5c60009170 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c6000c340 con 0x7f5c60009170 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 97933210) 0x7f5c7c004f30 con 0x7f5c60009170 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6789/0 conn(0x7f5c60009170 legacy=0x7f5c60009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.677+0000 7f5c75ffb700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6789/0 conn(0x7f5c60009170 legacy=0x7f5c60009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:12.696 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5c64036430 legacy=0x7f5c640388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 10 monclient: shutdown 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 >> v1:172.21.15.76:6789/0 conn(0x7f5c80139ce0 legacy=0x7f5c801d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c85b70700 1 -- 172.21.15.76:0/2706833403 reap_dead start 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 shutdown_connections 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 >> 172.21.15.76:0/2706833403 conn(0x7f5c800a3e50 msgr2=0x7f5c80070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 shutdown_connections 2024-09-16T06:53:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.681+0000 7f5c875d3700 1 -- 172.21.15.76:0/2706833403 wait complete. 2024-09-16T06:53:12.698 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:12.698 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:53:12.698 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.817+0000 7f6c16ba7700 10 monclient: get_monmap_and_config 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.817+0000 7f6c16ba7700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.817+0000 7f6c16ba7700 10 monclient: monmap: 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:12.823430+0000 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:12.823430+0000 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.818 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:12.819 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:12.819 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 Processor -- start 2024-09-16T06:53:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- start start 2024-09-16T06:53:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: init 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-b con 0x7f6c10097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-a con 0x7f6c10099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-c con 0x7f6c10139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c10098990 con 0x7f6c10099090 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c10097f40 con 0x7f6c10139500 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1009d610 con 0x7f6c10097b60 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient(hunting): authenticate will time out at 1739.956702s 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c153a4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6c10099090 0x7f6c1009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57116/0 (socket says 172.21.15.76:57116) 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c153a4700 1 -- 172.21.15.76:0/1211051197 learned_addr learned my addr 172.21.15.76:0/1211051197 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680617982 0 0) 0x7f6c10098990 con 0x7f6c10099090 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): my global_id is 264354 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bfc0035d0 con 0x7f6c10099090 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151046260 0 0) 0x7f6c1009d610 con 0x7f6c10097b60 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient(hunting): my global_id is 244571 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c10098990 con 0x7f6c10097b60 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181073794 0 0) 0x7f6bfc0035d0 con 0x7f6c10099090 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 >> v1:172.21.15.76:6790/0 conn(0x7f6c10097b60 legacy=0x7f6c1009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 >> v1:172.21.15.189:6789/0 conn(0x7f6c10139500 legacy=0x7f6c1013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 monclient: found mon.noname-a 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c100986b0 con 0x7f6c10099090 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c0c001610 con 0x7f6c10099090 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: dump: 2024-09-16T06:53:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 5 monclient: authenticate success, global_id 264354 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c0c0018f0 con 0x7f6c10099090 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 1 -- 172.21.15.76:0/1211051197 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c0c001c80 con 0x7f6c10099090 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 4 set_mon_vals no callback set 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c14ba3700 10 monclient: dump: 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: shutdown 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- 172.21.15.76:0/1211051197 >> v1:172.21.15.76:6789/0 conn(0x7f6c10099090 legacy=0x7f6c1009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- 172.21.15.76:0/1211051197 shutdown_connections 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- 172.21.15.76:0/1211051197 >> 172.21.15.76:0/1211051197 conn(0x7f6c100a0030 msgr2=0x7f6c100a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- 172.21.15.76:0/1211051197 shutdown_connections 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- 172.21.15.76:0/1211051197 wait complete. 2024-09-16T06:53:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: build_initial_monmap 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: monmap: 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 Processor -- start 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 1 -- start start 2024-09-16T06:53:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.821+0000 7f6c16ba7700 10 monclient: init 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-b con 0x7f6c10097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-a con 0x7f6c10099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient(hunting): picked mon.noname-c con 0x7f6c10139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1008e7d0 con 0x7f6c10099090 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c100a2bb0 con 0x7f6c10097b60 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1008f410 con 0x7f6c10139500 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient(hunting): authenticate will time out at 1739.960542s 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c153a4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6c10099090 0x7f6c101ca690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57120/0 (socket says 172.21.15.76:57120) 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c153a4700 1 -- 172.21.15.76:0/2894405180 learned_addr learned my addr 172.21.15.76:0/2894405180 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852763528 0 0) 0x7f6c1008f410 con 0x7f6c10139500 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): my global_id is 244574 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf40035d0 con 0x7f6c10139500 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982335728 0 0) 0x7f6c1008e7d0 con 0x7f6c10099090 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): my global_id is 264357 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1008f410 con 0x7f6c10099090 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190863542 0 0) 0x7f6c100a2bb0 con 0x7f6c10097b60 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): my global_id is 284125 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1008e7d0 con 0x7f6c10097b60 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694525146 0 0) 0x7f6c1008f410 con 0x7f6c10099090 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c100a2bb0 con 0x7f6c10099090 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c0c001430 con 0x7f6c10099090 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): dump: 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1424587485 0 0) 0x7f6bf40035d0 con 0x7f6c10139500 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c1008f410 con 0x7f6c10139500 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c08001b20 con 0x7f6c10139500 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient(hunting): dump: 2024-09-16T06:53:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2682894422 0 0) 0x7f6c100a2bb0 con 0x7f6c10099090 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 >> v1:172.21.15.76:6790/0 conn(0x7f6c10139500 legacy=0x7f6c101cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 >> v1:172.21.15.189:6789/0 conn(0x7f6c10097b60 legacy=0x7f6c10096110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: _finish_hunting 0 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 monclient: found mon.a 2024-09-16T06:53:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10090ba0 con 0x7f6c10099090 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: _finish_auth 0 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: _check_auth_tickets 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 5 monclient: authenticate success, global_id 264357 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _renew_subs 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1009c400 con 0x7f6c10099090 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _renew_subs 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c100908c0 con 0x7f6c10099090 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c0c004ca0 con 0x7f6c10099090 2024-09-16T06:53:12.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c07fff700 4 set_mon_vals no callback set 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6c0c004fb0 con 0x7f6c10099090 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 10 monclient: dump: 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:12.840 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6c0c00fa20 con 0x7f6c10099090 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1008f120 con 0x7f6c10099090 2024-09-16T06:53:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6c0c044050 con 0x7f6c10099090 2024-09-16T06:53:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.825+0000 7f6c15ba5700 10 monclient: get_auth_request con 0x7f6bf4036490 auth_method 0 2024-09-16T06:53:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.833+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6c0c014820 con 0x7f6c10099090 2024-09-16T06:53:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.833+0000 7f6c067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:12.833+0000 7f6c067fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.065+0000 7f6c16ba7700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:53:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.065+0000 7f6c16ba7700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:53:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.065+0000 7f6c16ba7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:13.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.065+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6c100672a0 con 0x7f6c10099090 2024-09-16T06:53:13.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=13,logm=43} crc {auth=1675136080,config=1097430493,health=923807535,kv=1001081567,logm=317926613}) 2024-09-16T06:53:13.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=13,logm=46} crc {auth=1068154807,config=1097430493,health=923807535,kv=1001081567,logm=3879226293}) 2024-09-16T06:53:13.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=13,logm=43} crc {auth=1675136080,config=1097430493,health=923807535,kv=1001081567,logm=317926613}) 2024-09-16T06:53:13.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=13,logm=48} crc {auth=2569185132,config=1097430493,health=923807535,kv=1001081567,logm=231516442}) 2024-09-16T06:53:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6c067fc700 1 -- 172.21.15.76:0/2894405180 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f6c0c012ec0 con 0x7f6c10099090 2024-09-16T06:53:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6c067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:53:13.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.069+0000 7f6c067fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6bf4036490 legacy=0x7f6bf4038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 10 monclient: shutdown 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 >> v1:172.21.15.76:6789/0 conn(0x7f6c10099090 legacy=0x7f6c101ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 shutdown_connections 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 >> 172.21.15.76:0/2894405180 conn(0x7f6c100a0030 msgr2=0x7f6c100a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 shutdown_connections 2024-09-16T06:53:13.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6c16ba7700 1 -- 172.21.15.76:0/2894405180 wait complete. 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3504429458}) 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419747285}) 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3504429458}) 2024-09-16T06:53:13.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.073+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191132404}) 2024-09-16T06:53:13.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444665601}) 2024-09-16T06:53:13.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345196006}) 2024-09-16T06:53:13.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444665601}) 2024-09-16T06:53:13.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.077+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780282491}) 2024-09-16T06:53:13.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146339758}) 2024-09-16T06:53:13.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553664121}) 2024-09-16T06:53:13.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146339758}) 2024-09-16T06:53:13.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2151408916}) 2024-09-16T06:53:13.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20303216}) 2024-09-16T06:53:13.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=587567814}) 2024-09-16T06:53:13.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20303216}) 2024-09-16T06:53:13.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.081+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435805896}) 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769889810}) 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743552176}) 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769889810}) 2024-09-16T06:53:13.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352611847}) 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047023021}) 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114627986}) 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047023021}) 2024-09-16T06:53:13.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943487170}) 2024-09-16T06:53:13.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226883956}) 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=222024267}) 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226883956}) 2024-09-16T06:53:13.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.085+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623468509}) 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851305682}) 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682067163}) 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851305682}) 2024-09-16T06:53:13.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824335645}) 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318060006}) 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104330780}) 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318060006}) 2024-09-16T06:53:13.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=460244526}) 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3743866682}) 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202217316}) 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3743866682}) 2024-09-16T06:53:13.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070501921}) 2024-09-16T06:53:13.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184155168}) 2024-09-16T06:53:13.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720411847}) 2024-09-16T06:53:13.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=184155168}) 2024-09-16T06:53:13.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.089+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704475284}) 2024-09-16T06:53:13.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695149333}) 2024-09-16T06:53:13.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259303650}) 2024-09-16T06:53:13.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695149333}) 2024-09-16T06:53:13.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105598934}) 2024-09-16T06:53:13.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040495409}) 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1681520877}) 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040495409}) 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=600940390}) 2024-09-16T06:53:13.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.094 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387585857}) 2024-09-16T06:53:13.094 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=317596600}) 2024-09-16T06:53:13.094 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.094 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387585857}) 2024-09-16T06:53:13.094 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.093+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523757135}) 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097810540}) 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870723126}) 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097810540}) 2024-09-16T06:53:13.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551194835}) 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865548659}) 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167910588}) 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865548659}) 2024-09-16T06:53:13.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2712733155}) 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900501638}) 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742517889}) 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900501638}) 2024-09-16T06:53:13.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.097+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789089969}) 2024-09-16T06:53:13.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2521221164}) 2024-09-16T06:53:13.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273474568}) 2024-09-16T06:53:13.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2521221164}) 2024-09-16T06:53:13.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323084992}) 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614897218}) 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440630620}) 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614897218}) 2024-09-16T06:53:13.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343293673}) 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673151038}) 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60315677}) 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673151038}) 2024-09-16T06:53:13.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281834709}) 2024-09-16T06:53:13.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2709020690}) 2024-09-16T06:53:13.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393333654}) 2024-09-16T06:53:13.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.101+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2709020690}) 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539601570}) 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669481887}) 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416637166}) 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669481887}) 2024-09-16T06:53:13.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845500798}) 2024-09-16T06:53:13.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899926235}) 2024-09-16T06:53:13.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725393650}) 2024-09-16T06:53:13.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899926235}) 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90118214}) 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3422719287}) 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959204348}) 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3422719287}) 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237733532}) 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479254352}) 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146417831}) 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.105+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479254352}) 2024-09-16T06:53:13.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400056982}) 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143398853}) 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238881353}) 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143398853}) 2024-09-16T06:53:13.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670797201}) 2024-09-16T06:53:13.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952133427}) 2024-09-16T06:53:13.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246357800}) 2024-09-16T06:53:13.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952133427}) 2024-09-16T06:53:13.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3524239686}) 2024-09-16T06:53:13.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584484721}) 2024-09-16T06:53:13.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216064973}) 2024-09-16T06:53:13.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584484721}) 2024-09-16T06:53:13.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.109+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071633857}) 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37607612}) 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079960249}) 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37607612}) 2024-09-16T06:53:13.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=868941193}) 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399847555}) 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793960044}) 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399847555}) 2024-09-16T06:53:13.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023715063}) 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567616713}) 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075571678}) 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567616713}) 2024-09-16T06:53:13.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122763986}) 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747239603}) 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584570202}) 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1747239603}) 2024-09-16T06:53:13.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.113+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250143094}) 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818676030}) 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823805695}) 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818676030}) 2024-09-16T06:53:13.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3886203846}) 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252574584}) 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822741455}) 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252574584}) 2024-09-16T06:53:13.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583064225}) 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421736470}) 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031553355}) 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421736470}) 2024-09-16T06:53:13.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2470496083}) 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826466677}) 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007575568}) 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826466677}) 2024-09-16T06:53:13.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.117+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417506960}) 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190790974}) 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688955100}) 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190790974}) 2024-09-16T06:53:13.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671788327}) 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143462901}) 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265372637}) 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143462901}) 2024-09-16T06:53:13.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=271599311}) 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296356036}) 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472176258}) 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296356036}) 2024-09-16T06:53:13.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590557578}) 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=979439891}) 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062442812}) 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=979439891}) 2024-09-16T06:53:13.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.121+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=149530514}) 2024-09-16T06:53:13.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621007739}) 2024-09-16T06:53:13.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997269098}) 2024-09-16T06:53:13.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621007739}) 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465530008}) 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563556422}) 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695785451}) 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563556422}) 2024-09-16T06:53:13.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615486289}) 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901031628}) 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905349719}) 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901031628}) 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116273443}) 2024-09-16T06:53:13.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2746872550}) 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12554142}) 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2746872550}) 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866946493}) 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954569487}) 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138067085}) 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954569487}) 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.125+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2851827963}) 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063058363}) 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149251936}) 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063058363}) 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469723434}) 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503900801}) 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2669635088}) 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503900801}) 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954234523}) 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160885470}) 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643546088}) 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160885470}) 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759036509}) 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419789062}) 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619540208}) 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419789062}) 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.129+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727068128}) 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935604289}) 2024-09-16T06:53:13.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742120862}) 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935604289}) 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170666737}) 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585875858}) 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250625894}) 2024-09-16T06:53:13.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585875858}) 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248713054}) 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641411356}) 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308521589}) 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641411356}) 2024-09-16T06:53:13.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837572553}) 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611907969}) 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052616071}) 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611907969}) 2024-09-16T06:53:13.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.133+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800396778}) 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211991113}) 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547171461}) 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211991113}) 2024-09-16T06:53:13.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067989684}) 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809722715}) 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796976795}) 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809722715}) 2024-09-16T06:53:13.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089711743}) 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814416221}) 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932770078}) 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814416221}) 2024-09-16T06:53:13.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020536168}) 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283313982}) 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683286469}) 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283313982}) 2024-09-16T06:53:13.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.137+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546503423}) 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535830982}) 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637750361}) 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535830982}) 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814934487}) 2024-09-16T06:53:13.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835899161}) 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581298662}) 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835899161}) 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092949565}) 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861713895}) 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405094799}) 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861713895}) 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2852875548}) 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695284286}) 2024-09-16T06:53:13.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408186032}) 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695284286}) 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.141+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935091599}) 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037390085}) 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649615052}) 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037390085}) 2024-09-16T06:53:13.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778092615}) 2024-09-16T06:53:13.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288163789}) 2024-09-16T06:53:13.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103997850}) 2024-09-16T06:53:13.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288163789}) 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1858822863}) 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003039454}) 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854180018}) 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003039454}) 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123253074}) 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=507671853}) 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773186189}) 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=507671853}) 2024-09-16T06:53:13.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.145+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273446920}) 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340703747}) 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719582863}) 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340703747}) 2024-09-16T06:53:13.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371185886}) 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215279399}) 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251149776}) 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215279399}) 2024-09-16T06:53:13.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165136607}) 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613940288}) 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583905907}) 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613940288}) 2024-09-16T06:53:13.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=127815185}) 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448108957}) 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3719247204}) 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448108957}) 2024-09-16T06:53:13.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366006209}) 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610683308}) 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502731041}) 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610683308}) 2024-09-16T06:53:13.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.149+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348649090}) 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571071944}) 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912205016}) 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571071944}) 2024-09-16T06:53:13.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=764045951}) 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232789796}) 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701581697}) 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232789796}) 2024-09-16T06:53:13.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524827322}) 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402669155}) 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476955455}) 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402669155}) 2024-09-16T06:53:13.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934175835}) 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3004884017}) 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617467040}) 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.153+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3004884017}) 2024-09-16T06:53:13.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889155742}) 2024-09-16T06:53:13.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864034575}) 2024-09-16T06:53:13.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912040631}) 2024-09-16T06:53:13.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864034575}) 2024-09-16T06:53:13.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607543038}) 2024-09-16T06:53:13.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244051609}) 2024-09-16T06:53:13.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421489581}) 2024-09-16T06:53:13.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244051609}) 2024-09-16T06:53:13.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133090224}) 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841465875}) 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884106924}) 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841465875}) 2024-09-16T06:53:13.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.157+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066689876}) 2024-09-16T06:53:13.159 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.159 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100375508}) 2024-09-16T06:53:13.159 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272500401}) 2024-09-16T06:53:13.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100375508}) 2024-09-16T06:53:13.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1303380265}) 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90601105}) 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756440218}) 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90601105}) 2024-09-16T06:53:13.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.161+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489008062}) 2024-09-16T06:53:13.162 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.162 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3526045766,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2005159828}) 2024-09-16T06:53:13.162 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2487258503,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=479929941}) 2024-09-16T06:53:13.162 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=3526045766,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2005159828}) 2024-09-16T06:53:13.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.165+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2596672845,mdsmap=1531151339,mgr=3454917385,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1863644171}) 2024-09-16T06:53:13.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=9,mon_config_key=20,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=2983526587,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2927302349}) 2024-09-16T06:53:13.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=3726517895,mon_config_key=1113509946,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2549239390}) 2024-09-16T06:53:13.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:13.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=9,mon_config_key=20,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=2983526587,mon_config_key=4279594037,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2927302349}) 2024-09-16T06:53:13.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:13.173+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=1473026123,mon_config_key=1714515979,monmap=1887442722,osd_metadata=2396766746,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=482556628}) 2024-09-16T06:53:14.086 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:53:14.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: get_monmap_and_config 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: monmap: 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:14.212387+0000 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:14.212387+0000 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:14.208 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 1 Processor -- start 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 1 -- start start 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: init 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-c con 0x7fc7a4132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-b con 0x7fc7a406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-a con 0x7fc7a406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a406c280 con 0x7fc7a406cf50 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a406bfa0 con 0x7fc7a4132330 2024-09-16T06:53:14.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a409ace0 con 0x7fc7a406d480 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient(hunting): authenticate will time out at 1741.346097s 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7aa9c2700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc7a406cf50 0x7fc7a413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59306/0 (socket says 172.21.15.76:59306) 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7aa9c2700 1 -- 172.21.15.76:0/3752298847 learned_addr learned my addr 172.21.15.76:0/3752298847 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615470242 0 0) 0x7fc7a406c280 con 0x7fc7a406cf50 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): my global_id is 264360 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7900035d0 con 0x7fc7a406cf50 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048697930 0 0) 0x7fc7a409ace0 con 0x7fc7a406d480 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): my global_id is 244577 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a406c280 con 0x7fc7a406d480 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187968844 0 0) 0x7fc7a406bfa0 con 0x7fc7a4132330 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient(hunting): my global_id is 284128 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a409ace0 con 0x7fc7a4132330 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2918734097 0 0) 0x7fc7a406c280 con 0x7fc7a406d480 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 >> v1:172.21.15.189:6789/0 conn(0x7fc7a4132330 legacy=0x7fc7a4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 >> v1:172.21.15.76:6789/0 conn(0x7fc7a406cf50 legacy=0x7fc7a413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 monclient: found mon.noname-b 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a409ae70 con 0x7fc7a406d480 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc78c001970 con 0x7fc7a406d480 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: dump: 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 5 monclient: authenticate success, global_id 244577 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc78c001c10 con 0x7fc7a406d480 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 1 -- 172.21.15.76:0/3752298847 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc78c001e40 con 0x7fc7a406d480 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7ab1c3700 4 set_mon_vals no callback set 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.209+0000 7fc7a91bf700 10 monclient: dump: 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.218 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 10 monclient: shutdown 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3752298847 >> v1:172.21.15.76:6790/0 conn(0x7fc7a406d480 legacy=0x7fc7a413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3752298847 shutdown_connections 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3752298847 >> 172.21.15.76:0/3752298847 conn(0x7fc7a40a41c0 msgr2=0x7fc7a40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3752298847 shutdown_connections 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3752298847 wait complete. 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 10 monclient: monmap: 2024-09-16T06:53:14.220 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:14.221 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 Processor -- start 2024-09-16T06:53:14.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 1 -- start start 2024-09-16T06:53:14.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.213+0000 7fc7ab1c3700 10 monclient: init 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-c con 0x7fc7a406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-b con 0x7fc7a406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient(hunting): picked mon.noname-a con 0x7fc7a4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a4130520 con 0x7fc7a4132330 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a409e8b0 con 0x7fc7a406d480 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a412f230 con 0x7fc7a406cf50 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient(hunting): authenticate will time out at 1741.352341s 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7aa1c1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc7a406cf50 0x7fc7a413e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47590/0 (socket says 172.21.15.76:47590) 2024-09-16T06:53:14.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7aa1c1700 1 -- 172.21.15.76:0/3609946175 learned_addr learned my addr 172.21.15.76:0/3609946175 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654485553 0 0) 0x7fc7a4130520 con 0x7fc7a4132330 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): my global_id is 264363 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7840035d0 con 0x7fc7a4132330 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414946196 0 0) 0x7fc7a412f230 con 0x7fc7a406cf50 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): my global_id is 244580 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a4130520 con 0x7fc7a406cf50 2024-09-16T06:53:14.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460354019 0 0) 0x7fc7a409e8b0 con 0x7fc7a406d480 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): my global_id is 284131 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a412f230 con 0x7fc7a406d480 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712619047 0 0) 0x7fc7840035d0 con 0x7fc7a4132330 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a409e8b0 con 0x7fc7a4132330 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748509540 0 0) 0x7fc7a4130520 con 0x7fc7a406cf50 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7840035d0 con 0x7fc7a406cf50 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc7940012b0 con 0x7fc7a4132330 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:14.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc79c0018f0 con 0x7fc7a406cf50 2024-09-16T06:53:14.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 435251852 0 0) 0x7fc7a412f230 con 0x7fc7a406d480 2024-09-16T06:53:14.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a4130520 con 0x7fc7a406d480 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc78c001d80 con 0x7fc7a406d480 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.231 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 719013518 0 0) 0x7fc7840035d0 con 0x7fc7a406cf50 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 >> v1:172.21.15.189:6789/0 conn(0x7fc7a406d480 legacy=0x7fc7a41ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 >> v1:172.21.15.76:6789/0 conn(0x7fc7a4132330 legacy=0x7fc7a41d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 monclient: found mon.c 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a4130ae0 con 0x7fc7a406cf50 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 5 monclient: authenticate success, global_id 244580 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _renew_subs 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a412f470 con 0x7fc7a406cf50 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _renew_subs 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a412ed80 con 0x7fc7a406cf50 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc79c001ac0 con 0x7fc7a406cf50 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc79c003970 con 0x7fc7a406cf50 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a3fff700 4 set_mon_vals no callback set 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 10 monclient: dump: 2024-09-16T06:53:14.233 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc79c003ad0 con 0x7fc7a406cf50 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc788005100 con 0x7fc7a406cf50 2024-09-16T06:53:14.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.217+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc79c043c70 con 0x7fc7a406cf50 2024-09-16T06:53:14.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.221+0000 7fc7a99c0700 10 monclient: get_auth_request con 0x7fc784036560 auth_method 0 2024-09-16T06:53:14.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.225+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc79c004280 con 0x7fc7a406cf50 2024-09-16T06:53:14.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.225+0000 7fc7a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.225+0000 7fc7a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:14.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc788005930 con 0x7fc7a406cf50 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7a2ffd700 1 -- 172.21.15.76:0/3609946175 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3787796381) 0x7fc79c0142e0 con 0x7fc7a406cf50 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:14.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:14.461 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:14.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc784036560 legacy=0x7fc784038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 10 monclient: shutdown 2024-09-16T06:53:14.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.461+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 >> v1:172.21.15.76:6790/0 conn(0x7fc7a406cf50 legacy=0x7fc7a413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.465+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 shutdown_connections 2024-09-16T06:53:14.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.465+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 >> 172.21.15.76:0/3609946175 conn(0x7fc7a40a41c0 msgr2=0x7fc7a409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.465+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 shutdown_connections 2024-09-16T06:53:14.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.465+0000 7fc7ab1c3700 1 -- 172.21.15.76:0/3609946175 wait complete. 2024-09-16T06:53:14.474 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:53:14.475 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:53:14.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:53:14.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:53:14.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 10 monclient: get_monmap_and_config 2024-09-16T06:53:14.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 10 monclient: monmap: 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:14.594734+0000 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:14.594734+0000 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:14.590 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 1 Processor -- start 2024-09-16T06:53:14.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 1 -- start start 2024-09-16T06:53:14.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.589+0000 7f82c0b6c700 10 monclient: init 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-b con 0x7f82bc132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-c con 0x7f82bc06d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-a con 0x7f82bc06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc06c280 con 0x7f82bc06cf50 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc06bfa0 con 0x7f82bc06d480 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc09ace0 con 0x7f82bc132330 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient(hunting): authenticate will time out at 1741.728246s 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82bb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f82bc132330 0x7f82bc132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47602/0 (socket says 172.21.15.76:47602) 2024-09-16T06:53:14.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82bb7fe700 1 -- 172.21.15.76:0/1164007175 learned_addr learned my addr 172.21.15.76:0/1164007175 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415347465 0 0) 0x7f82bc09ace0 con 0x7f82bc132330 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): my global_id is 244583 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a80035d0 con 0x7f82bc132330 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832406980 0 0) 0x7f82bc06c280 con 0x7f82bc06cf50 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient(hunting): my global_id is 264366 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc09ace0 con 0x7f82bc06cf50 2024-09-16T06:53:14.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3904133958 0 0) 0x7f82bc09ace0 con 0x7f82bc06cf50 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 >> v1:172.21.15.76:6790/0 conn(0x7f82bc132330 legacy=0x7f82bc132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 >> v1:172.21.15.189:6789/0 conn(0x7f82bc06d480 legacy=0x7f82bc13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 monclient: found mon.noname-a 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc09ae70 con 0x7f82bc06cf50 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82b40018f0 con 0x7f82bc06cf50 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 5 monclient: authenticate success, global_id 264366 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: dump: 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.597 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b4001ba0 con 0x7f82bc06cf50 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 1 -- 172.21.15.76:0/1164007175 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82b4001dd0 con 0x7f82bc06cf50 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82baffd700 10 monclient: dump: 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.598 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 4 set_mon_vals no callback set 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: shutdown 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- 172.21.15.76:0/1164007175 >> v1:172.21.15.76:6789/0 conn(0x7f82bc06cf50 legacy=0x7f82bc13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- 172.21.15.76:0/1164007175 shutdown_connections 2024-09-16T06:53:14.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- 172.21.15.76:0/1164007175 >> 172.21.15.76:0/1164007175 conn(0x7f82bc0a41c0 msgr2=0x7f82bc0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- 172.21.15.76:0/1164007175 shutdown_connections 2024-09-16T06:53:14.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 1 -- 172.21.15.76:0/1164007175 wait complete. 2024-09-16T06:53:14.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.593+0000 7f82c0b6c700 10 monclient: monmap: 2024-09-16T06:53:14.605 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 Processor -- start 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- start start 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: init 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-b con 0x7f82bc06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-a con 0x7f82bc06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient(hunting): picked mon.noname-c con 0x7f82bc132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc1302d0 con 0x7f82bc06d480 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc12df90 con 0x7f82bc06cf50 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc131500 con 0x7f82bc132330 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f82bc06d480 0x7f82bc12d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59330/0 (socket says 172.21.15.76:59330) 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b3fff700 1 -- 172.21.15.76:0/4277776233 learned_addr learned my addr 172.21.15.76:0/4277776233 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient(hunting): authenticate will time out at 1741.734129s 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170265748 0 0) 0x7f82bc1302d0 con 0x7f82bc06d480 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): my global_id is 264369 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0035d0 con 0x7f82bc06d480 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031700952 0 0) 0x7f82bc131500 con 0x7f82bc132330 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): my global_id is 244586 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc1302d0 con 0x7f82bc132330 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899193923 0 0) 0x7f82bc12df90 con 0x7f82bc06cf50 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): my global_id is 284137 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc131500 con 0x7f82bc06cf50 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2011336170 0 0) 0x7f82bc1302d0 con 0x7f82bc132330 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc12df90 con 0x7f82bc132330 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3421540658 0 0) 0x7f829c0035d0 con 0x7f82bc06d480 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc1302d0 con 0x7f82bc06d480 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82b4001d10 con 0x7f82bc132330 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): dump: 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.609 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82a4001910 con 0x7f82bc06d480 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): dump: 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.610 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854268730 0 0) 0x7f82bc131500 con 0x7f82bc06cf50 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f829c0035d0 con 0x7f82bc06cf50 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82ac0018d0 con 0x7f82bc06cf50 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient(hunting): dump: 2024-09-16T06:53:14.612 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.613 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.613 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.613 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.613 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.613 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.614 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.614 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.614 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.614 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 806550091 0 0) 0x7f82bc1302d0 con 0x7f82bc06d480 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6790/0 conn(0x7f82bc132330 legacy=0x7f82bc1d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.189:6789/0 conn(0x7f82bc06cf50 legacy=0x7f82bc12d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 monclient: found mon.a 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc09e610 con 0x7f82bc06d480 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 5 monclient: authenticate success, global_id 264369 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _renew_subs 2024-09-16T06:53:14.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc130b30 con 0x7f82bc06d480 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _renew_subs 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc12f610 con 0x7f82bc06d480 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a4001ca0 con 0x7f82bc06d480 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82a4003b10 con 0x7f82bc06d480 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82ba7fc700 4 set_mon_vals no callback set 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.597+0000 7f82b8ff9700 10 monclient: dump: 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.618 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82c0b6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f82a4003cf0 con 0x7f82bc06d480 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f82a0006590 con 0x7f82a0002830 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f82a4043e70 con 0x7f82bc06d480 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680668227 0 0) 0x7f82a0006590 con 0x7f82a0002830 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient(hunting): my global_id is 264372 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc1302d0 con 0x7f82a0002830 2024-09-16T06:53:14.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82bbfff700 10 monclient: get_auth_request con 0x7f829c036560 auth_method 0 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2227101728 0 0) 0x7f82bc1302d0 con 0x7f82a0002830 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a0006590 con 0x7f82a0002830 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82ac004a30 con 0x7f82a0002830 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient: dump: 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.620 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 975447392 0 0) 0x7f82a0006590 con 0x7f82a0002830 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82a000a190 con 0x7f82a0002830 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f82ac004d80 con 0x7f82a0002830 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6789/0 conn(0x7f82a0002830 legacy=0x7f82a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.601+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6789/0 conn(0x7f82a0002830 legacy=0x7f82a0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f82a0009e50 con 0x7f82a0009170 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083913686 0 0) 0x7f82a0009e50 con 0x7f82a0009170 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient(hunting): my global_id is 264375 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0006590 con 0x7f82a0009170 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3629720268 0 0) 0x7f82a0006590 con 0x7f82a0009170 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a0009e50 con 0x7f82a0009170 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f82ac005350 con 0x7f82a0009170 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient: dump: 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2655517510 0 0) 0x7f82a0009e50 con 0x7f82a0009170 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f82a000c340 con 0x7f82a0009170 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 628546537) 0x7f82ac005660 con 0x7f82a0009170 2024-09-16T06:53:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6789/0 conn(0x7f82a0009170 legacy=0x7f82a0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82b8ff9700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6789/0 conn(0x7f82a0009170 legacy=0x7f82a0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:14.626 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.627 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.628 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:14.629 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:14.630 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.631 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.632 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.634 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.635 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.636 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.637 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:53:14.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.639 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.639 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:14.639 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:14.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.189:6812/3098810674 conn(0x7f829c036560 legacy=0x7f829c038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 10 monclient: shutdown 2024-09-16T06:53:14.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 >> v1:172.21.15.76:6789/0 conn(0x7f82bc06d480 legacy=0x7f82bc12d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82bbfff700 1 -- 172.21.15.76:0/4277776233 reap_dead start 2024-09-16T06:53:14.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82bbfff700 1 -- 172.21.15.76:0/4277776233 reap_dead start 2024-09-16T06:53:14.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 shutdown_connections 2024-09-16T06:53:14.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 >> 172.21.15.76:0/4277776233 conn(0x7f82bc0a41c0 msgr2=0x7f82bc09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 shutdown_connections 2024-09-16T06:53:14.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.613+0000 7f82c0b6c700 1 -- 172.21.15.76:0/4277776233 wait complete. 2024-09-16T06:53:14.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:14.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:53:14.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:53:14.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: get_monmap_and_config 2024-09-16T06:53:14.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: monmap: 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:14.766849+0000 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:14.766849+0000 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:14.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 1 Processor -- start 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 1 -- start start 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: init 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-c con 0x7fa6b0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-b con 0x7fa6b0099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-a con 0x7fa6b0139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0098990 con 0x7fa6b0139500 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.761+0000 7fa6b624d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0097f40 con 0x7fa6b0097b60 2024-09-16T06:53:14.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b009d610 con 0x7fa6b0099090 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient(hunting): authenticate will time out at 1741.900013s 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b4a4a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa6b0099090 0x7fa6b009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47616/0 (socket says 172.21.15.76:47616) 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b4a4a700 1 -- 172.21.15.76:0/2923442965 learned_addr learned my addr 172.21.15.76:0/2923442965 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253465048 0 0) 0x7fa6b009d610 con 0x7fa6b0099090 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): my global_id is 244589 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6980035d0 con 0x7fa6b0099090 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215519666 0 0) 0x7fa6b0098990 con 0x7fa6b0139500 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient(hunting): my global_id is 264378 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b009d610 con 0x7fa6b0139500 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898678470 0 0) 0x7fa6980035d0 con 0x7fa6b0099090 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 >> v1:172.21.15.189:6789/0 conn(0x7fa6b0097b60 legacy=0x7fa6b009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 >> v1:172.21.15.76:6789/0 conn(0x7fa6b0139500 legacy=0x7fa6b013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 monclient: found mon.noname-b 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00986b0 con 0x7fa6b0099090 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 5 monclient: authenticate success, global_id 244589 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a4001610 con 0x7fa6b0099090 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: dump: 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a40018f0 con 0x7fa6b0099090 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 4 set_mon_vals no callback set 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 1 -- 172.21.15.76:0/2923442965 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a4001c80 con 0x7fa6b0099090 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6affff700 10 monclient: dump: 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient: shutdown 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- 172.21.15.76:0/2923442965 >> v1:172.21.15.76:6790/0 conn(0x7fa6b0099090 legacy=0x7fa6b009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- 172.21.15.76:0/2923442965 shutdown_connections 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- 172.21.15.76:0/2923442965 >> 172.21.15.76:0/2923442965 conn(0x7fa6b00a0030 msgr2=0x7fa6b00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- 172.21.15.76:0/2923442965 shutdown_connections 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 1 -- 172.21.15.76:0/2923442965 wait complete. 2024-09-16T06:53:14.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.765+0000 7fa6b624d700 10 monclient: monmap: 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:14.772 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 Processor -- start 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- start start 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: init 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-b con 0x7fa6b0097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-a con 0x7fa6b0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient(hunting): picked mon.noname-c con 0x7fa6b0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b009a540 con 0x7fa6b0099090 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b006e130 con 0x7fa6b0097b60 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0071140 con 0x7fa6b0139500 2024-09-16T06:53:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient(hunting): authenticate will time out at 1741.906136s 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b5a4c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa6b0139500 0x7fa6b01cf5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47618/0 (socket says 172.21.15.76:47618) 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b5a4c700 1 -- 172.21.15.76:0/2477226369 learned_addr learned my addr 172.21.15.76:0/2477226369 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704964167 0 0) 0x7fa6b009a540 con 0x7fa6b0099090 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): my global_id is 264381 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c0035d0 con 0x7fa6b0099090 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849539913 0 0) 0x7fa6b0071140 con 0x7fa6b0139500 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): my global_id is 244592 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b009a540 con 0x7fa6b0139500 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116634233 0 0) 0x7fa6b006e130 con 0x7fa6b0097b60 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): my global_id is 284143 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0071140 con 0x7fa6b0097b60 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745761232 0 0) 0x7fa69c0035d0 con 0x7fa6b0099090 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b006e130 con 0x7fa6b0099090 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a4001c00 con 0x7fa6b0099090 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): dump: 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 282654038 0 0) 0x7fa6b009a540 con 0x7fa6b0139500 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69c0035d0 con 0x7fa6b0139500 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a8001bb0 con 0x7fa6b0139500 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient(hunting): dump: 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1447153720 0 0) 0x7fa6b006e130 con 0x7fa6b0099090 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6790/0 conn(0x7fa6b0139500 legacy=0x7fa6b01cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.189:6789/0 conn(0x7fa6b0097b60 legacy=0x7fa6b0070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 monclient: found mon.a 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b006fc60 con 0x7fa6b0099090 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 5 monclient: authenticate success, global_id 264381 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _renew_subs 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b006ff40 con 0x7fa6b0099090 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _renew_subs 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b01d0d10 con 0x7fa6b0099090 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a4004ca0 con 0x7fa6b0099090 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a4001210 con 0x7fa6b0099090 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6af7fe700 4 set_mon_vals no callback set 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.769+0000 7fa6adffb700 10 monclient: dump: 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6b624d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa690006590 con 0x7fa690002830 2024-09-16T06:53:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa6a400f8c0 con 0x7fa6b0099090 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa6a4043eb0 con 0x7fa6b0099090 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274278027 0 0) 0x7fa690006590 con 0x7fa690002830 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient(hunting): my global_id is 244595 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b006e130 con 0x7fa690002830 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6b5a4c700 10 monclient: get_auth_request con 0x7fa69c036410 auth_method 0 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3987855016 0 0) 0x7fa6b006e130 con 0x7fa690002830 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa690006590 con 0x7fa690002830 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a0001ae0 con 0x7fa690002830 2024-09-16T06:53:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient: dump: 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1790739230 0 0) 0x7fa690006590 con 0x7fa690002830 2024-09-16T06:53:14.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa69000a190 con 0x7fa690002830 2024-09-16T06:53:14.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa6a00036a0 con 0x7fa690002830 2024-09-16T06:53:14.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:14.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6790/0 conn(0x7fa690002830 legacy=0x7fa690004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.773+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6790/0 conn(0x7fa690002830 legacy=0x7fa690004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa690009e50 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497851740 0 0) 0x7fa690009e50 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient(hunting): my global_id is 244598 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa690006590 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3246081644 0 0) 0x7fa690006590 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa690009e50 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa6a0003c20 con 0x7fa690009170 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient: dump: 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2760374732 0 0) 0x7fa690009e50 con 0x7fa690009170 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa69000c340 con 0x7fa690009170 2024-09-16T06:53:14.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2501812744) 0x7fa6a0003f00 con 0x7fa690009170 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6790/0 conn(0x7fa690009170 legacy=0x7fa690009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6adffb700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6790/0 conn(0x7fa690009170 legacy=0x7fa690009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:53:14.792 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.793 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.794 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:14.795 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.796 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:14.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.798 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.801 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.802 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.804 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:14.804 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:14.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa69c036410 legacy=0x7fa69c0388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 10 monclient: shutdown 2024-09-16T06:53:14.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 >> v1:172.21.15.76:6789/0 conn(0x7fa6b0099090 legacy=0x7fa6b01cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b5a4c700 1 -- 172.21.15.76:0/2477226369 reap_dead start 2024-09-16T06:53:14.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b5a4c700 1 -- 172.21.15.76:0/2477226369 reap_dead start 2024-09-16T06:53:14.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 shutdown_connections 2024-09-16T06:53:14.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 >> 172.21.15.76:0/2477226369 conn(0x7fa6b00a0030 msgr2=0x7fa6b00a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 shutdown_connections 2024-09-16T06:53:14.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.785+0000 7fa6b624d700 1 -- 172.21.15.76:0/2477226369 wait complete. 2024-09-16T06:53:14.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:14.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:53:14.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 10 monclient: monmap: 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.921 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:14.926667+0000 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:14.926667+0000 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:14.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 1 Processor -- start 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 1 -- start start 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.921+0000 7f7d4b59e700 10 monclient: init 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7d4c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7d4c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c098990 con 0x7f7d4c097b60 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c097f40 con 0x7f7d4c099090 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c09d610 con 0x7f7d4c139500 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient(hunting): authenticate will time out at 1742.060192s 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7d4c097b60 0x7f7d4c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59384/0 (socket says 172.21.15.76:59384) 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4a59c700 1 -- 172.21.15.76:0/468359407 learned_addr learned my addr 172.21.15.76:0/468359407 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495982911 0 0) 0x7f7d4c098990 con 0x7f7d4c097b60 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): my global_id is 264384 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d380035d0 con 0x7f7d4c097b60 2024-09-16T06:53:14.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540949373 0 0) 0x7f7d4c097f40 con 0x7f7d4c099090 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): my global_id is 284146 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c098990 con 0x7f7d4c099090 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799460065 0 0) 0x7f7d4c09d610 con 0x7f7d4c139500 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient(hunting): my global_id is 244601 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c097f40 con 0x7f7d4c139500 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292583599 0 0) 0x7f7d380035d0 con 0x7f7d4c097b60 2024-09-16T06:53:14.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c139500 legacy=0x7f7d4c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 >> v1:172.21.15.189:6789/0 conn(0x7f7d4c099090 legacy=0x7f7d4c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 monclient: found mon.noname-a 2024-09-16T06:53:14.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0986b0 con 0x7f7d4c097b60 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 5 monclient: authenticate success, global_id 264384 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d3c001610 con 0x7f7d4c097b60 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:14.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: dump: 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d3c0018f0 con 0x7f7d4c097b60 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 4 set_mon_vals no callback set 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 1 -- 172.21.15.76:0/468359407 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d3c001c80 con 0x7f7d4c097b60 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d49d9b700 10 monclient: dump: 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.933 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: shutdown 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- 172.21.15.76:0/468359407 >> v1:172.21.15.76:6789/0 conn(0x7f7d4c097b60 legacy=0x7f7d4c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- 172.21.15.76:0/468359407 shutdown_connections 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- 172.21.15.76:0/468359407 >> 172.21.15.76:0/468359407 conn(0x7f7d4c0a0030 msgr2=0x7f7d4c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- 172.21.15.76:0/468359407 shutdown_connections 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 1 -- 172.21.15.76:0/468359407 wait complete. 2024-09-16T06:53:14.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: build_initial_monmap 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.925+0000 7f7d4b59e700 10 monclient: monmap: 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:14.935 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 Processor -- start 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- start start 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: init 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7d4c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7d4c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:14.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c091310 con 0x7f7d4c097b60 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c13c270 con 0x7f7d4c099090 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c0a2cc0 con 0x7f7d4c139500 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient(hunting): authenticate will time out at 1742.065261s 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7d4c097b60 0x7f7d4c138a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59386/0 (socket says 172.21.15.76:59386) 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4a59c700 1 -- 172.21.15.76:0/1876265867 learned_addr learned my addr 172.21.15.76:0/1876265867 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164935974 0 0) 0x7f7d4c091310 con 0x7f7d4c097b60 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): my global_id is 264387 2024-09-16T06:53:14.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d2c0035d0 con 0x7f7d4c097b60 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881165568 0 0) 0x7f7d4c0a2cc0 con 0x7f7d4c139500 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): my global_id is 244604 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c091310 con 0x7f7d4c139500 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569283240 0 0) 0x7f7d4c13c270 con 0x7f7d4c099090 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): my global_id is 284149 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c0a2cc0 con 0x7f7d4c099090 2024-09-16T06:53:14.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682236102 0 0) 0x7f7d2c0035d0 con 0x7f7d4c097b60 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c13c270 con 0x7f7d4c097b60 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d3c001650 con 0x7f7d4c097b60 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): dump: 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.944 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602562873 0 0) 0x7f7d4c091310 con 0x7f7d4c139500 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d2c0035d0 con 0x7f7d4c139500 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163072373 0 0) 0x7f7d4c0a2cc0 con 0x7f7d4c099090 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d4c091310 con 0x7f7d4c099090 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2237519043 0 0) 0x7f7d4c13c270 con 0x7f7d4c097b60 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.76:6790/0 conn(0x7f7d4c139500 legacy=0x7f7d4c1d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6789/0 conn(0x7f7d4c099090 legacy=0x7f7d4c1ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:14.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 monclient: found mon.a 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c09a540 con 0x7f7d4c097b60 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 5 monclient: authenticate success, global_id 264387 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _renew_subs 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c0915f0 con 0x7f7d4c097b60 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _renew_subs 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:14.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c08fea0 con 0x7f7d4c097b60 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d3c004cc0 con 0x7f7d4c097b60 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d3c005050 con 0x7f7d4c097b60 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4959a700 4 set_mon_vals no callback set 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 10 monclient: dump: 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7d3c005230 con 0x7f7d4c097b60 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.929+0000 7f7d4b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d30006590 con 0x7f7d30002830 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7d3c0441b0 con 0x7f7d4c097b60 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d4ad9d700 10 monclient: get_auth_request con 0x7f7d2c036480 auth_method 0 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721196078 0 0) 0x7f7d30006590 con 0x7f7d30002830 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient(hunting): my global_id is 284152 2024-09-16T06:53:14.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c13c270 con 0x7f7d30002830 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2049117266 0 0) 0x7f7d4c13c270 con 0x7f7d30002830 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d30006590 con 0x7f7d30002830 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d34001d50 con 0x7f7d30002830 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient: dump: 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.949 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3265505653 0 0) 0x7f7d30006590 con 0x7f7d30002830 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d3000a190 con 0x7f7d30002830 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7d340059d0 con 0x7f7d30002830 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6789/0 conn(0x7f7d30002830 legacy=0x7f7d30004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.933+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6789/0 conn(0x7f7d30002830 legacy=0x7f7d30004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.941+0000 7f7d4b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.941+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d30009e50 con 0x7f7d30009170 2024-09-16T06:53:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145177369 0 0) 0x7f7d30009e50 con 0x7f7d30009170 2024-09-16T06:53:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient(hunting): my global_id is 284155 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d30006590 con 0x7f7d30009170 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1001280954 0 0) 0x7f7d30006590 con 0x7f7d30009170 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d30009e50 con 0x7f7d30009170 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d34003ef0 con 0x7f7d30009170 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient: dump: 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.953 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2884926262 0 0) 0x7f7d30009e50 con 0x7f7d30009170 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d3000c340 con 0x7f7d30009170 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1371456135) 0x7f7d34005b40 con 0x7f7d30009170 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6789/0 conn(0x7f7d30009170 legacy=0x7f7d30009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d437fe700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6789/0 conn(0x7f7d30009170 legacy=0x7f7d30009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 132, 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:14.955 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.956 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.957 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:14.958 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:14.959 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:14.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:14.961 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:14.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:14.966 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:14.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:14.966 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:14.966 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:14.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7d2c036480 legacy=0x7f7d2c038940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 10 monclient: shutdown 2024-09-16T06:53:14.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:14.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 >> v1:172.21.15.76:6789/0 conn(0x7f7d4c097b60 legacy=0x7f7d4c138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:14.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4ad9d700 1 -- 172.21.15.76:0/1876265867 reap_dead start 2024-09-16T06:53:14.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4ad9d700 1 -- 172.21.15.76:0/1876265867 reap_dead start 2024-09-16T06:53:14.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 shutdown_connections 2024-09-16T06:53:14.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 >> 172.21.15.76:0/1876265867 conn(0x7f7d4c0a0030 msgr2=0x7f7d4c139c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:14.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 shutdown_connections 2024-09-16T06:53:14.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:14.945+0000 7f7d4b59e700 1 -- 172.21.15.76:0/1876265867 wait complete. 2024-09-16T06:53:14.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:14.968 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:53:14.969 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:53:14.969 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:53:14.969 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:53:14.969 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:53:14.969 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:53:15.019 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:53:15.020 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:53:15.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.133+0000 7f7857fff700 10 monclient: get_monmap_and_config 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.133+0000 7f7857fff700 10 monclient: build_initial_monmap 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.133+0000 7f7857fff700 10 monclient: monmap: 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:15.139318+0000 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:15.139318+0000 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:15.134 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:15.135 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:15.135 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:15.135 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 Processor -- start 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- start start 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient: init 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-a con 0x7f7858132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-b con 0x7f785806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-c con 0x7f785806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785806c280 con 0x7f7858132330 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785806bfa0 con 0x7f785806cf50 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785809ace0 con 0x7f785806d480 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:15.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient(hunting): authenticate will time out at 1742.273136s 2024-09-16T06:53:15.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7856ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7858132330 legacy=0x7f7858132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:15.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f78567fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f785806d480 0x7f785813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47672/0 (socket says 172.21.15.76:47672) 2024-09-16T06:53:15.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f78567fc700 1 -- 172.21.15.76:0/3708204138 learned_addr learned my addr 172.21.15.76:0/3708204138 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841139885 0 0) 0x7f785809ace0 con 0x7f785806d480 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): my global_id is 244607 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78440035d0 con 0x7f785806d480 2024-09-16T06:53:15.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956965110 0 0) 0x7f785806bfa0 con 0x7f785806cf50 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient(hunting): my global_id is 284158 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785809ace0 con 0x7f785806cf50 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964799874 0 0) 0x7f78440035d0 con 0x7f785806d480 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 >> v1:172.21.15.189:6789/0 conn(0x7f785806cf50 legacy=0x7f785813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 >> v1:172.21.15.76:6789/0 conn(0x7f7858132330 legacy=0x7f7858132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 monclient: found mon.noname-b 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:15.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785809ae70 con 0x7f785806d480 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7840001610 con 0x7f785806d480 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 5 monclient: authenticate success, global_id 244607 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: dump: 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:15.141 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78400018f0 con 0x7f785806d480 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 1 -- 172.21.15.76:0/3708204138 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7840001ae0 con 0x7f785806d480 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 4 set_mon_vals no callback set 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:15.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7855ffb700 10 monclient: dump: 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 10 monclient: shutdown 2024-09-16T06:53:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- 172.21.15.76:0/3708204138 >> v1:172.21.15.76:6790/0 conn(0x7f785806d480 legacy=0x7f785813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- 172.21.15.76:0/3708204138 shutdown_connections 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- 172.21.15.76:0/3708204138 >> 172.21.15.76:0/3708204138 conn(0x7f78580a41c0 msgr2=0x7f78580a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- 172.21.15.76:0/3708204138 shutdown_connections 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.137+0000 7f7857fff700 1 -- 172.21.15.76:0/3708204138 wait complete. 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient: build_initial_monmap 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient: monmap: 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:15.144 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:15.145 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:15.145 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:15.145 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:15.145 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:15.145 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 1 Processor -- start 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 1 -- start start 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient: init 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-c con 0x7f785806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:15.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-b con 0x7f785806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient(hunting): picked mon.noname-a con 0x7f7858132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785813ede0 con 0x7f7858132330 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78581310d0 con 0x7f785806d480 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7858131550 con 0x7f785806cf50 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7857fff700 10 monclient(hunting): authenticate will time out at 1742.278707s 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f78577fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f7858132330 legacy=0x7f78581d2380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7856ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f785806cf50 0x7f78581c66a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47686/0 (socket says 172.21.15.76:47686) 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f7856ffd700 1 -- 172.21.15.76:0/1937872577 learned_addr learned my addr 172.21.15.76:0/1937872577 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959526132 0 0) 0x7f7858131550 con 0x7f785806cf50 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:15.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): my global_id is 244610 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78380035d0 con 0x7f785806cf50 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572454118 0 0) 0x7f78581310d0 con 0x7f785806d480 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): my global_id is 284161 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7858131550 con 0x7f785806d480 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670502128 0 0) 0x7f78380035d0 con 0x7f785806cf50 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78581310d0 con 0x7f785806cf50 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f78500045c0 con 0x7f785806cf50 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:15.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): dump: 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.141+0000 7f784f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 24728371 0 0) 0x7f78581310d0 con 0x7f785806cf50 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 >> v1:172.21.15.189:6789/0 conn(0x7f785806d480 legacy=0x7f78581cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:15.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 >> v1:172.21.15.76:6789/0 conn(0x7f7858132330 legacy=0x7f78581d2380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 monclient: found mon.c 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858130ba0 con 0x7f785806cf50 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 5 monclient: authenticate success, global_id 244610 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _renew_subs 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785812fb80 con 0x7f785806cf50 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _renew_subs 2024-09-16T06:53:15.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f785812eff0 con 0x7f785806cf50 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7850004850 con 0x7f785806cf50 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7850004be0 con 0x7f785806cf50 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7854ff9700 4 set_mon_vals no callback set 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 10 monclient: dump: 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:15.154 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7850004d40 con 0x7f785806cf50 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7850044240 con 0x7f785806cf50 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783c005100 con 0x7f785806cf50 2024-09-16T06:53:15.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.145+0000 7f78567fc700 10 monclient: get_auth_request con 0x7f7838035fd0 auth_method 0 2024-09-16T06:53:15.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.153+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7850014b80 con 0x7f785806cf50 2024-09-16T06:53:15.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.153+0000 7f784f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:15.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.153+0000 7f784f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:15.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:15.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:15.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:15.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f783c005930 con 0x7f785806cf50 2024-09-16T06:53:15.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f784f7fe700 1 -- 172.21.15.76:0/1937872577 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1496608930) 0x7f7850012ff0 con 0x7f785806cf50 2024-09-16T06:53:15.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f784f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:15.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f784f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:15.388 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7838035fd0 legacy=0x7f7838038490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 10 monclient: shutdown 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 >> v1:172.21.15.76:6790/0 conn(0x7f785806cf50 legacy=0x7f78581c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 shutdown_connections 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 >> 172.21.15.76:0/1937872577 conn(0x7f78580a41c0 msgr2=0x7f785809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 shutdown_connections 2024-09-16T06:53:15.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:15.389+0000 7f7857fff700 1 -- 172.21.15.76:0/1937872577 wait complete. 2024-09-16T06:53:18.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: get_monmap_and_config 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: build_initial_monmap 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: monmap: 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:18.530206+0000 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:18.530206+0000 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:18.525 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:18.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:18.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:18.526 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:18.526 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 1 Processor -- start 2024-09-16T06:53:18.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 1 -- start start 2024-09-16T06:53:18.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: init 2024-09-16T06:53:18.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-a con 0x7f5e88132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-b con 0x7f5e8806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-c con 0x7f5e8806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8806c280 con 0x7f5e88132330 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8806bfa0 con 0x7f5e8806cf50 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8809ace0 con 0x7f5e8806d480 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8e302700 10 monclient(hunting): authenticate will time out at 1745.663862s 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.525+0000 7f5e8d300700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5e88132330 legacy=0x7f5e88132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8caff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e8806d480 0x7f5e8813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47702/0 (socket says 172.21.15.76:47702) 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8caff700 1 -- 172.21.15.76:0/4204875875 learned_addr learned my addr 172.21.15.76:0/4204875875 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377272856 0 0) 0x7f5e8809ace0 con 0x7f5e8806d480 2024-09-16T06:53:18.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient(hunting): my global_id is 244613 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e740035d0 con 0x7f5e8806d480 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156502832 0 0) 0x7f5e740035d0 con 0x7f5e8806d480 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 >> v1:172.21.15.189:6789/0 conn(0x7f5e8806cf50 legacy=0x7f5e8813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 >> v1:172.21.15.76:6789/0 conn(0x7f5e88132330 legacy=0x7f5e88132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 monclient: found mon.noname-b 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:18.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8809ae70 con 0x7f5e8806d480 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e70001610 con 0x7f5e8806d480 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: dump: 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:18.535 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 5 monclient: authenticate success, global_id 244613 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e70001930 con 0x7f5e8806d480 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 1 -- 172.21.15.76:0/4204875875 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e70001b20 con 0x7f5e8806d480 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 4 set_mon_vals no callback set 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e87fff700 10 monclient: dump: 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: shutdown 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- 172.21.15.76:0/4204875875 >> v1:172.21.15.76:6790/0 conn(0x7f5e8806d480 legacy=0x7f5e8813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:18.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- 172.21.15.76:0/4204875875 shutdown_connections 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- 172.21.15.76:0/4204875875 >> 172.21.15.76:0/4204875875 conn(0x7f5e880a41c0 msgr2=0x7f5e880a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- 172.21.15.76:0/4204875875 shutdown_connections 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- 172.21.15.76:0/4204875875 wait complete. 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: build_initial_monmap 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: monmap: 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:18.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 Processor -- start 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- start start 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: init 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-b con 0x7f5e8806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-a con 0x7f5e8806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient(hunting): picked mon.noname-c con 0x7f5e88132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8812f870 con 0x7f5e8806d480 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e881ca650 con 0x7f5e8806cf50 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e88130c90 con 0x7f5e88132330 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:18.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8e302700 10 monclient(hunting): authenticate will time out at 1745.667595s 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8caff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5e8806d480 legacy=0x7f5e881cea40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8db01700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5e88132330 0x7f5e881d2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47706/0 (socket says 172.21.15.76:47706) 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.529+0000 7f5e8db01700 1 -- 172.21.15.76:0/2212466526 learned_addr learned my addr 172.21.15.76:0/2212466526 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351866158 0 0) 0x7f5e88130c90 con 0x7f5e88132330 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): my global_id is 244616 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e680035d0 con 0x7f5e88132330 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1565337553 0 0) 0x7f5e680035d0 con 0x7f5e88132330 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e88130c90 con 0x7f5e88132330 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e80001830 con 0x7f5e88132330 2024-09-16T06:53:18.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 291973076 0 0) 0x7f5e88130c90 con 0x7f5e88132330 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 >> v1:172.21.15.189:6789/0 conn(0x7f5e8806cf50 legacy=0x7f5e8813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 >> v1:172.21.15.76:6789/0 conn(0x7f5e8806d480 legacy=0x7f5e881cea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 monclient: found mon.c 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8812ec10 con 0x7f5e88132330 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:18.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 5 monclient: authenticate success, global_id 244616 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _renew_subs 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e88131620 con 0x7f5e88132330 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _renew_subs 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e881309b0 con 0x7f5e88132330 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e800036c0 con 0x7f5e88132330 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5e80003a50 con 0x7f5e88132330 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e877fe700 4 set_mon_vals no callback set 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 10 monclient: dump: 2024-09-16T06:53:18.543 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:18.544 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:18.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:18.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:18.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c005100 con 0x7f5e88132330 2024-09-16T06:53:18.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5e80003c30 con 0x7f5e88132330 2024-09-16T06:53:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.533+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5e80043ec0 con 0x7f5e88132330 2024-09-16T06:53:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.537+0000 7f5e8d300700 10 monclient: get_auth_request con 0x7f5e680360a0 auth_method 0 2024-09-16T06:53:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.541+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5e80014800 con 0x7f5e88132330 2024-09-16T06:53:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.541+0000 7f5e85ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.541+0000 7f5e85ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:18.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.765+0000 7f5e8e302700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:18.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e8e302700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:18.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e8e302700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:18.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e6c005930 con 0x7f5e88132330 2024-09-16T06:53:18.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e85ffb700 1 -- 172.21.15.76:0/2212466526 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 165199491) 0x7f5e80012c70 con 0x7f5e88132330 2024-09-16T06:53:18.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e85ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:18.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.769+0000 7f5e85ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:18.768 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5e680360a0 legacy=0x7f5e68038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 10 monclient: shutdown 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 >> v1:172.21.15.76:6790/0 conn(0x7f5e88132330 legacy=0x7f5e881d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 shutdown_connections 2024-09-16T06:53:18.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 >> 172.21.15.76:0/2212466526 conn(0x7f5e880a41c0 msgr2=0x7f5e8809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:18.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 shutdown_connections 2024-09-16T06:53:18.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:18.773+0000 7f5e8e302700 1 -- 172.21.15.76:0/2212466526 wait complete. 2024-09-16T06:53:21.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 10 monclient: get_monmap_and_config 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 10 monclient: build_initial_monmap 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 10 monclient: monmap: 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:21.910596+0000 2024-09-16T06:53:21.905 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:21.910596+0000 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:21.906 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 1 Processor -- start 2024-09-16T06:53:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 1 -- start start 2024-09-16T06:53:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.905+0000 7fd1ef221700 10 monclient: init 2024-09-16T06:53:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-a con 0x7fd1e8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-c con 0x7fd1e806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806c280 con 0x7fd1e8132330 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806bfa0 con 0x7fd1e806d480 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e809ace0 con 0x7fd1e806cf50 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ef221700 10 monclient(hunting): authenticate will time out at 1749.044441s 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1ee21f700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1e8132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1eea20700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 0x7fd1e813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47714/0 (socket says 172.21.15.76:47714) 2024-09-16T06:53:21.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:21.909+0000 7fd1eea20700 1 -- 172.21.15.76:0/3790532833 learned_addr learned my addr 172.21.15.76:0/3790532833 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:22.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:22.113+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1e8132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:22.517+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1e8132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:23.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:23.320+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1e8132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:24.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): tick 2024-09-16T06:53:24.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:53:24.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 monclient(hunting): continuing hunt 2024-09-16T06:53:24.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:53:24.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 legacy=0x7fd1e813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:24.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.189:6789/0 conn(0x7fd1e806d480 legacy=0x7fd1e813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:24.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1e8132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:24.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fd1e8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806c280 con 0x7fd1e8132330 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc00f390 con 0x7fd1cc003740 2024-09-16T06:53:24.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0105a0 con 0x7fd1e806cf50 2024-09-16T06:53:24.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1dbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:24.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:24.908+0000 7fd1eea20700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1cc00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:25.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:25.112+0000 7fd1eea20700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1cc00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:25.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:25.516+0000 7fd1eea20700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1cc00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:26.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:26.320+0000 7fd1eea20700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1cc00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): tick 2024-09-16T06:53:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:53:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 monclient(hunting): continuing hunt 2024-09-16T06:53:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:53:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 legacy=0x7fd1cc0093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.189:6789/0 conn(0x7fd1cc003740 legacy=0x7fd1cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1e8132330 legacy=0x7fd1cc00cbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc00f620 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fd1e806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0117b0 con 0x7fd1cc00f620 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806c280 con 0x7fd1cc003740 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc010310 con 0x7fd1e806cf50 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1cc00fa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:27.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:27.908+0000 7fd1dbfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:28.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:28.112+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1cc00fa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:28.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:28.516+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1cc00fa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:29.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.320+0000 7fd1ee21f700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1cc00fa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422854397 0 0) 0x7fd1e806c280 con 0x7fd1cc003740 2024-09-16T06:53:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 10 monclient(hunting): _negotiate 2024-09-16T06:53:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 10 monclient(hunting): my global_id is 284164 2024-09-16T06:53:29.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.640+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d40035d0 con 0x7fd1cc003740 2024-09-16T06:53:29.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953087507 0 0) 0x7fd1cc010310 con 0x7fd1e806cf50 2024-09-16T06:53:29.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient(hunting): _negotiate 2024-09-16T06:53:29.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:29.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:29.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient(hunting): my global_id is 244619 2024-09-16T06:53:29.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e806c280 con 0x7fd1e806cf50 2024-09-16T06:53:29.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814944759 0 0) 0x7fd1e806c280 con 0x7fd1e806cf50 2024-09-16T06:53:29.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.189:6789/0 conn(0x7fd1cc003740 legacy=0x7fd1cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:29.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1cc00fa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:29.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient: _finish_hunting 0 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 monclient: found mon.noname-b 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc00bbb0 con 0x7fd1e806cf50 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient: _finish_auth 0 2024-09-16T06:53:29.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 10 monclient: _check_auth_tickets 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.652+0000 7fd1ed21d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ef221700 5 monclient: authenticate success, global_id 244619 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd1dc001590 con 0x7fd1e806cf50 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ef221700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: dump: 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.657 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:29.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ef221700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:29.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc0017f0 con 0x7fd1e806cf50 2024-09-16T06:53:29.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:29.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 1 -- 172.21.15.76:0/3790532833 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd1dc001c00 con 0x7fd1e806cf50 2024-09-16T06:53:29.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ef221700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ef221700 4 set_mon_vals no callback set 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.656+0000 7fd1ed21d700 10 monclient: dump: 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:29.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: shutdown 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- 172.21.15.76:0/3790532833 >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 legacy=0x7fd1cc0093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- 172.21.15.76:0/3790532833 shutdown_connections 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- 172.21.15.76:0/3790532833 >> 172.21.15.76:0/3790532833 conn(0x7fd1e80a41c0 msgr2=0x7fd1e80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- 172.21.15.76:0/3790532833 shutdown_connections 2024-09-16T06:53:29.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- 172.21.15.76:0/3790532833 wait complete. 2024-09-16T06:53:29.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: build_initial_monmap 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: monmap: 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:29.667 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 Processor -- start 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 1 -- start start 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: init 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.660+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc00f620 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient(hunting): picked mon.noname-c con 0x7fd1e806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8122ca0 con 0x7fd1cc00f620 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80a6e90 con 0x7fd1cc003740 2024-09-16T06:53:29.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8124600 con 0x7fd1e806cf50 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient(hunting): authenticate will time out at 1756.800148s 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1eda1e700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1e81cee10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1eea20700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 0x7fd1e81d2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44830/0 (socket says 172.21.15.76:44830) 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1eea20700 1 -- 172.21.15.76:0/2701874885 learned_addr learned my addr 172.21.15.76:0/2701874885 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076719374 0 0) 0x7fd1e8124600 con 0x7fd1e806cf50 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): my global_id is 244622 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc010310 con 0x7fd1e806cf50 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148659094 0 0) 0x7fd1e80a6e90 con 0x7fd1cc003740 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:29.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): my global_id is 284167 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e8124600 con 0x7fd1cc003740 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 469938929 0 0) 0x7fd1cc010310 con 0x7fd1e806cf50 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e80a6e90 con 0x7fd1e806cf50 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd1e0001af0 con 0x7fd1e806cf50 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): dump: 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 240675728 0 0) 0x7fd1e80a6e90 con 0x7fd1e806cf50 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 >> v1:172.21.15.189:6789/0 conn(0x7fd1cc003740 legacy=0x7fd1e81c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 >> v1:172.21.15.76:6789/0 conn(0x7fd1cc00f620 legacy=0x7fd1e81cee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 monclient: found mon.c 2024-09-16T06:53:29.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80a73c0 con 0x7fd1e806cf50 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 5 monclient: authenticate success, global_id 244622 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _renew_subs 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e809e8b0 con 0x7fd1e806cf50 2024-09-16T06:53:29.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _renew_subs 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e8123a30 con 0x7fd1e806cf50 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1e0002040 con 0x7fd1e806cf50 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd1e0004db0 con 0x7fd1e806cf50 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 10 monclient: dump: 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:29.673 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1eca1c700 4 set_mon_vals no callback set 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e80a1290 con 0x7fd1e806cf50 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.664+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd1e000f820 con 0x7fd1e806cf50 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.668+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd1e0043f70 con 0x7fd1e806cf50 2024-09-16T06:53:29.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.668+0000 7fd1ee21f700 10 monclient: get_auth_request con 0x7fd1cc036fa0 auth_method 0 2024-09-16T06:53:29.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.672+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd1e0014680 con 0x7fd1e806cf50 2024-09-16T06:53:29.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.672+0000 7fd1daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:29.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.672+0000 7fd1daffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1ef221700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1ef221700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1ef221700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd1e8124120 con 0x7fd1e806cf50 2024-09-16T06:53:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1daffd700 1 -- 172.21.15.76:0/2701874885 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 2193277177) 0x7fd1e0012d20 con 0x7fd1e806cf50 2024-09-16T06:53:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:29.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.904+0000 7fd1daffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:29.905 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":136,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd1cc036fa0 legacy=0x7fd1cc039460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 10 monclient: shutdown 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 >> v1:172.21.15.76:6790/0 conn(0x7fd1e806cf50 legacy=0x7fd1e81d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 shutdown_connections 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 >> 172.21.15.76:0/2701874885 conn(0x7fd1e80a41c0 msgr2=0x7fd1e809ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 shutdown_connections 2024-09-16T06:53:29.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:29.908+0000 7fd1ef221700 1 -- 172.21.15.76:0/2701874885 wait complete. 2024-09-16T06:53:29.918 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:53:29.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:53:29.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:53:30.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: get_monmap_and_config 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: build_initial_monmap 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: monmap: 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:30.033670+0000 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:30.033670+0000 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 1 Processor -- start 2024-09-16T06:53:30.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 1 -- start start 2024-09-16T06:53:30.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: init 2024-09-16T06:53:30.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:30.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient(hunting): picked mon.noname-b con 0x7f613c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient(hunting): picked mon.noname-c con 0x7f613c09b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient(hunting): picked mon.noname-a con 0x7f613c138030 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c09f610 con 0x7f613c138030 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c09bfb0 con 0x7f613c09b6a0 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c09a880 con 0x7f613c09f230 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6143892700 10 monclient(hunting): authenticate will time out at 1757.167315s 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6141e2f700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f613c138030 legacy=0x7f613c13a420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f614162e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f613c09f230 0x7f613c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44836/0 (socket says 172.21.15.76:44836) 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f614162e700 1 -- 172.21.15.76:0/1600355474 learned_addr learned my addr 172.21.15.76:0/1600355474 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418325608 0 0) 0x7f613c09a880 con 0x7f613c09f230 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 10 monclient(hunting): my global_id is 244625 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.028+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61280034f0 con 0x7f613c09f230 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767533012 0 0) 0x7f61280034f0 con 0x7f613c09f230 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 >> v1:172.21.15.189:6789/0 conn(0x7f613c09b6a0 legacy=0x7f613c09a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 >> v1:172.21.15.76:6789/0 conn(0x7f613c138030 legacy=0x7f613c13a420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 monclient: found mon.noname-b 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c09d3c0 con 0x7f613c09f230 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:30.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6138001610 con 0x7f613c09f230 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 5 monclient: authenticate success, global_id 244625 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: dump: 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.036 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6138001930 con 0x7f613c09f230 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 1 -- 172.21.15.76:0/1600355474 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6138001cc0 con 0x7f613c09f230 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 4 set_mon_vals no callback set 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6133fff700 10 monclient: dump: 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.037 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.038 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 10 monclient: shutdown 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- 172.21.15.76:0/1600355474 >> v1:172.21.15.76:6790/0 conn(0x7f613c09f230 legacy=0x7f613c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- 172.21.15.76:0/1600355474 shutdown_connections 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- 172.21.15.76:0/1600355474 >> 172.21.15.76:0/1600355474 conn(0x7f613c0a3e50 msgr2=0x7f613c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- 172.21.15.76:0/1600355474 shutdown_connections 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- 172.21.15.76:0/1600355474 wait complete. 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 10 monclient: build_initial_monmap 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 10 monclient: monmap: 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:30.039 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 Processor -- start 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 1 -- start start 2024-09-16T06:53:30.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.032+0000 7f6143892700 10 monclient: init 2024-09-16T06:53:30.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:30.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:30.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient(hunting): picked mon.noname-b con 0x7f613c09b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient(hunting): picked mon.noname-c con 0x7f613c09f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient(hunting): picked mon.noname-a con 0x7f613c138030 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c06f8b0 con 0x7f613c138030 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c11a3c0 con 0x7f613c09b6a0 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6141e2f700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f613c138030 legacy=0x7f613c1ce200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c11b270 con 0x7f613c09f230 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient(hunting): authenticate will time out at 1757.172198s 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6140e2d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f613c09f230 0x7f613c06f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44848/0 (socket says 172.21.15.76:44848) 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6140e2d700 1 -- 172.21.15.76:0/2629111175 learned_addr learned my addr 172.21.15.76:0/2629111175 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911200757 0 0) 0x7f613c11b270 con 0x7f613c09f230 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): my global_id is 244628 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61200035d0 con 0x7f613c09f230 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890845478 0 0) 0x7f613c11a3c0 con 0x7f613c09b6a0 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): my global_id is 284173 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c11b270 con 0x7f613c09b6a0 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3297078434 0 0) 0x7f61200035d0 con 0x7f613c09f230 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f613c11a3c0 con 0x7f613c09f230 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f612c001930 con 0x7f613c09f230 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1881179880 0 0) 0x7f613c11a3c0 con 0x7f613c09f230 2024-09-16T06:53:30.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.189:6789/0 conn(0x7f613c09b6a0 legacy=0x7f613c06ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6789/0 conn(0x7f613c138030 legacy=0x7f613c1ce200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 monclient: found mon.c 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c11af90 con 0x7f613c09f230 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 5 monclient: authenticate success, global_id 244628 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _renew_subs 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c11acb0 con 0x7f613c09f230 2024-09-16T06:53:30.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _renew_subs 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c070db0 con 0x7f613c09f230 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f612c001cc0 con 0x7f613c09f230 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f612c003b10 con 0x7f613c09f230 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6132ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: dump: 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.056 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f612c003c70 con 0x7f613c09f230 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f611c006590 con 0x7f611c002830 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f612c043e50 con 0x7f613c09f230 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375690112 0 0) 0x7f611c006590 con 0x7f611c002830 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient(hunting): my global_id is 244631 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c11a3c0 con 0x7f611c002830 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6141e2f700 10 monclient: get_auth_request con 0x7f6120036220 auth_method 0 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2406995246 0 0) 0x7f613c11a3c0 con 0x7f611c002830 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f611c006590 con 0x7f611c002830 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6138001d00 con 0x7f611c002830 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 10 monclient: dump: 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.058 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 488478270 0 0) 0x7f611c006590 con 0x7f611c002830 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.036+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f611c00a190 con 0x7f611c002830 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.040+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6138001620 con 0x7f611c002830 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.040+0000 7f6131ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.040+0000 7f6131ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.040+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6790/0 conn(0x7f611c002830 legacy=0x7f611c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.040+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6790/0 conn(0x7f611c002830 legacy=0x7f611c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:30.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f611c009e50 con 0x7f611c009170 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392944221 0 0) 0x7f611c009e50 con 0x7f611c009170 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient(hunting): my global_id is 244634 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f611c006590 con 0x7f611c009170 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1396886174 0 0) 0x7f611c006590 con 0x7f611c009170 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f611c009e50 con 0x7f611c009170 2024-09-16T06:53:30.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6138004e50 con 0x7f611c009170 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient: dump: 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2489677050 0 0) 0x7f611c009e50 con 0x7f611c009170 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f611c00c340 con 0x7f611c009170 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 2008168309) 0x7f6138007150 con 0x7f611c009170 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6790/0 conn(0x7f611c009170 legacy=0x7f611c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6131ffb700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6790/0 conn(0x7f611c009170 legacy=0x7f611c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:30.062 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 136, 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:30.063 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:30.064 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.065 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:30.066 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.067 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.069 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.072 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.073 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:30.074 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6120036220 legacy=0x7f61200386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 10 monclient: shutdown 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 >> v1:172.21.15.76:6790/0 conn(0x7f613c09f230 legacy=0x7f613c06f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6141e2f700 1 -- 172.21.15.76:0/2629111175 reap_dead start 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6141e2f700 1 -- 172.21.15.76:0/2629111175 reap_dead start 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 shutdown_connections 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 >> 172.21.15.76:0/2629111175 conn(0x7f613c0a3e50 msgr2=0x7f613c0a0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:30.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 shutdown_connections 2024-09-16T06:53:30.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.048+0000 7f6143892700 1 -- 172.21.15.76:0/2629111175 wait complete. 2024-09-16T06:53:30.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:53:30.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:53:30.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: get_monmap_and_config 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: build_initial_monmap 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: monmap: 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:30.204568+0000 2024-09-16T06:53:30.199 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:30.204568+0000 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:30.200 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 1 Processor -- start 2024-09-16T06:53:30.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 1 -- start start 2024-09-16T06:53:30.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: init 2024-09-16T06:53:30.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-a con 0x7f55a009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-c con 0x7f55a0098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-b con 0x7f55a0099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a009d560 con 0x7f55a009ed60 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a009f140 con 0x7f55a0098be0 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a009d790 con 0x7f55a0099030 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient(hunting): authenticate will time out at 1757.337982s 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559ffff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f55a009ed60 legacy=0x7f55a00984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a4ef4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f55a0099030 0x7f55a013ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44860/0 (socket says 172.21.15.76:44860) 2024-09-16T06:53:30.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a4ef4700 1 -- 172.21.15.76:0/3253201071 learned_addr learned my addr 172.21.15.76:0/3253201071 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037887099 0 0) 0x7f55a009d790 con 0x7f55a0099030 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): my global_id is 244637 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c0035d0 con 0x7f55a0099030 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257629068 0 0) 0x7f55a009f140 con 0x7f55a0098be0 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient(hunting): my global_id is 284176 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a009d790 con 0x7f55a0098be0 2024-09-16T06:53:30.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699190135 0 0) 0x7f558c0035d0 con 0x7f55a0099030 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 >> v1:172.21.15.189:6789/0 conn(0x7f55a0098be0 legacy=0x7f55a01374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 >> v1:172.21.15.76:6789/0 conn(0x7f55a009ed60 legacy=0x7f55a00984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 monclient: found mon.noname-b 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a009f620 con 0x7f55a0099030 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: _finish_auth 0 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5598001990 con 0x7f55a0099030 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 5 monclient: authenticate success, global_id 244637 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:30.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: dump: 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:30.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5598001c70 con 0x7f55a0099030 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f55a56f5700 4 set_mon_vals no callback set 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 1 -- 172.21.15.76:0/3253201071 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5598002000 con 0x7f55a0099030 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.200+0000 7f559effd700 10 monclient: dump: 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: shutdown 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- 172.21.15.76:0/3253201071 >> v1:172.21.15.76:6790/0 conn(0x7f55a0099030 legacy=0x7f55a013ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- 172.21.15.76:0/3253201071 shutdown_connections 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- 172.21.15.76:0/3253201071 >> 172.21.15.76:0/3253201071 conn(0x7f55a00a0030 msgr2=0x7f55a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- 172.21.15.76:0/3253201071 shutdown_connections 2024-09-16T06:53:30.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- 172.21.15.76:0/3253201071 wait complete. 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: build_initial_monmap 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: monmap: 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:30.215 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 Processor -- start 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- start start 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: init 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-b con 0x7f55a0098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-c con 0x7f55a0099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient(hunting): picked mon.noname-a con 0x7f55a009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a00985f0 con 0x7f55a009ed60 2024-09-16T06:53:30.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a4ef4700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f55a009ed60 legacy=0x7f55a01d2210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a00962c0 con 0x7f55a0098be0 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a00927d0 con 0x7f55a0099030 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f55a56f5700 10 monclient(hunting): authenticate will time out at 1757.343778s 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f559f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f55a0099030 0x7f55a0090f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:44874/0 (socket says 172.21.15.76:44874) 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.204+0000 7f559f7fe700 1 -- 172.21.15.76:0/2852925620 learned_addr learned my addr 172.21.15.76:0/2852925620 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613882585 0 0) 0x7f55a00927d0 con 0x7f55a0099030 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): my global_id is 244640 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55800035d0 con 0x7f55a0099030 2024-09-16T06:53:30.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602576317 0 0) 0x7f55800035d0 con 0x7f55a0099030 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55a00927d0 con 0x7f55a0099030 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5590001850 con 0x7f55a0099030 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): dump: 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.218 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053264622 0 0) 0x7f55a00962c0 con 0x7f55a0098be0 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): my global_id is 284179 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55800035d0 con 0x7f55a0098be0 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4183990039 0 0) 0x7f55a00927d0 con 0x7f55a0099030 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6789/0 conn(0x7f55a0098be0 legacy=0x7f55a0090860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.76:6789/0 conn(0x7f55a009ed60 legacy=0x7f55a01d2210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 monclient: found mon.c 2024-09-16T06:53:30.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a0091680 con 0x7f55a0099030 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: _finish_auth 0 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 5 monclient: authenticate success, global_id 244640 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 10 monclient: _renew_subs 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a0098a10 con 0x7f55a0099030 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 10 monclient: _renew_subs 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a0094380 con 0x7f55a0099030 2024-09-16T06:53:30.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5590003740 con 0x7f55a0099030 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5590003ad0 con 0x7f55a0099030 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559e7fc700 4 set_mon_vals no callback set 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: dump: 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.221 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5584006590 con 0x7f5584002830 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5590003cb0 con 0x7f55a0099030 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5590043e40 con 0x7f55a0099030 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413194004 0 0) 0x7f5584006590 con 0x7f5584002830 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient(hunting): my global_id is 284182 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a00927d0 con 0x7f5584002830 2024-09-16T06:53:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f55a4ef4700 10 monclient: get_auth_request con 0x7f5580036180 auth_method 0 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3477141107 0 0) 0x7f55a00927d0 con 0x7f5584002830 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5584006590 con 0x7f5584002830 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5588004110 con 0x7f5584002830 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 10 monclient: dump: 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.223 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1034137326 0 0) 0x7f5584006590 con 0x7f5584002830 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.208+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f558400a190 con 0x7f5584002830 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.212+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5588006520 con 0x7f5584002830 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.212+0000 7f559cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.212+0000 7f559cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.212+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6789/0 conn(0x7f5584002830 legacy=0x7f5584004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.212+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6789/0 conn(0x7f5584002830 legacy=0x7f5584004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:30.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.216+0000 7f55a56f5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.216+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5584009e30 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789790350 0 0) 0x7f5584009e30 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient(hunting): my global_id is 284185 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5584006590 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 906538894 0 0) 0x7f5584006590 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5584009e30 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f55880049a0 con 0x7f5584009150 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:30.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient: dump: 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 372437971 0 0) 0x7f5584009e30 con 0x7f5584009150 2024-09-16T06:53:30.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f558400c340 con 0x7f5584009150 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 1434647570) 0x7f5588005d60 con 0x7f5584009150 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6789/0 conn(0x7f5584009150 legacy=0x7f55840095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f559cff9700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6789/0 conn(0x7f5584009150 legacy=0x7f55840095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:30.228 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 136, 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.230 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:30.231 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:30.232 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.233 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:30.234 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:30.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.237 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:30.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:30.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:30.239 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:30.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:30.239 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:30.239 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5580036180 legacy=0x7f5580038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 10 monclient: shutdown 2024-09-16T06:53:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 >> v1:172.21.15.76:6790/0 conn(0x7f55a0099030 legacy=0x7f55a0090f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:30.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a4ef4700 1 -- 172.21.15.76:0/2852925620 reap_dead start 2024-09-16T06:53:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a4ef4700 1 -- 172.21.15.76:0/2852925620 reap_dead start 2024-09-16T06:53:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 shutdown_connections 2024-09-16T06:53:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 >> 172.21.15.76:0/2852925620 conn(0x7f55a00a0030 msgr2=0x7f55a00972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 shutdown_connections 2024-09-16T06:53:30.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:30.220+0000 7f55a56f5700 1 -- 172.21.15.76:0/2852925620 wait complete. 2024-09-16T06:53:30.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:53:30.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:53:50.242 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:53:50.243 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:53:50.243 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:53:50.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:53:50.246 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:53:50.246 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:53:50.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: get_monmap_and_config 2024-09-16T06:53:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: monmap: 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:50.368810+0000 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:50.368810+0000 2024-09-16T06:53:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:50.365 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 1 Processor -- start 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 1 -- start start 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: init 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-b con 0x7f09c8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-a con 0x7f09c806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-c con 0x7f09c806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c806c280 con 0x7f09c806d480 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c806bfa0 con 0x7f09c806cf50 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c809ace0 con 0x7f09c8132330 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09cf1d7700 10 monclient(hunting): authenticate will time out at 1777.502124s 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09ce1d5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f09c8132330 0x7f09c8132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54538/0 (socket says 172.21.15.76:54538) 2024-09-16T06:53:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.364+0000 7f09ce1d5700 1 -- 172.21.15.76:0/563681450 learned_addr learned my addr 172.21.15.76:0/563681450 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972522886 0 0) 0x7f09c806bfa0 con 0x7f09c806cf50 2024-09-16T06:53:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient(hunting): my global_id is 284188 2024-09-16T06:53:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b40035d0 con 0x7f09c806cf50 2024-09-16T06:53:50.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289025492 0 0) 0x7f09b40035d0 con 0x7f09c806cf50 2024-09-16T06:53:50.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 >> v1:172.21.15.76:6790/0 conn(0x7f09c8132330 legacy=0x7f09c8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 >> v1:172.21.15.76:6789/0 conn(0x7f09c806d480 legacy=0x7f09c813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 monclient: found mon.noname-c 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c809ae70 con 0x7f09c806cf50 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f09c0001990 con 0x7f09c806cf50 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: dump: 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 5 monclient: authenticate success, global_id 284188 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:50.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c0001bb0 con 0x7f09c806cf50 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 1 -- 172.21.15.76:0/563681450 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f09c0001da0 con 0x7f09c806cf50 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 4 set_mon_vals no callback set 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cd1d3700 10 monclient: dump: 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 10 monclient: shutdown 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 1 -- 172.21.15.76:0/563681450 >> v1:172.21.15.189:6789/0 conn(0x7f09c806cf50 legacy=0x7f09c813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 1 -- 172.21.15.76:0/563681450 shutdown_connections 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 1 -- 172.21.15.76:0/563681450 >> 172.21.15.76:0/563681450 conn(0x7f09c80a41c0 msgr2=0x7f09c80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 1 -- 172.21.15.76:0/563681450 shutdown_connections 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 1 -- 172.21.15.76:0/563681450 wait complete. 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.392+0000 7f09cf1d7700 10 monclient: monmap: 2024-09-16T06:53:50.398 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 1 Processor -- start 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 1 -- start start 2024-09-16T06:53:50.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient: init 2024-09-16T06:53:50.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-c con 0x7f09c806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-b con 0x7f09c806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient(hunting): picked mon.noname-a con 0x7f09c8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c81393b0 con 0x7f09c8132330 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c81362a0 con 0x7f09c806d480 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c80a6c20 con 0x7f09c806cf50 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09cf1d7700 10 monclient(hunting): authenticate will time out at 1777.533512s 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09ce1d5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f09c806cf50 0x7f09c8135480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54540/0 (socket says 172.21.15.76:54540) 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09ce1d5700 1 -- 172.21.15.76:0/2366700195 learned_addr learned my addr 172.21.15.76:0/2366700195 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866679961 0 0) 0x7f09c80a6c20 con 0x7f09c806cf50 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): my global_id is 244643 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ac0035d0 con 0x7f09c806cf50 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439606771 0 0) 0x7f09ac0035d0 con 0x7f09c806cf50 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09c80a6c20 con 0x7f09c806cf50 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f09b80017a0 con 0x7f09c806cf50 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): dump: 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.396+0000 7f09beffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2956859023 0 0) 0x7f09c80a6c20 con 0x7f09c806cf50 2024-09-16T06:53:50.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 >> v1:172.21.15.189:6789/0 conn(0x7f09c806d480 legacy=0x7f09c8135b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 >> v1:172.21.15.76:6789/0 conn(0x7f09c8132330 legacy=0x7f09c81d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 monclient: found mon.c 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c80a7150 con 0x7f09c806cf50 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 5 monclient: authenticate success, global_id 244643 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 10 monclient: _renew_subs 2024-09-16T06:53:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c813dfd0 con 0x7f09c806cf50 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 10 monclient: _renew_subs 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c8136fe0 con 0x7f09c806cf50 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09b8004890 con 0x7f09c806cf50 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f09b8004c20 con 0x7f09c806cf50 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 10 monclient: dump: 2024-09-16T06:53:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cc9d2700 4 set_mon_vals no callback set 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f09b8004dc0 con 0x7f09c806cf50 2024-09-16T06:53:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f09b8043e70 con 0x7f09c806cf50 2024-09-16T06:53:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.400+0000 7f09cf1d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.404+0000 7f09cd9d4700 10 monclient: get_auth_request con 0x7f09ac0361c0 auth_method 0 2024-09-16T06:53:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.404+0000 7f09cf1d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.404+0000 7f09cf1d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.404+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b0005100 con 0x7f09c806cf50 2024-09-16T06:53:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.416+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f09b8014810 con 0x7f09c806cf50 2024-09-16T06:53:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.416+0000 7f09beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.416+0000 7f09beffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:50.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.648+0000 7f09cf1d7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.648+0000 7f09cf1d7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.648+0000 7f09cf1d7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.648+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f09b0005930 con 0x7f09c806cf50 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09beffd700 1 -- 172.21.15.76:0/2366700195 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 847030296) 0x7f09b8012c80 con 0x7f09c806cf50 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:50.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09beffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:50.651 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:50.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 >> v1:172.21.15.189:6812/3098810674 conn(0x7f09ac0361c0 legacy=0x7f09ac038680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 10 monclient: shutdown 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 >> v1:172.21.15.76:6790/0 conn(0x7f09c806cf50 legacy=0x7f09c8135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 shutdown_connections 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 >> 172.21.15.76:0/2366700195 conn(0x7f09c80a41c0 msgr2=0x7f09c8131890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 shutdown_connections 2024-09-16T06:53:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.652+0000 7f09cf1d7700 1 -- 172.21.15.76:0/2366700195 wait complete. 2024-09-16T06:53:50.665 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:53:50.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:53:50.665 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: get_monmap_and_config 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: monmap: 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:50.781544+0000 2024-09-16T06:53:50.776 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:50.781544+0000 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:50.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 1 Processor -- start 2024-09-16T06:53:50.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 1 -- start start 2024-09-16T06:53:50.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: init 2024-09-16T06:53:50.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-c con 0x7f85bc132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-a con 0x7f85bc06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-b con 0x7f85bc06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc06c280 con 0x7f85bc06d480 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc06bfa0 con 0x7f85bc132330 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc09ace0 con 0x7f85bc06cf50 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 10 monclient(hunting): authenticate will time out at 1777.914832s 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c0aae700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f85bc06d480 0x7f85bc13a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51236/0 (socket says 172.21.15.76:51236) 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c0aae700 1 -- 172.21.15.76:0/2870461165 learned_addr learned my addr 172.21.15.76:0/2870461165 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349607022 0 0) 0x7f85bc06c280 con 0x7f85bc06d480 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): my global_id is 294099 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a80035d0 con 0x7f85bc06d480 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058814297 0 0) 0x7f85bc09ace0 con 0x7f85bc06cf50 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): my global_id is 244646 2024-09-16T06:53:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc06c280 con 0x7f85bc06cf50 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882373898 0 0) 0x7f85bc06bfa0 con 0x7f85bc132330 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient(hunting): my global_id is 284191 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc09ace0 con 0x7f85bc132330 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203290245 0 0) 0x7f85a80035d0 con 0x7f85bc06d480 2024-09-16T06:53:50.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 >> v1:172.21.15.76:6790/0 conn(0x7f85bc06cf50 legacy=0x7f85bc13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 >> v1:172.21.15.189:6789/0 conn(0x7f85bc132330 legacy=0x7f85bc132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 monclient: found mon.noname-a 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc09ae70 con 0x7f85bc06d480 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85b8001610 con 0x7f85bc06d480 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:50.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85b3fff700 10 monclient: dump: 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.776+0000 7f85c22b1700 5 monclient: authenticate success, global_id 294099 2024-09-16T06:53:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b80018f0 con 0x7f85bc06d480 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 1 -- 172.21.15.76:0/2870461165 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85b8001ae0 con 0x7f85bc06d480 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85b3fff700 10 monclient: dump: 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.786 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 4 set_mon_vals no callback set 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 10 monclient: shutdown 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- 172.21.15.76:0/2870461165 >> v1:172.21.15.76:6789/0 conn(0x7f85bc06d480 legacy=0x7f85bc13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- 172.21.15.76:0/2870461165 shutdown_connections 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- 172.21.15.76:0/2870461165 >> 172.21.15.76:0/2870461165 conn(0x7f85bc0a41c0 msgr2=0x7f85bc0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- 172.21.15.76:0/2870461165 shutdown_connections 2024-09-16T06:53:50.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- 172.21.15.76:0/2870461165 wait complete. 2024-09-16T06:53:50.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 10 monclient: monmap: 2024-09-16T06:53:50.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 Processor -- start 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 1 -- start start 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.780+0000 7f85c22b1700 10 monclient: init 2024-09-16T06:53:50.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-b con 0x7f85bc06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-c con 0x7f85bc06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient(hunting): picked mon.noname-a con 0x7f85bc132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc138c80 con 0x7f85bc132330 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc139490 con 0x7f85bc06cf50 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc136330 con 0x7f85bc06d480 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient(hunting): authenticate will time out at 1777.920450s 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c0aae700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f85bc06d480 0x7f85bc135c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54546/0 (socket says 172.21.15.76:54546) 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c0aae700 1 -- 172.21.15.76:0/2397658824 learned_addr learned my addr 172.21.15.76:0/2397658824 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c1ab0700 1 --1- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f85bc132330 0x7f85bc1d2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51240/0 (socket says 172.21.15.76:51240) 2024-09-16T06:53:50.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78570899 0 0) 0x7f85bc136330 con 0x7f85bc06d480 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): my global_id is 244649 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a00035d0 con 0x7f85bc06d480 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876982208 0 0) 0x7f85bc138c80 con 0x7f85bc132330 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): my global_id is 294102 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc136330 con 0x7f85bc132330 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375959930 0 0) 0x7f85bc139490 con 0x7f85bc06cf50 2024-09-16T06:53:50.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): my global_id is 284194 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc138c80 con 0x7f85bc06cf50 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2467777790 0 0) 0x7f85a00035d0 con 0x7f85bc06d480 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85bc139490 con 0x7f85bc06d480 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85b8004890 con 0x7f85bc06d480 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.797 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920959551 0 0) 0x7f85bc136330 con 0x7f85bc132330 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85a00035d0 con 0x7f85bc132330 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85b4001c30 con 0x7f85bc132330 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient(hunting): dump: 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 980607262 0 0) 0x7f85bc139490 con 0x7f85bc06d480 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.189:6789/0 conn(0x7f85bc06cf50 legacy=0x7f85bc135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f85bc132330 legacy=0x7f85bc1d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 monclient: found mon.c 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc1370c0 con 0x7f85bc06d480 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 5 monclient: authenticate success, global_id 244649 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _renew_subs 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc09e910 con 0x7f85bc06d480 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _renew_subs 2024-09-16T06:53:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc137a10 con 0x7f85bc06d480 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b8004ca0 con 0x7f85bc06d480 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85b8005030 con 0x7f85bc06d480 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 10 monclient: dump: 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b2ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f859c006590 con 0x7f859c002830 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.784+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f85b8005210 con 0x7f85bc06d480 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f85b8044140 con 0x7f85bc06d480 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256713024 0 0) 0x7f859c006590 con 0x7f859c002830 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient(hunting): my global_id is 294105 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc139490 con 0x7f859c002830 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85c1ab0700 10 monclient: get_auth_request con 0x7f85a00363a0 auth_method 0 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3507491954 0 0) 0x7f85bc139490 con 0x7f859c002830 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f859c006590 con 0x7f859c002830 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85ac0017d0 con 0x7f859c002830 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient: dump: 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2558016130 0 0) 0x7f859c006590 con 0x7f859c002830 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f859c00a190 con 0x7f859c002830 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f85ac001bc0 con 0x7f859c002830 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f859c002830 legacy=0x7f859c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.788+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f859c002830 legacy=0x7f859c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f859c009e30 con 0x7f859c009150 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822039002 0 0) 0x7f859c009e30 con 0x7f859c009150 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient(hunting): my global_id is 294108 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859c006590 con 0x7f859c009150 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2699801466 0 0) 0x7f859c006590 con 0x7f859c009150 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f859c009e30 con 0x7f859c009150 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f85ac003a50 con 0x7f859c009150 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient: dump: 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 49676088 0 0) 0x7f859c009e30 con 0x7f859c009150 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f859c00c340 con 0x7f859c009150 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2044354126) 0x7f85ac004f30 con 0x7f859c009150 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f859c009150 legacy=0x7f859c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85b1ffb700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6789/0 conn(0x7f859c009150 legacy=0x7f859c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:50.809 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.810 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.811 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:50.812 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.813 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:50.814 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.818 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.819 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:50.820 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.189:6812/3098810674 conn(0x7f85a00363a0 legacy=0x7f85a0038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 10 monclient: shutdown 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 >> v1:172.21.15.76:6790/0 conn(0x7f85bc06d480 legacy=0x7f85bc135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c1ab0700 1 -- 172.21.15.76:0/2397658824 reap_dead start 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c1ab0700 1 -- 172.21.15.76:0/2397658824 reap_dead start 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 shutdown_connections 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 >> 172.21.15.76:0/2397658824 conn(0x7f85bc0a41c0 msgr2=0x7f85bc09dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 shutdown_connections 2024-09-16T06:53:50.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.800+0000 7f85c22b1700 1 -- 172.21.15.76:0/2397658824 wait complete. 2024-09-16T06:53:50.823 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:50.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:53:50.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.940+0000 7fcb96c46700 10 monclient: get_monmap_and_config 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.940+0000 7fcb96c46700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.940+0000 7fcb96c46700 10 monclient: monmap: 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:50.947400+0000 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:50.947400+0000 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.942 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.943 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:50.943 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:50.943 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 Processor -- start 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- start start 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient: init 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-a con 0x7fcb90097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-c con 0x7fcb90099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-b con 0x7fcb90139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb90098990 con 0x7fcb90097b60 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb90097f40 con 0x7fcb90099090 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9009d610 con 0x7fcb90139500 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient(hunting): authenticate will time out at 1778.081489s 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb95c44700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcb90097b60 0x7fcb9009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51274/0 (socket says 172.21.15.76:51274) 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb95c44700 1 -- 172.21.15.76:0/381200267 learned_addr learned my addr 172.21.15.76:0/381200267 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465185426 0 0) 0x7fcb90098990 con 0x7fcb90097b60 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): my global_id is 294111 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7c0035d0 con 0x7fcb90097b60 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133293562 0 0) 0x7fcb9009d610 con 0x7fcb90139500 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): my global_id is 244652 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90098990 con 0x7fcb90139500 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489157731 0 0) 0x7fcb90097f40 con 0x7fcb90099090 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient(hunting): my global_id is 284197 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9009d610 con 0x7fcb90099090 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526569349 0 0) 0x7fcb7c0035d0 con 0x7fcb90097b60 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 >> v1:172.21.15.76:6790/0 conn(0x7fcb90139500 legacy=0x7fcb9013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 >> v1:172.21.15.189:6789/0 conn(0x7fcb90099090 legacy=0x7fcb9009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 monclient: found mon.noname-a 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900986b0 con 0x7fcb90097b60 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb8c001610 con 0x7fcb90097b60 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: dump: 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 5 monclient: authenticate success, global_id 294111 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb8c001870 con 0x7fcb90097b60 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 1 -- 172.21.15.76:0/381200267 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb8c001c40 con 0x7fcb90097b60 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 4 set_mon_vals no callback set 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb94c42700 10 monclient: dump: 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 10 monclient: shutdown 2024-09-16T06:53:50.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- 172.21.15.76:0/381200267 >> v1:172.21.15.76:6789/0 conn(0x7fcb90097b60 legacy=0x7fcb9009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- 172.21.15.76:0/381200267 shutdown_connections 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.944+0000 7fcb96c46700 1 -- 172.21.15.76:0/381200267 >> 172.21.15.76:0/381200267 conn(0x7fcb900a0030 msgr2=0x7fcb900a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- 172.21.15.76:0/381200267 shutdown_connections 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- 172.21.15.76:0/381200267 wait complete. 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient: build_initial_monmap 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient: monmap: 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:50.954 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:50.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:50.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 Processor -- start 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- start start 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient: init 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:50.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-c con 0x7fcb90097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-b con 0x7fcb90099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient(hunting): picked mon.noname-a con 0x7fcb90139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb90070430 con 0x7fcb90139500 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb90071b90 con 0x7fcb90099090 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb90133ca0 con 0x7fcb90097b60 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96c46700 10 monclient(hunting): authenticate will time out at 1778.087750s 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96445700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcb90139500 0x7fcb901cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51290/0 (socket says 172.21.15.76:51290) 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb96445700 1 -- 172.21.15.76:0/3794431364 learned_addr learned my addr 172.21.15.76:0/3794431364 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.948+0000 7fcb95c44700 1 --1- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb90097b60 0x7fcb9006f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54574/0 (socket says 172.21.15.76:54574) 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403250256 0 0) 0x7fcb90070430 con 0x7fcb90139500 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): my global_id is 294114 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb740035d0 con 0x7fcb90139500 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645088017 0 0) 0x7fcb90133ca0 con 0x7fcb90097b60 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): my global_id is 244655 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90070430 con 0x7fcb90097b60 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963086316 0 0) 0x7fcb90071b90 con 0x7fcb90099090 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): my global_id is 284200 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90133ca0 con 0x7fcb90099090 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666735030 0 0) 0x7fcb90070430 con 0x7fcb90097b60 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb90071b90 con 0x7fcb90097b60 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb8c001b80 con 0x7fcb90097b60 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): dump: 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.962 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144928386 0 0) 0x7fcb740035d0 con 0x7fcb90139500 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb90070430 con 0x7fcb90139500 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb88001650 con 0x7fcb90139500 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): dump: 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.963 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901815873 0 0) 0x7fcb90071b90 con 0x7fcb90097b60 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.189:6789/0 conn(0x7fcb90099090 legacy=0x7fcb9006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6789/0 conn(0x7fcb90139500 legacy=0x7fcb901cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: _finish_hunting 0 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 monclient: found mon.c 2024-09-16T06:53:50.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb901330f0 con 0x7fcb90097b60 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: _finish_auth 0 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: _check_auth_tickets 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 5 monclient: authenticate success, global_id 244655 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 10 monclient: _renew_subs 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb900711c0 con 0x7fcb90097b60 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 10 monclient: _renew_subs 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb901340c0 con 0x7fcb90097b60 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb8c001190 con 0x7fcb90097b60 2024-09-16T06:53:50.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb8c004ce0 con 0x7fcb90097b60 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient: dump: 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.966 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb83fff700 4 set_mon_vals no callback set 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcb78006590 con 0x7fcb78002830 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fcb8c004e40 con 0x7fcb90097b60 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fcb8c043d50 con 0x7fcb90097b60 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683689842 0 0) 0x7fcb78006590 con 0x7fcb78002830 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 10 monclient(hunting): my global_id is 244658 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90071b90 con 0x7fcb78002830 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.952+0000 7fcb96445700 10 monclient: get_auth_request con 0x7fcb74036350 auth_method 0 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 908441222 0 0) 0x7fcb90071b90 con 0x7fcb78002830 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb78006590 con 0x7fcb78002830 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb840017d0 con 0x7fcb78002830 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 10 monclient: dump: 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.968 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2151640235 0 0) 0x7fcb78006590 con 0x7fcb78002830 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb7800a190 con 0x7fcb78002830 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fcb84001ab0 con 0x7fcb78002830 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb78002830 legacy=0x7fcb78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.956+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb78002830 legacy=0x7fcb78004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb96c46700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcb78009e30 con 0x7fcb78009150 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285446977 0 0) 0x7fcb78009e30 con 0x7fcb78009150 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:50.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient(hunting): my global_id is 244661 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb78006590 con 0x7fcb78009150 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 14817941 0 0) 0x7fcb78006590 con 0x7fcb78009150 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb78009e30 con 0x7fcb78009150 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fcb84003a30 con 0x7fcb78009150 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 10 monclient: dump: 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:50.972 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1264668908 0 0) 0x7fcb78009e30 con 0x7fcb78009150 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.964+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcb7800c340 con 0x7fcb78009150 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 2422469201) 0x7fcb84003bd0 con 0x7fcb78009150 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb827fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb827fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb78009150 legacy=0x7fcb780095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb827fc700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb78009150 legacy=0x7fcb780095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:50.974 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.975 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.976 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:50.977 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:50.978 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.979 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:50.981 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:50.982 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.983 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.984 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:50.985 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:50.986 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:50.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.189:6812/3098810674 conn(0x7fcb74036350 legacy=0x7fcb74038810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 10 monclient: shutdown 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 >> v1:172.21.15.76:6790/0 conn(0x7fcb90097b60 legacy=0x7fcb9006f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96445700 1 -- 172.21.15.76:0/3794431364 reap_dead start 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 shutdown_connections 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 >> 172.21.15.76:0/3794431364 conn(0x7fcb900a0030 msgr2=0x7fcb9012f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 shutdown_connections 2024-09-16T06:53:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:50.968+0000 7fcb96c46700 1 -- 172.21.15.76:0/3794431364 wait complete. 2024-09-16T06:53:50.988 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:50.988 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:53:50.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:53:51.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 10 monclient: get_monmap_and_config 2024-09-16T06:53:51.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 10 monclient: build_initial_monmap 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 10 monclient: monmap: 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:51.119085+0000 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:51.119085+0000 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:51.114 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 1 Processor -- start 2024-09-16T06:53:51.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 1 -- start start 2024-09-16T06:53:51.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.112+0000 7f38a359e700 10 monclient: init 2024-09-16T06:53:51.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f38a4132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f38a406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-a con 0x7f38a406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a406c280 con 0x7f38a406cf50 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a406bfa0 con 0x7f38a406d480 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a409ace0 con 0x7f38a4132330 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient(hunting): authenticate will time out at 1778.252433s 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a259c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f38a4132330 0x7f38a4132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54608/0 (socket says 172.21.15.76:54608) 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a259c700 1 -- 172.21.15.76:0/1293574208 learned_addr learned my addr 172.21.15.76:0/1293574208 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820437297 0 0) 0x7f38a409ace0 con 0x7f38a4132330 2024-09-16T06:53:51.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): my global_id is 244664 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38900035d0 con 0x7f38a4132330 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584084915 0 0) 0x7f38a406c280 con 0x7f38a406cf50 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): my global_id is 294117 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a409ace0 con 0x7f38a406cf50 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764569297 0 0) 0x7f38a406bfa0 con 0x7f38a406d480 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient(hunting): my global_id is 284203 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a406c280 con 0x7f38a406d480 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007898168 0 0) 0x7f38900035d0 con 0x7f38a4132330 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 >> v1:172.21.15.189:6789/0 conn(0x7f38a406d480 legacy=0x7f38a413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 >> v1:172.21.15.76:6789/0 conn(0x7f38a406cf50 legacy=0x7f38a413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 monclient: found mon.noname-b 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a409ae70 con 0x7f38a4132330 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: _finish_auth 0 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:53:51.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 5 monclient: authenticate success, global_id 244664 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3894001610 con 0x7f38a4132330 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: dump: 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.121 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3894001830 con 0x7f38a4132330 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 4 set_mon_vals no callback set 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 1 -- 172.21.15.76:0/1293574208 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3894001a20 con 0x7f38a4132330 2024-09-16T06:53:51.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a1d9b700 10 monclient: dump: 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: shutdown 2024-09-16T06:53:51.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:51.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- 172.21.15.76:0/1293574208 >> v1:172.21.15.76:6790/0 conn(0x7f38a4132330 legacy=0x7f38a4132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- 172.21.15.76:0/1293574208 shutdown_connections 2024-09-16T06:53:51.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- 172.21.15.76:0/1293574208 >> 172.21.15.76:0/1293574208 conn(0x7f38a40a41c0 msgr2=0x7f38a40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:51.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- 172.21.15.76:0/1293574208 shutdown_connections 2024-09-16T06:53:51.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 1 -- 172.21.15.76:0/1293574208 wait complete. 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: build_initial_monmap 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.116+0000 7f38a359e700 10 monclient: monmap: 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:51.128 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 Processor -- start 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- start start 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: init 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-a con 0x7f38a406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f38a406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f38a4132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a409e8b0 con 0x7f38a406cf50 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a41ca6e0 con 0x7f38a4132330 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a40a7050 con 0x7f38a406d480 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f38a406cf50 0x7f38a41c98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51294/0 (socket says 172.21.15.76:51294) 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:51.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f3899d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f38a406d480 0x7f38a41c9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54618/0 (socket says 172.21.15.76:54618) 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a259c700 1 -- 172.21.15.76:0/3325889190 learned_addr learned my addr 172.21.15.76:0/3325889190 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient(hunting): authenticate will time out at 1778.257834s 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795965659 0 0) 0x7f38a40a7050 con 0x7f38a406d480 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): my global_id is 244667 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38840035d0 con 0x7f38a406d480 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636220638 0 0) 0x7f38a41ca6e0 con 0x7f38a4132330 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): my global_id is 284206 2024-09-16T06:53:51.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a40a7050 con 0x7f38a4132330 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630019694 0 0) 0x7f38a409e8b0 con 0x7f38a406cf50 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): my global_id is 294120 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a41ca6e0 con 0x7f38a406cf50 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154465244 0 0) 0x7f38840035d0 con 0x7f38a406d480 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a409e8b0 con 0x7f38a406d480 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f388c0018d0 con 0x7f38a406d480 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): dump: 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049121833 0 0) 0x7f38a40a7050 con 0x7f38a4132330 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38840035d0 con 0x7f38a4132330 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1035944998 0 0) 0x7f38a409e8b0 con 0x7f38a406d480 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6789/0 conn(0x7f38a4132330 legacy=0x7f38a41d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.76:6789/0 conn(0x7f38a406cf50 legacy=0x7f38a41c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 monclient: found mon.c 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40a6b20 con 0x7f38a406d480 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 5 monclient: authenticate success, global_id 244667 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _renew_subs 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a412ea90 con 0x7f38a406d480 2024-09-16T06:53:51.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _renew_subs 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a412dec0 con 0x7f38a406d480 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c001c60 con 0x7f38a406d480 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f388c003b90 con 0x7f38a406d480 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f38a0d99700 4 set_mon_vals no callback set 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.120+0000 7f389b7fe700 10 monclient: dump: 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.134 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f38a359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3888006590 con 0x7f3888002830 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f388c003d70 con 0x7f38a406d480 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f388c043f20 con 0x7f38a406d480 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731235926 0 0) 0x7f3888006590 con 0x7f3888002830 2024-09-16T06:53:51.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient(hunting): my global_id is 284209 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a409e8b0 con 0x7f3888002830 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f38a2d9d700 10 monclient: get_auth_request con 0x7f38840361e0 auth_method 0 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3499902055 0 0) 0x7f38a409e8b0 con 0x7f3888002830 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3888006590 con 0x7f3888002830 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3894004a80 con 0x7f3888002830 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient: dump: 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.136 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2262461439 0 0) 0x7f3888006590 con 0x7f3888002830 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f388800a190 con 0x7f3888002830 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3894004db0 con 0x7f3888002830 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6789/0 conn(0x7f3888002830 legacy=0x7f3888004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.124+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6789/0 conn(0x7f3888002830 legacy=0x7f3888004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3888009e50 con 0x7f3888009170 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731092183 0 0) 0x7f3888009e50 con 0x7f3888009170 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient(hunting): my global_id is 284212 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888006590 con 0x7f3888009170 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 131252559 0 0) 0x7f3888006590 con 0x7f3888009170 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3888009e50 con 0x7f3888009170 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3894005370 con 0x7f3888009170 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient: dump: 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.140 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1671290641 0 0) 0x7f3888009e50 con 0x7f3888009170 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f388800c340 con 0x7f3888009170 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 4013796263) 0x7f38940066f0 con 0x7f3888009170 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6789/0 conn(0x7f3888009170 legacy=0x7f3888009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f389b7fe700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6789/0 conn(0x7f3888009170 legacy=0x7f3888009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:51.141 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:51.142 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:51.143 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:51.144 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.147 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.149 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.150 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.151 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:51.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:51.153 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:51.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:51.153 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:51.153 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.189:6812/3098810674 conn(0x7f38840361e0 legacy=0x7f38840386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 10 monclient: shutdown 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 >> v1:172.21.15.76:6790/0 conn(0x7f38a406d480 legacy=0x7f38a41c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a2d9d700 1 -- 172.21.15.76:0/3325889190 reap_dead start 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 shutdown_connections 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.136+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 >> 172.21.15.76:0/3325889190 conn(0x7f38a40a41c0 msgr2=0x7f38a409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.140+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 shutdown_connections 2024-09-16T06:53:51.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.140+0000 7f38a359e700 1 -- 172.21.15.76:0/3325889190 wait complete. 2024-09-16T06:53:51.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:51.157 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:53:51.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 10 monclient: get_monmap_and_config 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 10 monclient: build_initial_monmap 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 10 monclient: monmap: 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:51.278457+0000 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:51.278457+0000 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:51.273 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 1 Processor -- start 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 1 -- start start 2024-09-16T06:53:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.272+0000 7f1af4af3700 10 monclient: init 2024-09-16T06:53:51.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:51.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:51.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-c con 0x7f1af0070d10 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:51.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-b con 0x7f1af009e920 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-a con 0x7f1af0071800 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0073f50 con 0x7f1af0071800 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af009efc0 con 0x7f1af0070d10 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af009f2a0 con 0x7f1af009e920 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient(hunting): authenticate will time out at 1778.412747s 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aeeffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1af009e920 0x7f1af013a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54632/0 (socket says 172.21.15.76:54632) 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aeffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1af0071800 0x7f1af013db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51310/0 (socket says 172.21.15.76:51310) 2024-09-16T06:53:51.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aeeffd700 1 -- 172.21.15.76:0/352010226 learned_addr learned my addr 172.21.15.76:0/352010226 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361957225 0 0) 0x7f1af009f2a0 con 0x7f1af009e920 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): my global_id is 244670 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad40035d0 con 0x7f1af009e920 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013217478 0 0) 0x7f1af0073f50 con 0x7f1af0071800 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): my global_id is 294123 2024-09-16T06:53:51.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af009f2a0 con 0x7f1af0071800 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223097700 0 0) 0x7f1af009efc0 con 0x7f1af0070d10 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient(hunting): my global_id is 284215 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0073f50 con 0x7f1af0070d10 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1272094657 0 0) 0x7f1ad40035d0 con 0x7f1af009e920 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 >> v1:172.21.15.189:6789/0 conn(0x7f1af0070d10 legacy=0x7f1af00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 >> v1:172.21.15.76:6789/0 conn(0x7f1af0071800 legacy=0x7f1af013db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 monclient: found mon.noname-b 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:51.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00a4ab0 con 0x7f1af009e920 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: _finish_auth 0 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 5 monclient: authenticate success, global_id 244670 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1adc001650 con 0x7f1af009e920 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: dump: 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.285 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1adc0018e0 con 0x7f1af009e920 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 1 -- 172.21.15.76:0/352010226 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1adc001cb0 con 0x7f1af009e920 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:51.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 4 set_mon_vals no callback set 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1aee7fc700 10 monclient: dump: 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.287 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: shutdown 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- 172.21.15.76:0/352010226 >> v1:172.21.15.76:6790/0 conn(0x7f1af009e920 legacy=0x7f1af013a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- 172.21.15.76:0/352010226 shutdown_connections 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- 172.21.15.76:0/352010226 >> 172.21.15.76:0/352010226 conn(0x7f1af00a4430 msgr2=0x7f1af00a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- 172.21.15.76:0/352010226 shutdown_connections 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 1 -- 172.21.15.76:0/352010226 wait complete. 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: build_initial_monmap 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.276+0000 7f1af4af3700 10 monclient: monmap: 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 Processor -- start 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- start start 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: init 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-c con 0x7f1af0070d10 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:51.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-b con 0x7f1af0071800 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient(hunting): picked mon.noname-a con 0x7f1af0137d20 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af013e110 con 0x7f1af0137d20 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af009b0f0 con 0x7f1af0071800 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0138170 con 0x7f1af0070d10 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient(hunting): authenticate will time out at 1778.416614s 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1aef7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1af0070d10 0x7f1af01ca1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54640/0 (socket says 172.21.15.76:54640) 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1aef7fe700 1 -- 172.21.15.76:0/976353122 learned_addr learned my addr 172.21.15.76:0/976353122 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323357894 0 0) 0x7f1af013e110 con 0x7f1af0137d20 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): my global_id is 294126 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad80035d0 con 0x7f1af0137d20 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840741081 0 0) 0x7f1af0138170 con 0x7f1af0070d10 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): my global_id is 244673 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af013e110 con 0x7f1af0070d10 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925076344 0 0) 0x7f1af009b0f0 con 0x7f1af0071800 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): my global_id is 284218 2024-09-16T06:53:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0138170 con 0x7f1af0071800 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867832908 0 0) 0x7f1af013e110 con 0x7f1af0070d10 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af009b0f0 con 0x7f1af0070d10 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1ae0001870 con 0x7f1af0070d10 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): dump: 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.292 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299298827 0 0) 0x7f1ad80035d0 con 0x7f1af0137d20 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af013e110 con 0x7f1af0137d20 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1ae8001df0 con 0x7f1af0137d20 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): dump: 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.293 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3067212284 0 0) 0x7f1af0138170 con 0x7f1af0071800 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad80035d0 con 0x7f1af0071800 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1adc001610 con 0x7f1af0071800 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient(hunting): dump: 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.294 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3463738628 0 0) 0x7f1af013e110 con 0x7f1af0137d20 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 >> v1:172.21.15.76:6790/0 conn(0x7f1af0070d10 legacy=0x7f1af01ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 >> v1:172.21.15.189:6789/0 conn(0x7f1af0071800 legacy=0x7f1af01ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:51.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 monclient: found mon.a 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af01cafc0 con 0x7f1af0137d20 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 5 monclient: authenticate success, global_id 294126 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _renew_subs 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af013e340 con 0x7f1af0137d20 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _renew_subs 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:51.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af012fde0 con 0x7f1af0137d20 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae8004c10 con 0x7f1af0137d20 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1ae8004f20 con 0x7f1af0137d20 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1aedffb700 4 set_mon_vals no callback set 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 10 monclient: dump: 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:51.297 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc005100 con 0x7f1af0137d20 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.280+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1ae800f990 con 0x7f1af0137d20 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.284+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1ae8044120 con 0x7f1af0137d20 2024-09-16T06:53:51.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.284+0000 7f1aef7fe700 10 monclient: get_auth_request con 0x7f1ad8036590 auth_method 0 2024-09-16T06:53:51.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.292+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1ae80147d0 con 0x7f1af0137d20 2024-09-16T06:53:51.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.292+0000 7f1ae7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:51.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.292+0000 7f1ae7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.520+0000 7f1af4af3700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:53:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.520+0000 7f1af4af3700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:53:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.520+0000 7f1af4af3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.520+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1acc005bc0 con 0x7f1af0137d20 2024-09-16T06:53:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1ae7fff700 1 -- 172.21.15.76:0/976353122 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1ae8012e70 con 0x7f1af0137d20 2024-09-16T06:53:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1ae7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:53:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1ae7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:51.523 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.524 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2649670593,config=1097430493,health=1927468327,kv=4212347341,logm=2644599747}) 2024-09-16T06:53:51.524 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=3348760802,config=1097430493,health=2797650238,kv=2418047910,logm=3865922679}) 2024-09-16T06:53:51.524 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.524 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=12,logm=49} crc {auth=2649670593,config=1097430493,health=1927468327,kv=4212347341,logm=2644599747}) 2024-09-16T06:53:51.524 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=170796563,config=1097430493,health=2797650238,kv=3833402919,logm=123075585}) 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1ad8036590 legacy=0x7f1ad8038a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 10 monclient: shutdown 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 >> v1:172.21.15.76:6789/0 conn(0x7f1af0137d20 legacy=0x7f1af01d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 shutdown_connections 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 >> 172.21.15.76:0/976353122 conn(0x7f1af00a4430 msgr2=0x7f1af0132260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 shutdown_connections 2024-09-16T06:53:51.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f1af4af3700 1 -- 172.21.15.76:0/976353122 wait complete. 2024-09-16T06:53:51.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.524+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647007565}) 2024-09-16T06:53:51.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214342919}) 2024-09-16T06:53:51.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647007565}) 2024-09-16T06:53:51.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225421416}) 2024-09-16T06:53:51.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456431205}) 2024-09-16T06:53:51.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782892001}) 2024-09-16T06:53:51.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456431205}) 2024-09-16T06:53:51.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.528+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551480034}) 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257066649}) 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436009719}) 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257066649}) 2024-09-16T06:53:51.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.532+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2207751346}) 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252698367}) 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144064210}) 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252698367}) 2024-09-16T06:53:51.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3998940702}) 2024-09-16T06:53:51.537 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.537 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1395254747}) 2024-09-16T06:53:51.537 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839520664}) 2024-09-16T06:53:51.537 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.537 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1395254747}) 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172068309}) 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882129810}) 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614857915}) 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882129810}) 2024-09-16T06:53:51.538 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2446784393}) 2024-09-16T06:53:51.539 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.539 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758612653}) 2024-09-16T06:53:51.539 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.536+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13770743}) 2024-09-16T06:53:51.539 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.539 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758612653}) 2024-09-16T06:53:51.540 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987157205}) 2024-09-16T06:53:51.541 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:53:51.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484795244}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645494940}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484795244}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902858600}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766050023}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709862791}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766050023}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809216884}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=91478774}) 2024-09-16T06:53:51.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254489062}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=91478774}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.540+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890151588}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604282584}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750041875}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604282584}) 2024-09-16T06:53:51.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143950967}) 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586947959}) 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014782444}) 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586947959}) 2024-09-16T06:53:51.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372835858}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176455236}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856994465}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176455236}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011724462}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445821169}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021988397}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445821169}) 2024-09-16T06:53:51.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497181474}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555365710}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932313397}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555365710}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.544+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560782130}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326439632}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866732655}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326439632}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748897231}) 2024-09-16T06:53:51.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645100721}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259802639}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645100721}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917390224}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279719040}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375874461}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279719040}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=324071192}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060982320}) 2024-09-16T06:53:51.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486069805}) 2024-09-16T06:53:51.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060982320}) 2024-09-16T06:53:51.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.548+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4005491094}) 2024-09-16T06:53:51.557 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.557 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808593313}) 2024-09-16T06:53:51.557 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492697281}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808593313}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194024176}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102136060}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170786724}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102136060}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825237467}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802811576}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992906861}) 2024-09-16T06:53:51.558 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802811576}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572434805}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175303856}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987357802}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175303856}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.552+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3785609842}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433079052}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230647366}) 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.559 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433079052}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436821076}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846607300}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828001231}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846607300}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745422215}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495592720}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153924713}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495592720}) 2024-09-16T06:53:51.560 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010550551}) 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865837417}) 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341989593}) 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865837417}) 2024-09-16T06:53:51.561 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616890451}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121825508}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110667435}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121825508}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.556+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583634379}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905612576}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649129292}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905612576}) 2024-09-16T06:53:51.564 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3865114408}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726344344}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075957179}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=726344344}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111624718}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075154299}) 2024-09-16T06:53:51.565 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225453503}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075154299}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=157864422}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035762557}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563437233}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035762557}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.560+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037053878}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528029482}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633266405}) 2024-09-16T06:53:51.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528029482}) 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2297021075}) 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179452777}) 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1889621142}) 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179452777}) 2024-09-16T06:53:51.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010344193}) 2024-09-16T06:53:51.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483112028}) 2024-09-16T06:53:51.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546147710}) 2024-09-16T06:53:51.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483112028}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541859058}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189700433}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775927080}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189700433}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2643608521}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537351200}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920451476}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537351200}) 2024-09-16T06:53:51.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.564+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674298154}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131993124}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862628192}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131993124}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048280786}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663546016}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219042408}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663546016}) 2024-09-16T06:53:51.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884465144}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279554942}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419295431}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279554942}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2777334664}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726777503}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131661918}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726777503}) 2024-09-16T06:53:51.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.568+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3554036082}) 2024-09-16T06:53:51.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706356697}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114355280}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706356697}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044649868}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134206385}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118021467}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134206385}) 2024-09-16T06:53:51.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777286335}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780866396}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202958150}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780866396}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725921200}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472231806}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703859500}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472231806}) 2024-09-16T06:53:51.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.572+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1598380010}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004138824}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485310619}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004138824}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885740227}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833678599}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286922384}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833678599}) 2024-09-16T06:53:51.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027370745}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845649694}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684348577}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845649694}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3702896530}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220238224}) 2024-09-16T06:53:51.581 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299432379}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220238224}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825700557}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.576+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065661692}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589687677}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065661692}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545528387}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622064804}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829971537}) 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.582 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622064804}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801198416}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657284747}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794088870}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657284747}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=776159806}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744849670}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1284784422}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744849670}) 2024-09-16T06:53:51.583 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428797100}) 2024-09-16T06:53:51.585 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445438249}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349788461}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445438249}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.580+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698977530}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545731110}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2609051264}) 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.586 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545731110}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016026036}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847936551}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749394366}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847936551}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525794754}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838776177}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672825743}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838776177}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461986018}) 2024-09-16T06:53:51.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548312910}) 2024-09-16T06:53:51.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114212320}) 2024-09-16T06:53:51.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548312910}) 2024-09-16T06:53:51.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.584+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1413179808}) 2024-09-16T06:53:51.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2895717106}) 2024-09-16T06:53:51.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555407378}) 2024-09-16T06:53:51.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2895717106}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930267823}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320464821}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3483953789}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320464821}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293454891}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771475493}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853777523}) 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771475493}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991387207}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351421104}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512419288}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351421104}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159870341}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963727287}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.588+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286153173}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963727287}) 2024-09-16T06:53:51.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208069551}) 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572205941}) 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598604572}) 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572205941}) 2024-09-16T06:53:51.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10900817}) 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564047129}) 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121065843}) 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564047129}) 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787852333}) 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.596 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99511004}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380037771}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99511004}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.592+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766991926}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182610324}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547512313}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182610324}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262260964}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101943382}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688722146}) 2024-09-16T06:53:51.597 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101943382}) 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3815671713}) 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2338588608}) 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061269545}) 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2338588608}) 2024-09-16T06:53:51.598 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199568552}) 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243213889}) 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212803763}) 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243213889}) 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2362703155}) 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=690597746}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458041539}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=690597746}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.596+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424625697}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996400038}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101503099}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996400038}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470230245}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=885717887}) 2024-09-16T06:53:51.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849885118}) 2024-09-16T06:53:51.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=885717887}) 2024-09-16T06:53:51.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953399825}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435322311}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970351691}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435322311}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738862705}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.600+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137299497}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801773308}) 2024-09-16T06:53:51.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137299497}) 2024-09-16T06:53:51.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311020445}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868649062}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556934703}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868649062}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=673647056}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3969167288}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1746452648}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3969167288}) 2024-09-16T06:53:51.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757954634}) 2024-09-16T06:53:51.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.606 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862211352}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403067393}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862211352}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.604+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=528293967}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712055617}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598263237}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712055617}) 2024-09-16T06:53:51.607 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067775634}) 2024-09-16T06:53:51.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007935783}) 2024-09-16T06:53:51.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026820229}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007935783}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=544820440}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255194428}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663309214}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3255194428}) 2024-09-16T06:53:51.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123949765}) 2024-09-16T06:53:51.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180273060}) 2024-09-16T06:53:51.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944397160}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180273060}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.608+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279771451}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362314427}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751406682}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362314427}) 2024-09-16T06:53:51.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780215857}) 2024-09-16T06:53:51.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947175732}) 2024-09-16T06:53:51.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000448515}) 2024-09-16T06:53:51.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947175732}) 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556292293}) 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845969179}) 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372679667}) 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845969179}) 2024-09-16T06:53:51.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.612+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025395759}) 2024-09-16T06:53:51.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1030835935,mdsmap=1531151339,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1055817192,mon_config_key=1247158487,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1204363697}) 2024-09-16T06:53:51.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=13} crc {logm=2811208720,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1055817192,mon_config_key=3972503601,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1281731540}) 2024-09-16T06:53:51.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=1030835935,mdsmap=1531151339,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1055817192,mon_config_key=1247158487,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1204363697}) 2024-09-16T06:53:51.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.616+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=32,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=485551916,mdsmap=3140514009,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1131827956,mon_config_key=1816386675,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1303491483}) 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=1082231432}) 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=38} crc {osdmap=2558567096}) 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=1082231432}) 2024-09-16T06:53:51.621 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:53:51.620+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=43} crc {osdmap=327692507}) 2024-09-16T06:53:52.542 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:53:52.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.660+0000 7f480b321700 10 monclient: get_monmap_and_config 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.660+0000 7f480b321700 10 monclient: build_initial_monmap 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.660+0000 7f480b321700 10 monclient: monmap: 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:52.667440+0000 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:52.667440+0000 2024-09-16T06:53:52.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:52.663 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.663 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:52.663 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:52.663 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:52.663 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 Processor -- start 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- start start 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient: init 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient(hunting): picked mon.noname-c con 0x7f4804132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient(hunting): picked mon.noname-a con 0x7f480406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient(hunting): picked mon.noname-b con 0x7f480406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480406c280 con 0x7f480406d480 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480406bfa0 con 0x7f4804132330 2024-09-16T06:53:52.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480409ace0 con 0x7f480406cf50 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient(hunting): authenticate will time out at 1779.800725s 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f4809b1e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f480406d480 0x7f480413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51338/0 (socket says 172.21.15.76:51338) 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f4809b1e700 1 -- 172.21.15.76:0/3882931629 learned_addr learned my addr 172.21.15.76:0/3882931629 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38727775 0 0) 0x7f480406c280 con 0x7f480406d480 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _negotiate 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): my global_id is 294129 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ec0035d0 con 0x7f480406d480 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163592658 0 0) 0x7f480409ace0 con 0x7f480406cf50 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _negotiate 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:52.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient(hunting): my global_id is 244676 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480406c280 con 0x7f480406cf50 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045089997 0 0) 0x7f47ec0035d0 con 0x7f480406d480 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 >> v1:172.21.15.76:6790/0 conn(0x7f480406cf50 legacy=0x7f480413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 >> v1:172.21.15.189:6789/0 conn(0x7f4804132330 legacy=0x7f4804132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: _finish_hunting 0 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 monclient: found mon.noname-a 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480409ae70 con 0x7f480406d480 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: _finish_auth 0 2024-09-16T06:53:52.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: _check_auth_tickets 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47f8001990 con 0x7f480406d480 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: dump: 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 5 monclient: authenticate success, global_id 294129 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47f8001bb0 con 0x7f480406d480 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 1 -- 172.21.15.76:0/3882931629 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47f8001de0 con 0x7f480406d480 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480931d700 10 monclient: dump: 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.670 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 4 set_mon_vals no callback set 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 10 monclient: shutdown 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- 172.21.15.76:0/3882931629 >> v1:172.21.15.76:6789/0 conn(0x7f480406d480 legacy=0x7f480413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- 172.21.15.76:0/3882931629 shutdown_connections 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- 172.21.15.76:0/3882931629 >> 172.21.15.76:0/3882931629 conn(0x7f48040a41c0 msgr2=0x7f48040a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:52.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- 172.21.15.76:0/3882931629 shutdown_connections 2024-09-16T06:53:52.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.664+0000 7f480b321700 1 -- 172.21.15.76:0/3882931629 wait complete. 2024-09-16T06:53:52.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: build_initial_monmap 2024-09-16T06:53:52.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: monmap: 2024-09-16T06:53:52.676 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 Processor -- start 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- start start 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: init 2024-09-16T06:53:52.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient(hunting): picked mon.noname-c con 0x7f480406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient(hunting): picked mon.noname-a con 0x7f480406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient(hunting): picked mon.noname-b con 0x7f4804132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4804130b20 con 0x7f480406d480 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480412e2b0 con 0x7f4804132330 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4804131050 con 0x7f480406cf50 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient(hunting): authenticate will time out at 1779.805751s 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4809b1e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f480406d480 0x7f480412dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51350/0 (socket says 172.21.15.76:51350) 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480a31f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f480406cf50 0x7f480412d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54662/0 (socket says 172.21.15.76:54662) 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4809b1e700 1 -- 172.21.15.76:0/3273981339 learned_addr learned my addr 172.21.15.76:0/3273981339 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:52.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877508923 0 0) 0x7f4804131050 con 0x7f480406cf50 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): my global_id is 244679 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f00035d0 con 0x7f480406cf50 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717576464 0 0) 0x7f4804130b20 con 0x7f480406d480 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): my global_id is 294132 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4804131050 con 0x7f480406d480 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771341251 0 0) 0x7f480412e2b0 con 0x7f4804132330 2024-09-16T06:53:52.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): my global_id is 284224 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4804130b20 con 0x7f4804132330 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4204815803 0 0) 0x7f47f00035d0 con 0x7f480406cf50 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f480412e2b0 con 0x7f480406cf50 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc001910 con 0x7f480406cf50 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.680 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614243133 0 0) 0x7f4804131050 con 0x7f480406d480 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f00035d0 con 0x7f480406d480 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47f80017b0 con 0x7f480406d480 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.681 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2513616358 0 0) 0x7f4804130b20 con 0x7f4804132330 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4804131050 con 0x7f4804132330 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47f40018d0 con 0x7f4804132330 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient(hunting): dump: 2024-09-16T06:53:52.682 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1180704510 0 0) 0x7f480412e2b0 con 0x7f480406cf50 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 >> v1:172.21.15.189:6789/0 conn(0x7f4804132330 legacy=0x7f48041d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 >> v1:172.21.15.76:6789/0 conn(0x7f480406d480 legacy=0x7f480412dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:52.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 monclient: found mon.c 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480412f4e0 con 0x7f480406cf50 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 5 monclient: authenticate success, global_id 244679 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _renew_subs 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804132e60 con 0x7f480406cf50 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _renew_subs 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f480409e910 con 0x7f480406cf50 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47fc001cc0 con 0x7f480406cf50 2024-09-16T06:53:52.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc003b10 con 0x7f480406cf50 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4808b1c700 4 set_mon_vals no callback set 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.668+0000 7f4802ffd700 10 monclient: dump: 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:52.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f47fc003c70 con 0x7f480406cf50 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f480b321700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f480b321700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f480b321700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e4005100 con 0x7f480406cf50 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f47fc043dc0 con 0x7f480406cf50 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.672+0000 7f4809b1e700 10 monclient: get_auth_request con 0x7f47f00365c0 auth_method 0 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.680+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f47fc0146b0 con 0x7f480406cf50 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.680+0000 7f4802ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:52.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.680+0000 7f4802ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:52.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f480b321700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:52.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f480b321700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:52.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f480b321700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:52.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47e4005930 con 0x7f480406cf50 2024-09-16T06:53:52.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f4802ffd700 1 -- 172.21.15.76:0/3273981339 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1181900951) 0x7f47fc01ccf0 con 0x7f480406cf50 2024-09-16T06:53:52.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f4802ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:52.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.916+0000 7f4802ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:52.917 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:52.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 >> v1:172.21.15.189:6812/3098810674 conn(0x7f47f00365c0 legacy=0x7f47f0038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 10 monclient: shutdown 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 >> v1:172.21.15.76:6790/0 conn(0x7f480406cf50 legacy=0x7f480412d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 shutdown_connections 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 >> 172.21.15.76:0/3273981339 conn(0x7f48040a41c0 msgr2=0x7f480409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 shutdown_connections 2024-09-16T06:53:52.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:52.920+0000 7f480b321700 1 -- 172.21.15.76:0/3273981339 wait complete. 2024-09-16T06:53:52.930 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:53:52.930 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:53:52.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:53:52.931 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: get_monmap_and_config 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: monmap: 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.041 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:53.046317+0000 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:53.046317+0000 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 1 Processor -- start 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 1 -- start start 2024-09-16T06:53:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: init 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient(hunting): picked mon.noname-a con 0x7f834009de70 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient(hunting): picked mon.noname-b con 0x7f83400a35f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient(hunting): picked mon.noname-c con 0x7f834006d760 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834006b7a0 con 0x7f834009de70 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834006b4f0 con 0x7f834006d760 2024-09-16T06:53:53.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400a3eb0 con 0x7f83400a35f0 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.040+0000 7f8347248700 10 monclient(hunting): authenticate will time out at 1780.179886s 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8337fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f83400a35f0 0x7f834013a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54676/0 (socket says 172.21.15.76:54676) 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8337fff700 1 -- 172.21.15.76:0/1664484377 learned_addr learned my addr 172.21.15.76:0/1664484377 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922311964 0 0) 0x7f83400a3eb0 con 0x7f83400a35f0 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): my global_id is 244682 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83280035d0 con 0x7f83400a35f0 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157799104 0 0) 0x7f834006b7a0 con 0x7f834009de70 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): my global_id is 294135 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83400a3eb0 con 0x7f834009de70 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448470105 0 0) 0x7f834006b4f0 con 0x7f834006d760 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient(hunting): my global_id is 284227 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834006b7a0 con 0x7f834006d760 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965178614 0 0) 0x7f83280035d0 con 0x7f83400a35f0 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 >> v1:172.21.15.189:6789/0 conn(0x7f834006d760 legacy=0x7f834013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 >> v1:172.21.15.76:6789/0 conn(0x7f834009de70 legacy=0x7f834009e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 monclient: found mon.noname-b 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834006ab80 con 0x7f83400a35f0 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8330001610 con 0x7f83400a35f0 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 5 monclient: authenticate success, global_id 244682 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: dump: 2024-09-16T06:53:53.048 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8330001830 con 0x7f83400a35f0 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 1 -- 172.21.15.76:0/1664484377 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8330001bc0 con 0x7f83400a35f0 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 4 set_mon_vals no callback set 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f83377fe700 10 monclient: dump: 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 10 monclient: shutdown 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 1 -- 172.21.15.76:0/1664484377 >> v1:172.21.15.76:6790/0 conn(0x7f83400a35f0 legacy=0x7f834013a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 1 -- 172.21.15.76:0/1664484377 shutdown_connections 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 1 -- 172.21.15.76:0/1664484377 >> 172.21.15.76:0/1664484377 conn(0x7f83400a4cf0 msgr2=0x7f83400a5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 1 -- 172.21.15.76:0/1664484377 shutdown_connections 2024-09-16T06:53:53.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 1 -- 172.21.15.76:0/1664484377 wait complete. 2024-09-16T06:53:53.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.044+0000 7f8347248700 10 monclient: monmap: 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 Processor -- start 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- start start 2024-09-16T06:53:53.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: init 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient(hunting): picked mon.noname-b con 0x7f834006d760 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient(hunting): picked mon.noname-c con 0x7f834009de70 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient(hunting): picked mon.noname-a con 0x7f834013e420 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340072540 con 0x7f834013e420 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83401c70a0 con 0x7f834006d760 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340136ef0 con 0x7f834009de70 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient(hunting): authenticate will time out at 1780.185218s 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8337fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f834009de70 0x7f83401ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54684/0 (socket says 172.21.15.76:54684) 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8337fff700 1 -- 172.21.15.76:0/367727064 learned_addr learned my addr 172.21.15.76:0/367727064 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013956620 0 0) 0x7f8340136ef0 con 0x7f834009de70 2024-09-16T06:53:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): my global_id is 244685 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c0035d0 con 0x7f834009de70 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826756977 0 0) 0x7f8340072540 con 0x7f834013e420 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): my global_id is 294138 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340136ef0 con 0x7f834013e420 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562755623 0 0) 0x7f83401c70a0 con 0x7f834006d760 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): my global_id is 284230 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340072540 con 0x7f834006d760 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519711829 0 0) 0x7f832c0035d0 con 0x7f834009de70 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83401c70a0 con 0x7f834009de70 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2878281008 0 0) 0x7f8340136ef0 con 0x7f834013e420 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f832c0035d0 con 0x7f834013e420 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8330001510 con 0x7f834009de70 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): dump: 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.061 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8338001bb0 con 0x7f834013e420 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient(hunting): dump: 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3147114163 0 0) 0x7f832c0035d0 con 0x7f834013e420 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6790/0 conn(0x7f834009de70 legacy=0x7f83401ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.189:6789/0 conn(0x7f834006d760 legacy=0x7f834013dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 monclient: found mon.a 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83401371d0 con 0x7f834013e420 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 5 monclient: authenticate success, global_id 294138 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _renew_subs 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8340135fa0 con 0x7f834013e420 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _renew_subs 2024-09-16T06:53:53.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401c6650 con 0x7f834013e420 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8338003a20 con 0x7f834013e420 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8338003d30 con 0x7f834013e420 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f8336ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.048+0000 7f83357fa700 10 monclient: dump: 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8338003e90 con 0x7f834013e420 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8338042d90 con 0x7f834013e420 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f8347248700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8320006590 con 0x7f8320002830 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f8344fe4700 10 monclient: get_auth_request con 0x7f832c036500 auth_method 0 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932282559 0 0) 0x7f8320006590 con 0x7f8320002830 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient(hunting): my global_id is 294141 2024-09-16T06:53:53.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c0035d0 con 0x7f8320002830 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 485038646 0 0) 0x7f832c0035d0 con 0x7f8320002830 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8320006590 con 0x7f8320002830 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8330004c20 con 0x7f8320002830 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient: dump: 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1556636964 0 0) 0x7f8320006590 con 0x7f8320002830 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f832000a190 con 0x7f8320002830 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8330004ec0 con 0x7f8320002830 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6789/0 conn(0x7f8320002830 legacy=0x7f8320004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.052+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6789/0 conn(0x7f8320002830 legacy=0x7f8320004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8320009e50 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084840480 0 0) 0x7f8320009e50 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient(hunting): my global_id is 294144 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320006590 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 477457021 0 0) 0x7f8320006590 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8320009e50 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8330005520 con 0x7f8320009170 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient: dump: 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 255947264 0 0) 0x7f8320009e50 con 0x7f8320009170 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f832000c340 con 0x7f8320009170 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3192438226) 0x7f83300068a0 con 0x7f8320009170 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6789/0 conn(0x7f8320009170 legacy=0x7f8320009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83357fa700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6789/0 conn(0x7f8320009170 legacy=0x7f8320009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:53.074 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.075 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.076 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:53.077 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.078 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.079 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.080 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.081 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.082 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:53.083 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.084 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.085 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:53.085 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.189:6812/3098810674 conn(0x7f832c036500 legacy=0x7f832c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 10 monclient: shutdown 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 >> v1:172.21.15.76:6789/0 conn(0x7f834013e420 legacy=0x7f83401d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83457e5700 1 -- 172.21.15.76:0/367727064 reap_dead start 2024-09-16T06:53:53.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.060+0000 7f83457e5700 1 -- 172.21.15.76:0/367727064 reap_dead start 2024-09-16T06:53:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.064+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 shutdown_connections 2024-09-16T06:53:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.064+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 >> 172.21.15.76:0/367727064 conn(0x7f83400a4cf0 msgr2=0x7f834006ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.064+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 shutdown_connections 2024-09-16T06:53:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.064+0000 7f8347248700 1 -- 172.21.15.76:0/367727064 wait complete. 2024-09-16T06:53:53.087 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:53.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:53:53.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:53:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: get_monmap_and_config 2024-09-16T06:53:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: monmap: 2024-09-16T06:53:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:53.206717+0000 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:53.206717+0000 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:53.202 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 1 Processor -- start 2024-09-16T06:53:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 1 -- start start 2024-09-16T06:53:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: init 2024-09-16T06:53:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f460c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f460c099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f460c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c098990 con 0x7f460c097b60 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c097f40 con 0x7f460c139500 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c09d610 con 0x7f460c099090 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.200+0000 7f460b59e700 10 monclient(hunting): authenticate will time out at 1780.339969s 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f4609d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f460c099090 0x7f460c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54686/0 (socket says 172.21.15.76:54686) 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f4609d9b700 1 -- 172.21.15.76:0/83780556 learned_addr learned my addr 172.21.15.76:0/83780556 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460a59c700 1 --1- 172.21.15.76:0/83780556 >> v1:172.21.15.76:6789/0 conn(0x7f460c097b60 0x7f460c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51416/0 (socket says 172.21.15.76:51416) 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536581924 0 0) 0x7f460c09d610 con 0x7f460c099090 2024-09-16T06:53:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): my global_id is 244688 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f40035d0 con 0x7f460c099090 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285273469 0 0) 0x7f460c098990 con 0x7f460c097b60 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): my global_id is 294147 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c09d610 con 0x7f460c097b60 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035493691 0 0) 0x7f460c097f40 con 0x7f460c139500 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient(hunting): my global_id is 284233 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c098990 con 0x7f460c139500 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121599629 0 0) 0x7f460c09d610 con 0x7f460c097b60 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 >> v1:172.21.15.76:6790/0 conn(0x7f460c099090 legacy=0x7f460c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 >> v1:172.21.15.189:6789/0 conn(0x7f460c139500 legacy=0x7f460c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 monclient: found mon.noname-a 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0986b0 con 0x7f460c097b60 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46000015f0 con 0x7f460c097b60 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 5 monclient: authenticate success, global_id 294147 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: dump: 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.210 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4600001890 con 0x7f460c097b60 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 1 -- 172.21.15.76:0/83780556 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4600001c60 con 0x7f460c097b60 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 4 set_mon_vals no callback set 2024-09-16T06:53:53.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460959a700 10 monclient: dump: 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 10 monclient: shutdown 2024-09-16T06:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- 172.21.15.76:0/83780556 >> v1:172.21.15.76:6789/0 conn(0x7f460c097b60 legacy=0x7f460c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- 172.21.15.76:0/83780556 shutdown_connections 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- 172.21.15.76:0/83780556 >> 172.21.15.76:0/83780556 conn(0x7f460c0a0030 msgr2=0x7f460c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- 172.21.15.76:0/83780556 shutdown_connections 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- 172.21.15.76:0/83780556 wait complete. 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 10 monclient: monmap: 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 Processor -- start 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 1 -- start start 2024-09-16T06:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.204+0000 7f460b59e700 10 monclient: init 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f460c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f460c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f460c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c1354b0 con 0x7f460c099090 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0a2a90 con 0x7f460c097b60 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c133310 con 0x7f460c139500 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient(hunting): authenticate will time out at 1780.345269s 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f4609d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f460c099090 0x7f460c132c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51418/0 (socket says 172.21.15.76:51418) 2024-09-16T06:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f4609d9b700 1 -- 172.21.15.76:0/546435170 learned_addr learned my addr 172.21.15.76:0/546435170 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4655665 0 0) 0x7f460c133310 con 0x7f460c139500 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): my global_id is 244691 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ec0035d0 con 0x7f460c139500 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275140610 0 0) 0x7f460c1354b0 con 0x7f460c099090 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): my global_id is 294150 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c133310 con 0x7f460c099090 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025545978 0 0) 0x7f460c0a2a90 con 0x7f460c097b60 2024-09-16T06:53:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): my global_id is 284236 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c1354b0 con 0x7f460c097b60 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2871845617 0 0) 0x7f45ec0035d0 con 0x7f460c139500 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f460c0a2a90 con 0x7f460c139500 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46040017f0 con 0x7f460c139500 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): dump: 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3457957958 0 0) 0x7f460c0a2a90 con 0x7f460c139500 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.189:6789/0 conn(0x7f460c097b60 legacy=0x7f460c1324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6789/0 conn(0x7f460c099090 legacy=0x7f460c132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 monclient: found mon.c 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0a2fe0 con 0x7f460c139500 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 5 monclient: authenticate success, global_id 244691 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _renew_subs 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c09a540 con 0x7f460c139500 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _renew_subs 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46040036a0 con 0x7f460c139500 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c133b10 con 0x7f460c139500 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46040039b0 con 0x7f460c139500 2024-09-16T06:53:53.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45fbfff700 4 set_mon_vals no callback set 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient: dump: 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f460c0994e0 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269014050 0 0) 0x7f460c1354b0 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.208+0000 7f45faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient(hunting): my global_id is 244694 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c0a2a90 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2296261667 0 0) 0x7f460c0a2a90 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f460c1354b0 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4600001210 con 0x7f460c071550 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient: dump: 2024-09-16T06:53:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f460400e420 con 0x7f460c139500 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f4604042aa0 con 0x7f460c139500 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 298470582 0 0) 0x7f460c1354b0 con 0x7f460c071550 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c135260 con 0x7f460c071550 2024-09-16T06:53:53.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f4609d9b700 10 monclient: get_auth_request con 0x7f45ec036550 auth_method 0 2024-09-16T06:53:53.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f46000014a0 con 0x7f460c071550 2024-09-16T06:53:53.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:53.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6790/0 conn(0x7f460c071550 legacy=0x7f460c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.212+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6790/0 conn(0x7f460c071550 legacy=0x7f460c06ce80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f460b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f460c135010 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429835958 0 0) 0x7f460c135010 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient(hunting): my global_id is 244697 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c1354b0 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1200411471 0 0) 0x7f460c1354b0 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f460c135010 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4600004cc0 con 0x7f460c02c6b0 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient: dump: 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2525738455 0 0) 0x7f460c135010 con 0x7f460c02c6b0 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f460c0973c0 con 0x7f460c02c6b0 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 3319451107) 0x7f4600007030 con 0x7f460c02c6b0 2024-09-16T06:53:53.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6790/0 conn(0x7f460c02c6b0 legacy=0x7f460c06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.220+0000 7f45faffd700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6790/0 conn(0x7f460c02c6b0 legacy=0x7f460c06d590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:53.231 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.232 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.233 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:53.234 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:53.235 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.236 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.238 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.240 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.241 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:53:53.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:53.243 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.189:6812/3098810674 conn(0x7f45ec036550 legacy=0x7f45ec038a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 10 monclient: shutdown 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 >> v1:172.21.15.76:6790/0 conn(0x7f460c139500 legacy=0x7f460c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460ad9d700 1 -- 172.21.15.76:0/546435170 reap_dead start 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460ad9d700 1 -- 172.21.15.76:0/546435170 reap_dead start 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 shutdown_connections 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 >> 172.21.15.76:0/546435170 conn(0x7f460c0a0030 msgr2=0x7f460c0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 shutdown_connections 2024-09-16T06:53:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.224+0000 7f460b59e700 1 -- 172.21.15.76:0/546435170 wait complete. 2024-09-16T06:53:53.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:53.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:53:53.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:53:53.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: get_monmap_and_config 2024-09-16T06:53:53.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: monmap: 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:53.365902+0000 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:53.365902+0000 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:53.361 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 1 Processor -- start 2024-09-16T06:53:53.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 1 -- start start 2024-09-16T06:53:53.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: init 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-a con 0x7fc334132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-b con 0x7fc33406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-c con 0x7fc33406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33406c280 con 0x7fc334132330 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33406bfa0 con 0x7fc33406cf50 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33409ace0 con 0x7fc33406d480 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.360+0000 7fc33936e700 10 monclient(hunting): authenticate will time out at 1780.499984s 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc32bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc33406d480 0x7fc33413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54710/0 (socket says 172.21.15.76:54710) 2024-09-16T06:53:53.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc32bfff700 1 -- 172.21.15.76:0/1861095332 learned_addr learned my addr 172.21.15.76:0/1861095332 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc333fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc334132330 0x7fc334132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51422/0 (socket says 172.21.15.76:51422) 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102604264 0 0) 0x7fc33409ace0 con 0x7fc33406d480 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): my global_id is 244700 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3200035d0 con 0x7fc33406d480 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312991315 0 0) 0x7fc33406c280 con 0x7fc334132330 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient(hunting): my global_id is 294153 2024-09-16T06:53:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33409ace0 con 0x7fc334132330 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1148747984 0 0) 0x7fc3200035d0 con 0x7fc33406d480 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 >> v1:172.21.15.189:6789/0 conn(0x7fc33406cf50 legacy=0x7fc33413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 >> v1:172.21.15.76:6789/0 conn(0x7fc334132330 legacy=0x7fc334132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 monclient: found mon.noname-b 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33409ae70 con 0x7fc33406d480 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 5 monclient: authenticate success, global_id 244700 2024-09-16T06:53:53.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc31c001610 con 0x7fc33406d480 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: dump: 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc31c0018b0 con 0x7fc33406d480 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 1 -- 172.21.15.76:0/1861095332 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc31c001aa0 con 0x7fc33406d480 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 4 set_mon_vals no callback set 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc3337fe700 10 monclient: dump: 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: shutdown 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- 172.21.15.76:0/1861095332 >> v1:172.21.15.76:6790/0 conn(0x7fc33406d480 legacy=0x7fc33413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- 172.21.15.76:0/1861095332 shutdown_connections 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- 172.21.15.76:0/1861095332 >> 172.21.15.76:0/1861095332 conn(0x7fc3340a41c0 msgr2=0x7fc3340a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- 172.21.15.76:0/1861095332 shutdown_connections 2024-09-16T06:53:53.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- 172.21.15.76:0/1861095332 wait complete. 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: monmap: 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.376 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 Processor -- start 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 1 -- start start 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: init 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.364+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-a con 0x7fc33406cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-c con 0x7fc33406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient(hunting): picked mon.noname-b con 0x7fc334132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33412dc40 con 0x7fc33406cf50 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33409e8d0 con 0x7fc334132330 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340a1740 con 0x7fc33406d480 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient(hunting): authenticate will time out at 1780.504112s 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc32bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc33406d480 0x7fc3341cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54720/0 (socket says 172.21.15.76:54720) 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc32bfff700 1 -- 172.21.15.76:0/1513366874 learned_addr learned my addr 172.21.15.76:0/1513366874 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc333fff700 1 --1- 172.21.15.76:0/1513366874 >> v1:172.21.15.76:6789/0 conn(0x7fc33406cf50 0x7fc3341c6940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51426/0 (socket says 172.21.15.76:51426) 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908831059 0 0) 0x7fc3340a1740 con 0x7fc33406d480 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): my global_id is 244703 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3140035d0 con 0x7fc33406d480 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836439303 0 0) 0x7fc33412dc40 con 0x7fc33406cf50 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): my global_id is 294156 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3340a1740 con 0x7fc33406cf50 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977787046 0 0) 0x7fc33409e8d0 con 0x7fc334132330 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): my global_id is 284242 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33412dc40 con 0x7fc334132330 2024-09-16T06:53:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700674873 0 0) 0x7fc3140035d0 con 0x7fc33406d480 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc33409e8d0 con 0x7fc33406d480 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc31c004890 con 0x7fc33406d480 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): dump: 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.380 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3729741818 0 0) 0x7fc3340a1740 con 0x7fc33406cf50 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3140035d0 con 0x7fc33406cf50 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc324001850 con 0x7fc33406cf50 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): dump: 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.381 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2235841030 0 0) 0x7fc33409e8d0 con 0x7fc33406d480 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6789/0 conn(0x7fc334132330 legacy=0x7fc3341d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.76:6789/0 conn(0x7fc33406cf50 legacy=0x7fc3341c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 monclient: found mon.c 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334131340 con 0x7fc33406d480 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 5 monclient: authenticate success, global_id 244703 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient: _renew_subs 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33412cc20 con 0x7fc33406d480 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient: _renew_subs 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3341c7050 con 0x7fc33406d480 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc31c004ca0 con 0x7fc33406d480 2024-09-16T06:53:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc31c004fb0 con 0x7fc33406d480 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc332ffd700 4 set_mon_vals no callback set 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient: dump: 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.384 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc318006590 con 0x7fc318002830 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc31c00fa20 con 0x7fc33406d480 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc31c043fd0 con 0x7fc33406d480 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709969630 0 0) 0x7fc318006590 con 0x7fc318002830 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 10 monclient(hunting): my global_id is 284245 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33409e8d0 con 0x7fc318002830 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.368+0000 7fc338b6d700 10 monclient: get_auth_request con 0x7fc314036470 auth_method 0 2024-09-16T06:53:53.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 695054580 0 0) 0x7fc33409e8d0 con 0x7fc318002830 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc318006590 con 0x7fc318002830 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc324004890 con 0x7fc318002830 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 10 monclient: dump: 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3844414849 0 0) 0x7fc318006590 con 0x7fc318002830 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc31800a190 con 0x7fc318002830 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc3240058c0 con 0x7fc318002830 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6789/0 conn(0x7fc318002830 legacy=0x7fc318004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.372+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6789/0 conn(0x7fc318002830 legacy=0x7fc318004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:53:53.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc318009e50 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178946699 0 0) 0x7fc318009e50 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient(hunting): my global_id is 284248 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc318006590 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 891697549 0 0) 0x7fc318006590 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc318009e50 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc324004ed0 con 0x7fc318009170 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient: dump: 2024-09-16T06:53:53.389 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3905600085 0 0) 0x7fc318009e50 con 0x7fc318009170 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc31800c340 con 0x7fc318009170 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1948868638) 0x7fc324005af0 con 0x7fc318009170 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:53:53.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6789/0 conn(0x7fc318009170 legacy=0x7fc318009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc3317fa700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6789/0 conn(0x7fc318009170 legacy=0x7fc318009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 138, 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:53:53.391 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.392 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.393 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:53:53.394 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:53:53.396 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:53:53.396 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:53:53.396 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.396 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:53:53.397 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:53:53.398 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.399 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:53:53.401 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:53:53.402 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:53:53.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:53:53.404 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:53:53.404 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:53:53.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc314036470 legacy=0x7fc314038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 10 monclient: shutdown 2024-09-16T06:53:53.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 >> v1:172.21.15.76:6790/0 conn(0x7fc33406d480 legacy=0x7fc3341cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc338b6d700 1 -- 172.21.15.76:0/1513366874 reap_dead start 2024-09-16T06:53:53.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.384+0000 7fc338b6d700 1 -- 172.21.15.76:0/1513366874 reap_dead start 2024-09-16T06:53:53.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.388+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 shutdown_connections 2024-09-16T06:53:53.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.388+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 >> 172.21.15.76:0/1513366874 conn(0x7fc3340a41c0 msgr2=0x7fc334129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.388+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 shutdown_connections 2024-09-16T06:53:53.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.388+0000 7fc33936e700 1 -- 172.21.15.76:0/1513366874 wait complete. 2024-09-16T06:53:53.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:53:53.406 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:53:53.406 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:53:53.407 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:53:53.407 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:53:53.407 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:53:53.407 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:53:53.459 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:53:53.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:53:53.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: get_monmap_and_config 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: monmap: 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:53.578371+0000 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:53.578371+0000 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 1 Processor -- start 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 1 -- start start 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: init 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-c con 0x7fc1f809f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-a con 0x7fc1f809c460 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-b con 0x7fc1f809fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f809a040 con 0x7fc1f809c460 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f809ac80 con 0x7fc1f809f230 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f809def0 con 0x7fc1f809fd20 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1ff386700 10 monclient(hunting): authenticate will time out at 1780.711759s 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.572+0000 7fc1fc921700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc1f809c460 legacy=0x7fc1f813ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1fd923700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1f809fd20 0x7fc1f8142160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54726/0 (socket says 172.21.15.76:54726) 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1fd923700 1 -- 172.21.15.76:0/2002101230 learned_addr learned my addr 172.21.15.76:0/2002101230 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736365964 0 0) 0x7fc1f809def0 con 0x7fc1f809fd20 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): my global_id is 244706 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e40035d0 con 0x7fc1f809fd20 2024-09-16T06:53:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359559411 0 0) 0x7fc1e8001110 con 0x7fc1f809f230 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient(hunting): my global_id is 284251 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f809def0 con 0x7fc1f809f230 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379610237 0 0) 0x7fc1e40035d0 con 0x7fc1f809fd20 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 >> v1:172.21.15.189:6789/0 conn(0x7fc1f809f230 legacy=0x7fc1f809f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 >> v1:172.21.15.76:6789/0 conn(0x7fc1f809c460 legacy=0x7fc1f813ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 monclient: found mon.noname-b 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f809a410 con 0x7fc1f809fd20 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1f0001610 con 0x7fc1f809fd20 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: dump: 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 5 monclient: authenticate success, global_id 244706 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f00018f0 con 0x7fc1f809fd20 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 1 -- 172.21.15.76:0/2002101230 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1f0001c80 con 0x7fc1f809fd20 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1effff700 10 monclient: dump: 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 4 set_mon_vals no callback set 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 10 monclient: shutdown 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- 172.21.15.76:0/2002101230 >> v1:172.21.15.76:6790/0 conn(0x7fc1f809fd20 legacy=0x7fc1f8142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- 172.21.15.76:0/2002101230 shutdown_connections 2024-09-16T06:53:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- 172.21.15.76:0/2002101230 >> 172.21.15.76:0/2002101230 conn(0x7fc1f80a3e50 msgr2=0x7fc1f80a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- 172.21.15.76:0/2002101230 shutdown_connections 2024-09-16T06:53:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- 172.21.15.76:0/2002101230 wait complete. 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 10 monclient: build_initial_monmap 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 10 monclient: monmap: 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 Processor -- start 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.576+0000 7fc1ff386700 1 -- start start 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: init 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-a con 0x7fc1f809c460 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-c con 0x7fc1f809f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient(hunting): picked mon.noname-b con 0x7fc1f809fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1fd122700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fc1f809c460 legacy=0x7fc1f8134fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f8136920 con 0x7fc1f809c460 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f8135dd0 con 0x7fc1f809fd20 2024-09-16T06:53:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1fc921700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc1f809f230 0x7fc1f81356c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54734/0 (socket says 172.21.15.76:54734) 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1fc921700 1 -- 172.21.15.76:0/1394927602 learned_addr learned my addr 172.21.15.76:0/1394927602 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f8137d40 con 0x7fc1f809f230 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient(hunting): authenticate will time out at 1780.717505s 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534666533 0 0) 0x7fc1f8137d40 con 0x7fc1f809f230 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): my global_id is 244709 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc0035d0 con 0x7fc1f809f230 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018248602 0 0) 0x7fc1f8135dd0 con 0x7fc1f809fd20 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): my global_id is 284254 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f8137d40 con 0x7fc1f809fd20 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972113831 0 0) 0x7fc1dc0035d0 con 0x7fc1f809f230 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f8135dd0 con 0x7fc1f809f230 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1f4004580 con 0x7fc1f809f230 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): dump: 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1576588524 0 0) 0x7fc1f8135dd0 con 0x7fc1f809f230 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 >> v1:172.21.15.189:6789/0 conn(0x7fc1f809fd20 legacy=0x7fc1f81d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 >> v1:172.21.15.76:6789/0 conn(0x7fc1f809c460 legacy=0x7fc1f8134fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 monclient: found mon.c 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8138410 con 0x7fc1f809f230 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 5 monclient: authenticate success, global_id 244709 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _renew_subs 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f81386f0 con 0x7fc1f809f230 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _renew_subs 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f8139800 con 0x7fc1f809f230 2024-09-16T06:53:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f4004860 con 0x7fc1f809f230 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc1f4004bf0 con 0x7fc1f809f230 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1ef7fe700 4 set_mon_vals no callback set 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.580+0000 7fc1edffb700 10 monclient: dump: 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:53.594 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1ff386700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1ff386700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1ff386700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f80678b0 con 0x7fc1f809f230 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc1f4004e10 con 0x7fc1f809f230 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc1f40441c0 con 0x7fc1f809f230 2024-09-16T06:53:53.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.584+0000 7fc1fd122700 10 monclient: get_auth_request con 0x7fc1dc035fa0 auth_method 0 2024-09-16T06:53:53.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.592+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc1f4012f70 con 0x7fc1f809f230 2024-09-16T06:53:53.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.592+0000 7fc1edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:53.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.592+0000 7fc1edffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:53.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1ff386700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:53.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1ff386700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:53.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1ff386700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:53.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc1f8069620 con 0x7fc1f809f230 2024-09-16T06:53:53.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1edffb700 1 -- 172.21.15.76:0/1394927602 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4270014376) 0x7fc1f4012f70 con 0x7fc1f809f230 2024-09-16T06:53:53.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1edffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:53.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.824+0000 7fc1edffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:53.825 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:53.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc1dc035fa0 legacy=0x7fc1dc038460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 10 monclient: shutdown 2024-09-16T06:53:53.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:53.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 >> v1:172.21.15.76:6790/0 conn(0x7fc1f809f230 legacy=0x7fc1f81356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:53.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 shutdown_connections 2024-09-16T06:53:53.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 >> 172.21.15.76:0/1394927602 conn(0x7fc1f80a3e50 msgr2=0x7fc1f809c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:53.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 shutdown_connections 2024-09-16T06:53:53.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:53.828+0000 7fc1ff386700 1 -- 172.21.15.76:0/1394927602 wait complete. 2024-09-16T06:53:56.840 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: build_initial_monmap 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: monmap: 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:53:56.965172+0000 2024-09-16T06:53:56.960 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:53:56.965172+0000 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 1 Processor -- start 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 1 -- start start 2024-09-16T06:53:56.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: init 2024-09-16T06:53:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:53:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fbb20097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fbb20099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fbb20139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb20098990 con 0x7fbb20097b60 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb20097f40 con 0x7fbb20139500 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2009d610 con 0x7fbb20099090 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient(hunting): authenticate will time out at 1784.098400s 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1effd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fbb20097b60 legacy=0x7fbb2009ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1e7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fbb20099090 0x7fbb2009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57650/0 (socket says 172.21.15.76:57650) 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1e7fc700 1 -- 172.21.15.76:0/2929499388 learned_addr learned my addr 172.21.15.76:0/2929499388 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044551037 0 0) 0x7fbb2009d610 con 0x7fbb20099090 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:53:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient(hunting): my global_id is 244712 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0c0035d0 con 0x7fbb20099090 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804978699 0 0) 0x7fbb0c0035d0 con 0x7fbb20099090 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 >> v1:172.21.15.189:6789/0 conn(0x7fbb20139500 legacy=0x7fbb2013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 >> v1:172.21.15.76:6789/0 conn(0x7fbb20097b60 legacy=0x7fbb2009ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 monclient: found mon.noname-b 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200986b0 con 0x7fbb20099090 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: _finish_auth 0 2024-09-16T06:53:56.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbb08001990 con 0x7fbb20099090 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 5 monclient: authenticate success, global_id 244712 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: dump: 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:56.969 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb08001cb0 con 0x7fbb20099090 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1ffff700 4 set_mon_vals no callback set 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 1 -- 172.21.15.76:0/2929499388 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbb08002080 con 0x7fbb20099090 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:56.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.960+0000 7fbb1dffb700 10 monclient: dump: 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: shutdown 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/2929499388 >> v1:172.21.15.76:6790/0 conn(0x7fbb20099090 legacy=0x7fbb2009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:56.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/2929499388 shutdown_connections 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/2929499388 >> 172.21.15.76:0/2929499388 conn(0x7fbb200a0030 msgr2=0x7fbb200a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/2929499388 shutdown_connections 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/2929499388 wait complete. 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: build_initial_monmap 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: monmap: 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:53:56.972 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 Processor -- start 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- start start 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: init 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fbb20097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fbb20099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fbb20139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2009a540 con 0x7fbb20099090 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2006ef40 con 0x7fbb20097b60 2024-09-16T06:53:56.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2006ccb0 con 0x7fbb20139500 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1e7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fbb20099090 legacy=0x7fbb2006c5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient(hunting): authenticate will time out at 1784.102039s 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb20097b60 0x7fbb2006be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:45278/0 (socket says 172.21.15.76:45278) 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1effd700 1 -- 172.21.15.76:0/920587615 learned_addr learned my addr 172.21.15.76:0/920587615 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1f7fe700 1 --1- 172.21.15.76:0/920587615 >> v1:172.21.15.76:6790/0 conn(0x7fbb20139500 0x7fbb201cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57656/0 (socket says 172.21.15.76:57656) 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289904891 0 0) 0x7fbb2006ccb0 con 0x7fbb20139500 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): _negotiate 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): my global_id is 244715 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb000035d0 con 0x7fbb20139500 2024-09-16T06:53:56.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543425092 0 0) 0x7fbb000035d0 con 0x7fbb20139500 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb2006ccb0 con 0x7fbb20139500 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbb18001870 con 0x7fbb20139500 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): dump: 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:56.975 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 804631878 0 0) 0x7fbb2006ccb0 con 0x7fbb20139500 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 >> v1:172.21.15.189:6789/0 conn(0x7fbb20097b60 legacy=0x7fbb2006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 >> v1:172.21.15.76:6789/0 conn(0x7fbb20099090 legacy=0x7fbb2006c5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: _finish_hunting 0 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 monclient: found mon.c 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200a2c10 con 0x7fbb20139500 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: _finish_auth 0 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: _check_auth_tickets 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:53:56.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 5 monclient: authenticate success, global_id 244715 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _renew_subs 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2006f500 con 0x7fbb20139500 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _renew_subs 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2006d130 con 0x7fbb20139500 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb180036c0 con 0x7fbb20139500 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fbb18003a50 con 0x7fbb20139500 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb1d7fa700 4 set_mon_vals no callback set 2024-09-16T06:53:56.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.964+0000 7fbb17fff700 10 monclient: dump: 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:53:56.978 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb1ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fbb18003c30 con 0x7fbb20139500 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb1ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb1ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb04005100 con 0x7fbb20139500 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fbb18043ea0 con 0x7fbb20139500 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.968+0000 7fbb1effd700 10 monclient: get_auth_request con 0x7fbb000360d0 auth_method 0 2024-09-16T06:53:56.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.976+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fbb180147e0 con 0x7fbb20139500 2024-09-16T06:53:56.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.976+0000 7fbb17fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:53:56.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:56.976+0000 7fbb17fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:53:57.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb1ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb1ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb1ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbb04005930 con 0x7fbb20139500 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb17fff700 1 -- 172.21.15.76:0/920587615 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2926571913) 0x7fbb18012c50 con 0x7fbb20139500 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb17fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:53:57.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.208+0000 7fbb17fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:53:57.209 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:53:57.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 >> v1:172.21.15.189:6812/3098810674 conn(0x7fbb000360d0 legacy=0x7fbb00038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:57.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 10 monclient: shutdown 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 >> v1:172.21.15.76:6790/0 conn(0x7fbb20139500 legacy=0x7fbb201cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 shutdown_connections 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 >> 172.21.15.76:0/920587615 conn(0x7fbb200a0030 msgr2=0x7fbb2013aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 shutdown_connections 2024-09-16T06:53:57.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:53:57.212+0000 7fbb1ffff700 1 -- 172.21.15.76:0/920587615 wait complete. 2024-09-16T06:54:00.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: get_monmap_and_config 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: build_initial_monmap 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: monmap: 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:00.346222+0000 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:00.346222+0000 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:00.341 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 1 Processor -- start 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 1 -- start start 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: init 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe6780973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe678097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe6781400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678099350 con 0x7fe6781400d0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67809f690 con 0x7fe6780973c0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678099bf0 con 0x7fe678097eb0 2024-09-16T06:54:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:00.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe67759e700 10 monclient(hunting): authenticate will time out at 1787.479499s 2024-09-16T06:54:00.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe676d9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:00.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe675d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe678097eb0 0x7fe67813e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:57670/0 (socket says 172.21.15.76:57670) 2024-09-16T06:54:00.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.343+0000 7fe675d9b700 1 -- 172.21.15.76:0/4036954187 learned_addr learned my addr 172.21.15.76:0/4036954187 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:00.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.547+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:00.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:00.951+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:01.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:01.755+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781424c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:03.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): tick 2024-09-16T06:54:03.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:54:03.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 monclient(hunting): continuing hunt 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6790/0 conn(0x7fe678097eb0 legacy=0x7fe67813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.189:6789/0 conn(0x7fe6780973c0 legacy=0x7fe6780977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781424c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe658003740 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe6781400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe6780973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678098360 con 0x7fe6781400d0 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678099350 con 0x7fe6780973c0 2024-09-16T06:54:03.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658010450 con 0x7fe658003740 2024-09-16T06:54:03.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe675d9b700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658009470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:03.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.343+0000 7fe667fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:03.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.547+0000 7fe675d9b700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658009470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:03.951+0000 7fe675d9b700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658009470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:04.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:04.755+0000 7fe675d9b700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658009470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): tick 2024-09-16T06:54:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:54:06.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 monclient(hunting): continuing hunt 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6790/0 conn(0x7fe658003740 legacy=0x7fe658005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.189:6789/0 conn(0x7fe6780973c0 legacy=0x7fe65800cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658009470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe65800f4d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe6580105e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe6781400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658011660 con 0x7fe6781400d0 2024-09-16T06:54:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678098360 con 0x7fe65800f4d0 2024-09-16T06:54:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580110f0 con 0x7fe6580105e0 2024-09-16T06:54:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe667fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:06.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.347+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658006360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:06.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.551+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658006360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:06.955+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658006360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:07.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844667985 0 0) 0x7fe678098360 con 0x7fe65800f4d0 2024-09-16T06:54:07.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 10 monclient(hunting): _negotiate 2024-09-16T06:54:07.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:07.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:07.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 10 monclient(hunting): my global_id is 284257 2024-09-16T06:54:07.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.755+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6600035d0 con 0x7fe65800f4d0 2024-09-16T06:54:07.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.759+0000 7fe676d9d700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658006360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:07.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256321156 0 0) 0x7fe6580110f0 con 0x7fe6580105e0 2024-09-16T06:54:07.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient(hunting): _negotiate 2024-09-16T06:54:07.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:07.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:07.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient(hunting): my global_id is 244718 2024-09-16T06:54:07.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe678098360 con 0x7fe6580105e0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793794490 0 0) 0x7fe678098360 con 0x7fe6580105e0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.189:6789/0 conn(0x7fe65800f4d0 legacy=0x7fe65800f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6789/0 conn(0x7fe6781400d0 legacy=0x7fe658006360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: _finish_hunting 0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 monclient: found mon.noname-b 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658011280 con 0x7fe6580105e0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: _finish_auth 0 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: _check_auth_tickets 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:07.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67759e700 5 monclient: authenticate success, global_id 244718 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe668001960 con 0x7fe6580105e0 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67559a700 10 monclient: dump: 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:07.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:07.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.763+0000 7fe67759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe668001c00 con 0x7fe6580105e0 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 4 set_mon_vals no callback set 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 1 -- 172.21.15.76:0/4036954187 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe668001fd0 con 0x7fe6580105e0 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67559a700 10 monclient: dump: 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:07.763 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:07.764 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 10 monclient: shutdown 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 1 -- 172.21.15.76:0/4036954187 >> v1:172.21.15.76:6790/0 conn(0x7fe6580105e0 legacy=0x7fe6580109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 1 -- 172.21.15.76:0/4036954187 shutdown_connections 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 1 -- 172.21.15.76:0/4036954187 >> 172.21.15.76:0/4036954187 conn(0x7fe6780a0030 msgr2=0x7fe6780a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:07.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 1 -- 172.21.15.76:0/4036954187 shutdown_connections 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 1 -- 172.21.15.76:0/4036954187 wait complete. 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 10 monclient: build_initial_monmap 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.767+0000 7fe67759e700 10 monclient: monmap: 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:07.766 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:07.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 1 Processor -- start 2024-09-16T06:54:07.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 1 -- start start 2024-09-16T06:54:07.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient: init 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe65800f4d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe6580105e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe675d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fe6580105e0 legacy=0x7fe6781d3490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:07.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe6781400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678072660 con 0x7fe6580105e0 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67813fae0 con 0x7fe6781400d0 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe65800f4d0 0x7fe6781ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:45082/0 (socket says 172.21.15.76:45082) 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67659c700 1 -- 172.21.15.76:0/3813868726 learned_addr learned my addr 172.21.15.76:0/3813868726 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67809a540 con 0x7fe65800f4d0 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 10 monclient(hunting): authenticate will time out at 1794.906678s 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243005142 0 0) 0x7fe67809a540 con 0x7fe65800f4d0 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): my global_id is 244721 2024-09-16T06:54:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658011660 con 0x7fe65800f4d0 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040808073 0 0) 0x7fe658011660 con 0x7fe65800f4d0 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe67809a540 con 0x7fe65800f4d0 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe66c001780 con 0x7fe65800f4d0 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): dump: 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:07.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151059872 0 0) 0x7fe678099350 con 0x7fe6781400d0 2024-09-16T06:54:07.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient(hunting): my global_id is 284260 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658011660 con 0x7fe6781400d0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2338672916 0 0) 0x7fe67809a540 con 0x7fe65800f4d0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 >> v1:172.21.15.189:6789/0 conn(0x7fe6781400d0 legacy=0x7fe6781d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 >> v1:172.21.15.76:6789/0 conn(0x7fe6580105e0 legacy=0x7fe6781d3490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 monclient: found mon.c 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67813ee70 con 0x7fe65800f4d0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:07.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe666ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.771+0000 7fe67759e700 5 monclient: authenticate success, global_id 244721 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _renew_subs 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67806ca30 con 0x7fe65800f4d0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _renew_subs 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67806bdb0 con 0x7fe65800f4d0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe66c0036a0 con 0x7fe65800f4d0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe66c003a30 con 0x7fe65800f4d0 2024-09-16T06:54:07.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe674d99700 4 set_mon_vals no callback set 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 10 monclient: dump: 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:07.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe66c003c10 con 0x7fe65800f4d0 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fe66c043d60 con 0x7fe65800f4d0 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:07.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.775+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe678094fc0 con 0x7fe65800f4d0 2024-09-16T06:54:07.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.787+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fe66c0146a0 con 0x7fe65800f4d0 2024-09-16T06:54:07.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.787+0000 7fe666ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:07.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:07.787+0000 7fe666ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:08.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe67759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe67759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe67759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6780672a0 con 0x7fe65800f4d0 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe666ffd700 1 -- 172.21.15.76:0/3813868726 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1691874137) 0x7fe66c012b10 con 0x7fe65800f4d0 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe666ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:08.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.023+0000 7fe666ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:08.022 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":142,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe6580376b0 legacy=0x7fe658039b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 10 monclient: shutdown 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 >> v1:172.21.15.76:6790/0 conn(0x7fe65800f4d0 legacy=0x7fe6781ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 shutdown_connections 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 >> 172.21.15.76:0/3813868726 conn(0x7fe6780a0030 msgr2=0x7fe67809c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 shutdown_connections 2024-09-16T06:54:08.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.027+0000 7fe67759e700 1 -- 172.21.15.76:0/3813868726 wait complete. 2024-09-16T06:54:08.035 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:54:08.035 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:54:08.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: monmap: 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:08.151994+0000 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:08.151994+0000 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.158 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 Processor -- start 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- start start 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: init 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3d4c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3d4c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3d4c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c06c280 con 0x7f3d4c06d480 2024-09-16T06:54:08.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c06bfa0 con 0x7f3d4c06cf50 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c09ace0 con 0x7f3d4c132330 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient(hunting): authenticate will time out at 1795.285279s 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4a7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3d4c06d480 legacy=0x7f3d4c13a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3d4c132330 0x7f3d4c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:45090/0 (socket says 172.21.15.76:45090) 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4affd700 1 -- 172.21.15.76:0/3282470682 learned_addr learned my addr 172.21.15.76:0/3282470682 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58858750 0 0) 0x7f3d4c09ace0 con 0x7f3d4c132330 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): my global_id is 244724 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d340035d0 con 0x7f3d4c132330 2024-09-16T06:54:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364427253 0 0) 0x7f3d4c06bfa0 con 0x7f3d4c06cf50 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient(hunting): my global_id is 284263 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d4c09ace0 con 0x7f3d4c06cf50 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490611355 0 0) 0x7f3d340035d0 con 0x7f3d4c132330 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 >> v1:172.21.15.189:6789/0 conn(0x7f3d4c06cf50 legacy=0x7f3d4c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 >> v1:172.21.15.76:6789/0 conn(0x7f3d4c06d480 legacy=0x7f3d4c13a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 monclient: found mon.noname-b 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c09ae70 con 0x7f3d4c132330 2024-09-16T06:54:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: _finish_auth 0 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 5 monclient: authenticate success, global_id 244724 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d3c001990 con 0x7f3d4c132330 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: dump: 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d3c001c70 con 0x7f3d4c132330 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 4 set_mon_vals no callback set 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 1 -- 172.21.15.76:0/3282470682 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d3c001e60 con 0x7f3d4c132330 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d49ffb700 10 monclient: dump: 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: shutdown 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- 172.21.15.76:0/3282470682 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c132330 legacy=0x7f3d4c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- 172.21.15.76:0/3282470682 shutdown_connections 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- 172.21.15.76:0/3282470682 >> 172.21.15.76:0/3282470682 conn(0x7f3d4c0a41c0 msgr2=0x7f3d4c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- 172.21.15.76:0/3282470682 shutdown_connections 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 1 -- 172.21.15.76:0/3282470682 wait complete. 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.151+0000 7f3d4bfff700 10 monclient: monmap: 2024-09-16T06:54:08.165 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 Processor -- start 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- start start 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: init 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:08.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3d4c06cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3d4c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3d4c132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c1ca690 con 0x7f3d4c06d480 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c12f010 con 0x7f3d4c132330 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c1311d0 con 0x7f3d4c06cf50 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient(hunting): authenticate will time out at 1795.289212s 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4a7fc700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3d4c06d480 legacy=0x7f3d4c1cea90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 0x7f3d4c13e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:45092/0 (socket says 172.21.15.76:45092) 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 -- 172.21.15.76:0/1312763670 learned_addr learned my addr 172.21.15.76:0/1312763670 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 legacy=0x7f3d4c13e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:54:08.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 legacy=0x7f3d4c13e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 --1- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 0x7f3d4c13e3a0 :-1 s=OPENED pgs=1646 cs=1 l=1).handle_message read tag failed 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4affd700 1 --1- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 0x7f3d4c13e3a0 :-1 s=OPENED pgs=1646 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034433057 0 0) 0x7f3d4c12f010 con 0x7f3d4c132330 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): my global_id is 284266 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d380035d0 con 0x7f3d4c132330 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255307555 0 0) 0x7f3d380035d0 con 0x7f3d4c132330 2024-09-16T06:54:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d4c12f010 con 0x7f3d4c132330 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d440018b0 con 0x7f3d4c132330 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): dump: 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1133544318 0 0) 0x7f3d4c12f010 con 0x7f3d4c132330 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d4c06cf50 legacy=0x7f3d4c13e3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6789/0 conn(0x7f3d4c06d480 legacy=0x7f3d4c1cea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: _finish_hunting 0 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 monclient: found mon.b 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c1316e0 con 0x7f3d4c132330 2024-09-16T06:54:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: _finish_auth 0 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: _check_auth_tickets 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 5 monclient: authenticate success, global_id 284266 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _renew_subs 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c130bf0 con 0x7f3d4c132330 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _renew_subs 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c12eb00 con 0x7f3d4c132330 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d440036c0 con 0x7f3d4c132330 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d497fa700 4 set_mon_vals no callback set 2024-09-16T06:54:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d440039d0 con 0x7f3d4c132330 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.155+0000 7f3d337fe700 10 monclient: dump: 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.172 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d4bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d2c006590 con 0x7f3d2c002830 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3d4400e440 con 0x7f3d4c132330 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3d44042a90 con 0x7f3d4c132330 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131862802 0 0) 0x7f3d2c006590 con 0x7f3d2c002830 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient(hunting): my global_id is 244727 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d4c12f010 con 0x7f3d2c002830 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d4a7fc700 10 monclient: get_auth_request con 0x7f3d38036080 auth_method 0 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 72275962 0 0) 0x7f3d4c12f010 con 0x7f3d2c002830 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d2c006590 con 0x7f3d2c002830 2024-09-16T06:54:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d3c001950 con 0x7f3d2c002830 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient: dump: 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 521254052 0 0) 0x7f3d2c006590 con 0x7f3d2c002830 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d2c00a190 con 0x7f3d2c002830 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3d3c007480 con 0x7f3d2c002830 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d2c002830 legacy=0x7f3d2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.159+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d2c002830 legacy=0x7f3d2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:08.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:08.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d2c009e50 con 0x7f3d2c009170 2024-09-16T06:54:08.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774771674 0 0) 0x7f3d2c009e50 con 0x7f3d2c009170 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient(hunting): my global_id is 244730 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d2c006590 con 0x7f3d2c009170 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2092435483 0 0) 0x7f3d2c006590 con 0x7f3d2c009170 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d2c009e50 con 0x7f3d2c009170 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3d3c0054a0 con 0x7f3d2c009170 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient: dump: 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.177 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1442878488 0 0) 0x7f3d2c009e50 con 0x7f3d2c009170 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d2c00c340 con 0x7f3d2c009170 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 1674578987) 0x7f3d3c0075f0 con 0x7f3d2c009170 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:08.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d2c009170 legacy=0x7f3d2c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d337fe700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.76:6790/0 conn(0x7f3d2c009170 legacy=0x7f3d2c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 142, 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:08.191 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:08.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3d38036080 legacy=0x7f3d38038540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 10 monclient: shutdown 2024-09-16T06:54:08.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:08.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 >> v1:172.21.15.189:6789/0 conn(0x7f3d4c132330 legacy=0x7f3d4c1d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4b7fe700 1 -- 172.21.15.76:0/1312763670 reap_dead start 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.171+0000 7f3d4b7fe700 1 -- 172.21.15.76:0/1312763670 reap_dead start 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.175+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 shutdown_connections 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.175+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 >> 172.21.15.76:0/1312763670 conn(0x7f3d4c0a41c0 msgr2=0x7f3d4c09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.175+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 shutdown_connections 2024-09-16T06:54:08.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.175+0000 7f3d4bfff700 1 -- 172.21.15.76:0/1312763670 wait complete. 2024-09-16T06:54:08.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:54:08.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:54:08.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:54:08.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: get_monmap_and_config 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: build_initial_monmap 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: monmap: 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:08.321216+0000 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:08.321216+0000 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:08.316 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:08.317 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:08.317 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 1 Processor -- start 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 1 -- start start 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: init 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient(hunting): picked mon.noname-c con 0x7f112809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient(hunting): picked mon.noname-a con 0x7f1128098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient(hunting): picked mon.noname-b con 0x7f1128099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112809d560 con 0x7f1128098be0 2024-09-16T06:54:08.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112809f140 con 0x7f112809ed60 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112809d790 con 0x7f1128099030 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 10 monclient(hunting): authenticate will time out at 1795.454662s 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f1127fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f1128098be0 legacy=0x7f11281374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d001700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1128099030 0x7f112813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:45114/0 (socket says 172.21.15.76:45114) 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d001700 1 -- 172.21.15.76:0/535120177 learned_addr learned my addr 172.21.15.76:0/535120177 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176301964 0 0) 0x7f112809d790 con 0x7f1128099030 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): my global_id is 244733 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11100035d0 con 0x7f1128099030 2024-09-16T06:54:08.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227004906 0 0) 0x7f112809f140 con 0x7f112809ed60 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient(hunting): my global_id is 284269 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112809d790 con 0x7f112809ed60 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051816504 0 0) 0x7f11100035d0 con 0x7f1128099030 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 >> v1:172.21.15.189:6789/0 conn(0x7f112809ed60 legacy=0x7f11280984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 >> v1:172.21.15.76:6789/0 conn(0x7f1128098be0 legacy=0x7f11281374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient: _finish_hunting 0 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 monclient: found mon.noname-b 2024-09-16T06:54:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112809f620 con 0x7f1128099030 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient: _finish_auth 0 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient: _check_auth_tickets 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 5 monclient: authenticate success, global_id 244733 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f112d802700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1120001990 con 0x7f1128099030 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.319+0000 7f11277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: dump: 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.325 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1120001c70 con 0x7f1128099030 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 1 -- 172.21.15.76:0/535120177 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1120002000 con 0x7f1128099030 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 4 set_mon_vals no callback set 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f11277fe700 10 monclient: dump: 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.326 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 10 monclient: shutdown 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- 172.21.15.76:0/535120177 >> v1:172.21.15.76:6790/0 conn(0x7f1128099030 legacy=0x7f112813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- 172.21.15.76:0/535120177 shutdown_connections 2024-09-16T06:54:08.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- 172.21.15.76:0/535120177 >> 172.21.15.76:0/535120177 conn(0x7f11280a0030 msgr2=0x7f11280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- 172.21.15.76:0/535120177 shutdown_connections 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- 172.21.15.76:0/535120177 wait complete. 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 10 monclient: build_initial_monmap 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 10 monclient: monmap: 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:08.328 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:08.329 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 Processor -- start 2024-09-16T06:54:08.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 1 -- start start 2024-09-16T06:54:08.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.323+0000 7f112d802700 10 monclient: init 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient(hunting): picked mon.noname-c con 0x7f1128098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient(hunting): picked mon.noname-b con 0x7f1128099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient(hunting): picked mon.noname-a con 0x7f112809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1128097030 con 0x7f112809ed60 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112809e850 con 0x7f1128099030 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112809a540 con 0x7f1128098be0 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d001700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f112809ed60 legacy=0x7f11281cf330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient(hunting): authenticate will time out at 1795.461200s 2024-09-16T06:54:08.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112c800700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1128098be0 0x7f1128096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:45120/0 (socket says 172.21.15.76:45120) 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112c800700 1 -- 172.21.15.76:0/4197686965 learned_addr learned my addr 172.21.15.76:0/4197686965 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977979725 0 0) 0x7f112809a540 con 0x7f1128098be0 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): my global_id is 244736 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11140035d0 con 0x7f1128098be0 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811688527 0 0) 0x7f112809e850 con 0x7f1128099030 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): my global_id is 284272 2024-09-16T06:54:08.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112809a540 con 0x7f1128099030 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 436311316 0 0) 0x7f11140035d0 con 0x7f1128098be0 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f112809e850 con 0x7f1128098be0 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1118001910 con 0x7f1128098be0 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): dump: 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.338 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567250997 0 0) 0x7f112809a540 con 0x7f1128099030 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11140035d0 con 0x7f1128099030 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f111c004170 con 0x7f1128099030 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:08.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient(hunting): dump: 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2307664211 0 0) 0x7f112809e850 con 0x7f1128098be0 2024-09-16T06:54:08.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6789/0 conn(0x7f1128099030 legacy=0x7f11281cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.76:6789/0 conn(0x7f112809ed60 legacy=0x7f11281cf330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: _finish_hunting 0 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 monclient: found mon.c 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128091b20 con 0x7f1128098be0 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: _finish_auth 0 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: _check_auth_tickets 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 5 monclient: authenticate success, global_id 244736 2024-09-16T06:54:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _renew_subs 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1128090de0 con 0x7f1128098be0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _renew_subs 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1128090210 con 0x7f1128098be0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1118001ca0 con 0x7f1128098be0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1118003b10 con 0x7f1128098be0 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f1126ffd700 4 set_mon_vals no callback set 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f11257fa700 10 monclient: dump: 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.327+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1108006590 con 0x7f1108002830 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1118003cf0 con 0x7f1128098be0 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1118043f60 con 0x7f1128098be0 2024-09-16T06:54:08.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f112d001700 10 monclient: get_auth_request con 0x7f11140362a0 auth_method 0 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404779282 0 0) 0x7f1108006590 con 0x7f1108002830 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient(hunting): my global_id is 284275 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112809e850 con 0x7f1108002830 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 68249430 0 0) 0x7f112809e850 con 0x7f1108002830 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1108006590 con 0x7f1108002830 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f111c0047f0 con 0x7f1108002830 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient: dump: 2024-09-16T06:54:08.344 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.345 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3870645574 0 0) 0x7f1108006590 con 0x7f1108002830 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f110800a190 con 0x7f1108002830 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f111c007480 con 0x7f1108002830 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6789/0 conn(0x7f1108002830 legacy=0x7f1108004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.331+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6789/0 conn(0x7f1108002830 legacy=0x7f1108004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:08.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:08.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1108009e50 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691242124 0 0) 0x7f1108009e50 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient(hunting): my global_id is 284278 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1108006590 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2547245485 0 0) 0x7f1108006590 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1108009e50 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f111c0054e0 con 0x7f1108009170 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:08.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient: dump: 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3936624925 0 0) 0x7f1108009e50 con 0x7f1108009170 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f110800c340 con 0x7f1108009170 2024-09-16T06:54:08.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 3431089137) 0x7f111c0075f0 con 0x7f1108009170 2024-09-16T06:54:08.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:08.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:08.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6789/0 conn(0x7f1108009170 legacy=0x7f1108009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f11257fa700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6789/0 conn(0x7f1108009170 legacy=0x7f1108009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:08.350 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 142, 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:08.365 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:08.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.189:6812/3098810674 conn(0x7f11140362a0 legacy=0x7f1114038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 10 monclient: shutdown 2024-09-16T06:54:08.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:08.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 >> v1:172.21.15.76:6790/0 conn(0x7f1128098be0 legacy=0x7f1128096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d001700 1 -- 172.21.15.76:0/4197686965 reap_dead start 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d001700 1 -- 172.21.15.76:0/4197686965 reap_dead start 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 shutdown_connections 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.343+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 >> 172.21.15.76:0/4197686965 conn(0x7f11280a0030 msgr2=0x7f112806e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.347+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 shutdown_connections 2024-09-16T06:54:08.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:08.347+0000 7f112d802700 1 -- 172.21.15.76:0/4197686965 wait complete. 2024-09-16T06:54:08.367 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:54:08.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:54:28.368 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:54:28.368 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:54:28.369 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:54:28.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:54:28.372 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:54:28.372 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:54:28.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.487+0000 7fc89359e700 10 monclient: get_monmap_and_config 2024-09-16T06:54:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.487+0000 7fc89359e700 10 monclient: build_initial_monmap 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.487+0000 7fc89359e700 10 monclient: monmap: 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:28.491093+0000 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:28.491093+0000 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:28.486 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 1 Processor -- start 2024-09-16T06:54:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 1 -- start start 2024-09-16T06:54:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient: init 2024-09-16T06:54:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-b con 0x7fc894132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-a con 0x7fc89406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-c con 0x7fc89406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89406c280 con 0x7fc89406d480 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89406bfa0 con 0x7fc89406cf50 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89409ace0 con 0x7fc894132330 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient(hunting): authenticate will time out at 1815.625041s 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc891d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc89406d480 0x7fc89413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43086/0 (socket says 172.21.15.76:43086) 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc891d9b700 1 -- 172.21.15.76:0/1187254550 learned_addr learned my addr 172.21.15.76:0/1187254550 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206066368 0 0) 0x7fc89409ace0 con 0x7fc894132330 2024-09-16T06:54:28.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): my global_id is 244739 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc87c0035d0 con 0x7fc894132330 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527420956 0 0) 0x7fc89406bfa0 con 0x7fc89406cf50 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient(hunting): my global_id is 284281 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89409ace0 con 0x7fc89406cf50 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219912900 0 0) 0x7fc87c0035d0 con 0x7fc894132330 2024-09-16T06:54:28.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 >> v1:172.21.15.189:6789/0 conn(0x7fc89406cf50 legacy=0x7fc89413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 >> v1:172.21.15.76:6789/0 conn(0x7fc89406d480 legacy=0x7fc89413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: _finish_hunting 0 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 monclient: found mon.noname-b 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89409ae70 con 0x7fc894132330 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: _finish_auth 0 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: _check_auth_tickets 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc888001590 con 0x7fc894132330 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:28.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: dump: 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 5 monclient: authenticate success, global_id 244739 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:28.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc888001870 con 0x7fc894132330 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 1 -- 172.21.15.76:0/1187254550 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc888001a60 con 0x7fc894132330 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89159a700 10 monclient: dump: 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.494 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:28.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.491+0000 7fc89359e700 4 set_mon_vals no callback set 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: shutdown 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- 172.21.15.76:0/1187254550 >> v1:172.21.15.76:6790/0 conn(0x7fc894132330 legacy=0x7fc894132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- 172.21.15.76:0/1187254550 shutdown_connections 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- 172.21.15.76:0/1187254550 >> 172.21.15.76:0/1187254550 conn(0x7fc8940a41c0 msgr2=0x7fc8940a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- 172.21.15.76:0/1187254550 shutdown_connections 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- 172.21.15.76:0/1187254550 wait complete. 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: build_initial_monmap 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: monmap: 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:28.500 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 Processor -- start 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- start start 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: init 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:28.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-c con 0x7fc89406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-a con 0x7fc89406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient(hunting): picked mon.noname-b con 0x7fc894132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89409e8b0 con 0x7fc89406d480 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc894131720 con 0x7fc894132330 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8941311f0 con 0x7fc89406cf50 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc89359e700 10 monclient(hunting): authenticate will time out at 1815.630457s 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc891d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc89406d480 0x7fc89413e560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43102/0 (socket says 172.21.15.76:43102) 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc891d9b700 1 -- 172.21.15.76:0/3153817345 learned_addr learned my addr 172.21.15.76:0/3153817345 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813741573 0 0) 0x7fc8941311f0 con 0x7fc89406cf50 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): my global_id is 244742 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8740035d0 con 0x7fc89406cf50 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467489246 0 0) 0x7fc894131720 con 0x7fc894132330 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): my global_id is 284284 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8941311f0 con 0x7fc894132330 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892416487 0 0) 0x7fc8740035d0 con 0x7fc89406cf50 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc894131720 con 0x7fc89406cf50 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8880019e0 con 0x7fc89406cf50 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): dump: 2024-09-16T06:54:28.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910441325 0 0) 0x7fc8941311f0 con 0x7fc894132330 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8740035d0 con 0x7fc894132330 2024-09-16T06:54:28.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc8840018b0 con 0x7fc894132330 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.495+0000 7fc882ffd700 10 monclient(hunting): dump: 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027232113 0 0) 0x7fc894131720 con 0x7fc89406cf50 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.189:6789/0 conn(0x7fc894132330 legacy=0x7fc8941d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6789/0 conn(0x7fc89406d480 legacy=0x7fc89413e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 monclient: found mon.c 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89412ed00 con 0x7fc89406cf50 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 5 monclient: authenticate success, global_id 244742 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 monclient: _renew_subs 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89412fa60 con 0x7fc89406cf50 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 monclient: _renew_subs 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89413ee50 con 0x7fc89406cf50 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc89406cf50 legacy=0x7fc8941c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc89406cf50 legacy=0x7fc8941c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 --1- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc89406cf50 0x7fc8941c98c0 :-1 s=OPENED pgs=1655 cs=1 l=1).handle_message read tag failed 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 --1- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc89406cf50 0x7fc8941c98c0 :-1 s=OPENED pgs=1655 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6790/0 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc89406cf50 legacy=0x7fc8941c98c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient(hunting): picked mon.c con 0x7fc874003ab0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:28.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient(hunting): picked mon.b con 0x7fc894132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient(hunting): picked mon.a con 0x7fc89406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc894131720 con 0x7fc89406d480 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc8941311f0 con 0x7fc894132330 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc8740122b0 con 0x7fc874003ab0 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89359e700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc874003ab0 legacy=0x7fc874003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc874003ab0 legacy=0x7fc874003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 --1- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc874003ab0 0x7fc874003e90 :-1 s=OPENED pgs=1656 cs=1 l=1).handle_message read tag failed 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc89259c700 1 --1- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc874003ab0 0x7fc874003e90 :-1 s=OPENED pgs=1656 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:54:28.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.499+0000 7fc882ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470034039 0 0) 0x7fc894131720 con 0x7fc89406d480 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc874011280 con 0x7fc89406d480 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164197448 0 0) 0x7fc8941311f0 con 0x7fc894132330 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fc894131720 con 0x7fc894132330 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3231413756 0 0) 0x7fc874011280 con 0x7fc89406d480 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6790/0 conn(0x7fc874003ab0 legacy=0x7fc874003e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.189:6789/0 conn(0x7fc894132330 legacy=0x7fc87400aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 monclient: found mon.a 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fc8740135f0 con 0x7fc89406d480 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc894132ed0 con 0x7fc89406d480 2024-09-16T06:54:28.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8941311f0 con 0x7fc89406d480 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc884001d80 con 0x7fc89406d480 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: dump: 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.527 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc884003840 con 0x7fc89406d480 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:28.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc890d99700 4 set_mon_vals no callback set 2024-09-16T06:54:28.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc88400e2c0 con 0x7fc89406d480 2024-09-16T06:54:28.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.527+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 6 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc884042740 con 0x7fc89406d480 2024-09-16T06:54:28.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.531+0000 7fc89259c700 10 monclient: get_auth_request con 0x7fc89406cf50 auth_method 0 2024-09-16T06:54:28.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.535+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc884012f90 con 0x7fc89406d480 2024-09-16T06:54:28.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.535+0000 7fc882ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.535+0000 7fc882ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:28.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.547+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc884012db0 con 0x7fc89406d480 2024-09-16T06:54:28.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.547+0000 7fc882ffd700 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T06:54:28.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc89359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:28.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc89359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:28.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc89359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:28.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc894132ed0 con 0x7fc89406d480 2024-09-16T06:54:28.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc882ffd700 1 -- 172.21.15.76:0/3153817345 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1572013516) 0x7fc884012f90 con 0x7fc89406d480 2024-09-16T06:54:28.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc882ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:28.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.771+0000 7fc882ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:28.768 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:28.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc89406cf50 legacy=0x7fc874044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 10 monclient: shutdown 2024-09-16T06:54:28.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:28.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 >> v1:172.21.15.76:6789/0 conn(0x7fc89406d480 legacy=0x7fc87400e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 shutdown_connections 2024-09-16T06:54:28.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 >> 172.21.15.76:0/3153817345 conn(0x7fc8940a41c0 msgr2=0x7fc89409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:28.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 shutdown_connections 2024-09-16T06:54:28.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.775+0000 7fc89359e700 1 -- 172.21.15.76:0/3153817345 wait complete. 2024-09-16T06:54:28.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:54:28.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:54:28.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: get_monmap_and_config 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: build_initial_monmap 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: monmap: 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:28.897199+0000 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:28.897199+0000 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:28.892 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:28.893 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 1 Processor -- start 2024-09-16T06:54:28.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 1 -- start start 2024-09-16T06:54:28.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: init 2024-09-16T06:54:28.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:28.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-b con 0x7fdd5409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-c con 0x7fdd54098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-a con 0x7fdd54099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5409d560 con 0x7fdd54099030 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5409f140 con 0x7fdd54098be0 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5409d790 con 0x7fdd5409ed60 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd5884a700 10 monclient(hunting): authenticate will time out at 1816.030738s 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd53fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdd54099030 0x7fdd5413ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43116/0 (socket says 172.21.15.76:43116) 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd53fff700 1 -- 172.21.15.76:0/525031543 learned_addr learned my addr 172.21.15.76:0/525031543 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862938508 0 0) 0x7fdd5409d790 con 0x7fdd5409ed60 2024-09-16T06:54:28.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): my global_id is 244745 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd340035d0 con 0x7fdd5409ed60 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643841014 0 0) 0x7fdd5409f140 con 0x7fdd54098be0 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 10 monclient(hunting): my global_id is 284287 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.895+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5409d790 con 0x7fdd54098be0 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027354316 0 0) 0x7fdd5409d790 con 0x7fdd54098be0 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 >> v1:172.21.15.76:6790/0 conn(0x7fdd5409ed60 legacy=0x7fdd540984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 >> v1:172.21.15.76:6789/0 conn(0x7fdd54099030 legacy=0x7fdd5413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: _finish_hunting 0 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 monclient: found mon.noname-c 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5409f620 con 0x7fdd54098be0 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: _finish_auth 0 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: _check_auth_tickets 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd3c0015f0 con 0x7fdd54098be0 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: dump: 2024-09-16T06:54:28.899 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 5 monclient: authenticate success, global_id 284287 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd3c001910 con 0x7fdd54098be0 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 1 -- 172.21.15.76:0/525031543 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd3c001ca0 con 0x7fdd54098be0 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 4 set_mon_vals no callback set 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd527fc700 10 monclient: dump: 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.901 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: shutdown 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- 172.21.15.76:0/525031543 >> v1:172.21.15.189:6789/0 conn(0x7fdd54098be0 legacy=0x7fdd541374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- 172.21.15.76:0/525031543 shutdown_connections 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- 172.21.15.76:0/525031543 >> 172.21.15.76:0/525031543 conn(0x7fdd540a0030 msgr2=0x7fdd540a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- 172.21.15.76:0/525031543 shutdown_connections 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- 172.21.15.76:0/525031543 wait complete. 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: build_initial_monmap 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: monmap: 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:28.902 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 Processor -- start 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- start start 2024-09-16T06:54:28.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: init 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-c con 0x7fdd54098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-b con 0x7fdd54099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient(hunting): picked mon.noname-a con 0x7fdd5409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd54093e50 con 0x7fdd5409ed60 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5409a540 con 0x7fdd54099030 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd54092420 con 0x7fdd54098be0 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd5884a700 10 monclient(hunting): authenticate will time out at 1816.035215s 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd537fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdd54098be0 0x7fdd54091600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40878/0 (socket says 172.21.15.76:40878) 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.899+0000 7fdd537fe700 1 -- 172.21.15.76:0/3602271778 learned_addr learned my addr 172.21.15.76:0/3602271778 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326010571 0 0) 0x7fdd54092420 con 0x7fdd54098be0 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): my global_id is 244748 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd380035d0 con 0x7fdd54098be0 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339228688 0 0) 0x7fdd380035d0 con 0x7fdd54098be0 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd54092420 con 0x7fdd54098be0 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd440018f0 con 0x7fdd54098be0 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): dump: 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3872680143 0 0) 0x7fdd54092420 con 0x7fdd54098be0 2024-09-16T06:54:28.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.189:6789/0 conn(0x7fdd54099030 legacy=0x7fdd54091d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6789/0 conn(0x7fdd5409ed60 legacy=0x7fdd541d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 monclient: found mon.c 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540a1250 con 0x7fdd54098be0 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: _finish_auth 0 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 5 monclient: authenticate success, global_id 244748 2024-09-16T06:54:28.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 10 monclient: _renew_subs 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd54095290 con 0x7fdd54098be0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 10 monclient: _renew_subs 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd54093b60 con 0x7fdd54098be0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd44001c70 con 0x7fdd54098be0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd44003b10 con 0x7fdd54098be0 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd51ffb700 4 set_mon_vals no callback set 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:28.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 10 monclient: dump: 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.903+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdd44003c70 con 0x7fdd54098be0 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd5884a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd2c006590 con 0x7fdd2c002830 2024-09-16T06:54:28.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fdd44043ed0 con 0x7fdd54098be0 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd53fff700 10 monclient: get_auth_request con 0x7fdd38036150 auth_method 0 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502322590 0 0) 0x7fdd2c006590 con 0x7fdd2c002830 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient(hunting): my global_id is 304098 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54092420 con 0x7fdd2c002830 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2678653913 0 0) 0x7fdd54092420 con 0x7fdd2c002830 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd2c006590 con 0x7fdd2c002830 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd3c0015b0 con 0x7fdd2c002830 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient: dump: 2024-09-16T06:54:28.914 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2215829081 0 0) 0x7fdd2c006590 con 0x7fdd2c002830 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd2c00a190 con 0x7fdd2c002830 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fdd3c001410 con 0x7fdd2c002830 2024-09-16T06:54:28.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:28.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:28.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6789/0 conn(0x7fdd2c002830 legacy=0x7fdd2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.907+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6789/0 conn(0x7fdd2c002830 legacy=0x7fdd2c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:28.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd5884a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:28.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdd2c009e50 con 0x7fdd2c009170 2024-09-16T06:54:28.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143498716 0 0) 0x7fdd2c009e50 con 0x7fdd2c009170 2024-09-16T06:54:28.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:28.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 10 monclient(hunting): my global_id is 304101 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.915+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd2c006590 con 0x7fdd2c009170 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1731355750 0 0) 0x7fdd2c006590 con 0x7fdd2c009170 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd2c009e50 con 0x7fdd2c009170 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd3c003a30 con 0x7fdd2c009170 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 10 monclient: dump: 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.918 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 890447902 0 0) 0x7fdd2c009e50 con 0x7fdd2c009170 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd2c00c340 con 0x7fdd2c009170 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 207639707) 0x7fdd3c004f30 con 0x7fdd2c009170 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6789/0 conn(0x7fdd2c009170 legacy=0x7fdd2c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd4bfff700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6789/0 conn(0x7fdd2c009170 legacy=0x7fdd2c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:28.920 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:28.921 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:28.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:28.923 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:28.924 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:28.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:28.929 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:28.930 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdd38036150 legacy=0x7fdd38038610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 10 monclient: shutdown 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 >> v1:172.21.15.76:6790/0 conn(0x7fdd54098be0 legacy=0x7fdd54091600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd53fff700 1 -- 172.21.15.76:0/3602271778 reap_dead start 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd53fff700 1 -- 172.21.15.76:0/3602271778 reap_dead start 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 shutdown_connections 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 >> 172.21.15.76:0/3602271778 conn(0x7fdd540a0030 msgr2=0x7fdd540972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:28.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 shutdown_connections 2024-09-16T06:54:28.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:28.919+0000 7fdd5884a700 1 -- 172.21.15.76:0/3602271778 wait complete. 2024-09-16T06:54:28.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:28.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:54:28.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:54:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.047+0000 7fc0c49e5700 10 monclient: get_monmap_and_config 2024-09-16T06:54:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.047+0000 7fc0c49e5700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.047+0000 7fc0c49e5700 10 monclient: monmap: 2024-09-16T06:54:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:29.051294+0000 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:29.051294+0000 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 Processor -- start 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- start start 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient: init 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-b con 0x7fc0c0097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-a con 0x7fc0c0099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-c con 0x7fc0c0139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0098990 con 0x7fc0c0099090 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0097f40 con 0x7fc0c0139500 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c009d610 con 0x7fc0c0097b60 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient(hunting): authenticate will time out at 1816.184842s 2024-09-16T06:54:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0b7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc0c0099090 0x7fc0c009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43158/0 (socket says 172.21.15.76:43158) 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0bf7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fc0c0097b60 0x7fc0c009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40882/0 (socket says 172.21.15.76:40882) 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0b7fff700 1 -- 172.21.15.76:0/1203998825 learned_addr learned my addr 172.21.15.76:0/1203998825 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603669887 0 0) 0x7fc0c0098990 con 0x7fc0c0099090 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): my global_id is 304104 2024-09-16T06:54:29.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac0035d0 con 0x7fc0c0099090 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354688915 0 0) 0x7fc0c009d610 con 0x7fc0c0097b60 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): my global_id is 244751 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0098990 con 0x7fc0c0097b60 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268746911 0 0) 0x7fc0c0097f40 con 0x7fc0c0139500 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient(hunting): my global_id is 284290 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c009d610 con 0x7fc0c0139500 2024-09-16T06:54:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878647815 0 0) 0x7fc0c0098990 con 0x7fc0c0097b60 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 >> v1:172.21.15.189:6789/0 conn(0x7fc0c0139500 legacy=0x7fc0c013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 >> v1:172.21.15.76:6789/0 conn(0x7fc0c0099090 legacy=0x7fc0c009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 monclient: found mon.noname-b 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00986b0 con 0x7fc0c0097b60 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 5 monclient: authenticate success, global_id 244751 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0b00015f0 con 0x7fc0c0097b60 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: dump: 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b0001810 con 0x7fc0c0097b60 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 4 set_mon_vals no callback set 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 1 -- 172.21.15.76:0/1203998825 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0b0001c20 con 0x7fc0c0097b60 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0beffd700 10 monclient: dump: 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 10 monclient: shutdown 2024-09-16T06:54:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1203998825 >> v1:172.21.15.76:6790/0 conn(0x7fc0c0097b60 legacy=0x7fc0c009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1203998825 shutdown_connections 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1203998825 >> 172.21.15.76:0/1203998825 conn(0x7fc0c00a0030 msgr2=0x7fc0c00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1203998825 shutdown_connections 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.051+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1203998825 wait complete. 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: monmap: 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:29.067 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 Processor -- start 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- start start 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: init 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:54:29.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-a con 0x7fc0c0097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-b con 0x7fc0c0099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient(hunting): picked mon.noname-c con 0x7fc0c0139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c006d5b0 con 0x7fc0c0097b60 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c006f0a0 con 0x7fc0c0099090 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c006fc50 con 0x7fc0c0139500 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient(hunting): authenticate will time out at 1816.189571s 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bf7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc0c0097b60 0x7fc0c006c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43172/0 (socket says 172.21.15.76:43172) 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bf7fe700 1 -- 172.21.15.76:0/1760400709 learned_addr learned my addr 172.21.15.76:0/1760400709 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207439048 0 0) 0x7fc0c006d5b0 con 0x7fc0c0097b60 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): my global_id is 304107 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a00035d0 con 0x7fc0c0097b60 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134926193 0 0) 0x7fc0c006fc50 con 0x7fc0c0139500 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): my global_id is 244754 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c006d5b0 con 0x7fc0c0139500 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159041469 0 0) 0x7fc0c006f0a0 con 0x7fc0c0099090 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): my global_id is 284293 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c006fc50 con 0x7fc0c0099090 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335161306 0 0) 0x7fc0a00035d0 con 0x7fc0c0097b60 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c006f0a0 con 0x7fc0c0097b60 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0b0001be0 con 0x7fc0c0097b60 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): dump: 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.071 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962233733 0 0) 0x7fc0c006d5b0 con 0x7fc0c0139500 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a00035d0 con 0x7fc0c0139500 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0b8001bb0 con 0x7fc0c0139500 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient(hunting): dump: 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.072 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 516213428 0 0) 0x7fc0c006fc50 con 0x7fc0c0099090 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c006d5b0 con 0x7fc0c0099090 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4262649033 0 0) 0x7fc0c006f0a0 con 0x7fc0c0097b60 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6790/0 conn(0x7fc0c0139500 legacy=0x7fc0c01cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.189:6789/0 conn(0x7fc0c0099090 legacy=0x7fc0c006cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 monclient: found mon.a 2024-09-16T06:54:29.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c006ff30 con 0x7fc0c0097b60 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 5 monclient: authenticate success, global_id 304107 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _renew_subs 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c0070210 con 0x7fc0c0097b60 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _renew_subs 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c006e920 con 0x7fc0c0097b60 2024-09-16T06:54:29.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b0001210 con 0x7fc0c0097b60 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0b0004d60 con 0x7fc0c0097b60 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0be7fc700 4 set_mon_vals no callback set 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.055+0000 7fc0bcff9700 10 monclient: dump: 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.075 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0c49e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0a4006590 con 0x7fc0a4002830 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc0b0004f40 con 0x7fc0c0097b60 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc0b0043f70 con 0x7fc0c0097b60 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822306721 0 0) 0x7fc0a4006590 con 0x7fc0a4002830 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient(hunting): my global_id is 244757 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c006f0a0 con 0x7fc0a4002830 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bffff700 10 monclient: get_auth_request con 0x7fc0a00364d0 auth_method 0 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 358547813 0 0) 0x7fc0c006f0a0 con 0x7fc0a4002830 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a4006590 con 0x7fc0a4002830 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0a8004890 con 0x7fc0a4002830 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient: dump: 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.077 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2144399164 0 0) 0x7fc0a4006590 con 0x7fc0a4002830 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0a400a190 con 0x7fc0a4002830 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fc0a8006b10 con 0x7fc0a4002830 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6790/0 conn(0x7fc0a4002830 legacy=0x7fc0a4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.059+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6790/0 conn(0x7fc0a4002830 legacy=0x7fc0a4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc0a4009e50 con 0x7fc0a4009170 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610008874 0 0) 0x7fc0a4009e50 con 0x7fc0a4009170 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient(hunting): my global_id is 244760 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4006590 con 0x7fc0a4009170 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2962951093 0 0) 0x7fc0a4006590 con 0x7fc0a4009170 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0a4009e50 con 0x7fc0a4009170 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc0a8004fd0 con 0x7fc0a4009170 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient: dump: 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1118985267 0 0) 0x7fc0a4009e50 con 0x7fc0a4009170 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc0a400c340 con 0x7fc0a4009170 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 835035260) 0x7fc0a8006350 con 0x7fc0a4009170 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6790/0 conn(0x7fc0a4009170 legacy=0x7fc0a4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bcff9700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6790/0 conn(0x7fc0a4009170 legacy=0x7fc0a4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:29.083 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.084 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:29.086 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.087 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.088 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:29.089 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.090 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:29.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.093 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.095 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:29.095 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:29.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc0a00364d0 legacy=0x7fc0a0038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 10 monclient: shutdown 2024-09-16T06:54:29.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 >> v1:172.21.15.76:6789/0 conn(0x7fc0c0097b60 legacy=0x7fc0c006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bffff700 1 -- 172.21.15.76:0/1760400709 reap_dead start 2024-09-16T06:54:29.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0bffff700 1 -- 172.21.15.76:0/1760400709 reap_dead start 2024-09-16T06:54:29.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 shutdown_connections 2024-09-16T06:54:29.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 >> 172.21.15.76:0/1760400709 conn(0x7fc0c00a0030 msgr2=0x7fc0c00a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.071+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 shutdown_connections 2024-09-16T06:54:29.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.075+0000 7fc0c49e5700 1 -- 172.21.15.76:0/1760400709 wait complete. 2024-09-16T06:54:29.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:29.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:54:29.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: get_monmap_and_config 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: monmap: 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:29.221366+0000 2024-09-16T06:54:29.216 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:29.221366+0000 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:29.217 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 1 Processor -- start 2024-09-16T06:54:29.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 1 -- start start 2024-09-16T06:54:29.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: init 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient(hunting): picked mon.noname-c con 0x7f193409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient(hunting): picked mon.noname-b con 0x7f1934098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient(hunting): picked mon.noname-a con 0x7f1934099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193409d560 con 0x7f1934099030 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193409f140 con 0x7f193409ed60 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193409d790 con 0x7f1934098be0 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 10 monclient(hunting): authenticate will time out at 1816.354637s 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f192bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1934098be0 0x7f19341374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40934/0 (socket says 172.21.15.76:40934) 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f192bfff700 1 -- 172.21.15.76:0/2299868553 learned_addr learned my addr 172.21.15.76:0/2299868553 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819009595 0 0) 0x7f193409d790 con 0x7f1934098be0 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): my global_id is 244763 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19200035d0 con 0x7f1934098be0 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441430459 0 0) 0x7f193409d560 con 0x7f1934099030 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient(hunting): my global_id is 304110 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193409d790 con 0x7f1934099030 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287085851 0 0) 0x7f193409d790 con 0x7f1934099030 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 >> v1:172.21.15.76:6790/0 conn(0x7f1934098be0 legacy=0x7f19341374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 >> v1:172.21.15.189:6789/0 conn(0x7f193409ed60 legacy=0x7f19340984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 monclient: found mon.noname-a 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193409f620 con 0x7f1934099030 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 5 monclient: authenticate success, global_id 304110 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f192c001990 con 0x7f1934099030 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f1931d9b700 10 monclient: dump: 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.224 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.219+0000 7f193359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f192c001bb0 con 0x7f1934099030 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 1 -- 172.21.15.76:0/2299868553 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f192c001f80 con 0x7f1934099030 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 4 set_mon_vals no callback set 2024-09-16T06:54:29.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f1931d9b700 10 monclient: dump: 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: shutdown 2024-09-16T06:54:29.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- 172.21.15.76:0/2299868553 >> v1:172.21.15.76:6789/0 conn(0x7f1934099030 legacy=0x7f193413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- 172.21.15.76:0/2299868553 shutdown_connections 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- 172.21.15.76:0/2299868553 >> 172.21.15.76:0/2299868553 conn(0x7f19340a0030 msgr2=0x7f19340a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- 172.21.15.76:0/2299868553 shutdown_connections 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- 172.21.15.76:0/2299868553 wait complete. 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: monmap: 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.227 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:29.228 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:29.228 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 Processor -- start 2024-09-16T06:54:29.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- start start 2024-09-16T06:54:29.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: init 2024-09-16T06:54:29.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient(hunting): picked mon.noname-a con 0x7f1934098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient(hunting): picked mon.noname-c con 0x7f1934099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient(hunting): picked mon.noname-b con 0x7f193409ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193408f530 con 0x7f1934098be0 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193409a540 con 0x7f193409ed60 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1934090e90 con 0x7f1934099030 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1934098be0 0x7f1934096400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43178/0 (socket says 172.21.15.76:43178) 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193259c700 1 -- 172.21.15.76:0/3043661524 learned_addr learned my addr 172.21.15.76:0/3043661524 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.223+0000 7f193359e700 10 monclient(hunting): authenticate will time out at 1816.359863s 2024-09-16T06:54:29.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12476549 0 0) 0x7f1934090e90 con 0x7f1934099030 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): my global_id is 244766 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19140035d0 con 0x7f1934099030 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221395810 0 0) 0x7f193408f530 con 0x7f1934098be0 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): my global_id is 304113 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1934090e90 con 0x7f1934098be0 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587128590 0 0) 0x7f193409a540 con 0x7f193409ed60 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): my global_id is 284299 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193408f530 con 0x7f193409ed60 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938208649 0 0) 0x7f19140035d0 con 0x7f1934099030 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f193409a540 con 0x7f1934099030 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510878774 0 0) 0x7f1934090e90 con 0x7f1934098be0 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19140035d0 con 0x7f1934098be0 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1924001720 con 0x7f1934099030 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): dump: 2024-09-16T06:54:29.236 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f191c001850 con 0x7f1934098be0 2024-09-16T06:54:29.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): dump: 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202164583 0 0) 0x7f193408f530 con 0x7f193409ed60 2024-09-16T06:54:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1934090e90 con 0x7f193409ed60 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f192c0017b0 con 0x7f193409ed60 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): dump: 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.239 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4150231375 0 0) 0x7f193409a540 con 0x7f1934099030 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6789/0 conn(0x7f193409ed60 legacy=0x7f19341cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.76:6789/0 conn(0x7f1934098be0 legacy=0x7f1934096400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 monclient: found mon.c 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340a2cc0 con 0x7f1934099030 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 5 monclient: authenticate success, global_id 244766 2024-09-16T06:54:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 10 monclient: _renew_subs 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19340984d0 con 0x7f1934099030 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 10 monclient: _renew_subs 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f193408eb30 con 0x7f1934099030 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1924001940 con 0x7f1934099030 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1924004a30 con 0x7f1934099030 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient: dump: 2024-09-16T06:54:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193159a700 4 set_mon_vals no callback set 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1918006590 con 0x7f1918002830 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f192400f4a0 con 0x7f1934099030 2024-09-16T06:54:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1924043cf0 con 0x7f1934099030 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f1932d9d700 10 monclient: get_auth_request con 0x7f1914036610 auth_method 0 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194687702 0 0) 0x7f1918006590 con 0x7f1918002830 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 10 monclient(hunting): my global_id is 284302 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.227+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193409a540 con 0x7f1918002830 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4047317548 0 0) 0x7f193409a540 con 0x7f1918002830 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1918006590 con 0x7f1918002830 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f191c0036c0 con 0x7f1918002830 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 10 monclient: dump: 2024-09-16T06:54:29.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1453160473 0 0) 0x7f1918006590 con 0x7f1918002830 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f191800a190 con 0x7f1918002830 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f191c005850 con 0x7f1918002830 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:29.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6789/0 conn(0x7f1918002830 legacy=0x7f1918004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.231+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6789/0 conn(0x7f1918002830 legacy=0x7f1918004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:29.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1918009e30 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252305502 0 0) 0x7f1918009e30 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient(hunting): my global_id is 284305 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1918006590 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 999126268 0 0) 0x7f1918006590 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1918009e30 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f191c003e30 con 0x7f1918009150 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient: dump: 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1765399048 0 0) 0x7f1918009e30 con 0x7f1918009150 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f191800c340 con 0x7f1918009150 2024-09-16T06:54:29.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 2592579954) 0x7f191c004190 con 0x7f1918009150 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6789/0 conn(0x7f1918009150 legacy=0x7f19180095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f192affd700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6789/0 conn(0x7f1918009150 legacy=0x7f19180095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:29.248 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.249 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.250 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.252 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:29.253 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:29.254 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.256 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:29.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.259 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:29.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:29.262 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:29.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:29.262 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:29.262 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:29.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1914036610 legacy=0x7f1914038ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 10 monclient: shutdown 2024-09-16T06:54:29.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 >> v1:172.21.15.76:6790/0 conn(0x7f1934099030 legacy=0x7f19341ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.243+0000 7f1932d9d700 1 -- 172.21.15.76:0/3043661524 reap_dead start 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.247+0000 7f1932d9d700 1 -- 172.21.15.76:0/3043661524 reap_dead start 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.247+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 shutdown_connections 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.247+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 >> 172.21.15.76:0/3043661524 conn(0x7f19340a0030 msgr2=0x7f193406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.247+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 shutdown_connections 2024-09-16T06:54:29.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.247+0000 7f193359e700 1 -- 172.21.15.76:0/3043661524 wait complete. 2024-09-16T06:54:29.264 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:29.264 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:54:29.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 10 monclient: get_monmap_and_config 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 10 monclient: monmap: 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:29.389811+0000 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:29.389811+0000 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 1 Processor -- start 2024-09-16T06:54:29.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 1 -- start start 2024-09-16T06:54:29.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.387+0000 7f8d352bf700 10 monclient: init 2024-09-16T06:54:29.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-c con 0x7f8d30131910 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-a con 0x7f8d301347f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-b con 0x7f8d30138300 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d30132400 con 0x7f8d301347f0 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3009fcd0 con 0x7f8d30131910 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3009b300 con 0x7f8d30138300 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient(hunting): authenticate will time out at 1816.524759s 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d34abe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8d30138300 0x7f8d3013a6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40944/0 (socket says 172.21.15.76:40944) 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d34abe700 1 -- 172.21.15.76:0/3530770498 learned_addr learned my addr 172.21.15.76:0/3530770498 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175920987 0 0) 0x7f8d30132400 con 0x7f8d301347f0 2024-09-16T06:54:29.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): my global_id is 304116 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d180034f0 con 0x7f8d301347f0 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802311268 0 0) 0x7f8d3009fcd0 con 0x7f8d30131910 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): my global_id is 284308 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d30132400 con 0x7f8d30131910 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053488307 0 0) 0x7f8d3009b300 con 0x7f8d30138300 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient(hunting): my global_id is 244769 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3009fcd0 con 0x7f8d30138300 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276569390 0 0) 0x7f8d180034f0 con 0x7f8d301347f0 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 >> v1:172.21.15.76:6790/0 conn(0x7f8d30138300 legacy=0x7f8d3013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 >> v1:172.21.15.189:6789/0 conn(0x7f8d30131910 legacy=0x7f8d30131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 monclient: found mon.noname-a 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3009b7e0 con 0x7f8d301347f0 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d24001610 con 0x7f8d301347f0 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: dump: 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.392 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 5 monclient: authenticate success, global_id 304116 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d240018f0 con 0x7f8d301347f0 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 1 -- 172.21.15.76:0/3530770498 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d24001c80 con 0x7f8d301347f0 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d2effd700 10 monclient: dump: 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.393 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:29.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 4 set_mon_vals no callback set 2024-09-16T06:54:29.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 10 monclient: shutdown 2024-09-16T06:54:29.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- 172.21.15.76:0/3530770498 >> v1:172.21.15.76:6789/0 conn(0x7f8d301347f0 legacy=0x7f8d30136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- 172.21.15.76:0/3530770498 shutdown_connections 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- 172.21.15.76:0/3530770498 >> 172.21.15.76:0/3530770498 conn(0x7f8d300a4430 msgr2=0x7f8d300a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- 172.21.15.76:0/3530770498 shutdown_connections 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.391+0000 7f8d352bf700 1 -- 172.21.15.76:0/3530770498 wait complete. 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient: build_initial_monmap 2024-09-16T06:54:29.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient: monmap: 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 1 Processor -- start 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 1 -- start start 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient: init 2024-09-16T06:54:29.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-c con 0x7f8d30131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-a con 0x7f8d301347f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient(hunting): picked mon.noname-b con 0x7f8d30138300 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3006e0f0 con 0x7f8d301347f0 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3009d490 con 0x7f8d30138300 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d30137d00 con 0x7f8d30131910 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d352bf700 10 monclient(hunting): authenticate will time out at 1816.531465s 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8d30131910 0x7f8d30070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40948/0 (socket says 172.21.15.76:40948) 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2ffff700 1 -- 172.21.15.76:0/963727675 learned_addr learned my addr 172.21.15.76:0/963727675 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:29.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310451809 0 0) 0x7f8d3006e0f0 con 0x7f8d301347f0 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): my global_id is 304119 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c0035d0 con 0x7f8d301347f0 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847361447 0 0) 0x7f8d30137d00 con 0x7f8d30131910 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.395+0000 7f8d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): my global_id is 244772 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3006e0f0 con 0x7f8d30131910 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013682176 0 0) 0x7f8d3009fcd0 con 0x7f8d30138300 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:29.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): my global_id is 284311 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d30137d00 con 0x7f8d30138300 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1041772798 0 0) 0x7f8d1c0035d0 con 0x7f8d301347f0 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3009fcd0 con 0x7f8d301347f0 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1561099163 0 0) 0x7f8d3006e0f0 con 0x7f8d30131910 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d1c0035d0 con 0x7f8d30131910 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d240015d0 con 0x7f8d301347f0 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): dump: 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.404 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d20001830 con 0x7f8d30131910 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): dump: 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757562286 0 0) 0x7f8d30137d00 con 0x7f8d30138300 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3006e0f0 con 0x7f8d30138300 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d28001a90 con 0x7f8d30138300 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient(hunting): dump: 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.407 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.408 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1242223798 0 0) 0x7f8d1c0035d0 con 0x7f8d30131910 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 >> v1:172.21.15.189:6789/0 conn(0x7f8d30138300 legacy=0x7f8d301ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 >> v1:172.21.15.76:6789/0 conn(0x7f8d301347f0 legacy=0x7f8d301caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 monclient: found mon.c 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3009b0f0 con 0x7f8d30131910 2024-09-16T06:54:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: _finish_auth 0 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 5 monclient: authenticate success, global_id 244772 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _renew_subs 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3006f610 con 0x7f8d30131910 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _renew_subs 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d301378f0 con 0x7f8d30131910 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d200036a0 con 0x7f8d30131910 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d20001990 con 0x7f8d30131910 2024-09-16T06:54:29.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 10 monclient: dump: 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2dffb700 4 set_mon_vals no callback set 2024-09-16T06:54:29.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8d200039f0 con 0x7f8d30131910 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8d20043a70 con 0x7f8d30131910 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d10005100 con 0x7f8d30131910 2024-09-16T06:54:29.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.399+0000 7f8d2f7fe700 10 monclient: get_auth_request con 0x7f8d1c036690 auth_method 0 2024-09-16T06:54:29.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.407+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8d20014490 con 0x7f8d30131910 2024-09-16T06:54:29.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.407+0000 7f8d2cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:29.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.407+0000 7f8d2cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:29.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.639+0000 7f8d352bf700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:54:29.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.639+0000 7f8d352bf700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:54:29.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.639+0000 7f8d352bf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:29.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.639+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8d10005bc0 con 0x7f8d30131910 2024-09-16T06:54:29.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d2cff9700 1 -- 172.21.15.76:0/963727675 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f8d200140a0 con 0x7f8d30131910 2024-09-16T06:54:29.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d2cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:54:29.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d2cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=12,kv=13,logm=48} crc {auth=4245899021,config=1228700967,health=2509884219,kv=1001081567,logm=2048421261}) 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=3291599931,config=1097430493,health=2509884220,kv=1477047248,logm=1392056109}) 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=12,kv=13,logm=48} crc {auth=4245899021,config=1228700967,health=2509884219,kv=1001081567,logm=2048421261}) 2024-09-16T06:54:29.640 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=13,logm=48} crc {auth=1924351873,config=1097430493,health=188528523,kv=1001081567,logm=2909263913}) 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8d1c036690 legacy=0x7f8d1c038b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 10 monclient: shutdown 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 >> v1:172.21.15.76:6790/0 conn(0x7f8d30131910 legacy=0x7f8d30070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 shutdown_connections 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 >> 172.21.15.76:0/963727675 conn(0x7f8d300a4430 msgr2=0x7f8d300a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 shutdown_connections 2024-09-16T06:54:29.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:29.643+0000 7f8d352bf700 1 -- 172.21.15.76:0/963727675 wait complete. 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127758266}) 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4009273084}) 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127758266}) 2024-09-16T06:54:29.643 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991067375}) 2024-09-16T06:54:29.645 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095621070}) 2024-09-16T06:54:29.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3818085694}) 2024-09-16T06:54:29.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095621070}) 2024-09-16T06:54:29.646 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.647+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950609244}) 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788385575}) 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008941293}) 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788385575}) 2024-09-16T06:54:29.649 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048494637}) 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474842099}) 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220830399}) 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.651+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474842099}) 2024-09-16T06:54:29.650 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397937516}) 2024-09-16T06:54:29.651 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.651 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415849914}) 2024-09-16T06:54:29.651 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769794395}) 2024-09-16T06:54:29.651 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415849914}) 2024-09-16T06:54:29.652 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165041389}) 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820675397}) 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095571388}) 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820675397}) 2024-09-16T06:54:29.653 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1066210632}) 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672933202}) 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299253690}) 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672933202}) 2024-09-16T06:54:29.654 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.655+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751038346}) 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412529651}) 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385943428}) 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412529651}) 2024-09-16T06:54:29.655 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649629545}) 2024-09-16T06:54:29.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977394127}) 2024-09-16T06:54:29.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29619318}) 2024-09-16T06:54:29.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.656 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977394127}) 2024-09-16T06:54:29.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580433515}) 2024-09-16T06:54:29.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770595280}) 2024-09-16T06:54:29.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=317496248}) 2024-09-16T06:54:29.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770595280}) 2024-09-16T06:54:29.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.659+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096476267}) 2024-09-16T06:54:29.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743031990}) 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231615345}) 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743031990}) 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088822433}) 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.659 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258230054}) 2024-09-16T06:54:29.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911547347}) 2024-09-16T06:54:29.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258230054}) 2024-09-16T06:54:29.660 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=534956514}) 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3980747660}) 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426227315}) 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3980747660}) 2024-09-16T06:54:29.661 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511280697}) 2024-09-16T06:54:29.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189276664}) 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221617597}) 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189276664}) 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.663+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009706315}) 2024-09-16T06:54:29.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109171582}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472851240}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109171582}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676904620}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15039242}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178465387}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15039242}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022083787}) 2024-09-16T06:54:29.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235821308}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695073435}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235821308}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225771989}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52176242}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710837317}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52176242}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.667+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391484793}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911214859}) 2024-09-16T06:54:29.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664401514}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911214859}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356475094}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038132957}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167900340}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038132957}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549201884}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784119378}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348439387}) 2024-09-16T06:54:29.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784119378}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498592195}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=834511678}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569794605}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=834511678}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416134375}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.671+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3073880836}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728513922}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3073880836}) 2024-09-16T06:54:29.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033485157}) 2024-09-16T06:54:29.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315782744}) 2024-09-16T06:54:29.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345271948}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315782744}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194585243}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399876063}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408093543}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399876063}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3699789747}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752960071}) 2024-09-16T06:54:29.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949989981}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752960071}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878667576}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703648305}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243525953}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703648305}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.675+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544237580}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710201171}) 2024-09-16T06:54:29.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2648038788}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710201171}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990861289}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114619163}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1761847549}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114619163}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997297931}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621397908}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4196448899}) 2024-09-16T06:54:29.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621397908}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4095787429}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101344602}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500677423}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101344602}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.679+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605944671}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581896228}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377285218}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581896228}) 2024-09-16T06:54:29.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236262838}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082560966}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278332816}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082560966}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=722103535}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759104031}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387873389}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759104031}) 2024-09-16T06:54:29.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663365966}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763705046}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91741681}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763705046}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.683+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816581881}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849286875}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496500730}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849286875}) 2024-09-16T06:54:29.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=75247304}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218000885}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533793300}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218000885}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3178750476}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2602208098}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141710003}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2602208098}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632591165}) 2024-09-16T06:54:29.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276062650}) 2024-09-16T06:54:29.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803729282}) 2024-09-16T06:54:29.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276062650}) 2024-09-16T06:54:29.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.687+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026925847}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995611343}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395583954}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995611343}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101526339}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027936389}) 2024-09-16T06:54:29.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287385723}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027936389}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153039030}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966368379}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916119749}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966368379}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618046086}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385632514}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956582608}) 2024-09-16T06:54:29.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385632514}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.691+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107172622}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006757968}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278867450}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006757968}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950174954}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850452435}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757383944}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850452435}) 2024-09-16T06:54:29.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135591242}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863379735}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688674157}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863379735}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453863351}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569664651}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034144066}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569664651}) 2024-09-16T06:54:29.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.695+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550643474}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1772746346}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622906492}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1772746346}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937363423}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873161477}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107625286}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873161477}) 2024-09-16T06:54:29.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079226789}) 2024-09-16T06:54:29.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122449350}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974508826}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122449350}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.699+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500219914}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540797861}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3686928911}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540797861}) 2024-09-16T06:54:29.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350449133}) 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317144443}) 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992804180}) 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317144443}) 2024-09-16T06:54:29.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322186972}) 2024-09-16T06:54:29.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437666929}) 2024-09-16T06:54:29.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701968902}) 2024-09-16T06:54:29.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437666929}) 2024-09-16T06:54:29.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.703+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733045571}) 2024-09-16T06:54:29.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451418689}) 2024-09-16T06:54:29.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410980621}) 2024-09-16T06:54:29.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451418689}) 2024-09-16T06:54:29.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103797258}) 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665972619}) 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251334626}) 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665972619}) 2024-09-16T06:54:29.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538082031}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3651964011}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907817976}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3651964011}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.707+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2202969798}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912948094}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732967406}) 2024-09-16T06:54:29.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912948094}) 2024-09-16T06:54:29.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885044195}) 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906781127}) 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052253614}) 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906781127}) 2024-09-16T06:54:29.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395198527}) 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790176143}) 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338765335}) 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790176143}) 2024-09-16T06:54:29.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.711+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1419416358}) 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703303967}) 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977158797}) 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703303967}) 2024-09-16T06:54:29.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4185134583}) 2024-09-16T06:54:29.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485583807}) 2024-09-16T06:54:29.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607481576}) 2024-09-16T06:54:29.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485583807}) 2024-09-16T06:54:29.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398762986}) 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1181269357}) 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616261001}) 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1181269357}) 2024-09-16T06:54:29.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.715+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298292758}) 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2109068993}) 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662648773}) 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2109068993}) 2024-09-16T06:54:29.715 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791482684}) 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950807796}) 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676109692}) 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950807796}) 2024-09-16T06:54:29.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1453319639}) 2024-09-16T06:54:29.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447369316}) 2024-09-16T06:54:29.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459083157}) 2024-09-16T06:54:29.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447369316}) 2024-09-16T06:54:29.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.719+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140352343}) 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602216757}) 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003129756}) 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602216757}) 2024-09-16T06:54:29.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457265407}) 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009764541}) 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229266308}) 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009764541}) 2024-09-16T06:54:29.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324175546}) 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594714862}) 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396457503}) 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594714862}) 2024-09-16T06:54:29.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135636542}) 2024-09-16T06:54:29.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201414213}) 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941173087}) 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.723+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201414213}) 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752258674}) 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470969359}) 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094136570}) 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470969359}) 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410152593}) 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163784795}) 2024-09-16T06:54:29.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346227921}) 2024-09-16T06:54:29.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163784795}) 2024-09-16T06:54:29.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612320130}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810651211}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163821235}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810651211}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838623420}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591082732}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3064161419}) 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591082732}) 2024-09-16T06:54:29.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.727+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576946780}) 2024-09-16T06:54:29.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618216202}) 2024-09-16T06:54:29.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244765320}) 2024-09-16T06:54:29.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618216202}) 2024-09-16T06:54:29.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344578215}) 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557217654}) 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874052212}) 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557217654}) 2024-09-16T06:54:29.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.731+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629796174}) 2024-09-16T06:54:29.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464378770}) 2024-09-16T06:54:29.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2866596200}) 2024-09-16T06:54:29.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464378770}) 2024-09-16T06:54:29.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287498650}) 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465660058}) 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695933676}) 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465660058}) 2024-09-16T06:54:29.732 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3287692211}) 2024-09-16T06:54:29.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56184892}) 2024-09-16T06:54:29.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702939256}) 2024-09-16T06:54:29.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56184892}) 2024-09-16T06:54:29.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.735+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016693479}) 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908864092}) 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72147305}) 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908864092}) 2024-09-16T06:54:29.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328837019}) 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485578131}) 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623998555}) 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485578131}) 2024-09-16T06:54:29.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604172619}) 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642864121}) 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164693495}) 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642864121}) 2024-09-16T06:54:29.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3486257094}) 2024-09-16T06:54:29.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302535182}) 2024-09-16T06:54:29.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44300784}) 2024-09-16T06:54:29.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.739+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302535182}) 2024-09-16T06:54:29.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548895009}) 2024-09-16T06:54:29.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011956621}) 2024-09-16T06:54:29.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939044744}) 2024-09-16T06:54:29.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011956621}) 2024-09-16T06:54:29.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864270787}) 2024-09-16T06:54:29.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055168465}) 2024-09-16T06:54:29.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725878965}) 2024-09-16T06:54:29.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055168465}) 2024-09-16T06:54:29.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3974944599}) 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748961218}) 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846863155}) 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748961218}) 2024-09-16T06:54:29.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.743+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178501309}) 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078452500}) 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685577369}) 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078452500}) 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419864215}) 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192427317}) 2024-09-16T06:54:29.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633346745}) 2024-09-16T06:54:29.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192427317}) 2024-09-16T06:54:29.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455483166}) 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187562320}) 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865490178}) 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187562320}) 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2687863996}) 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935248231}) 2024-09-16T06:54:29.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523571024}) 2024-09-16T06:54:29.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935248231}) 2024-09-16T06:54:29.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.747+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=300424085}) 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4083902105}) 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391122952}) 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4083902105}) 2024-09-16T06:54:29.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908585442}) 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609231808}) 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584455774}) 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609231808}) 2024-09-16T06:54:29.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.751+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221977624}) 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=36} crc {logm=2763125398,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1190290531,mon_config_key=3158179092,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=3590317730}) 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=39} crc {logm=2763125398,mdsmap=3140514009,mgr=1018678713,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=27762313,mon_config_key=321043148,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=1587767677}) 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=36} crc {logm=2763125398,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1190290531,mon_config_key=3158179092,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=3590317730}) 2024-09-16T06:54:29.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.755+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=46346761,mdsmap=1531151339,mgr=278123252,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2780637732,mon_config_key=3366812847,monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1564638089}) 2024-09-16T06:54:29.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=1404438588}) 2024-09-16T06:54:29.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=3742194964}) 2024-09-16T06:54:29.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:54:29.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=1404438588}) 2024-09-16T06:54:29.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:54:29.763+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=18} crc {osdmap=37580365}) 2024-09-16T06:54:30.666 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:54:30.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: get_monmap_and_config 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: build_initial_monmap 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: monmap: 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:30.790420+0000 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:30.790420+0000 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:30.785 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:30.786 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:30.786 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:30.786 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 1 Processor -- start 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 1 -- start start 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: init 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient(hunting): picked mon.noname-c con 0x7f024c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:30.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient(hunting): picked mon.noname-b con 0x7f024c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient(hunting): picked mon.noname-a con 0x7f024c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09d560 con 0x7f024c099030 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09f140 con 0x7f024c09ed60 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09d790 con 0x7f024c098be0 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.787+0000 7f025230d700 10 monclient(hunting): authenticate will time out at 1817.923779s 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0251b0c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f024c099030 0x7f024c13ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43208/0 (socket says 172.21.15.76:43208) 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0251b0c700 1 -- 172.21.15.76:0/551314845 learned_addr learned my addr 172.21.15.76:0/551314845 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0250b0a700 1 --1- 172.21.15.76:0/551314845 >> v1:172.21.15.76:6790/0 conn(0x7f024c098be0 0x7f024c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40962/0 (socket says 172.21.15.76:40962) 2024-09-16T06:54:30.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537698250 0 0) 0x7f024c09d790 con 0x7f024c098be0 2024-09-16T06:54:30.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:30.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): my global_id is 244775 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02380035d0 con 0x7f024c098be0 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582756812 0 0) 0x7f024c09d560 con 0x7f024c099030 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): my global_id is 304122 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c09d790 con 0x7f024c099030 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635533485 0 0) 0x7f024c09f140 con 0x7f024c09ed60 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:30.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient(hunting): my global_id is 284314 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c09d560 con 0x7f024c09ed60 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 174408528 0 0) 0x7f02380035d0 con 0x7f024c098be0 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 >> v1:172.21.15.189:6789/0 conn(0x7f024c09ed60 legacy=0x7f024c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 >> v1:172.21.15.76:6789/0 conn(0x7f024c099030 legacy=0x7f024c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: _finish_hunting 0 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 monclient: found mon.noname-b 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c09f620 con 0x7f024c098be0 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: _finish_auth 0 2024-09-16T06:54:30.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: _check_auth_tickets 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 5 monclient: authenticate success, global_id 244775 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0248001610 con 0x7f024c098be0 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: dump: 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:30.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02480018f0 con 0x7f024c098be0 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 1 -- 172.21.15.76:0/551314845 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0248001c80 con 0x7f024c098be0 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 4 set_mon_vals no callback set 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f0243fff700 10 monclient: dump: 2024-09-16T06:54:30.795 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 10 monclient: shutdown 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:30.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 1 -- 172.21.15.76:0/551314845 >> v1:172.21.15.76:6790/0 conn(0x7f024c098be0 legacy=0x7f024c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:30.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 1 -- 172.21.15.76:0/551314845 shutdown_connections 2024-09-16T06:54:30.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 1 -- 172.21.15.76:0/551314845 >> 172.21.15.76:0/551314845 conn(0x7f024c0a0030 msgr2=0x7f024c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:30.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 1 -- 172.21.15.76:0/551314845 shutdown_connections 2024-09-16T06:54:30.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.791+0000 7f025230d700 1 -- 172.21.15.76:0/551314845 wait complete. 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient: build_initial_monmap 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient: monmap: 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:30.801 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 1 Processor -- start 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 1 -- start start 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient: init 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient(hunting): picked mon.noname-c con 0x7f024c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient(hunting): picked mon.noname-b con 0x7f024c099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient(hunting): picked mon.noname-a con 0x7f024c09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c139520 con 0x7f024c09ed60 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0a14e0 con 0x7f024c099030 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025130b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f024c098be0 0x7f024c13a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40972/0 (socket says 172.21.15.76:40972) 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025130b700 1 -- 172.21.15.76:0/1140478275 learned_addr learned my addr 172.21.15.76:0/1140478275 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c09a540 con 0x7f024c098be0 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f025230d700 10 monclient(hunting): authenticate will time out at 1817.930697s 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455642783 0 0) 0x7f024c09a540 con 0x7f024c098be0 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): my global_id is 244778 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02300035d0 con 0x7f024c098be0 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009823175 0 0) 0x7f02300035d0 con 0x7f024c098be0 2024-09-16T06:54:30.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c09a540 con 0x7f024c098be0 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f023c001990 con 0x7f024c098be0 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): dump: 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:30.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.795+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882982336 0 0) 0x7f024c09a540 con 0x7f024c098be0 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 >> v1:172.21.15.189:6789/0 conn(0x7f024c099030 legacy=0x7f024c1cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 >> v1:172.21.15.76:6789/0 conn(0x7f024c09ed60 legacy=0x7f024c1cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 monclient: found mon.c 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c139dc0 con 0x7f024c098be0 2024-09-16T06:54:30.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: _finish_auth 0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 5 monclient: authenticate success, global_id 244778 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _renew_subs 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c098900 con 0x7f024c098be0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _renew_subs 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c138800 con 0x7f024c098be0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f023c001cb0 con 0x7f024c098be0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f023c003b10 con 0x7f024c098be0 2024-09-16T06:54:30.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f02437fe700 4 set_mon_vals no callback set 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f0241ffb700 10 monclient: dump: 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:30.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.799+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c005100 con 0x7f024c098be0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.803+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f023c003cf0 con 0x7f024c098be0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.803+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f023c043ea0 con 0x7f024c098be0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.803+0000 7f0250b0a700 10 monclient: get_auth_request con 0x7f023003a560 auth_method 0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.807+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f023c0147e0 con 0x7f024c098be0 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.807+0000 7f0241ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:30.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:30.807+0000 7f0241ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:31.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f025230d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f025230d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f025230d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f022c0053b0 con 0x7f024c098be0 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f0241ffb700 1 -- 172.21.15.76:0/1140478275 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 700304707) 0x7f023c0147e0 con 0x7f024c098be0 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f0241ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:31.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.039+0000 7f0241ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:31.038 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:31.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 >> v1:172.21.15.189:6812/3098810674 conn(0x7f023003a560 legacy=0x7f023003ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 10 monclient: shutdown 2024-09-16T06:54:31.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 >> v1:172.21.15.76:6790/0 conn(0x7f024c098be0 legacy=0x7f024c13a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 shutdown_connections 2024-09-16T06:54:31.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 >> 172.21.15.76:0/1140478275 conn(0x7f024c0a0030 msgr2=0x7f024c13ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 shutdown_connections 2024-09-16T06:54:31.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.043+0000 7f025230d700 1 -- 172.21.15.76:0/1140478275 wait complete. 2024-09-16T06:54:31.051 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:54:31.051 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:54:31.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:54:31.051 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: get_monmap_and_config 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: monmap: 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:31.169478+0000 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:31.169478+0000 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.164 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:31.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:31.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 1 Processor -- start 2024-09-16T06:54:31.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 1 -- start start 2024-09-16T06:54:31.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: init 2024-09-16T06:54:31.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb94809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb948098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb948099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94809d560 con 0x7fb948099030 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94809f140 con 0x7fb94809ed60 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94809d790 con 0x7fb948098be0 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 10 monclient(hunting): authenticate will time out at 1818.302721s 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb93dd9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb948098be0 0x7fb9481374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40988/0 (socket says 172.21.15.76:40988) 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb946d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb948099030 0x7fb94813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43234/0 (socket says 172.21.15.76:43234) 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb93dd9b700 1 -- 172.21.15.76:0/1686081516 learned_addr learned my addr 172.21.15.76:0/1686081516 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918551477 0 0) 0x7fb94809d560 con 0x7fb948099030 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): my global_id is 304125 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9340035d0 con 0x7fb948099030 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309044111 0 0) 0x7fb94809d790 con 0x7fb948098be0 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient(hunting): my global_id is 244781 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94809d560 con 0x7fb948098be0 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490077940 0 0) 0x7fb9340035d0 con 0x7fb948099030 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 >> v1:172.21.15.76:6790/0 conn(0x7fb948098be0 legacy=0x7fb9481374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 >> v1:172.21.15.189:6789/0 conn(0x7fb94809ed60 legacy=0x7fb9480984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 monclient: found mon.noname-a 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94809f620 con 0x7fb948099030 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 5 monclient: authenticate success, global_id 304125 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb94759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb940001990 con 0x7fb948099030 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.167+0000 7fb945d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: dump: 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.171 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb940001c70 con 0x7fb948099030 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 1 -- 172.21.15.76:0/1686081516 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb940002040 con 0x7fb948099030 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 4 set_mon_vals no callback set 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb945d9b700 10 monclient: dump: 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: shutdown 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/1686081516 >> v1:172.21.15.76:6789/0 conn(0x7fb948099030 legacy=0x7fb94813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/1686081516 shutdown_connections 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/1686081516 >> 172.21.15.76:0/1686081516 conn(0x7fb9480a0030 msgr2=0x7fb9480a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/1686081516 shutdown_connections 2024-09-16T06:54:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/1686081516 wait complete. 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: monmap: 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:31.174 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 Processor -- start 2024-09-16T06:54:31.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- start start 2024-09-16T06:54:31.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: init 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb948098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb948099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb94809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb948095070 con 0x7fb94809ed60 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9480922f0 con 0x7fb948099030 2024-09-16T06:54:31.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9480986c0 con 0x7fb948098be0 2024-09-16T06:54:31.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient(hunting): authenticate will time out at 1818.306304s 2024-09-16T06:54:31.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb946d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb94809ed60 0x7fb9481d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43250/0 (socket says 172.21.15.76:43250) 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb948098be0 0x7fb9480914d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41000/0 (socket says 172.21.15.76:41000) 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb946d9d700 1 -- 172.21.15.76:0/3506968417 learned_addr learned my addr 172.21.15.76:0/3506968417 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337840194 0 0) 0x7fb948095070 con 0x7fb94809ed60 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): my global_id is 304128 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9280035d0 con 0x7fb94809ed60 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572370657 0 0) 0x7fb9480986c0 con 0x7fb948098be0 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): my global_id is 244784 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb948095070 con 0x7fb948098be0 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511564836 0 0) 0x7fb9480922f0 con 0x7fb948099030 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): my global_id is 284320 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9480986c0 con 0x7fb948099030 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3831711090 0 0) 0x7fb9280035d0 con 0x7fb94809ed60 2024-09-16T06:54:31.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9480922f0 con 0x7fb94809ed60 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb940001950 con 0x7fb94809ed60 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): dump: 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.185 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1186850223 0 0) 0x7fb948095070 con 0x7fb948098be0 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9280035d0 con 0x7fb948098be0 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9380018b0 con 0x7fb948098be0 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient(hunting): dump: 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.186 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372132327 0 0) 0x7fb9480922f0 con 0x7fb94809ed60 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6790/0 conn(0x7fb948098be0 legacy=0x7fb9480914d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.189:6789/0 conn(0x7fb948099030 legacy=0x7fb948091be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 monclient: found mon.a 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94809a540 con 0x7fb94809ed60 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb93f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 5 monclient: authenticate success, global_id 304128 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _renew_subs 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9480a1250 con 0x7fb94809ed60 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _renew_subs 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.171+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb948093940 con 0x7fb94809ed60 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb940001570 con 0x7fb94809ed60 2024-09-16T06:54:31.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb940005100 con 0x7fb94809ed60 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: dump: 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.189 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb94759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb94559a700 4 set_mon_vals no callback set 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb9400052e0 con 0x7fb94809ed60 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fb940044390 con 0x7fb94809ed60 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb948096f20 con 0x7fb948096b40 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb94659c700 10 monclient: get_auth_request con 0x7fb928036490 auth_method 0 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615847144 0 0) 0x7fb948096f20 con 0x7fb948096b40 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient(hunting): my global_id is 304131 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9480922f0 con 0x7fb948096b40 2024-09-16T06:54:31.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3447700208 0 0) 0x7fb9480922f0 con 0x7fb948096b40 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb948096f20 con 0x7fb948096b40 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb9300016e0 con 0x7fb948096b40 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: dump: 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.191 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3491700444 0 0) 0x7fb948096f20 con 0x7fb948096b40 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9480a1610 con 0x7fb948096b40 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fb930001ad0 con 0x7fb948096b40 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6789/0 conn(0x7fb948096b40 legacy=0x7fb948061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.175+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6789/0 conn(0x7fb948096b40 legacy=0x7fb948061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb94759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb94802cc30 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535427613 0 0) 0x7fb94802cc30 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient(hunting): my global_id is 304134 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb948096f20 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 701063969 0 0) 0x7fb948096f20 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb94802cc30 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb930004c20 con 0x7fb94802c820 2024-09-16T06:54:31.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient: dump: 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.195 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2509466088 0 0) 0x7fb94802cc30 con 0x7fb94802c820 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb94809aaa0 con 0x7fb94802c820 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3421681415) 0x7fb930005f30 con 0x7fb94802c820 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6789/0 conn(0x7fb94802c820 legacy=0x7fb9480705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.187+0000 7fb93f7fe700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6789/0 conn(0x7fb94802c820 legacy=0x7fb9480705d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:31.196 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.197 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.198 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:31.199 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.200 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.201 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.202 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.204 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:31.205 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.206 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.207 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:31.207 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb928036490 legacy=0x7fb928038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 10 monclient: shutdown 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb946d9d700 1 -- 172.21.15.76:0/3506968417 reap_dead start 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 >> v1:172.21.15.76:6789/0 conn(0x7fb94809ed60 legacy=0x7fb9481d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 shutdown_connections 2024-09-16T06:54:31.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 >> 172.21.15.76:0/3506968417 conn(0x7fb9480a0030 msgr2=0x7fb94806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 shutdown_connections 2024-09-16T06:54:31.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.191+0000 7fb94759e700 1 -- 172.21.15.76:0/3506968417 wait complete. 2024-09-16T06:54:31.209 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:31.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:54:31.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: get_monmap_and_config 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: monmap: 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:31.328547+0000 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:31.328547+0000 2024-09-16T06:54:31.323 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.324 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.324 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.324 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:31.324 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:31.324 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 1 Processor -- start 2024-09-16T06:54:31.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 1 -- start start 2024-09-16T06:54:31.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: init 2024-09-16T06:54:31.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-a con 0x7ff8f0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-b con 0x7ff8f006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-c con 0x7ff8f006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f006c280 con 0x7ff8f0132330 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f006bfa0 con 0x7ff8f006cf50 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f009ace0 con 0x7ff8f006d480 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8effff700 10 monclient(hunting): authenticate will time out at 1818.463074s 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8ee7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff8f006d480 0x7ff8f013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41008/0 (socket says 172.21.15.76:41008) 2024-09-16T06:54:31.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8ee7fc700 1 -- 172.21.15.76:0/4028205881 learned_addr learned my addr 172.21.15.76:0/4028205881 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479845558 0 0) 0x7ff8f009ace0 con 0x7ff8f006d480 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): my global_id is 244787 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc0035d0 con 0x7ff8f006d480 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368382811 0 0) 0x7ff8f006c280 con 0x7ff8f0132330 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): my global_id is 304137 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f009ace0 con 0x7ff8f0132330 2024-09-16T06:54:31.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831525608 0 0) 0x7ff8f006bfa0 con 0x7ff8f006cf50 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 10 monclient(hunting): my global_id is 284323 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.327+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f006c280 con 0x7ff8f006cf50 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2250777357 0 0) 0x7ff8f009ace0 con 0x7ff8f0132330 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 >> v1:172.21.15.76:6790/0 conn(0x7ff8f006d480 legacy=0x7ff8f013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 >> v1:172.21.15.189:6789/0 conn(0x7ff8f006cf50 legacy=0x7ff8f013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 monclient: found mon.noname-a 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f009ae70 con 0x7ff8f0132330 2024-09-16T06:54:31.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8e00015f0 con 0x7ff8f0132330 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 5 monclient: authenticate success, global_id 304137 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: dump: 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.338 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e0001910 con 0x7ff8f0132330 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 1 -- 172.21.15.76:0/4028205881 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8e0001b00 con 0x7ff8f0132330 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 4 set_mon_vals no callback set 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8edffb700 10 monclient: dump: 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: shutdown 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- 172.21.15.76:0/4028205881 >> v1:172.21.15.76:6789/0 conn(0x7ff8f0132330 legacy=0x7ff8f0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- 172.21.15.76:0/4028205881 shutdown_connections 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- 172.21.15.76:0/4028205881 >> 172.21.15.76:0/4028205881 conn(0x7ff8f00a41c0 msgr2=0x7ff8f00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- 172.21.15.76:0/4028205881 shutdown_connections 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- 172.21.15.76:0/4028205881 wait complete. 2024-09-16T06:54:31.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: monmap: 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 Processor -- start 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- start start 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: init 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-a con 0x7ff8f006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-c con 0x7ff8f006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient(hunting): picked mon.noname-b con 0x7ff8f0132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f0131720 con 0x7ff8f006cf50 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f012ff00 con 0x7ff8f0132330 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f01311d0 con 0x7ff8f006d480 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8effff700 10 monclient(hunting): authenticate will time out at 1818.467223s 2024-09-16T06:54:31.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8eeffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff8f006cf50 0x7ff8f0129110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43296/0 (socket says 172.21.15.76:43296) 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8eeffd700 1 -- 172.21.15.76:0/2773894801 learned_addr learned my addr 172.21.15.76:0/2773894801 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898772991 0 0) 0x7ff8f0131720 con 0x7ff8f006cf50 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): my global_id is 304140 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d00035d0 con 0x7ff8f006cf50 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63575756 0 0) 0x7ff8f01311d0 con 0x7ff8f006d480 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 10 monclient(hunting): my global_id is 244790 2024-09-16T06:54:31.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f0131720 con 0x7ff8f006d480 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.331+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917975697 0 0) 0x7ff8f012ff00 con 0x7ff8f0132330 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): my global_id is 284326 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f01311d0 con 0x7ff8f0132330 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2000422352 0 0) 0x7ff8d00035d0 con 0x7ff8f006cf50 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f012ff00 con 0x7ff8f006cf50 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8e00036a0 con 0x7ff8f006cf50 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): dump: 2024-09-16T06:54:31.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358101614 0 0) 0x7ff8f0131720 con 0x7ff8f006d480 2024-09-16T06:54:31.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8d00035d0 con 0x7ff8f006d480 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8d80018d0 con 0x7ff8f006d480 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): dump: 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.347 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512646806 0 0) 0x7ff8f012ff00 con 0x7ff8f006cf50 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6790/0 conn(0x7ff8f006d480 legacy=0x7ff8f01cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.189:6789/0 conn(0x7ff8f0132330 legacy=0x7ff8f01d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 monclient: found mon.a 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f009e8b0 con 0x7ff8f006cf50 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 5 monclient: authenticate success, global_id 304140 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 10 monclient: _renew_subs 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f012ea50 con 0x7ff8f006cf50 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 10 monclient: _renew_subs 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f012ec90 con 0x7ff8f006cf50 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff8f00a55c0 con 0x7ff8f0130970 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e0003bc0 con 0x7ff8f006cf50 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8e0003f50 con 0x7ff8f006cf50 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: dump: 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8ed7fa700 4 set_mon_vals no callback set 2024-09-16T06:54:31.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff8e0004170 con 0x7ff8f006cf50 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff8e0043e50 con 0x7ff8f006cf50 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524321617 0 0) 0x7ff8f00a55c0 con 0x7ff8f0130970 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient(hunting): my global_id is 244793 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f012ff00 con 0x7ff8f0130970 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3562301176 0 0) 0x7ff8f012ff00 con 0x7ff8f0130970 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f00a55c0 con 0x7ff8f0130970 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8d8004a30 con 0x7ff8f0130970 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 10 monclient: dump: 2024-09-16T06:54:31.351 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8ef7fe700 10 monclient: get_auth_request con 0x7ff8d00361c0 auth_method 0 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 186897837 0 0) 0x7ff8f00a55c0 con 0x7ff8f0130970 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.335+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8f00a3450 con 0x7ff8f0130970 2024-09-16T06:54:31.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.339+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7ff8d8005980 con 0x7ff8f0130970 2024-09-16T06:54:31.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.339+0000 7ff8e77fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.339+0000 7ff8e77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:31.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.339+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6790/0 conn(0x7ff8f0130970 legacy=0x7ff8f009b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.339+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6790/0 conn(0x7ff8f0130970 legacy=0x7ff8f009b500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8effff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff8f0129c00 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337325653 0 0) 0x7ff8f0129c00 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient(hunting): my global_id is 244796 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f00a55c0 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4130370868 0 0) 0x7ff8f00a55c0 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f0129c00 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff8d80050b0 con 0x7ff8f0129820 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient: dump: 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3362823677 0 0) 0x7ff8f0129c00 con 0x7ff8f0129820 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff8f00672a0 con 0x7ff8f0129820 2024-09-16T06:54:31.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4214 (unknown 0 0 1685888974) 0x7ff8d8005460 con 0x7ff8f0129820 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6790/0 conn(0x7ff8f0129820 legacy=0x7ff8f002c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.347+0000 7ff8e77fe700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6790/0 conn(0x7ff8f0129820 legacy=0x7ff8f002c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:31.357 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:31.360 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.361 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.362 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.363 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.364 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.365 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.366 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.367 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.368 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:31.369 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:31.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff8d00361c0 legacy=0x7ff8d0038680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 10 monclient: shutdown 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 >> v1:172.21.15.76:6789/0 conn(0x7ff8f006cf50 legacy=0x7ff8f0129110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8ef7fe700 1 -- 172.21.15.76:0/2773894801 reap_dead start 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8ef7fe700 1 -- 172.21.15.76:0/2773894801 reap_dead start 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 shutdown_connections 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 >> 172.21.15.76:0/2773894801 conn(0x7ff8f00a41c0 msgr2=0x7ff8f00a0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 shutdown_connections 2024-09-16T06:54:31.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.351+0000 7ff8effff700 1 -- 172.21.15.76:0/2773894801 wait complete. 2024-09-16T06:54:31.371 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:31.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:54:31.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: get_monmap_and_config 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: monmap: 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:31.493516+0000 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:31.493516+0000 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:31.489 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 1 Processor -- start 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 1 -- start start 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: init 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-a con 0x7fe5a4098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a409d560 con 0x7fe5a4098be0 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a409f140 con 0x7fe5a4099030 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a409d790 con 0x7fe5a409ed60 2024-09-16T06:54:31.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a8fc6700 10 monclient(hunting): authenticate will time out at 1818.626945s 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe5a409ed60 0x7fe5a40984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41038/0 (socket says 172.21.15.76:41038) 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe59bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe5a4098be0 0x7fe5a41374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43304/0 (socket says 172.21.15.76:43304) 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a37fe700 1 -- 172.21.15.76:0/1218849397 learned_addr learned my addr 172.21.15.76:0/1218849397 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131116169 0 0) 0x7fe5a409d560 con 0x7fe5a4098be0 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): my global_id is 304143 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5900035d0 con 0x7fe5a4098be0 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239140714 0 0) 0x7fe5a409d790 con 0x7fe5a409ed60 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): my global_id is 244799 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a409d560 con 0x7fe5a409ed60 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958844953 0 0) 0x7fe5a409f140 con 0x7fe5a4099030 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 10 monclient(hunting): my global_id is 284329 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.491+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a409d790 con 0x7fe5a4099030 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799767861 0 0) 0x7fe5900035d0 con 0x7fe5a4098be0 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 >> v1:172.21.15.76:6790/0 conn(0x7fe5a409ed60 legacy=0x7fe5a40984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 >> v1:172.21.15.189:6789/0 conn(0x7fe5a4099030 legacy=0x7fe5a413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a409f620 con 0x7fe5a4098be0 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe58c001610 con 0x7fe5a4098be0 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: dump: 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.496 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 5 monclient: authenticate success, global_id 304143 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe58c0018f0 con 0x7fe5a4098be0 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 1 -- 172.21.15.76:0/1218849397 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe58c001c80 con 0x7fe5a4098be0 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 4 set_mon_vals no callback set 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a2ffd700 10 monclient: dump: 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 10 monclient: shutdown 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1218849397 >> v1:172.21.15.76:6789/0 conn(0x7fe5a4098be0 legacy=0x7fe5a41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1218849397 shutdown_connections 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1218849397 >> 172.21.15.76:0/1218849397 conn(0x7fe5a40a0030 msgr2=0x7fe5a40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1218849397 shutdown_connections 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1218849397 wait complete. 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 10 monclient: monmap: 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:31.499 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:31.500 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 Processor -- start 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 1 -- start start 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.495+0000 7fe5a8fc6700 10 monclient: init 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a4098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient(hunting): picked mon.noname-a con 0x7fe5a409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40a1570 con 0x7fe5a409ed60 2024-09-16T06:54:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a4096040 con 0x7fe5a4099030 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a408e370 con 0x7fe5a4098be0 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient(hunting): authenticate will time out at 1818.632399s 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe5a409ed60 0x7fe5a41ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:43312/0 (socket says 172.21.15.76:43312) 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe5a4098be0 0x7fe5a408d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41052/0 (socket says 172.21.15.76:41052) 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a3fff700 1 -- 172.21.15.76:0/1134754107 learned_addr learned my addr 172.21.15.76:0/1134754107 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019155973 0 0) 0x7fe5a409d790 con 0x7fe5a409ed60 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): my global_id is 304146 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5840035d0 con 0x7fe5a409ed60 2024-09-16T06:54:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590146521 0 0) 0x7fe5a408e370 con 0x7fe5a4098be0 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): my global_id is 244802 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a409d790 con 0x7fe5a4098be0 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820955009 0 0) 0x7fe5a4096040 con 0x7fe5a4099030 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): my global_id is 284332 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a408e370 con 0x7fe5a4099030 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490540846 0 0) 0x7fe5840035d0 con 0x7fe5a409ed60 2024-09-16T06:54:31.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a4096040 con 0x7fe5a409ed60 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe59c001af0 con 0x7fe5a409ed60 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): dump: 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439377538 0 0) 0x7fe5a409d790 con 0x7fe5a4098be0 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5840035d0 con 0x7fe5a4098be0 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe5940018b0 con 0x7fe5a4098be0 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): dump: 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.510 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230359802 0 0) 0x7fe5a408e370 con 0x7fe5a4099030 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a409d790 con 0x7fe5a4099030 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe58c0015d0 con 0x7fe5a4099030 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient(hunting): dump: 2024-09-16T06:54:31.511 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241818543 0 0) 0x7fe5a4096040 con 0x7fe5a409ed60 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.76:6790/0 conn(0x7fe5a4098be0 legacy=0x7fe5a408d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6789/0 conn(0x7fe5a4099030 legacy=0x7fe5a408dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 monclient: found mon.a 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4098910 con 0x7fe5a409ed60 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 5 monclient: authenticate success, global_id 304146 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _renew_subs 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a409a540 con 0x7fe5a409ed60 2024-09-16T06:54:31.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _renew_subs 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a4090aa0 con 0x7fe5a409ed60 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe59c003a40 con 0x7fe5a409ed60 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe59c003d50 con 0x7fe5a409ed60 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a1ffb700 4 set_mon_vals no callback set 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 10 monclient: dump: 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.514 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe59c003eb0 con 0x7fe5a409ed60 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe588006590 con 0x7fe588002830 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.499+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fe59c043ef0 con 0x7fe5a409ed60 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe59bfff700 10 monclient: get_auth_request con 0x7fe584036540 auth_method 0 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436235512 0 0) 0x7fe588006590 con 0x7fe588002830 2024-09-16T06:54:31.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient(hunting): my global_id is 284335 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a4096040 con 0x7fe588002830 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2743371450 0 0) 0x7fe5a4096040 con 0x7fe588002830 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe588006590 con 0x7fe588002830 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe594004c20 con 0x7fe588002830 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient: dump: 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.516 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 512179190 0 0) 0x7fe588006590 con 0x7fe588002830 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe58800a190 con 0x7fe588002830 2024-09-16T06:54:31.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe594004fb0 con 0x7fe588002830 2024-09-16T06:54:31.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:31.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6789/0 conn(0x7fe588002830 legacy=0x7fe588004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.503+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6789/0 conn(0x7fe588002830 legacy=0x7fe588004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.511+0000 7fe5a8fc6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.511+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe588009e30 con 0x7fe588009150 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978393754 0 0) 0x7fe588009e30 con 0x7fe588009150 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient(hunting): my global_id is 284338 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe588006590 con 0x7fe588009150 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 637177391 0 0) 0x7fe588006590 con 0x7fe588009150 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe588009e30 con 0x7fe588009150 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe594005530 con 0x7fe588009150 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient: dump: 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1627530703 0 0) 0x7fe588009e30 con 0x7fe588009150 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe58800c340 con 0x7fe588009150 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 26438347) 0x7fe594006900 con 0x7fe588009150 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6789/0 conn(0x7fe588009150 legacy=0x7fe5880095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a0ff9700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6789/0 conn(0x7fe588009150 legacy=0x7fe5880095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 144, 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:31.533 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:31.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe584036540 legacy=0x7fe584038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 10 monclient: shutdown 2024-09-16T06:54:31.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 >> v1:172.21.15.76:6789/0 conn(0x7fe5a409ed60 legacy=0x7fe5a41ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a3fff700 1 -- 172.21.15.76:0/1134754107 reap_dead start 2024-09-16T06:54:31.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a3fff700 1 -- 172.21.15.76:0/1134754107 reap_dead start 2024-09-16T06:54:31.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 shutdown_connections 2024-09-16T06:54:31.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 >> 172.21.15.76:0/1134754107 conn(0x7fe5a40a0030 msgr2=0x7fe5a406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 shutdown_connections 2024-09-16T06:54:31.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.515+0000 7fe5a8fc6700 1 -- 172.21.15.76:0/1134754107 wait complete. 2024-09-16T06:54:31.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:54:31.535 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:54:31.535 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:54:31.536 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:54:31.536 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:54:31.536 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:54:31.536 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:54:31.575 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:54:31.575 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:54:31.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: get_monmap_and_config 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: monmap: 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.687 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:31.692802+0000 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:31.692802+0000 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 1 Processor -- start 2024-09-16T06:54:31.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 1 -- start start 2024-09-16T06:54:31.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: init 2024-09-16T06:54:31.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:54:31.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-a con 0x7f6cd4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-c con 0x7f6cd406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd406c280 con 0x7f6cd4132330 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd406bfa0 con 0x7f6cd406d480 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cd91d0700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6cd4132330 legacy=0x7f6cd4132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd409ace0 con 0x7f6cd406cf50 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cda1d2700 10 monclient(hunting): authenticate will time out at 1818.827708s 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cd99d1700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6cd406cf50 0x7f6cd413da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41066/0 (socket says 172.21.15.76:41066) 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.691+0000 7f6cd99d1700 1 -- 172.21.15.76:0/800426814 learned_addr learned my addr 172.21.15.76:0/800426814 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270299863 0 0) 0x7f6cd409ace0 con 0x7f6cd406cf50 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): my global_id is 244805 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc00035d0 con 0x7f6cd406cf50 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648704955 0 0) 0x7f6cd406bfa0 con 0x7f6cd406d480 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient(hunting): my global_id is 284341 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd409ace0 con 0x7f6cd406d480 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 800640269 0 0) 0x7f6cc00035d0 con 0x7f6cd406cf50 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 >> v1:172.21.15.189:6789/0 conn(0x7f6cd406d480 legacy=0x7f6cd413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 >> v1:172.21.15.76:6789/0 conn(0x7f6cd4132330 legacy=0x7f6cd4132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 monclient: found mon.noname-b 2024-09-16T06:54:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd409ae70 con 0x7f6cd406cf50 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 5 monclient: authenticate success, global_id 244805 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ccc001990 con 0x7f6cd406cf50 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: dump: 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ccc001bb0 con 0x7f6cd406cf50 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 4 set_mon_vals no callback set 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 1 -- 172.21.15.76:0/800426814 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6ccc001da0 con 0x7f6cd406cf50 2024-09-16T06:54:31.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6ccbfff700 10 monclient: dump: 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: shutdown 2024-09-16T06:54:31.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- 172.21.15.76:0/800426814 >> v1:172.21.15.76:6790/0 conn(0x7f6cd406cf50 legacy=0x7f6cd413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- 172.21.15.76:0/800426814 shutdown_connections 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- 172.21.15.76:0/800426814 >> 172.21.15.76:0/800426814 conn(0x7f6cd40a41c0 msgr2=0x7f6cd40a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- 172.21.15.76:0/800426814 shutdown_connections 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- 172.21.15.76:0/800426814 wait complete. 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: build_initial_monmap 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: monmap: 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:31.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 Processor -- start 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- start start 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: init 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-c con 0x7f6cd406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient(hunting): picked mon.noname-a con 0x7f6cd4132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd41311b0 con 0x7f6cd4132330 2024-09-16T06:54:31.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd409e8b0 con 0x7f6cd406d480 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd413ed10 con 0x7f6cd406cf50 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cda1d2700 10 monclient(hunting): authenticate will time out at 1818.831884s 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.695+0000 7f6cd99d1700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f6cd4132330 legacy=0x7f6cd413e600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cd91d0700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f6cd406cf50 0x7f6cd4130a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41078/0 (socket says 172.21.15.76:41078) 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cd91d0700 1 -- 172.21.15.76:0/4001072878 learned_addr learned my addr 172.21.15.76:0/4001072878 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129399516 0 0) 0x7f6cd413ed10 con 0x7f6cd406cf50 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): my global_id is 244808 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb80035d0 con 0x7f6cd406cf50 2024-09-16T06:54:31.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927695810 0 0) 0x7f6cd409e8b0 con 0x7f6cd406d480 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): my global_id is 284344 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd413ed10 con 0x7f6cd406d480 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051459644 0 0) 0x7f6cb80035d0 con 0x7f6cd406cf50 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cd409e8b0 con 0x7f6cd406cf50 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6cd0004200 con 0x7f6cd406cf50 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): dump: 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.705 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489883235 0 0) 0x7f6cd409e8b0 con 0x7f6cd406cf50 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 >> v1:172.21.15.189:6789/0 conn(0x7f6cd406d480 legacy=0x7f6cd413def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 >> v1:172.21.15.76:6789/0 conn(0x7f6cd4132330 legacy=0x7f6cd413e600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 monclient: found mon.c 2024-09-16T06:54:31.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd412dfa0 con 0x7f6cd406cf50 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: _finish_auth 0 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 5 monclient: authenticate success, global_id 244808 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _renew_subs 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd412eae0 con 0x7f6cd406cf50 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _renew_subs 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd41d2e90 con 0x7f6cd406cf50 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cd0004470 con 0x7f6cd406cf50 2024-09-16T06:54:31.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6cd0004800 con 0x7f6cd406cf50 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6ccaffd700 4 set_mon_vals no callback set 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 10 monclient: dump: 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:31.708 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:31.709 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:31.709 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6cd0004960 con 0x7f6cd406cf50 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4005100 con 0x7f6cd406cf50 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6cd0043e60 con 0x7f6cd406cf50 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.699+0000 7f6cd89cf700 10 monclient: get_auth_request con 0x7f6cb80361d0 auth_method 0 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.707+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6cd0005160 con 0x7f6cd406cf50 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.707+0000 7f6cc9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.707+0000 7f6cc9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:31.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:31.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:31.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:31.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6cb4005930 con 0x7f6cd406cf50 2024-09-16T06:54:31.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cc9ffb700 1 -- 172.21.15.76:0/4001072878 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2437714556) 0x7f6cd0012bc0 con 0x7f6cd406cf50 2024-09-16T06:54:31.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cc9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:31.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cc9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:31.939 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6cb80361d0 legacy=0x7f6cb8038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 10 monclient: shutdown 2024-09-16T06:54:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 >> v1:172.21.15.76:6790/0 conn(0x7f6cd406cf50 legacy=0x7f6cd4130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 shutdown_connections 2024-09-16T06:54:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 >> 172.21.15.76:0/4001072878 conn(0x7f6cd40a41c0 msgr2=0x7f6cd409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 shutdown_connections 2024-09-16T06:54:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:31.943+0000 7f6cda1d2700 1 -- 172.21.15.76:0/4001072878 wait complete. 2024-09-16T06:54:34.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.074+0000 7f72aaeb0700 10 monclient: get_monmap_and_config 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.074+0000 7f72aaeb0700 10 monclient: build_initial_monmap 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.074+0000 7f72aaeb0700 10 monclient: monmap: 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:35.079699+0000 2024-09-16T06:54:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:35.079699+0000 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 1 Processor -- start 2024-09-16T06:54:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 1 -- start start 2024-09-16T06:54:35.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient: init 2024-09-16T06:54:35.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-c con 0x7f72a409abe0 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-b con 0x7f72a4133210 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-a con 0x7f72a409afc0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40a0c70 con 0x7f72a409afc0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a409b430 con 0x7f72a409abe0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aa6af700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f72a409afc0 legacy=0x7f72a413c0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a406b1b0 con 0x7f72a4133210 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 10 monclient(hunting): authenticate will time out at 1822.214696s 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a96ad700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f72a4133210 0x7f72a40a0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42976/0 (socket says 172.21.15.76:42976) 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a96ad700 1 -- 172.21.15.76:0/3247288759 learned_addr learned my addr 172.21.15.76:0/3247288759 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:35.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870536666 0 0) 0x7f72a406b1b0 con 0x7f72a4133210 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient(hunting): _negotiate 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient(hunting): my global_id is 244811 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72900035d0 con 0x7f72a4133210 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870524666 0 0) 0x7f72900035d0 con 0x7f72a4133210 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 >> v1:172.21.15.189:6789/0 conn(0x7f72a409abe0 legacy=0x7f72a409fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 >> v1:172.21.15.76:6789/0 conn(0x7f72a409afc0 legacy=0x7f72a413c0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: _finish_hunting 0 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 monclient: found mon.noname-b 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a406b490 con 0x7f72a4133210 2024-09-16T06:54:35.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: _finish_auth 0 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: _check_auth_tickets 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 5 monclient: authenticate success, global_id 244811 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7294001610 con 0x7f72a4133210 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72a8eac700 10 monclient: dump: 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:35.081 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.078+0000 7f72aaeb0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7294001930 con 0x7f72a4133210 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 4 set_mon_vals no callback set 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 1 -- 172.21.15.76:0/3247288759 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7294001d00 con 0x7f72a4133210 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:35.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72a8eac700 10 monclient: dump: 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:35.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:35.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 10 monclient: shutdown 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3247288759 >> v1:172.21.15.76:6790/0 conn(0x7f72a4133210 legacy=0x7f72a40a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3247288759 shutdown_connections 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3247288759 >> 172.21.15.76:0/3247288759 conn(0x7f72a40a1120 msgr2=0x7f72a40a1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3247288759 shutdown_connections 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3247288759 wait complete. 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 10 monclient: build_initial_monmap 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 10 monclient: monmap: 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:35.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 Processor -- start 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 1 -- start start 2024-09-16T06:54:35.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.082+0000 7f72aaeb0700 10 monclient: init 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-a con 0x7f72a409abe0 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-c con 0x7f72a409afc0 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient(hunting): picked mon.noname-b con 0x7f72a412af20 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72a9eae700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f72a409abe0 legacy=0x7f72a41c7eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a409e7f0 con 0x7f72a409abe0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a412dc90 con 0x7f72a412af20 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a409b630 con 0x7f72a409afc0 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient(hunting): authenticate will time out at 1822.221141s 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72a96ad700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f72a409afc0 0x7f72a41cce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42990/0 (socket says 172.21.15.76:42990) 2024-09-16T06:54:35.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72a96ad700 1 -- 172.21.15.76:0/3158675011 learned_addr learned my addr 172.21.15.76:0/3158675011 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235840887 0 0) 0x7f72a409b630 con 0x7f72a409afc0 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): my global_id is 244814 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72880035d0 con 0x7f72a409afc0 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1940477536 0 0) 0x7f72880035d0 con 0x7f72a409afc0 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72a409b630 con 0x7f72a409afc0 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7294001c40 con 0x7f72a409afc0 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): dump: 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:35.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2989763302 0 0) 0x7f72a409b630 con 0x7f72a409afc0 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 >> v1:172.21.15.189:6789/0 conn(0x7f72a412af20 legacy=0x7f72a41d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 >> v1:172.21.15.76:6789/0 conn(0x7f72a409abe0 legacy=0x7f72a41c7eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 monclient: found mon.c 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4099ca0 con 0x7f72a409afc0 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 5 monclient: authenticate success, global_id 244814 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _renew_subs 2024-09-16T06:54:35.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a412c490 con 0x7f72a409afc0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _renew_subs 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a412d130 con 0x7f72a409afc0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7294004d10 con 0x7f72a409afc0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f72940050a0 con 0x7f72a409afc0 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729b7fe700 4 set_mon_vals no callback set 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 10 monclient: dump: 2024-09-16T06:54:35.095 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.086+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7294005200 con 0x7f72a409afc0 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f72aaeb0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f72aaeb0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f72aaeb0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c005100 con 0x7f72a409afc0 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f72940440e0 con 0x7f72a409afc0 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.090+0000 7f72a9eae700 10 monclient: get_auth_request con 0x7f72880360f0 auth_method 0 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.098+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7294014b40 con 0x7f72a409afc0 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.098+0000 7f729a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.098+0000 7f729a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:35.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.326+0000 7f72aaeb0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.326+0000 7f72aaeb0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.326+0000 7f72aaeb0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.326+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f728c005930 con 0x7f72a409afc0 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.330+0000 7f729a7fc700 1 -- 172.21.15.76:0/3158675011 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3248512093) 0x7f729403be90 con 0x7f72a409afc0 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.330+0000 7f729a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:35.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.330+0000 7f729a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:35.327 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:35.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 >> v1:172.21.15.189:6812/3098810674 conn(0x7f72880360f0 legacy=0x7f72880385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:35.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 10 monclient: shutdown 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 >> v1:172.21.15.76:6790/0 conn(0x7f72a409afc0 legacy=0x7f72a41cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 shutdown_connections 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 >> 172.21.15.76:0/3158675011 conn(0x7f72a40a1120 msgr2=0x7f72a4099170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 shutdown_connections 2024-09-16T06:54:35.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:35.334+0000 7f72aaeb0700 1 -- 172.21.15.76:0/3158675011 wait complete. 2024-09-16T06:54:38.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:54:38.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.462+0000 7f51693c9700 10 monclient: get_monmap_and_config 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.462+0000 7f51693c9700 10 monclient: build_initial_monmap 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.462+0000 7f51693c9700 10 monclient: monmap: 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:38.467377+0000 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:38.467377+0000 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:38.463 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:38.463 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 1 Processor -- start 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 1 -- start start 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient: init 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:38.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-a con 0x7f5164132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-b con 0x7f516406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-c con 0x7f516406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516406c280 con 0x7f5164132330 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516406bfa0 con 0x7f516406cf50 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516409ace0 con 0x7f516406d480 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51693c9700 10 monclient(hunting): authenticate will time out at 1825.601345s 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f5163fff700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f5164132330 legacy=0x7f5164132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51637fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f516406d480 0x7f516413a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42994/0 (socket says 172.21.15.76:42994) 2024-09-16T06:54:38.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.466+0000 7f51637fe700 1 -- 172.21.15.76:0/3534149549 learned_addr learned my addr 172.21.15.76:0/3534149549 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:38.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:38.670+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5164132330 legacy=0x7f5164132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:39.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:39.074+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5164132330 legacy=0x7f5164132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:39.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:39.878+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5164132330 legacy=0x7f5164132710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:41.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): tick 2024-09-16T06:54:41.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:54:41.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6790/0 conn(0x7f516406d480 legacy=0x7f516413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.189:6789/0 conn(0x7f516406cf50 legacy=0x7f516413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5164132330 legacy=0x7f5164132710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f5144003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f516406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f516406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516406c280 con 0x7f5144003740 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f5144005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514400f360 con 0x7f516406cf50 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5144010570 con 0x7f516406d480 2024-09-16T06:54:41.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.466+0000 7f5161ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:41.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:41.670+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f5144005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:42.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:42.074+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f5144005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:42.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:42.878+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f5144005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): tick 2024-09-16T06:54:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 monclient(hunting): continuing hunt 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6790/0 conn(0x7f516406d480 legacy=0x7f5144009380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.189:6789/0 conn(0x7f516406cf50 legacy=0x7f514400cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f5144005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f5144003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f514400fd30 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f516406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:44.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5144011780 con 0x7f5144003740 2024-09-16T06:54:44.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516406c280 con 0x7f514400fd30 2024-09-16T06:54:44.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5144010330 con 0x7f516406d480 2024-09-16T06:54:44.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5161ffb700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:44.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.466+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f514400f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:44.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:44.670+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f514400f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:45.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.074+0000 7f5163fff700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f514400f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:45.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.854+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116417846 0 0) 0x7f516406c280 con 0x7f514400fd30 2024-09-16T06:54:45.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:45.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:45.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:45.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient(hunting): my global_id is 284347 2024-09-16T06:54:45.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51500035d0 con 0x7f514400fd30 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777381043 0 0) 0x7f51500035d0 con 0x7f514400fd30 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6790/0 conn(0x7f516406d480 legacy=0x7f5144009380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.76:6789/0 conn(0x7f5144003740 legacy=0x7f514400f620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 monclient: found mon.noname-c 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:54:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514400cbe0 con 0x7f514400fd30 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f515c001910 con 0x7f514400fd30 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 5 monclient: authenticate success, global_id 284347 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:45.857 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f515c001b30 con 0x7f514400fd30 2024-09-16T06:54:45.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:45.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 1 -- 172.21.15.76:0/3534149549 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f515c001f40 con 0x7f514400fd30 2024-09-16T06:54:45.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:45.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 4 set_mon_vals no callback set 2024-09-16T06:54:45.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:45.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:45.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T06:54:45.864 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:45.864 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:45.864 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:45.865 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:45.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 10 monclient: shutdown 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 1 -- 172.21.15.76:0/3534149549 >> v1:172.21.15.189:6789/0 conn(0x7f514400fd30 legacy=0x7f5144010700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 1 -- 172.21.15.76:0/3534149549 shutdown_connections 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 1 -- 172.21.15.76:0/3534149549 >> 172.21.15.76:0/3534149549 conn(0x7f51640a41c0 msgr2=0x7f51640a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 1 -- 172.21.15.76:0/3534149549 shutdown_connections 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.858+0000 7f51693c9700 1 -- 172.21.15.76:0/3534149549 wait complete. 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient: build_initial_monmap 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient: monmap: 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:45.870 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 1 Processor -- start 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 1 -- start start 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient: init 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-b con 0x7f5144003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-c con 0x7f514400fd30 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:45.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient(hunting): picked mon.noname-a con 0x7f516406d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:45.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51641246e0 con 0x7f516406d480 2024-09-16T06:54:45.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640a6e90 con 0x7f5144003740 2024-09-16T06:54:45.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640a7350 con 0x7f514400fd30 2024-09-16T06:54:45.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:45.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51693c9700 10 monclient(hunting): authenticate will time out at 1832.999418s 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f5168bc8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f516406d480 legacy=0x7f51641d2780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51637fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f514400fd30 0x7f51641cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53736/0 (socket says 172.21.15.76:53736) 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.862+0000 7f51637fe700 1 -- 172.21.15.76:0/3463319457 learned_addr learned my addr 172.21.15.76:0/3463319457 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219637316 0 0) 0x7f51640a7350 con 0x7f514400fd30 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 10 monclient(hunting): my global_id is 244817 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.866+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514400d040 con 0x7f514400fd30 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190104581 0 0) 0x7f514400d040 con 0x7f514400fd30 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51640a7350 con 0x7f514400fd30 2024-09-16T06:54:45.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f515c001f40 con 0x7f514400fd30 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): dump: 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:45.874 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836378310 0 0) 0x7f51640a6e90 con 0x7f5144003740 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient(hunting): my global_id is 284350 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514400d040 con 0x7f5144003740 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1676297019 0 0) 0x7f51640a7350 con 0x7f514400fd30 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 >> v1:172.21.15.189:6789/0 conn(0x7f5144003740 legacy=0x7f51641c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:45.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 >> v1:172.21.15.76:6789/0 conn(0x7f516406d480 legacy=0x7f51641d2780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 monclient: found mon.c 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516409e8b0 con 0x7f514400fd30 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: _finish_auth 0 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 5 monclient: authenticate success, global_id 244817 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _renew_subs 2024-09-16T06:54:45.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5164124400 con 0x7f514400fd30 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _renew_subs 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5164123b10 con 0x7f514400fd30 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f515c004fb0 con 0x7f514400fd30 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f515c0052c0 con 0x7f514400fd30 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 10 monclient: dump: 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:45.877 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51627fc700 4 set_mon_vals no callback set 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f515c005420 con 0x7f514400fd30 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5148005100 con 0x7f514400fd30 2024-09-16T06:54:45.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f515c044300 con 0x7f514400fd30 2024-09-16T06:54:45.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.870+0000 7f5163fff700 10 monclient: get_auth_request con 0x7f51440373a0 auth_method 0 2024-09-16T06:54:45.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.882+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f515c014d00 con 0x7f514400fd30 2024-09-16T06:54:45.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.882+0000 7f5160ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:45.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:45.882+0000 7f5160ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:46.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f51693c9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:54:46.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f51693c9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:46.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f51693c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5148005930 con 0x7f514400fd30 2024-09-16T06:54:46.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f5160ff9700 1 -- 172.21.15.76:0/3463319457 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3350904328) 0x7f515c013170 con 0x7f514400fd30 2024-09-16T06:54:46.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f5160ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:54:46.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.118+0000 7f5160ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:46.117 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":148,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 >> v1:172.21.15.189:6812/3098810674 conn(0x7f51440373a0 legacy=0x7f5144039860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 10 monclient: shutdown 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 >> v1:172.21.15.76:6790/0 conn(0x7f514400fd30 legacy=0x7f51641cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 shutdown_connections 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 >> 172.21.15.76:0/3463319457 conn(0x7f51640a41c0 msgr2=0x7f51640a14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 shutdown_connections 2024-09-16T06:54:46.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.122+0000 7f51693c9700 1 -- 172.21.15.76:0/3463319457 wait complete. 2024-09-16T06:54:46.131 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:54:46.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:54:46.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:54:46.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: get_monmap_and_config 2024-09-16T06:54:46.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: build_initial_monmap 2024-09-16T06:54:46.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: monmap: 2024-09-16T06:54:46.244 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:46.244 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:46.249956+0000 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:46.249956+0000 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:46.245 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 1 Processor -- start 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 1 -- start start 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: init 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-a con 0x7faa80097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-c con 0x7faa80099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-b con 0x7faa80139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa80098990 con 0x7faa80097b60 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa80097f40 con 0x7faa80099090 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8009d610 con 0x7faa80139500 2024-09-16T06:54:46.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa86df7700 10 monclient(hunting): authenticate will time out at 1833.383161s 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa85df5700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7faa80097b60 legacy=0x7faa8009ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa865f6700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faa80139500 0x7faa8013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53738/0 (socket says 172.21.15.76:53738) 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa865f6700 1 -- 172.21.15.76:0/3126974917 learned_addr learned my addr 172.21.15.76:0/3126974917 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191527286 0 0) 0x7faa8009d610 con 0x7faa80139500 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 10 monclient(hunting): my global_id is 244820 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.246+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa680035d0 con 0x7faa80139500 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383851594 0 0) 0x7faa80097f40 con 0x7faa80099090 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient(hunting): my global_id is 284353 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8009d610 con 0x7faa80099090 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404144048 0 0) 0x7faa680035d0 con 0x7faa80139500 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 >> v1:172.21.15.189:6789/0 conn(0x7faa80099090 legacy=0x7faa8009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 >> v1:172.21.15.76:6789/0 conn(0x7faa80097b60 legacy=0x7faa8009ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: _finish_hunting 0 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 monclient: found mon.noname-b 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa800986b0 con 0x7faa80139500 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: _finish_auth 0 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: _check_auth_tickets 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:46.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa70001610 con 0x7faa80139500 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 5 monclient: authenticate success, global_id 244820 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: dump: 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.251 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa700018f0 con 0x7faa80139500 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 1 -- 172.21.15.76:0/3126974917 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa70001c80 con 0x7faa80139500 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 4 set_mon_vals no callback set 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa84df3700 10 monclient: dump: 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: shutdown 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- 172.21.15.76:0/3126974917 >> v1:172.21.15.76:6790/0 conn(0x7faa80139500 legacy=0x7faa8013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- 172.21.15.76:0/3126974917 shutdown_connections 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- 172.21.15.76:0/3126974917 >> 172.21.15.76:0/3126974917 conn(0x7faa800a0030 msgr2=0x7faa800a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- 172.21.15.76:0/3126974917 shutdown_connections 2024-09-16T06:54:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- 172.21.15.76:0/3126974917 wait complete. 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: build_initial_monmap 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: monmap: 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:46.259 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 Processor -- start 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- start start 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: init 2024-09-16T06:54:46.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-c con 0x7faa80097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-b con 0x7faa80099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient(hunting): picked mon.noname-a con 0x7faa80139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8008c580 con 0x7faa80139500 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa801cace0 con 0x7faa80099090 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa801392d0 con 0x7faa80097b60 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa86df7700 10 monclient(hunting): authenticate will time out at 1833.387904s 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.250+0000 7faa865f6700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7faa80139500 legacy=0x7faa801d2120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa85df5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faa80097b60 0x7faa801c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53748/0 (socket says 172.21.15.76:53748) 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa85df5700 1 -- 172.21.15.76:0/786698492 learned_addr learned my addr 172.21.15.76:0/786698492 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238342249 0 0) 0x7faa801392d0 con 0x7faa80097b60 2024-09-16T06:54:46.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): my global_id is 244823 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c0035d0 con 0x7faa80097b60 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088424329 0 0) 0x7faa6c0035d0 con 0x7faa80097b60 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa801392d0 con 0x7faa80097b60 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa78004200 con 0x7faa80097b60 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): dump: 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474915207 0 0) 0x7faa801cace0 con 0x7faa80099090 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): my global_id is 284356 2024-09-16T06:54:46.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c0035d0 con 0x7faa80099090 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727568468 0 0) 0x7faa801392d0 con 0x7faa80097b60 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.189:6789/0 conn(0x7faa80099090 legacy=0x7faa801ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6789/0 conn(0x7faa80139500 legacy=0x7faa801d2120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: _finish_hunting 0 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 monclient: found mon.c 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa80096f10 con 0x7faa80097b60 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: _finish_auth 0 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: _check_auth_tickets 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 5 monclient: authenticate success, global_id 244823 2024-09-16T06:54:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 10 monclient: _renew_subs 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa800a1250 con 0x7faa80097b60 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 10 monclient: _renew_subs 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8009a540 con 0x7faa80097b60 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa780044b0 con 0x7faa80097b60 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa78004840 con 0x7faa80097b60 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa77fff700 4 set_mon_vals no callback set 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: dump: 2024-09-16T06:54:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa60006590 con 0x7faa60002830 2024-09-16T06:54:46.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faa780049a0 con 0x7faa80097b60 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7faa78043f20 con 0x7faa80097b60 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064177262 0 0) 0x7faa60006590 con 0x7faa60002830 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient(hunting): my global_id is 244826 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa801392d0 con 0x7faa60002830 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa865f6700 10 monclient: get_auth_request con 0x7faa6c036220 auth_method 0 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4030990679 0 0) 0x7faa801392d0 con 0x7faa60002830 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa60006590 con 0x7faa60002830 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa7c001ac0 con 0x7faa60002830 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 10 monclient: dump: 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3344488603 0 0) 0x7faa60006590 con 0x7faa60002830 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.254+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa6000a190 con 0x7faa60002830 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.258+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7faa7c001eb0 con 0x7faa60002830 2024-09-16T06:54:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.258+0000 7faa767fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:46.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.258+0000 7faa767fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:46.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.258+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6790/0 conn(0x7faa60002830 legacy=0x7faa60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.258+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6790/0 conn(0x7faa60002830 legacy=0x7faa60004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa86df7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa60009e50 con 0x7faa60009170 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578846486 0 0) 0x7faa60009e50 con 0x7faa60009170 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient(hunting): my global_id is 244829 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60006590 con 0x7faa60009170 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4213198552 0 0) 0x7faa60006590 con 0x7faa60009170 2024-09-16T06:54:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa60009e50 con 0x7faa60009170 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa7c003c40 con 0x7faa60009170 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 10 monclient: dump: 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.273 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3891652662 0 0) 0x7faa60009e50 con 0x7faa60009170 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.266+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa6000c340 con 0x7faa60009170 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 1520495695) 0x7faa7c003eb0 con 0x7faa60009170 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa767fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa767fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6790/0 conn(0x7faa60009170 legacy=0x7faa60009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa767fc700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6790/0 conn(0x7faa60009170 legacy=0x7faa60009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:46.274 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 148, 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.275 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.276 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:46.277 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:46.278 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.279 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.281 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:46.282 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.283 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:54:46.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.285 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:46.286 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:46.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.189:6812/3098810674 conn(0x7faa6c036220 legacy=0x7faa6c0386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 10 monclient: shutdown 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 >> v1:172.21.15.76:6790/0 conn(0x7faa80097b60 legacy=0x7faa801c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa865f6700 1 -- 172.21.15.76:0/786698492 reap_dead start 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa865f6700 1 -- 172.21.15.76:0/786698492 reap_dead start 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 shutdown_connections 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 >> 172.21.15.76:0/786698492 conn(0x7faa800a0030 msgr2=0x7faa800702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 shutdown_connections 2024-09-16T06:54:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.270+0000 7faa86df7700 1 -- 172.21.15.76:0/786698492 wait complete. 2024-09-16T06:54:46.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:54:46.289 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:54:46.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.402+0000 7f3ce15e9700 10 monclient: get_monmap_and_config 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.402+0000 7f3ce15e9700 10 monclient: build_initial_monmap 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.402+0000 7f3ce15e9700 10 monclient: monmap: 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:54:46.407766+0000 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:54:46.407766+0000 2024-09-16T06:54:46.402 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:46.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:46.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:54:46.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:54:46.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 Processor -- start 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- start start 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient: init 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-c con 0x7f3cdc097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-b con 0x7f3cdc099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-a con 0x7f3cdc139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:46.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc098990 con 0x7f3cdc139500 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc097f40 con 0x7f3cdc097b60 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc09d610 con 0x7f3cdc099090 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient(hunting): authenticate will time out at 1833.541612s 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce0de8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3cdc139500 legacy=0x7f3cdc13b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3cdc099090 0x7f3cdc09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53766/0 (socket says 172.21.15.76:53766) 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdb7fe700 1 -- 172.21.15.76:0/4107185866 learned_addr learned my addr 172.21.15.76:0/4107185866 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622936136 0 0) 0x7f3cdc09d610 con 0x7f3cdc099090 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): my global_id is 244832 2024-09-16T06:54:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc80035d0 con 0x7f3cdc099090 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301311270 0 0) 0x7f3cdc097f40 con 0x7f3cdc097b60 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient(hunting): my global_id is 284359 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc09d610 con 0x7f3cdc097b60 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298886208 0 0) 0x7f3cc80035d0 con 0x7f3cdc099090 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 >> v1:172.21.15.189:6789/0 conn(0x7f3cdc097b60 legacy=0x7f3cdc09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 >> v1:172.21.15.76:6789/0 conn(0x7f3cdc139500 legacy=0x7f3cdc13b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: _finish_hunting 0 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 monclient: found mon.noname-b 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:46.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc0986b0 con 0x7f3cdc099090 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: _finish_auth 0 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: _check_auth_tickets 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 5 monclient: authenticate success, global_id 244832 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3cc4001610 con 0x7f3cdc099090 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: dump: 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.409 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cc40018f0 con 0x7f3cdc099090 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient: get_monmap_and_config success 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 1 -- 172.21.15.76:0/4107185866 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3cc4001c80 con 0x7f3cdc099090 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 4 set_mon_vals no callback set 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3cdaffd700 10 monclient: dump: 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 10 monclient: shutdown 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- 172.21.15.76:0/4107185866 >> v1:172.21.15.76:6790/0 conn(0x7f3cdc099090 legacy=0x7f3cdc09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- 172.21.15.76:0/4107185866 shutdown_connections 2024-09-16T06:54:46.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.406+0000 7f3ce15e9700 1 -- 172.21.15.76:0/4107185866 >> 172.21.15.76:0/4107185866 conn(0x7f3cdc0a0030 msgr2=0x7f3cdc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:46.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- 172.21.15.76:0/4107185866 shutdown_connections 2024-09-16T06:54:46.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- 172.21.15.76:0/4107185866 wait complete. 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient: build_initial_monmap 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient: monmap: 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:54:46.417 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 Processor -- start 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- start start 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient: init 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient: _reopen_session rank -1 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-c con 0x7f3cdc097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-b con 0x7f3cdc099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient(hunting): picked mon.noname-a con 0x7f3cdc139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:54:46.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc071140 con 0x7f3cdc139500 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc09a540 con 0x7f3cdc099090 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cdc139260 con 0x7f3cdc097b60 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient(hunting): _renew_subs 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce0de8700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f3cdc139500 legacy=0x7f3cdc1cfef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3ce15e9700 10 monclient(hunting): authenticate will time out at 1833.546965s 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cdbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3cdc097b60 0x7f3cdc136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53780/0 (socket says 172.21.15.76:53780) 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cdbfff700 1 -- 172.21.15.76:0/605929748 learned_addr learned my addr 172.21.15.76:0/605929748 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551034125 0 0) 0x7f3cdc139260 con 0x7f3cdc097b60 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): my global_id is 244835 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc0035d0 con 0x7f3cdc097b60 2024-09-16T06:54:46.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223407041 0 0) 0x7f3cdc09a540 con 0x7f3cdc099090 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): my global_id is 284362 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc139260 con 0x7f3cdc099090 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788487793 0 0) 0x7f3cbc0035d0 con 0x7f3cdc097b60 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cdc09a540 con 0x7f3cdc097b60 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3ccc001910 con 0x7f3cdc097b60 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): dump: 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.420 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.410+0000 7f3cd8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 206545092 0 0) 0x7f3cdc09a540 con 0x7f3cdc097b60 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6789/0 conn(0x7f3cdc099090 legacy=0x7f3cdc137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.76:6789/0 conn(0x7f3cdc139500 legacy=0x7f3cdc1cfef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:54:46.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 monclient: found mon.c 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc06cb40 con 0x7f3cdc097b60 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 5 monclient: authenticate success, global_id 244835 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 10 monclient: _renew_subs 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc06be90 con 0x7f3cdc097b60 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 10 monclient: _renew_subs 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc1388f0 con 0x7f3cdc097b60 2024-09-16T06:54:46.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ccc001ca0 con 0x7f3cdc097b60 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3ccc003b10 con 0x7f3cdc097b60 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cda7fc700 4 set_mon_vals no callback set 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: dump: 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3cc0006590 con 0x7f3cc0002830 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3ccc003cf0 con 0x7f3cdc097b60 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3ccc043e30 con 0x7f3cdc097b60 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036282035 0 0) 0x7f3cc0006590 con 0x7f3cc0002830 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient(hunting): my global_id is 284365 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc09a540 con 0x7f3cc0002830 2024-09-16T06:54:46.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce0de8700 10 monclient: get_auth_request con 0x7f3cbc0361b0 auth_method 0 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce0de8700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3cbc0361b0 legacy=0x7f3cbc038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce0de8700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3cbc0361b0 legacy=0x7f3cbc038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce0de8700 1 --1- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3cbc0361b0 0x7f3cbc038670 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3ce0de8700 1 --1- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3cbc0361b0 0x7f3cbc038670 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3cbc0361b0 legacy=0x7f3cbc038670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1139183676 0 0) 0x7f3cdc09a540 con 0x7f3cc0002830 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cc0006590 con 0x7f3cc0002830 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3cc4001d00 con 0x7f3cc0002830 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 10 monclient: dump: 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.425 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3396290225 0 0) 0x7f3cc0006590 con 0x7f3cc0002830 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cc000a190 con 0x7f3cc0002830 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.414+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f3cc4001620 con 0x7f3cc0002830 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.418+0000 7f3cd8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.418+0000 7f3cd8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.418+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6789/0 conn(0x7f3cc0002830 legacy=0x7f3cc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.418+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6789/0 conn(0x7f3cc0002830 legacy=0x7f3cc0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:46.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:54:46.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3cc0009e50 con 0x7f3cc0009170 2024-09-16T06:54:46.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993984815 0 0) 0x7f3cc0009e50 con 0x7f3cc0009170 2024-09-16T06:54:46.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:54:46.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:54:46.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:54:46.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient(hunting): my global_id is 284368 2024-09-16T06:54:46.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc0006590 con 0x7f3cc0009170 2024-09-16T06:54:46.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3217592274 0 0) 0x7f3cc0006590 con 0x7f3cc0009170 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cc0009e50 con 0x7f3cc0009170 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3cc4004dc0 con 0x7f3cc0009170 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient: dump: 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:54:46.429 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3587669349 0 0) 0x7f3cc0009e50 con 0x7f3cc0009170 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3cc000c340 con 0x7f3cc0009170 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3869 (unknown 0 0 529311711) 0x7f3cc4005060 con 0x7f3cc0009170 2024-09-16T06:54:46.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6789/0 conn(0x7f3cc0009170 legacy=0x7f3cc0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3cd8ff9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.189:6789/0 conn(0x7f3cc0009170 legacy=0x7f3cc0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce0de8700 1 -- 172.21.15.76:0/605929748 reap_dead start 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 148, 2024-09-16T06:54:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:54:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:54:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:54:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:54:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:54:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:54:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:54:46.442 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 10 monclient: shutdown 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 >> v1:172.21.15.76:6790/0 conn(0x7f3cdc097b60 legacy=0x7f3cdc136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 shutdown_connections 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 >> 172.21.15.76:0/605929748 conn(0x7f3cdc0a0030 msgr2=0x7f3cdc0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 shutdown_connections 2024-09-16T06:54:46.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:54:46.430+0000 7f3ce15e9700 1 -- 172.21.15.76:0/605929748 wait complete. 2024-09-16T06:54:46.444 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:54:46.444 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:55:06.445 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:55:06.445 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:55:06.446 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:55:06.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:55:06.449 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:55:06.450 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:55:06.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 10 monclient: get_monmap_and_config 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 10 monclient: build_initial_monmap 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 10 monclient: monmap: 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:06.566866+0000 2024-09-16T06:55:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:06.566866+0000 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 1 Processor -- start 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 1 -- start start 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.562+0000 7f2620c75700 10 monclient: init 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-a con 0x7f261c09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-b con 0x7f261c09d520 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-c con 0x7f261c09fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c09b520 con 0x7f261c09f230 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c071f30 con 0x7f261c09fd20 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c09b0e0 con 0x7f261c09d520 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient(hunting): authenticate will time out at 1853.700291s 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 0x7f261c09f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38920/0 (socket says 172.21.15.76:38920) 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 -- 172.21.15.76:0/1225425891 learned_addr learned my addr 172.21.15.76:0/1225425891 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 -- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 legacy=0x7f261c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 -- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 legacy=0x7f261c09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 --1- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 0x7f261c09f610 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261a59c700 1 --1- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 0x7f261c09f610 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730544616 0 0) 0x7f261c09b0e0 con 0x7f261c09d520 2024-09-16T06:55:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): my global_id is 244838 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26080035d0 con 0x7f261c09d520 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440630148 0 0) 0x7f261c071f30 con 0x7f261c09fd20 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient(hunting): my global_id is 284371 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c09b0e0 con 0x7f261c09fd20 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2270135596 0 0) 0x7f26080035d0 con 0x7f261c09d520 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 >> v1:172.21.15.189:6789/0 conn(0x7f261c09fd20 legacy=0x7f261c13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 legacy=0x7f261c09f610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: _finish_hunting 0 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 monclient: found mon.noname-b 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c09c7d0 con 0x7f261c09d520 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: _finish_auth 0 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: _check_auth_tickets 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f260c001610 con 0x7f261c09d520 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 5 monclient: authenticate success, global_id 244838 2024-09-16T06:55:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: dump: 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f260c001880 con 0x7f261c09d520 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 1 -- 172.21.15.76:0/1225425891 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f260c001c50 con 0x7f261c09d520 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 4 set_mon_vals no callback set 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f261959a700 10 monclient: dump: 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: shutdown 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- 172.21.15.76:0/1225425891 >> v1:172.21.15.76:6790/0 conn(0x7f261c09d520 legacy=0x7f261c137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- 172.21.15.76:0/1225425891 shutdown_connections 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- 172.21.15.76:0/1225425891 >> 172.21.15.76:0/1225425891 conn(0x7f261c0a3e50 msgr2=0x7f261c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- 172.21.15.76:0/1225425891 shutdown_connections 2024-09-16T06:55:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- 172.21.15.76:0/1225425891 wait complete. 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: build_initial_monmap 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: monmap: 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 Processor -- start 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 1 -- start start 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.566+0000 7f2620c75700 10 monclient: init 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-a con 0x7f261c09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-b con 0x7f261c09fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient(hunting): picked mon.noname-c con 0x7f261c130b00 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c072450 con 0x7f261c09f230 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c11fdd0 con 0x7f261c09fd20 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c130fe0 con 0x7f261c130b00 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient(hunting): authenticate will time out at 1853.704190s 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f261a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 0x7f261c1c5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38926/0 (socket says 172.21.15.76:38926) 2024-09-16T06:55:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f261a59c700 1 -- 172.21.15.76:0/3879452554 learned_addr learned my addr 172.21.15.76:0/3879452554 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f261ad9d700 1 --1- 172.21.15.76:0/3879452554 >> v1:172.21.15.76:6790/0 conn(0x7f261c130b00 0x7f261c1cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36462/0 (socket says 172.21.15.76:36462) 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399809210 0 0) 0x7f261c130fe0 con 0x7f261c130b00 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): my global_id is 244841 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc0035d0 con 0x7f261c130b00 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094399239 0 0) 0x7f261c11fdd0 con 0x7f261c09fd20 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): my global_id is 284374 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c130fe0 con 0x7f261c09fd20 2024-09-16T06:55:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728428704 0 0) 0x7f25fc0035d0 con 0x7f261c130b00 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f261c11fdd0 con 0x7f261c130b00 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2604001830 con 0x7f261c130b00 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): dump: 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.584 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290673143 0 0) 0x7f261c130fe0 con 0x7f261c09fd20 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25fc0035d0 con 0x7f261c09fd20 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f260c0015d0 con 0x7f261c09fd20 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient(hunting): dump: 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.585 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3952113733 0 0) 0x7f261c11fdd0 con 0x7f261c130b00 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 >> v1:172.21.15.189:6789/0 conn(0x7f261c09fd20 legacy=0x7f261c1303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 >> v1:172.21.15.76:6789/0 conn(0x7f261c09f230 legacy=0x7f261c1c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 monclient: found mon.c 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0a29b0 con 0x7f261c130b00 2024-09-16T06:55:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: _finish_auth 0 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 5 monclient: authenticate success, global_id 244841 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _renew_subs 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c11faf0 con 0x7f261c130b00 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _renew_subs 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c11da70 con 0x7f261c130b00 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26040036c0 con 0x7f261c130b00 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2604003a50 con 0x7f261c130b00 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2618d99700 4 set_mon_vals no callback set 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 10 monclient: dump: 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.588 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2600005100 con 0x7f261c130b00 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2604003bb0 con 0x7f261c130b00 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2604043d90 con 0x7f261c130b00 2024-09-16T06:55:06.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.570+0000 7f261a59c700 10 monclient: get_auth_request con 0x7f25fc036270 auth_method 0 2024-09-16T06:55:06.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.578+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f2604014750 con 0x7f261c130b00 2024-09-16T06:55:06.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.578+0000 7f2612ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:06.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.578+0000 7f2612ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:06.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2620c75700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:06.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2620c75700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:06.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2620c75700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2600005930 con 0x7f261c130b00 2024-09-16T06:55:06.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2612ffd700 1 -- 172.21.15.76:0/3879452554 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1847786642) 0x7f2604012bc0 con 0x7f261c130b00 2024-09-16T06:55:06.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2612ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:06.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.806+0000 7f2612ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:06.806 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 >> v1:172.21.15.189:6812/3098810674 conn(0x7f25fc036270 legacy=0x7f25fc038730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 10 monclient: shutdown 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 >> v1:172.21.15.76:6790/0 conn(0x7f261c130b00 legacy=0x7f261c1cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 shutdown_connections 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 >> 172.21.15.76:0/3879452554 conn(0x7f261c0a3e50 msgr2=0x7f261c09e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 shutdown_connections 2024-09-16T06:55:06.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.810+0000 7f2620c75700 1 -- 172.21.15.76:0/3879452554 wait complete. 2024-09-16T06:55:06.820 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:55:06.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:55:06.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:55:06.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 10 monclient: get_monmap_and_config 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 10 monclient: build_initial_monmap 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 10 monclient: monmap: 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:06.930378+0000 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:06.930378+0000 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:06.925 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 1 Processor -- start 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 1 -- start start 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.926+0000 7f1428844700 10 monclient: init 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient(hunting): picked mon.noname-a con 0x7f14240a04e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient(hunting): picked mon.noname-c con 0x7f14240a0fd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient(hunting): picked mon.noname-b con 0x7f14240a1420 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142406c400 con 0x7f14240a04e0 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142406c9a0 con 0x7f14240a0fd0 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142406c1c0 con 0x7f14240a1420 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:06.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient(hunting): authenticate will time out at 1854.064462s 2024-09-16T06:55:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1422d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f14240a04e0 0x7f14240a08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38942/0 (socket says 172.21.15.76:38942) 2024-09-16T06:55:06.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1422d9d700 1 -- 172.21.15.76:0/980221231 learned_addr learned my addr 172.21.15.76:0/980221231 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444020304 0 0) 0x7f142406c1c0 con 0x7f14240a1420 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): my global_id is 244844 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f140c0035d0 con 0x7f14240a1420 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800105031 0 0) 0x7f142406c9a0 con 0x7f14240a0fd0 2024-09-16T06:55:06.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient(hunting): my global_id is 284377 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142406c1c0 con 0x7f14240a0fd0 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577379640 0 0) 0x7f142406c1c0 con 0x7f14240a0fd0 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 >> v1:172.21.15.76:6790/0 conn(0x7f14240a1420 legacy=0x7f142413dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 >> v1:172.21.15.76:6789/0 conn(0x7f14240a04e0 legacy=0x7f14240a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 monclient: found mon.noname-c 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142409f8f0 con 0x7f14240a0fd0 2024-09-16T06:55:06.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: _finish_auth 0 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 5 monclient: authenticate success, global_id 284377 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14180015f0 con 0x7f14240a0fd0 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: dump: 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.932 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1418001910 con 0x7f14240a0fd0 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 1 -- 172.21.15.76:0/980221231 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1418001ca0 con 0x7f14240a0fd0 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 4 set_mon_vals no callback set 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:06.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1421d9b700 10 monclient: dump: 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: shutdown 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:06.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- 172.21.15.76:0/980221231 >> v1:172.21.15.189:6789/0 conn(0x7f14240a0fd0 legacy=0x7f142413a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- 172.21.15.76:0/980221231 shutdown_connections 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- 172.21.15.76:0/980221231 >> 172.21.15.76:0/980221231 conn(0x7f14240a5870 msgr2=0x7f14240a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- 172.21.15.76:0/980221231 shutdown_connections 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 1 -- 172.21.15.76:0/980221231 wait complete. 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: build_initial_monmap 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.930+0000 7f1428844700 10 monclient: monmap: 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:06.935 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:06.936 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:06.936 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 1 Processor -- start 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 1 -- start start 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient: init 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient(hunting): picked mon.noname-a con 0x7f14240a04e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient(hunting): picked mon.noname-b con 0x7f14240a0fd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient(hunting): picked mon.noname-c con 0x7f14240a1420 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1424098f00 con 0x7f14240a04e0 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142409d430 con 0x7f14240a0fd0 2024-09-16T06:55:06.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142409c640 con 0x7f14240a1420 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1428844700 10 monclient(hunting): authenticate will time out at 1854.069532s 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1422d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f14240a04e0 0x7f1424139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38950/0 (socket says 172.21.15.76:38950) 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f1422d9d700 1 -- 172.21.15.76:0/487674851 learned_addr learned my addr 172.21.15.76:0/487674851 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174558216 0 0) 0x7f142409c640 con 0x7f14240a1420 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): my global_id is 244847 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14100035d0 con 0x7f14240a1420 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760215672 0 0) 0x7f142409d430 con 0x7f14240a0fd0 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): my global_id is 284380 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142409c640 con 0x7f14240a0fd0 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373393516 0 0) 0x7f14100035d0 con 0x7f14240a1420 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f142409d430 con 0x7f14240a1420 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f141c0018d0 con 0x7f14240a1420 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.942 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129399583 0 0) 0x7f1424098f00 con 0x7f14240a04e0 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient(hunting): my global_id is 314097 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14100035d0 con 0x7f14240a04e0 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273021595 0 0) 0x7f142409d430 con 0x7f14240a1420 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.189:6789/0 conn(0x7f14240a0fd0 legacy=0x7f1424139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6789/0 conn(0x7f14240a04e0 legacy=0x7f1424139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 monclient: found mon.c 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142413a580 con 0x7f14240a1420 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.934+0000 7f140b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 5 monclient: authenticate success, global_id 244847 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 10 monclient: _renew_subs 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142409cee0 con 0x7f14240a1420 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 10 monclient: _renew_subs 2024-09-16T06:55:06.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f142413cd60 con 0x7f14240a1420 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f142413f3e0 con 0x7f142406f820 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f141c0036c0 con 0x7f14240a1420 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f141c003a50 con 0x7f14240a1420 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f142159a700 4 set_mon_vals no callback set 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: dump: 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.945 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f141c003bb0 con 0x7f14240a1420 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f141c042a90 con 0x7f14240a1420 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945368058 0 0) 0x7f142413f3e0 con 0x7f142406f820 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient(hunting): my global_id is 314100 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142409d430 con 0x7f142406f820 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f142259c700 10 monclient: get_auth_request con 0x7f1410036390 auth_method 0 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1494962307 0 0) 0x7f142409d430 con 0x7f142406f820 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f142413f3e0 con 0x7f142406f820 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1414001aa0 con 0x7f142406f820 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: dump: 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3224974779 0 0) 0x7f142413f3e0 con 0x7f142406f820 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1424098d80 con 0x7f142406f820 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1414001ed0 con 0x7f142406f820 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6789/0 conn(0x7f142406f820 legacy=0x7f142406fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.938+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6789/0 conn(0x7f142406f820 legacy=0x7f142406fc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.946+0000 7f1428844700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.946+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f142413c090 con 0x7f1424070310 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632583684 0 0) 0x7f142413c090 con 0x7f1424070310 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient(hunting): my global_id is 314103 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142413f3e0 con 0x7f1424070310 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 311124672 0 0) 0x7f142413f3e0 con 0x7f1424070310 2024-09-16T06:55:06.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f142413c090 con 0x7f1424070310 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1414005390 con 0x7f1424070310 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient: dump: 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:06.951 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2149121459 0 0) 0x7f142413c090 con 0x7f1424070310 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f142413b2c0 con 0x7f1424070310 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 1987356778) 0x7f1414006f30 con 0x7f1424070310 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6789/0 conn(0x7f1424070310 legacy=0x7f142402c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f140b7fe700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6789/0 conn(0x7f1424070310 legacy=0x7f142402c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:06.952 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:06.964 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1410036390 legacy=0x7f1410038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 10 monclient: shutdown 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 >> v1:172.21.15.76:6790/0 conn(0x7f14240a1420 legacy=0x7f14241d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f142359e700 1 -- 172.21.15.76:0/487674851 reap_dead start 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f142359e700 1 -- 172.21.15.76:0/487674851 reap_dead start 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 shutdown_connections 2024-09-16T06:55:06.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 >> 172.21.15.76:0/487674851 conn(0x7f14240a5870 msgr2=0x7f14240a41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:06.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 shutdown_connections 2024-09-16T06:55:06.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:06.950+0000 7f1428844700 1 -- 172.21.15.76:0/487674851 wait complete. 2024-09-16T06:55:06.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:06.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:55:06.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: get_monmap_and_config 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: monmap: 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:07.088603+0000 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:07.088603+0000 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.084 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:07.084 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:07.084 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 1 Processor -- start 2024-09-16T06:55:07.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 1 -- start start 2024-09-16T06:55:07.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: init 2024-09-16T06:55:07.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f68d809ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f68d8098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f68d8099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809d560 con 0x7f68d8099030 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809f140 con 0x7f68d8098be0 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d809d790 con 0x7f68d809ed60 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient(hunting): authenticate will time out at 1854.222122s 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f68d8099030 0x7f68d813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38990/0 (socket says 172.21.15.76:38990) 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d77fe700 1 -- 172.21.15.76:0/4127067248 learned_addr learned my addr 172.21.15.76:0/4127067248 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d6ffd700 1 --1- 172.21.15.76:0/4127067248 >> v1:172.21.15.76:6790/0 conn(0x7f68d809ed60 0x7f68d80984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36500/0 (socket says 172.21.15.76:36500) 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894267853 0 0) 0x7f68d809d560 con 0x7f68d8099030 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): my global_id is 314106 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c40035d0 con 0x7f68d8099030 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726860577 0 0) 0x7f68d809d790 con 0x7f68d809ed60 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): my global_id is 244850 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d809d560 con 0x7f68d809ed60 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897407856 0 0) 0x7f68d809f140 con 0x7f68d8098be0 2024-09-16T06:55:07.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient(hunting): my global_id is 284383 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d809d790 con 0x7f68d8098be0 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437783714 0 0) 0x7f68c40035d0 con 0x7f68d8099030 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 >> v1:172.21.15.76:6790/0 conn(0x7f68d809ed60 legacy=0x7f68d80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 >> v1:172.21.15.189:6789/0 conn(0x7f68d8098be0 legacy=0x7f68d81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 monclient: found mon.noname-a 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d809f620 con 0x7f68d8099030 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68d0001990 con 0x7f68d8099030 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: dump: 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.091 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 5 monclient: authenticate success, global_id 314106 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68d0001c70 con 0x7f68d8099030 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 1 -- 172.21.15.76:0/4127067248 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68d0002000 con 0x7f68d8099030 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d7fff700 4 set_mon_vals no callback set 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.086+0000 7f68d67fc700 10 monclient: dump: 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: shutdown 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/4127067248 >> v1:172.21.15.76:6789/0 conn(0x7f68d8099030 legacy=0x7f68d813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/4127067248 shutdown_connections 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/4127067248 >> 172.21.15.76:0/4127067248 conn(0x7f68d80a0030 msgr2=0x7f68d80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/4127067248 shutdown_connections 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/4127067248 wait complete. 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: monmap: 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.094 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 Processor -- start 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- start start 2024-09-16T06:55:07.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: init 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f68d8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f68d8099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f68d809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d81c65b0 con 0x7f68d8099030 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d808f030 con 0x7f68d809ed60 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d80a1570 con 0x7f68d8098be0 2024-09-16T06:55:07.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient(hunting): authenticate will time out at 1854.226274s 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f68d8099030 0x7f68d81ca990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39000/0 (socket says 172.21.15.76:39000) 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cffff700 1 -- 172.21.15.76:0/813452066 learned_addr learned my addr 172.21.15.76:0/813452066 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150109731 0 0) 0x7f68d81c65b0 con 0x7f68d8099030 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): my global_id is 314109 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b80035d0 con 0x7f68d8099030 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441520198 0 0) 0x7f68d80a1570 con 0x7f68d8098be0 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): my global_id is 244853 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d81c65b0 con 0x7f68d8098be0 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540878186 0 0) 0x7f68d808f030 con 0x7f68d809ed60 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): my global_id is 284386 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d80a1570 con 0x7f68d809ed60 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682523616 0 0) 0x7f68b80035d0 con 0x7f68d8099030 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68d808f030 con 0x7f68d8099030 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68c8001990 con 0x7f68d8099030 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:07.103 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047647994 0 0) 0x7f68d81c65b0 con 0x7f68d8098be0 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b80035d0 con 0x7f68d8098be0 2024-09-16T06:55:07.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68c0001830 con 0x7f68d8098be0 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3528146184 0 0) 0x7f68d808f030 con 0x7f68d8099030 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6790/0 conn(0x7f68d8098be0 legacy=0x7f68d80963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.189:6789/0 conn(0x7f68d809ed60 legacy=0x7f68d81ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 monclient: found mon.a 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d809eab0 con 0x7f68d8099030 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 5 monclient: authenticate success, global_id 314109 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _renew_subs 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:07.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d809a540 con 0x7f68d8099030 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _renew_subs 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d8090e50 con 0x7f68d8099030 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c8001d40 con 0x7f68d8099030 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68c8003b10 con 0x7f68d8099030 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68cf7fe700 10 monclient: dump: 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.107 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.090+0000 7f68d5ffb700 4 set_mon_vals no callback set 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f68c8003c70 con 0x7f68d8099030 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f68c8043ec0 con 0x7f68d8099030 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f68bc006590 con 0x7f68bc002830 2024-09-16T06:55:07.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68d6ffd700 10 monclient: get_auth_request con 0x7f68b80364e0 auth_method 0 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776139184 0 0) 0x7f68bc006590 con 0x7f68bc002830 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient(hunting): my global_id is 244856 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d808f030 con 0x7f68bc002830 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 362690455 0 0) 0x7f68d808f030 con 0x7f68bc002830 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68bc006590 con 0x7f68bc002830 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68d00017b0 con 0x7f68bc002830 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient: dump: 2024-09-16T06:55:07.109 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3581865797 0 0) 0x7f68bc006590 con 0x7f68bc002830 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68bc00a190 con 0x7f68bc002830 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f68d0001fa0 con 0x7f68bc002830 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:07.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6790/0 conn(0x7f68bc002830 legacy=0x7f68bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.094+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6790/0 conn(0x7f68bc002830 legacy=0x7f68bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f68bc009e30 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436594299 0 0) 0x7f68bc009e30 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient(hunting): my global_id is 244859 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc006590 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1908978257 0 0) 0x7f68bc006590 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68bc009e30 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f68d00054a0 con 0x7f68bc009150 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient: dump: 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3768051038 0 0) 0x7f68bc009e30 con 0x7f68bc009150 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f68bc00c340 con 0x7f68bc009150 2024-09-16T06:55:07.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 274189536) 0x7f68d0006fe0 con 0x7f68bc009150 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6790/0 conn(0x7f68bc009150 legacy=0x7f68bc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68cf7fe700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6790/0 conn(0x7f68bc009150 legacy=0x7f68bc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:55:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:55:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:07.124 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:07.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.189:6812/3098810674 conn(0x7f68b80364e0 legacy=0x7f68b80389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 10 monclient: shutdown 2024-09-16T06:55:07.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d77fe700 1 -- 172.21.15.76:0/813452066 reap_dead start 2024-09-16T06:55:07.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 >> v1:172.21.15.76:6789/0 conn(0x7f68d8099030 legacy=0x7f68d81ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 shutdown_connections 2024-09-16T06:55:07.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 >> 172.21.15.76:0/813452066 conn(0x7f68d80a0030 msgr2=0x7f68d806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 shutdown_connections 2024-09-16T06:55:07.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.102+0000 7f68d7fff700 1 -- 172.21.15.76:0/813452066 wait complete. 2024-09-16T06:55:07.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:07.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:55:07.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:55:07.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: get_monmap_and_config 2024-09-16T06:55:07.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: monmap: 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:07.248213+0000 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:07.248213+0000 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:07.243 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 Processor -- start 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- start start 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: init 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-a con 0x7f29c0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-c con 0x7f29c006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-b con 0x7f29c006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c006c280 con 0x7f29c0132330 2024-09-16T06:55:07.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c006bfa0 con 0x7f29c006d480 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c009ace0 con 0x7f29c006cf50 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient(hunting): authenticate will time out at 1854.381681s 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f29c0132330 0x7f29c0132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39012/0 (socket says 172.21.15.76:39012) 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bf59e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f29c006cf50 0x7f29c013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36516/0 (socket says 172.21.15.76:36516) 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bed9d700 1 -- 172.21.15.76:0/2792175828 learned_addr learned my addr 172.21.15.76:0/2792175828 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084800488 0 0) 0x7f29c006c280 con 0x7f29c0132330 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): my global_id is 314112 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a00035d0 con 0x7f29c0132330 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829829728 0 0) 0x7f29c009ace0 con 0x7f29c006cf50 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient(hunting): my global_id is 244862 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c006c280 con 0x7f29c006cf50 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383123659 0 0) 0x7f29a00035d0 con 0x7f29c0132330 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 >> v1:172.21.15.76:6790/0 conn(0x7f29c006cf50 legacy=0x7f29c013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 >> v1:172.21.15.189:6789/0 conn(0x7f29c006d480 legacy=0x7f29c013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 monclient: found mon.noname-a 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c009ae70 con 0x7f29c0132330 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 5 monclient: authenticate success, global_id 314112 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b0001610 con 0x7f29c0132330 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: dump: 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29b0001830 con 0x7f29c0132330 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 4 set_mon_vals no callback set 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 1 -- 172.21.15.76:0/2792175828 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b0001a20 con 0x7f29c0132330 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29bdd9b700 10 monclient: dump: 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.253 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.253 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.253 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.253 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 10 monclient: shutdown 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- 172.21.15.76:0/2792175828 >> v1:172.21.15.76:6789/0 conn(0x7f29c0132330 legacy=0x7f29c0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- 172.21.15.76:0/2792175828 shutdown_connections 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- 172.21.15.76:0/2792175828 >> 172.21.15.76:0/2792175828 conn(0x7f29c00a41c0 msgr2=0x7f29c00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- 172.21.15.76:0/2792175828 shutdown_connections 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.246+0000 7f29c4a4a700 1 -- 172.21.15.76:0/2792175828 wait complete. 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: monmap: 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:07.255 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 Processor -- start 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- start start 2024-09-16T06:55:07.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: init 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-c con 0x7f29c006cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-b con 0x7f29c006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient(hunting): picked mon.noname-a con 0x7f29c0132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c012e6f0 con 0x7f29c0132330 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c013ed20 con 0x7f29c006d480 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c0131570 con 0x7f29c006cf50 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient(hunting): authenticate will time out at 1854.385413s 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29bed9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f29c006cf50 0x7f29c013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36520/0 (socket says 172.21.15.76:36520) 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29bed9d700 1 -- 172.21.15.76:0/1208279892 learned_addr learned my addr 172.21.15.76:0/1208279892 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310209628 0 0) 0x7f29c0131570 con 0x7f29c006cf50 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): my global_id is 244865 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a40035d0 con 0x7f29c006cf50 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104610571 0 0) 0x7f29c012e6f0 con 0x7f29c0132330 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): my global_id is 314115 2024-09-16T06:55:07.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c0131570 con 0x7f29c0132330 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568328864 0 0) 0x7f29c013ed20 con 0x7f29c006d480 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): my global_id is 284392 2024-09-16T06:55:07.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c012e6f0 con 0x7f29c006d480 2024-09-16T06:55:07.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684538341 0 0) 0x7f29a40035d0 con 0x7f29c006cf50 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29c013ed20 con 0x7f29c006cf50 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b0001510 con 0x7f29c006cf50 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): dump: 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.265 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3513902962 0 0) 0x7f29c0131570 con 0x7f29c0132330 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29a40035d0 con 0x7f29c0132330 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b80019b0 con 0x7f29c0132330 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient(hunting): dump: 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.266 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3648652436 0 0) 0x7f29c013ed20 con 0x7f29c006cf50 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6789/0 conn(0x7f29c006d480 legacy=0x7f29c01ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.76:6789/0 conn(0x7f29c0132330 legacy=0x7f29c01d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 monclient: found mon.c 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0131060 con 0x7f29c006cf50 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 5 monclient: authenticate success, global_id 244865 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _renew_subs 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c0133040 con 0x7f29c006cf50 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _renew_subs 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c012ee00 con 0x7f29c006cf50 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29b0004b40 con 0x7f29c006cf50 2024-09-16T06:55:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b0004e50 con 0x7f29c006cf50 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29bd59a700 4 set_mon_vals no callback set 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.250+0000 7f29b77fe700 10 monclient: dump: 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.269 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f29b000f8c0 con 0x7f29c006cf50 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f29b0043e80 con 0x7f29c006cf50 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29c4a4a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2998006590 con 0x7f2998002830 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29be59c700 10 monclient: get_auth_request con 0x7f29a4036490 auth_method 0 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550726616 0 0) 0x7f2998006590 con 0x7f2998002830 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient(hunting): my global_id is 284395 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c013ed20 con 0x7f2998002830 2024-09-16T06:55:07.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1548781566 0 0) 0x7f29c013ed20 con 0x7f2998002830 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2998006590 con 0x7f2998002830 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b8004db0 con 0x7f2998002830 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient: dump: 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.271 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3298985168 0 0) 0x7f2998006590 con 0x7f2998002830 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f299800a190 con 0x7f2998002830 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f29b8007070 con 0x7f2998002830 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6789/0 conn(0x7f2998002830 legacy=0x7f2998004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.254+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6789/0 conn(0x7f2998002830 legacy=0x7f2998004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29c4a4a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2998009e50 con 0x7f2998009170 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444714102 0 0) 0x7f2998009e50 con 0x7f2998009170 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient(hunting): my global_id is 284398 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998006590 con 0x7f2998009170 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2286069176 0 0) 0x7f2998006590 con 0x7f2998009170 2024-09-16T06:55:07.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2998009e50 con 0x7f2998009170 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f29b80054f0 con 0x7f2998009170 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.266+0000 7f29b77fe700 10 monclient: dump: 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.275 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 93133753 0 0) 0x7f2998009e50 con 0x7f2998009170 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f299800c340 con 0x7f2998009170 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3322403981) 0x7f29b80071e0 con 0x7f2998009170 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6789/0 conn(0x7f2998009170 legacy=0x7f2998009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29b77fe700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6789/0 conn(0x7f2998009170 legacy=0x7f2998009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:07.276 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:07.277 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:07.278 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:07.279 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:07.280 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:07.281 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:07.285 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.286 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:07.287 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:07.288 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:07.289 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:07.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.189:6812/3098810674 conn(0x7f29a4036490 legacy=0x7f29a4038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 10 monclient: shutdown 2024-09-16T06:55:07.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 >> v1:172.21.15.76:6790/0 conn(0x7f29c006cf50 legacy=0x7f29c013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29bf59e700 1 -- 172.21.15.76:0/1208279892 reap_dead start 2024-09-16T06:55:07.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 shutdown_connections 2024-09-16T06:55:07.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 >> 172.21.15.76:0/1208279892 conn(0x7f29c00a41c0 msgr2=0x7f29c009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 shutdown_connections 2024-09-16T06:55:07.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.270+0000 7f29c4a4a700 1 -- 172.21.15.76:0/1208279892 wait complete. 2024-09-16T06:55:07.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:07.291 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:55:07.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: get_monmap_and_config 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: monmap: 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:07.409756+0000 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:07.409756+0000 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.405 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.405 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:07.405 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:07.405 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 1 Processor -- start 2024-09-16T06:55:07.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 1 -- start start 2024-09-16T06:55:07.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: init 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-b con 0x7f98a00973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-a con 0x7f98a0097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-c con 0x7f98a01400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a0099350 con 0x7f98a0097eb0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a009f690 con 0x7f98a01400d0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a0099bf0 con 0x7f98a00973c0 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a5cb4700 10 monclient(hunting): authenticate will time out at 1854.543031s 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f98a0097eb0 0x7f98a013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39018/0 (socket says 172.21.15.76:39018) 2024-09-16T06:55:07.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f98a4cb2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f98a00973c0 0x7f98a00977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36524/0 (socket says 172.21.15.76:36524) 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989ffff700 1 -- 172.21.15.76:0/639710667 learned_addr learned my addr 172.21.15.76:0/639710667 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689179022 0 0) 0x7f98a0099bf0 con 0x7f98a00973c0 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): my global_id is 244868 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c0035d0 con 0x7f98a00973c0 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865025808 0 0) 0x7f98a0099350 con 0x7f98a0097eb0 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): my global_id is 314118 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0099bf0 con 0x7f98a0097eb0 2024-09-16T06:55:07.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865090592 0 0) 0x7f98a009f690 con 0x7f98a01400d0 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 10 monclient(hunting): my global_id is 284401 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.406+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0099350 con 0x7f98a01400d0 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647577170 0 0) 0x7f988c0035d0 con 0x7f98a00973c0 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 >> v1:172.21.15.189:6789/0 conn(0x7f98a01400d0 legacy=0x7f98a01424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 >> v1:172.21.15.76:6789/0 conn(0x7f98a0097eb0 legacy=0x7f98a013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a0098360 con 0x7f98a00973c0 2024-09-16T06:55:07.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 5 monclient: authenticate success, global_id 244868 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9888001610 con 0x7f98a00973c0 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: dump: 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.413 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98880018f0 con 0x7f98a00973c0 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 1 -- 172.21.15.76:0/639710667 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9888001cc0 con 0x7f98a00973c0 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 4 set_mon_vals no callback set 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f989f7fe700 10 monclient: dump: 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 10 monclient: shutdown 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- 172.21.15.76:0/639710667 >> v1:172.21.15.76:6790/0 conn(0x7f98a00973c0 legacy=0x7f98a00977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- 172.21.15.76:0/639710667 shutdown_connections 2024-09-16T06:55:07.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- 172.21.15.76:0/639710667 >> 172.21.15.76:0/639710667 conn(0x7f98a00a0030 msgr2=0x7f98a00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- 172.21.15.76:0/639710667 shutdown_connections 2024-09-16T06:55:07.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- 172.21.15.76:0/639710667 wait complete. 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 10 monclient: build_initial_monmap 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 10 monclient: monmap: 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:07.421 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 Processor -- start 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 1 -- start start 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.410+0000 7f98a5cb4700 10 monclient: init 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-b con 0x7f98a00973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-a con 0x7f98a0097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:07.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient(hunting): picked mon.noname-c con 0x7f98a01400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a0072660 con 0x7f98a0097eb0 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a013fc90 con 0x7f98a00973c0 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a009eab0 con 0x7f98a01400d0 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient(hunting): authenticate will time out at 1854.549232s 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f98a0097eb0 0x7f98a013ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39022/0 (socket says 172.21.15.76:39022) 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989ffff700 1 -- 172.21.15.76:0/298336945 learned_addr learned my addr 172.21.15.76:0/298336945 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090553745 0 0) 0x7f98a009eab0 con 0x7f98a01400d0 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): my global_id is 244871 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98800035d0 con 0x7f98a01400d0 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152722016 0 0) 0x7f98a0072660 con 0x7f98a0097eb0 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): my global_id is 314121 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a009eab0 con 0x7f98a0097eb0 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155782156 0 0) 0x7f98a013fc90 con 0x7f98a00973c0 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): my global_id is 284404 2024-09-16T06:55:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0072660 con 0x7f98a00973c0 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4076086839 0 0) 0x7f98800035d0 con 0x7f98a01400d0 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98a013fc90 con 0x7f98a01400d0 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f98980018b0 con 0x7f98a01400d0 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): dump: 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.426 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814304223 0 0) 0x7f98a009eab0 con 0x7f98a0097eb0 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98800035d0 con 0x7f98a0097eb0 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9890001d50 con 0x7f98a0097eb0 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient(hunting): dump: 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.427 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2626216798 0 0) 0x7f98a013fc90 con 0x7f98a01400d0 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 >> v1:172.21.15.189:6789/0 conn(0x7f98a00973c0 legacy=0x7f98a013c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 >> v1:172.21.15.76:6789/0 conn(0x7f98a0097eb0 legacy=0x7f98a013ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 monclient: found mon.c 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:07.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a009a540 con 0x7f98a01400d0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 5 monclient: authenticate success, global_id 244871 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _renew_subs 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a006c810 con 0x7f98a01400d0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _renew_subs 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a013e070 con 0x7f98a01400d0 2024-09-16T06:55:07.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98980036c0 con 0x7f98a01400d0 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f98980039d0 con 0x7f98a01400d0 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989effd700 4 set_mon_vals no callback set 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.414+0000 7f989d7fa700 10 monclient: dump: 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:07.430 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f98a5cb4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f98a5cb4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f98a5cb4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9884005100 con 0x7f98a01400d0 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f989800e440 con 0x7f98a01400d0 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f9898042be0 con 0x7f98a01400d0 2024-09-16T06:55:07.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.418+0000 7f98a4cb2700 10 monclient: get_auth_request con 0x7f98800363a0 auth_method 0 2024-09-16T06:55:07.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.426+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f98980132f0 con 0x7f98a01400d0 2024-09-16T06:55:07.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.426+0000 7f989d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:07.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.426+0000 7f989d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:07.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f98a5cb4700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:55:07.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f98a5cb4700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:55:07.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f98a5cb4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:07.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9884005bc0 con 0x7f98a01400d0 2024-09-16T06:55:07.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f989d7fa700 1 -- 172.21.15.76:0/298336945 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f9898011990 con 0x7f98a01400d0 2024-09-16T06:55:07.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f989d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:55:07.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f989d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:07.657 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3719929415,config=1097430493,health=193804072,kv=1953775313,logm=3084915514}) 2024-09-16T06:55:07.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=2093098329,config=1097430493,health=193804072,kv=71370927,logm=984519358}) 2024-09-16T06:55:07.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3719929415,config=1097430493,health=193804072,kv=1953775313,logm=3084915514}) 2024-09-16T06:55:07.658 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.658+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=2489143139,config=1097430493,health=193804072,kv=1588170259,logm=1126260458}) 2024-09-16T06:55:07.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 >> v1:172.21.15.189:6812/3098810674 conn(0x7f98800363a0 legacy=0x7f9880038860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 10 monclient: shutdown 2024-09-16T06:55:07.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:07.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 >> v1:172.21.15.76:6790/0 conn(0x7f98a01400d0 legacy=0x7f98a01d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:07.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 shutdown_connections 2024-09-16T06:55:07.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 >> 172.21.15.76:0/298336945 conn(0x7f98a00a0030 msgr2=0x7f98a0096210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:07.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 shutdown_connections 2024-09-16T06:55:07.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f98a5cb4700 1 -- 172.21.15.76:0/298336945 wait complete. 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516858968}) 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594496289}) 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516858968}) 2024-09-16T06:55:07.662 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.662+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=790342117}) 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930118485}) 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1924837653}) 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930118485}) 2024-09-16T06:55:07.665 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.666+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=473817108}) 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=417884451}) 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851090493}) 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=417884451}) 2024-09-16T06:55:07.667 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250917667}) 2024-09-16T06:55:07.668 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451616729}) 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179351220}) 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451616729}) 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125865141}) 2024-09-16T06:55:07.669 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339976254}) 2024-09-16T06:55:07.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812661055}) 2024-09-16T06:55:07.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339976254}) 2024-09-16T06:55:07.670 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.670+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792857443}) 2024-09-16T06:55:07.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:55:07.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141114284}) 2024-09-16T06:55:07.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771913606}) 2024-09-16T06:55:07.671 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141114284}) 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062667257}) 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3088257829}) 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253618341}) 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.672 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3088257829}) 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966744359}) 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512909362}) 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3266033514}) 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.673 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512909362}) 2024-09-16T06:55:07.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.674+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975434501}) 2024-09-16T06:55:07.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.674 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376078679}) 2024-09-16T06:55:07.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229427312}) 2024-09-16T06:55:07.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376078679}) 2024-09-16T06:55:07.675 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414753228}) 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213539790}) 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522470580}) 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213539790}) 2024-09-16T06:55:07.676 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163519288}) 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470308648}) 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610358182}) 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470308648}) 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=429261283}) 2024-09-16T06:55:07.677 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758305553}) 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102311843}) 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758305553}) 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.678+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=733746195}) 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.678 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994629864}) 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24729160}) 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994629864}) 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608257642}) 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.679 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862912677}) 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544368968}) 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862912677}) 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202388263}) 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.680 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055977379}) 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287153773}) 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055977379}) 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=681544013}) 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.681 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907084470}) 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779897871}) 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907084470}) 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.682+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603309307}) 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.682 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=195316745}) 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744708515}) 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=195316745}) 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797172742}) 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450756230}) 2024-09-16T06:55:07.683 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355509439}) 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450756230}) 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1362167147}) 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773660733}) 2024-09-16T06:55:07.684 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804955514}) 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773660733}) 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3495356747}) 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280497665}) 2024-09-16T06:55:07.685 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741312793}) 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280497665}) 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.686+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005015621}) 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1912224014}) 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2814185033}) 2024-09-16T06:55:07.686 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1912224014}) 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660118631}) 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467947685}) 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888141867}) 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467947685}) 2024-09-16T06:55:07.687 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472627588}) 2024-09-16T06:55:07.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910215131}) 2024-09-16T06:55:07.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098813883}) 2024-09-16T06:55:07.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.688 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910215131}) 2024-09-16T06:55:07.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498754427}) 2024-09-16T06:55:07.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959247018}) 2024-09-16T06:55:07.689 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208056984}) 2024-09-16T06:55:07.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959247018}) 2024-09-16T06:55:07.690 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.690+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=134039281}) 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627109740}) 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3854910528}) 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627109740}) 2024-09-16T06:55:07.691 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346728740}) 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649409658}) 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838545854}) 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649409658}) 2024-09-16T06:55:07.692 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691791965}) 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=446999700}) 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104681472}) 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=446999700}) 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177545477}) 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.693 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319602791}) 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146131751}) 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319602791}) 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.694+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399104367}) 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.694 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008442726}) 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091376590}) 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008442726}) 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314068210}) 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792858543}) 2024-09-16T06:55:07.695 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739934076}) 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792858543}) 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657460252}) 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982738484}) 2024-09-16T06:55:07.696 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281723951}) 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982738484}) 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3008632640}) 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446495772}) 2024-09-16T06:55:07.697 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333637295}) 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446495772}) 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.698+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293463538}) 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808415025}) 2024-09-16T06:55:07.698 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124442913}) 2024-09-16T06:55:07.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808415025}) 2024-09-16T06:55:07.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621376042}) 2024-09-16T06:55:07.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.699 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011735919}) 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960436003}) 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011735919}) 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236548227}) 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.700 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566251789}) 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529221772}) 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566251789}) 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219957302}) 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.701 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226890704}) 2024-09-16T06:55:07.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347843915}) 2024-09-16T06:55:07.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226890704}) 2024-09-16T06:55:07.702 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.702+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777895268}) 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84519973}) 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56306945}) 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84519973}) 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1784343398}) 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.703 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425371766}) 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512118192}) 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425371766}) 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570667102}) 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.704 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020411961}) 2024-09-16T06:55:07.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101877358}) 2024-09-16T06:55:07.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020411961}) 2024-09-16T06:55:07.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2130578750}) 2024-09-16T06:55:07.705 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538419170}) 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205764892}) 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538419170}) 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.706+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1054598717}) 2024-09-16T06:55:07.706 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654630313}) 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752145330}) 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654630313}) 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1220396049}) 2024-09-16T06:55:07.707 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757224785}) 2024-09-16T06:55:07.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410094453}) 2024-09-16T06:55:07.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757224785}) 2024-09-16T06:55:07.708 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988655588}) 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729374201}) 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242114265}) 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729374201}) 2024-09-16T06:55:07.709 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655595710}) 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403408377}) 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151106757}) 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403408377}) 2024-09-16T06:55:07.710 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.710+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515788356}) 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1129081877}) 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13344519}) 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1129081877}) 2024-09-16T06:55:07.711 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514829855}) 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680246124}) 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327374162}) 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680246124}) 2024-09-16T06:55:07.712 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972360180}) 2024-09-16T06:55:07.713 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469219164}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492416831}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469219164}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478616143}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70248972}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794125075}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70248972}) 2024-09-16T06:55:07.714 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.714+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678139122}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457657611}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207009332}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457657611}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751653441}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820507095}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994734962}) 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.716 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820507095}) 2024-09-16T06:55:07.717 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830614771}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763930458}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816863711}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763930458}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813301884}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067923042}) 2024-09-16T06:55:07.718 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823895502}) 2024-09-16T06:55:07.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067923042}) 2024-09-16T06:55:07.719 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.718+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898366099}) 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385806148}) 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831304508}) 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385806148}) 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300161581}) 2024-09-16T06:55:07.720 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919876853}) 2024-09-16T06:55:07.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781195369}) 2024-09-16T06:55:07.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919876853}) 2024-09-16T06:55:07.721 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1701488866}) 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114364167}) 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348627726}) 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114364167}) 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.722+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296593357}) 2024-09-16T06:55:07.722 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886149899}) 2024-09-16T06:55:07.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902319202}) 2024-09-16T06:55:07.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886149899}) 2024-09-16T06:55:07.723 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1598715298}) 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132171467}) 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500277537}) 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132171467}) 2024-09-16T06:55:07.724 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905803975}) 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470253966}) 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312626578}) 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470253966}) 2024-09-16T06:55:07.725 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2111498017}) 2024-09-16T06:55:07.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1823371232}) 2024-09-16T06:55:07.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218812660}) 2024-09-16T06:55:07.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.726 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1823371232}) 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.726+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1386523667}) 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042022879}) 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454375083}) 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042022879}) 2024-09-16T06:55:07.727 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342704572}) 2024-09-16T06:55:07.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011598346}) 2024-09-16T06:55:07.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261886492}) 2024-09-16T06:55:07.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.728 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011598346}) 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353956616}) 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503598060}) 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415677888}) 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503598060}) 2024-09-16T06:55:07.729 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1164530563}) 2024-09-16T06:55:07.730 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613837606}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513613191}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613837606}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.730+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495831187}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=692435432}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407972786}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=692435432}) 2024-09-16T06:55:07.731 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502784491}) 2024-09-16T06:55:07.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857370173}) 2024-09-16T06:55:07.733 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522586581}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857370173}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445226240}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745119163}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379365754}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745119163}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346007232}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2105471325}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562733330}) 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.734 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2105471325}) 2024-09-16T06:55:07.735 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.734+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081597238}) 2024-09-16T06:55:07.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867527981}) 2024-09-16T06:55:07.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588796562}) 2024-09-16T06:55:07.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.736 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867527981}) 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76968197}) 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108233114}) 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318763679}) 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108233114}) 2024-09-16T06:55:07.737 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3493433597}) 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484453058}) 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521419726}) 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484453058}) 2024-09-16T06:55:07.738 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.738+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922457168}) 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940590732}) 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416040002}) 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940590732}) 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565028781}) 2024-09-16T06:55:07.739 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189396038}) 2024-09-16T06:55:07.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611098554}) 2024-09-16T06:55:07.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189396038}) 2024-09-16T06:55:07.740 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749345912}) 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592655802}) 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079585620}) 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592655802}) 2024-09-16T06:55:07.741 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.742+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4241758846}) 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290582948}) 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674351628}) 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290582948}) 2024-09-16T06:55:07.742 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180363179}) 2024-09-16T06:55:07.743 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.743 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947679324}) 2024-09-16T06:55:07.743 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306915800}) 2024-09-16T06:55:07.743 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.743 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947679324}) 2024-09-16T06:55:07.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910790509}) 2024-09-16T06:55:07.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509153448}) 2024-09-16T06:55:07.744 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196434970}) 2024-09-16T06:55:07.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509153448}) 2024-09-16T06:55:07.745 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563982198}) 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1980636504}) 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191970164}) 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1980636504}) 2024-09-16T06:55:07.746 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.746+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010081402}) 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334257108}) 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305212768}) 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334257108}) 2024-09-16T06:55:07.747 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386466192}) 2024-09-16T06:55:07.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186566703}) 2024-09-16T06:55:07.748 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008769696}) 2024-09-16T06:55:07.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186566703}) 2024-09-16T06:55:07.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272098133}) 2024-09-16T06:55:07.749 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.750 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945200393}) 2024-09-16T06:55:07.750 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710513908}) 2024-09-16T06:55:07.750 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.750 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945200393}) 2024-09-16T06:55:07.750 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.750+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499659013}) 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410556152}) 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145365861}) 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410556152}) 2024-09-16T06:55:07.751 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407505247}) 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816534524}) 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097143934}) 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816534524}) 2024-09-16T06:55:07.752 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779853902}) 2024-09-16T06:55:07.753 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.753 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561683249}) 2024-09-16T06:55:07.753 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2017870836}) 2024-09-16T06:55:07.754 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.754 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561683249}) 2024-09-16T06:55:07.754 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.754+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584951298}) 2024-09-16T06:55:07.754 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.755 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983605602}) 2024-09-16T06:55:07.755 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908117115}) 2024-09-16T06:55:07.755 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.755 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983605602}) 2024-09-16T06:55:07.755 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884777301}) 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1233680479}) 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123263216}) 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1233680479}) 2024-09-16T06:55:07.756 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2028639529}) 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2379713757}) 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852619854}) 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2379713757}) 2024-09-16T06:55:07.757 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.758+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152637386}) 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202361601}) 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888284836}) 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202361601}) 2024-09-16T06:55:07.759 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=886434135}) 2024-09-16T06:55:07.760 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262918880}) 2024-09-16T06:55:07.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677326948}) 2024-09-16T06:55:07.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262918880}) 2024-09-16T06:55:07.761 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.762+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1171178539}) 2024-09-16T06:55:07.762 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.762 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1831188726}) 2024-09-16T06:55:07.762 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917947080}) 2024-09-16T06:55:07.762 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.763 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1831188726}) 2024-09-16T06:55:07.763 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381981183}) 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177024340}) 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2939730398}) 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177024340}) 2024-09-16T06:55:07.764 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231167418}) 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540399998}) 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157232916}) 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540399998}) 2024-09-16T06:55:07.766 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.766+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727424691}) 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3381493566}) 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1506221578}) 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3381493566}) 2024-09-16T06:55:07.767 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274856}) 2024-09-16T06:55:07.768 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.768 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3457004241}) 2024-09-16T06:55:07.768 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978934433}) 2024-09-16T06:55:07.768 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.769 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3457004241}) 2024-09-16T06:55:07.769 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472214682}) 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319125012}) 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996573847}) 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319125012}) 2024-09-16T06:55:07.770 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.770+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190686347}) 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885201114}) 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225340034}) 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885201114}) 2024-09-16T06:55:07.771 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333774627}) 2024-09-16T06:55:07.772 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848089516}) 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481726406}) 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848089516}) 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112170235}) 2024-09-16T06:55:07.773 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.774 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1334465617}) 2024-09-16T06:55:07.774 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140684912}) 2024-09-16T06:55:07.774 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.774 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1334465617}) 2024-09-16T06:55:07.774 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.774+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317336016}) 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=2785681015,mdsmap=1531151339,mgr=278123252,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1297859398,mon_config_key=3670262951,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2498045953}) 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=2785681015,mdsmap=1531151339,mgr=3975174500,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=605771444,mon_config_key=4279594037,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4104304176}) 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=2785681015,mdsmap=1531151339,mgr=278123252,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=1297859398,mon_config_key=3670262951,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2498045953}) 2024-09-16T06:55:07.776 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.778+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=2785681015,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2416316247,mon_config_key=2697620603,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2100396250}) 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=13} crc {osdmap=4164150591}) 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=2514999911}) 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=13} crc {osdmap=4164150591}) 2024-09-16T06:55:07.782 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:07.786+0000 7f6d1a767700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=2514999911}) 2024-09-16T06:55:08.672 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:55:08.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.789+0000 7f5ebd0e0700 10 monclient: get_monmap_and_config 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.789+0000 7f5ebd0e0700 10 monclient: build_initial_monmap 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.789+0000 7f5ebd0e0700 10 monclient: monmap: 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:08.795541+0000 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:08.795541+0000 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:08.790 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:08.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:08.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 Processor -- start 2024-09-16T06:55:08.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- start start 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient: init 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-c con 0x7f5eb8132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-a con 0x7f5eb806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-b con 0x7f5eb806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb806c280 con 0x7f5eb806d480 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb806bfa0 con 0x7f5eb8132330 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb809ace0 con 0x7f5eb806cf50 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient(hunting): authenticate will time out at 1855.929380s 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebc8df700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5eb806cf50 0x7f5eb813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36536/0 (socket says 172.21.15.76:36536) 2024-09-16T06:55:08.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5eb806d480 0x7f5eb813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39036/0 (socket says 172.21.15.76:39036) 2024-09-16T06:55:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebc8df700 1 -- 172.21.15.76:0/887782805 learned_addr learned my addr 172.21.15.76:0/887782805 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453368687 0 0) 0x7f5eb806c280 con 0x7f5eb806d480 2024-09-16T06:55:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): my global_id is 314124 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea40035d0 con 0x7f5eb806d480 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987104828 0 0) 0x7f5eb809ace0 con 0x7f5eb806cf50 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): my global_id is 244874 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb806c280 con 0x7f5eb806cf50 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948021432 0 0) 0x7f5eb806bfa0 con 0x7f5eb8132330 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient(hunting): my global_id is 284407 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb809ace0 con 0x7f5eb8132330 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164675895 0 0) 0x7f5ea40035d0 con 0x7f5eb806d480 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 >> v1:172.21.15.76:6790/0 conn(0x7f5eb806cf50 legacy=0x7f5eb813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 >> v1:172.21.15.189:6789/0 conn(0x7f5eb8132330 legacy=0x7f5eb8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 monclient: found mon.noname-a 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:08.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb809ae70 con 0x7f5eb806d480 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: _finish_auth 0 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 5 monclient: authenticate success, global_id 314124 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ea0001610 con 0x7f5eb806d480 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: dump: 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea00018f0 con 0x7f5eb806d480 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 1 -- 172.21.15.76:0/887782805 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ea0001ae0 con 0x7f5eb806d480 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 4 set_mon_vals no callback set 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:08.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5eb6ffd700 10 monclient: dump: 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 10 monclient: shutdown 2024-09-16T06:55:08.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/887782805 >> v1:172.21.15.76:6789/0 conn(0x7f5eb806d480 legacy=0x7f5eb813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/887782805 shutdown_connections 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/887782805 >> 172.21.15.76:0/887782805 conn(0x7f5eb80a41c0 msgr2=0x7f5eb80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.793+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/887782805 shutdown_connections 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/887782805 wait complete. 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: build_initial_monmap 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: monmap: 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 Processor -- start 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- start start 2024-09-16T06:55:08.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: init 2024-09-16T06:55:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-b con 0x7f5eb806cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-c con 0x7f5eb806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:08.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient(hunting): picked mon.noname-a con 0x7f5eb8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb809e8b0 con 0x7f5eb8132330 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb812cc00 con 0x7f5eb806cf50 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb812e270 con 0x7f5eb806d480 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient(hunting): authenticate will time out at 1855.934110s 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb77fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5eb806d480 0x7f5eb812b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36546/0 (socket says 172.21.15.76:36546) 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb77fe700 1 -- 172.21.15.76:0/3410990376 learned_addr learned my addr 172.21.15.76:0/3410990376 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415647178 0 0) 0x7f5eb812e270 con 0x7f5eb806d480 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): my global_id is 244877 2024-09-16T06:55:08.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e980035d0 con 0x7f5eb806d480 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610283084 0 0) 0x7f5eb809e8b0 con 0x7f5eb8132330 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): my global_id is 314127 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb812e270 con 0x7f5eb8132330 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593327884 0 0) 0x7f5eb812cc00 con 0x7f5eb806cf50 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): my global_id is 284410 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb809e8b0 con 0x7f5eb806cf50 2024-09-16T06:55:08.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957444970 0 0) 0x7f5e980035d0 con 0x7f5eb806d480 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb812cc00 con 0x7f5eb806d480 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ea0004890 con 0x7f5eb806d480 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): dump: 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188863484 0 0) 0x7f5eb809e8b0 con 0x7f5eb806cf50 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e980035d0 con 0x7f5eb806cf50 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ea80018b0 con 0x7f5eb806cf50 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient(hunting): dump: 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.810 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201290718 0 0) 0x7f5eb812e270 con 0x7f5eb8132330 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb809e8b0 con 0x7f5eb8132330 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601313431 0 0) 0x7f5eb812cc00 con 0x7f5eb806d480 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 >> v1:172.21.15.189:6789/0 conn(0x7f5eb806cf50 legacy=0x7f5eb812b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 >> v1:172.21.15.76:6789/0 conn(0x7f5eb8132330 legacy=0x7f5eb81d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 monclient: found mon.c 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:08.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb812c0a0 con 0x7f5eb806d480 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient: _finish_auth 0 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5eb4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 5 monclient: authenticate success, global_id 244877 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: _renew_subs 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.797+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb812c620 con 0x7f5eb806d480 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 10 monclient: _renew_subs 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb8131260 con 0x7f5eb806d480 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea0004ca0 con 0x7f5eb806d480 2024-09-16T06:55:08.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:08.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5ea0004fb0 con 0x7f5eb806d480 2024-09-16T06:55:08.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:08.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:08.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb67fc700 4 set_mon_vals no callback set 2024-09-16T06:55:08.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 10 monclient: dump: 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:08.816 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c005100 con 0x7f5eb806d480 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5ea000fa20 con 0x7f5eb806d480 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5ea0044050 con 0x7f5eb806d480 2024-09-16T06:55:08.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.801+0000 7f5eb7fff700 10 monclient: get_auth_request con 0x7f5e980364a0 auth_method 0 2024-09-16T06:55:08.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.809+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5ea0010f20 con 0x7f5eb806d480 2024-09-16T06:55:08.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.809+0000 7f5eb4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:08.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:08.809+0000 7f5eb4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:09.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:09.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:09.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e9c005930 con 0x7f5eb806d480 2024-09-16T06:55:09.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5eb4ff9700 1 -- 172.21.15.76:0/3410990376 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 439104541) 0x7f5ea00146a0 con 0x7f5eb806d480 2024-09-16T06:55:09.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5eb4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:09.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5eb4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:09.044 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:09.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5e980364a0 legacy=0x7f5e98038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 10 monclient: shutdown 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 >> v1:172.21.15.76:6790/0 conn(0x7f5eb806d480 legacy=0x7f5eb812b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 shutdown_connections 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 >> 172.21.15.76:0/3410990376 conn(0x7f5eb80a41c0 msgr2=0x7f5eb8129f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 shutdown_connections 2024-09-16T06:55:09.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.045+0000 7f5ebd0e0700 1 -- 172.21.15.76:0/3410990376 wait complete. 2024-09-16T06:55:09.057 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:55:09.058 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:55:09.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:55:09.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: get_monmap_and_config 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: monmap: 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:09.177714+0000 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:09.177714+0000 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.173 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.173 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:09.173 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:09.173 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 1 Processor -- start 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 1 -- start start 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: init 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:09.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-a con 0x7f298c0207b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-b con 0x7f298c0229f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-c con 0x7f298c021e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c021bf0 con 0x7f298c0207b0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c01c340 con 0x7f298c021e70 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c01d500 con 0x7f298c0229f0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299b1f8700 10 monclient(hunting): authenticate will time out at 1856.311154s 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299a1f6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f298c0207b0 0x7f298c020b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39060/0 (socket says 172.21.15.76:39060) 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.173+0000 7f299a1f6700 1 -- 172.21.15.76:0/367557395 learned_addr learned my addr 172.21.15.76:0/367557395 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229535577 0 0) 0x7f298c021bf0 con 0x7f298c0207b0 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): my global_id is 314130 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29800035d0 con 0x7f298c0207b0 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203011736 0 0) 0x7f298c01d500 con 0x7f298c0229f0 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): my global_id is 244880 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c021bf0 con 0x7f298c0229f0 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496632785 0 0) 0x7f298c01c340 con 0x7f298c021e70 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient(hunting): my global_id is 284413 2024-09-16T06:55:09.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c01d500 con 0x7f298c021e70 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 539921357 0 0) 0x7f29800035d0 con 0x7f298c0207b0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 >> v1:172.21.15.76:6790/0 conn(0x7f298c0229f0 legacy=0x7f298c0bcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 >> v1:172.21.15.189:6789/0 conn(0x7f298c021e70 legacy=0x7f298c0c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 monclient: found mon.noname-a 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c01d920 con 0x7f298c0207b0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2984001610 con 0x7f298c0207b0 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: dump: 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 5 monclient: authenticate success, global_id 314130 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29840018f0 con 0x7f298c0207b0 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 1 -- 172.21.15.76:0/367557395 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2984001c80 con 0x7f298c0207b0 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f29991f4700 10 monclient: dump: 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.181 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 4 set_mon_vals no callback set 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 10 monclient: shutdown 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 1 -- 172.21.15.76:0/367557395 >> v1:172.21.15.76:6789/0 conn(0x7f298c0207b0 legacy=0x7f298c020b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 1 -- 172.21.15.76:0/367557395 shutdown_connections 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 1 -- 172.21.15.76:0/367557395 >> 172.21.15.76:0/367557395 conn(0x7f298c005b40 msgr2=0x7f298c005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 1 -- 172.21.15.76:0/367557395 shutdown_connections 2024-09-16T06:55:09.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 1 -- 172.21.15.76:0/367557395 wait complete. 2024-09-16T06:55:09.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.177+0000 7f299b1f8700 10 monclient: monmap: 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 Processor -- start 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- start start 2024-09-16T06:55:09.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: init 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-b con 0x7f298c0207b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-a con 0x7f298c021e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient(hunting): picked mon.noname-c con 0x7f298c0229f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c01f7a0 con 0x7f298c021e70 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c146220 con 0x7f298c0207b0 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c0204a0 con 0x7f298c0229f0 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient(hunting): authenticate will time out at 1856.317342s 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f29999f5700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f298c021e70 0x7f298c150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39070/0 (socket says 172.21.15.76:39070) 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299a9f7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f298c0229f0 0x7f298c154570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36572/0 (socket says 172.21.15.76:36572) 2024-09-16T06:55:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f29999f5700 1 -- 172.21.15.76:0/2392565477 learned_addr learned my addr 172.21.15.76:0/2392565477 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159073910 0 0) 0x7f298c0204a0 con 0x7f298c0229f0 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): my global_id is 244883 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29780035d0 con 0x7f298c0229f0 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366559040 0 0) 0x7f298c01f7a0 con 0x7f298c021e70 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): my global_id is 314133 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c0204a0 con 0x7f298c021e70 2024-09-16T06:55:09.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047085065 0 0) 0x7f298c146220 con 0x7f298c0207b0 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): my global_id is 284416 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c01f7a0 con 0x7f298c0207b0 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321571362 0 0) 0x7f29780035d0 con 0x7f298c0229f0 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f298c146220 con 0x7f298c0229f0 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2994069400 con 0x7f298c0229f0 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): dump: 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.190 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2608960347 0 0) 0x7f298c0204a0 con 0x7f298c021e70 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29780035d0 con 0x7f298c021e70 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2990001950 con 0x7f298c021e70 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient(hunting): dump: 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2632199365 0 0) 0x7f298c146220 con 0x7f298c0229f0 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.189:6789/0 conn(0x7f298c0207b0 legacy=0x7f298c145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6789/0 conn(0x7f298c021e70 legacy=0x7f298c150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 monclient: found mon.c 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c12a940 con 0x7f298c0229f0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 5 monclient: authenticate success, global_id 244883 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _renew_subs 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c12d470 con 0x7f298c0229f0 2024-09-16T06:55:09.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _renew_subs 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c12bf20 con 0x7f298c0229f0 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f299404ec20 con 0x7f298c0229f0 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2994066de0 con 0x7f298c0229f0 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f29989f3700 4 set_mon_vals no callback set 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f298affd700 10 monclient: dump: 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.194 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.181+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f297c006590 con 0x7f297c002830 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2994066f40 con 0x7f298c0229f0 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f29940bd910 con 0x7f298c0229f0 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111411832 0 0) 0x7f297c006590 con 0x7f297c002830 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient(hunting): my global_id is 314136 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c146220 con 0x7f297c002830 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4093405061 0 0) 0x7f298c146220 con 0x7f297c002830 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f297c006590 con 0x7f297c002830 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2984001cc0 con 0x7f297c002830 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient: dump: 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f29999f5700 10 monclient: get_auth_request con 0x7f297803a800 auth_method 0 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3354060923 0 0) 0x7f297c006590 con 0x7f297c002830 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f297c00a190 con 0x7f297c002830 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2984006b90 con 0x7f297c002830 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6789/0 conn(0x7f297c002830 legacy=0x7f297c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.185+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6789/0 conn(0x7f297c002830 legacy=0x7f297c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.193+0000 7f299b1f8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f297c009e30 con 0x7f297c009150 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864078107 0 0) 0x7f297c009e30 con 0x7f297c009150 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient(hunting): my global_id is 314139 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f297c006590 con 0x7f297c009150 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 434526671 0 0) 0x7f297c006590 con 0x7f297c009150 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f297c009e30 con 0x7f297c009150 2024-09-16T06:55:09.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2984005090 con 0x7f297c009150 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient: dump: 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.200 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2140227968 0 0) 0x7f297c009e30 con 0x7f297c009150 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f297c00c340 con 0x7f297c009150 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3880 (unknown 0 0 3982517203) 0x7f2984006470 con 0x7f297c009150 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6789/0 conn(0x7f297c009150 legacy=0x7f297c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f298affd700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6789/0 conn(0x7f297c009150 legacy=0x7f297c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:09.201 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.202 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.203 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:09.204 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.205 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:09.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:09.209 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.210 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:09.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:09.212 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:09.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.189:6812/3098810674 conn(0x7f297803a800 legacy=0x7f297803ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 10 monclient: shutdown 2024-09-16T06:55:09.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 >> v1:172.21.15.76:6790/0 conn(0x7f298c0229f0 legacy=0x7f298c154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299a9f7700 1 -- 172.21.15.76:0/2392565477 reap_dead start 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 shutdown_connections 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 >> 172.21.15.76:0/2392565477 conn(0x7f298c005b40 msgr2=0x7f298c01db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 shutdown_connections 2024-09-16T06:55:09.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.197+0000 7f299b1f8700 1 -- 172.21.15.76:0/2392565477 wait complete. 2024-09-16T06:55:09.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:09.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:55:09.215 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:55:09.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.333+0000 7f6667fff700 10 monclient: get_monmap_and_config 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.333+0000 7f6667fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.333+0000 7f6667fff700 10 monclient: monmap: 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:09.339443+0000 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:09.339443+0000 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:09.334 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:09.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 Processor -- start 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- start start 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: init 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:09.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6668099e60 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-b con 0x7f66681354d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6668138fe0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6668098b70 con 0x7f6668099e60 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666809a240 con 0x7f6668138fe0 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6668098890 con 0x7f66681354d0 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient(hunting): authenticate will time out at 1856.472750s 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6666ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6668099e60 0x7f6668098f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39094/0 (socket says 172.21.15.76:39094) 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f66667fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f66681354d0 0x7f66681378c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36582/0 (socket says 172.21.15.76:36582) 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6666ffd700 1 -- 172.21.15.76:0/795545683 learned_addr learned my addr 172.21.15.76:0/795545683 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389544585 0 0) 0x7f6668098b70 con 0x7f6668099e60 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): my global_id is 314142 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66500035d0 con 0x7f6668099e60 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935036610 0 0) 0x7f6668098890 con 0x7f66681354d0 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): my global_id is 244886 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668098b70 con 0x7f66681354d0 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506175510 0 0) 0x7f666809a240 con 0x7f6668138fe0 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient(hunting): my global_id is 284419 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668098890 con 0x7f6668138fe0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547222060 0 0) 0x7f6668098b70 con 0x7f66681354d0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 >> v1:172.21.15.189:6789/0 conn(0x7f6668138fe0 legacy=0x7f666813b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 >> v1:172.21.15.76:6789/0 conn(0x7f6668099e60 legacy=0x7f6668098f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 monclient: found mon.noname-b 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666809fa00 con 0x7f66681354d0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f665c0015f0 con 0x7f66681354d0 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 5 monclient: authenticate success, global_id 244886 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: dump: 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.342 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f665c001910 con 0x7f66681354d0 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 1 -- 172.21.15.76:0/795545683 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f665c001ca0 con 0x7f66681354d0 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 4 set_mon_vals no callback set 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6665ffb700 10 monclient: dump: 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: shutdown 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- 172.21.15.76:0/795545683 >> v1:172.21.15.76:6790/0 conn(0x7f66681354d0 legacy=0x7f66681378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- 172.21.15.76:0/795545683 shutdown_connections 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- 172.21.15.76:0/795545683 >> 172.21.15.76:0/795545683 conn(0x7f66680a0270 msgr2=0x7f66680a0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- 172.21.15.76:0/795545683 shutdown_connections 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 1 -- 172.21.15.76:0/795545683 wait complete. 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.337+0000 7f6667fff700 10 monclient: monmap: 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:09.345 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 Processor -- start 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- start start 2024-09-16T06:55:09.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: init 2024-09-16T06:55:09.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6668099e60 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-a con 0x7f66681354d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6668138fe0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66681299e0 con 0x7f66681354d0 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666812ad80 con 0x7f6668099e60 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666812b060 con 0x7f6668138fe0 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient(hunting): authenticate will time out at 1856.477301s 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f66667fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f66681354d0 0x7f6668128b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39098/0 (socket says 172.21.15.76:39098) 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f66667fc700 1 -- 172.21.15.76:0/1840966447 learned_addr learned my addr 172.21.15.76:0/1840966447 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958546735 0 0) 0x7f66681299e0 con 0x7f66681354d0 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): my global_id is 314145 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66540035d0 con 0x7f66681354d0 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129300010 0 0) 0x7f666812b060 con 0x7f6668138fe0 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): my global_id is 244889 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66681299e0 con 0x7f6668138fe0 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119989915 0 0) 0x7f66540035d0 con 0x7f66681354d0 2024-09-16T06:55:09.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f666812b060 con 0x7f66681354d0 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f665c0015f0 con 0x7f66681354d0 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): dump: 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.353 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356254369 0 0) 0x7f666812ad80 con 0x7f6668099e60 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): my global_id is 284422 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66540035d0 con 0x7f6668099e60 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2413723930 0 0) 0x7f66681299e0 con 0x7f6668138fe0 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f666812ad80 con 0x7f6668138fe0 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6660001c30 con 0x7f6668138fe0 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient(hunting): dump: 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 719337727 0 0) 0x7f666812b060 con 0x7f66681354d0 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6790/0 conn(0x7f6668138fe0 legacy=0x7f66681cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.189:6789/0 conn(0x7f6668099e60 legacy=0x7f6668128410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 monclient: found mon.a 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666812b340 con 0x7f66681354d0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 5 monclient: authenticate success, global_id 314145 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _renew_subs 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666812a9c0 con 0x7f66681354d0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _renew_subs 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6668129230 con 0x7f66681354d0 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f665c004be0 con 0x7f66681354d0 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f665c004ef0 con 0x7f66681354d0 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f66657fa700 4 set_mon_vals no callback set 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.341+0000 7f664ffff700 10 monclient: dump: 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.357 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f665c005050 con 0x7f66681354d0 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f6667fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6648006590 con 0x7f6648002830 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f665c043f60 con 0x7f66681354d0 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275652856 0 0) 0x7f6648006590 con 0x7f6648002830 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient(hunting): my global_id is 244892 2024-09-16T06:55:09.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666812b060 con 0x7f6648002830 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f66677fe700 10 monclient: get_auth_request con 0x7f6654036450 auth_method 0 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3033990265 0 0) 0x7f666812b060 con 0x7f6648002830 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6648006590 con 0x7f6648002830 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6658001850 con 0x7f6648002830 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient: dump: 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.359 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 834440715 0 0) 0x7f6648006590 con 0x7f6648002830 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f664800a190 con 0x7f6648002830 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6658001b60 con 0x7f6648002830 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6790/0 conn(0x7f6648002830 legacy=0x7f6648004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.345+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6790/0 conn(0x7f6648002830 legacy=0x7f6648004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f6667fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6648009e50 con 0x7f6648009170 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607395649 0 0) 0x7f6648009e50 con 0x7f6648009170 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient(hunting): my global_id is 244895 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648006590 con 0x7f6648009170 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3128541237 0 0) 0x7f6648006590 con 0x7f6648009170 2024-09-16T06:55:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6648009e50 con 0x7f6648009170 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6658004c20 con 0x7f6648009170 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient: dump: 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1752889948 0 0) 0x7f6648009e50 con 0x7f6648009170 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f664800c340 con 0x7f6648009170 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 2238503375) 0x7f6658004e30 con 0x7f6648009170 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6790/0 conn(0x7f6648009170 legacy=0x7f6648009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.357+0000 7f664ffff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6790/0 conn(0x7f6648009170 legacy=0x7f6648009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:09.364 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:09.365 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.366 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:09.367 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.368 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.369 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.370 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:09.372 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.374 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:09.375 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6654036450 legacy=0x7f6654038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 10 monclient: shutdown 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 >> v1:172.21.15.76:6789/0 conn(0x7f66681354d0 legacy=0x7f6668128b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f66677fe700 1 -- 172.21.15.76:0/1840966447 reap_dead start 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f66677fe700 1 -- 172.21.15.76:0/1840966447 reap_dead start 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 shutdown_connections 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 >> 172.21.15.76:0/1840966447 conn(0x7f66680a0270 msgr2=0x7f666812e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 shutdown_connections 2024-09-16T06:55:09.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.361+0000 7f6667fff700 1 -- 172.21.15.76:0/1840966447 wait complete. 2024-09-16T06:55:09.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:09.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:55:09.378 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:55:09.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: get_monmap_and_config 2024-09-16T06:55:09.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: monmap: 2024-09-16T06:55:09.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:09.508273+0000 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:09.508273+0000 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:09.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:09.505 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 Processor -- start 2024-09-16T06:55:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- start start 2024-09-16T06:55:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: init 2024-09-16T06:55:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-a con 0x7f8c2409b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-c con 0x7f8c240964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-b con 0x7f8c24138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24098560 con 0x7f8c2409b1c0 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24099ec0 con 0x7f8c240964f0 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24097030 con 0x7f8c24138870 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient(hunting): authenticate will time out at 1856.641756s 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c2359e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8c24138870 0x7f8c2413ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36618/0 (socket says 172.21.15.76:36618) 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c2359e700 1 -- 172.21.15.76:0/78652132 learned_addr learned my addr 172.21.15.76:0/78652132 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c22d9d700 1 -- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6789/0 conn(0x7f8c2409b1c0 legacy=0x7f8c2409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c22d9d700 1 -- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6789/0 conn(0x7f8c2409b1c0 legacy=0x7f8c2409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c22d9d700 1 --1- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6789/0 conn(0x7f8c2409b1c0 0x7f8c2409ed60 :-1 s=OPENED pgs=1227 cs=1 l=1).handle_message read tag failed 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c22d9d700 1 --1- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6789/0 conn(0x7f8c2409b1c0 0x7f8c2409ed60 :-1 s=OPENED pgs=1227 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:55:09.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598655499 0 0) 0x7f8c24097030 con 0x7f8c24138870 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient(hunting): my global_id is 244898 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c0035d0 con 0x7f8c24138870 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2338090742 0 0) 0x7f8c0c0035d0 con 0x7f8c24138870 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 >> v1:172.21.15.189:6789/0 conn(0x7f8c240964f0 legacy=0x7f8c24096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6789/0 conn(0x7f8c2409b1c0 legacy=0x7f8c2409ed60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 monclient: found mon.noname-b 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2409b5a0 con 0x7f8c24138870 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 5 monclient: authenticate success, global_id 244898 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c1c001990 con 0x7f8c24138870 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: dump: 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.516 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c1c001bb0 con 0x7f8c24138870 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 4 set_mon_vals no callback set 2024-09-16T06:55:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 1 -- 172.21.15.76:0/78652132 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c1c001f80 con 0x7f8c24138870 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c21d9b700 10 monclient: dump: 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 10 monclient: shutdown 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- 172.21.15.76:0/78652132 >> v1:172.21.15.76:6790/0 conn(0x7f8c24138870 legacy=0x7f8c2413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- 172.21.15.76:0/78652132 shutdown_connections 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- 172.21.15.76:0/78652132 >> 172.21.15.76:0/78652132 conn(0x7f8c240a0030 msgr2=0x7f8c240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- 172.21.15.76:0/78652132 shutdown_connections 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.505+0000 7f8c28a4b700 1 -- 172.21.15.76:0/78652132 wait complete. 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: monmap: 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:09.519 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 Processor -- start 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- start start 2024-09-16T06:55:09.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: init 2024-09-16T06:55:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-a con 0x7f8c240964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-b con 0x7f8c2409b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient(hunting): picked mon.noname-c con 0x7f8c24138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2409a540 con 0x7f8c240964f0 2024-09-16T06:55:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c241c60d0 con 0x7f8c2409b1c0 2024-09-16T06:55:09.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240a2f80 con 0x7f8c24138870 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient(hunting): authenticate will time out at 1856.646060s 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c2359e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8c24138870 0x7f8c241d0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:36634/0 (socket says 172.21.15.76:36634) 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c2359e700 1 -- 172.21.15.76:0/3690057868 learned_addr learned my addr 172.21.15.76:0/3690057868 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394105513 0 0) 0x7f8c240a2f80 con 0x7f8c24138870 2024-09-16T06:55:09.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): my global_id is 244901 2024-09-16T06:55:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c100035d0 con 0x7f8c24138870 2024-09-16T06:55:09.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695516829 0 0) 0x7f8c2409a540 con 0x7f8c240964f0 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): my global_id is 314148 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c240a2f80 con 0x7f8c240964f0 2024-09-16T06:55:09.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966779208 0 0) 0x7f8c241c60d0 con 0x7f8c2409b1c0 2024-09-16T06:55:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): my global_id is 284428 2024-09-16T06:55:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2409a540 con 0x7f8c2409b1c0 2024-09-16T06:55:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552080478 0 0) 0x7f8c100035d0 con 0x7f8c24138870 2024-09-16T06:55:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c241c60d0 con 0x7f8c24138870 2024-09-16T06:55:09.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c1c001ec0 con 0x7f8c24138870 2024-09-16T06:55:09.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:09.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:09.528 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.528 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.530 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.530 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.530 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:09.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1837995245 0 0) 0x7f8c240a2f80 con 0x7f8c240964f0 2024-09-16T06:55:09.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c100035d0 con 0x7f8c240964f0 2024-09-16T06:55:09.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c180018a0 con 0x7f8c240964f0 2024-09-16T06:55:09.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.533 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.533 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.533 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.533 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151418617 0 0) 0x7f8c2409a540 con 0x7f8c2409b1c0 2024-09-16T06:55:09.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c240a2f80 con 0x7f8c2409b1c0 2024-09-16T06:55:09.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c140017f0 con 0x7f8c2409b1c0 2024-09-16T06:55:09.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:09.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.536 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1501130886 0 0) 0x7f8c241c60d0 con 0x7f8c24138870 2024-09-16T06:55:09.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6789/0 conn(0x7f8c2409b1c0 legacy=0x7f8c241ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.76:6789/0 conn(0x7f8c240964f0 legacy=0x7f8c24131870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 monclient: found mon.c 2024-09-16T06:55:09.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c241292d0 con 0x7f8c24138870 2024-09-16T06:55:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 5 monclient: authenticate success, global_id 244901 2024-09-16T06:55:09.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _renew_subs 2024-09-16T06:55:09.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c24128a30 con 0x7f8c24138870 2024-09-16T06:55:09.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _renew_subs 2024-09-16T06:55:09.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:09.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c24133320 con 0x7f8c24138870 2024-09-16T06:55:09.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c1c001550 con 0x7f8c24138870 2024-09-16T06:55:09.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c1c005060 con 0x7f8c24138870 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c20d99700 4 set_mon_vals no callback set 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.509+0000 7f8c0b7fe700 10 monclient: dump: 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.542 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.543 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c28a4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c04006590 con 0x7f8c04002830 2024-09-16T06:55:09.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8c1c00fad0 con 0x7f8c24138870 2024-09-16T06:55:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8c1c044280 con 0x7f8c24138870 2024-09-16T06:55:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c2259c700 10 monclient: get_auth_request con 0x7f8c10036500 auth_method 0 2024-09-16T06:55:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532587160 0 0) 0x7f8c04006590 con 0x7f8c04002830 2024-09-16T06:55:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient(hunting): my global_id is 284431 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c241c60d0 con 0x7f8c04002830 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1311224662 0 0) 0x7f8c241c60d0 con 0x7f8c04002830 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c04006590 con 0x7f8c04002830 2024-09-16T06:55:09.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c18003a70 con 0x7f8c04002830 2024-09-16T06:55:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 10 monclient: dump: 2024-09-16T06:55:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.547 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.548 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.549 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.549 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.549 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1952105913 0 0) 0x7f8c04006590 con 0x7f8c04002830 2024-09-16T06:55:09.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.513+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c0400a190 con 0x7f8c04002830 2024-09-16T06:55:09.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.517+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8c18005870 con 0x7f8c04002830 2024-09-16T06:55:09.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.517+0000 7f8c0b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.517+0000 7f8c0b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:09.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.517+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6789/0 conn(0x7f8c04002830 legacy=0x7f8c04004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.517+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6789/0 conn(0x7f8c04002830 legacy=0x7f8c04004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c28a4b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:09.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8c04009e50 con 0x7f8c04009170 2024-09-16T06:55:09.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779062587 0 0) 0x7f8c04009e50 con 0x7f8c04009170 2024-09-16T06:55:09.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient(hunting): my global_id is 284434 2024-09-16T06:55:09.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04006590 con 0x7f8c04009170 2024-09-16T06:55:09.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2130985973 0 0) 0x7f8c04006590 con 0x7f8c04009170 2024-09-16T06:55:09.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c04009e50 con 0x7f8c04009170 2024-09-16T06:55:09.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8c18004100 con 0x7f8c04009170 2024-09-16T06:55:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.525+0000 7f8c0b7fe700 10 monclient: dump: 2024-09-16T06:55:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.558 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.559 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.559 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1585272596 0 0) 0x7f8c04009e50 con 0x7f8c04009170 2024-09-16T06:55:09.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8c0400c340 con 0x7f8c04009170 2024-09-16T06:55:09.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1396492706) 0x7f8c180043c0 con 0x7f8c04009170 2024-09-16T06:55:09.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:09.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:09.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6789/0 conn(0x7f8c04009170 legacy=0x7f8c04009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c0b7fe700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6789/0 conn(0x7f8c04009170 legacy=0x7f8c04009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:09.561 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:09.561 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:09.561 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:09.561 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:09.561 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 150, 2024-09-16T06:55:09.562 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:09.562 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:09.562 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:09.562 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:09.562 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.563 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:09.563 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.563 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:09.563 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:09.563 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.564 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.564 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.564 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.564 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.564 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.565 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.565 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.565 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.565 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.565 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.566 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.567 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.567 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.567 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.567 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.567 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.568 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.568 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:09.568 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.569 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:09.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:09.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:09.570 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:09.570 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:09.570 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:09.571 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:09.571 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:09.571 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:09.571 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:09.571 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:09.572 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:09.572 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:09.572 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:09.572 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:09.572 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.573 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:09.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.574 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:09.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.574 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:09.574 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:09.575 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.576 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.576 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.576 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.576 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:09.577 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.579 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.580 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:09.581 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:09.582 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8c10036500 legacy=0x7f8c100389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 10 monclient: shutdown 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c2359e700 1 -- 172.21.15.76:0/3690057868 reap_dead start 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 >> v1:172.21.15.76:6790/0 conn(0x7f8c24138870 legacy=0x7f8c241d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 shutdown_connections 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 >> 172.21.15.76:0/3690057868 conn(0x7f8c240a0030 msgr2=0x7f8c2409dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 shutdown_connections 2024-09-16T06:55:09.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.529+0000 7f8c28a4b700 1 -- 172.21.15.76:0/3690057868 wait complete. 2024-09-16T06:55:09.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:09.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:55:09.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:55:09.585 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:55:09.585 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:55:09.585 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:55:09.585 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:55:09.633 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:55:09.633 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:55:09.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:09.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.745+0000 7f33b2f0b700 10 monclient: get_monmap_and_config 2024-09-16T06:55:09.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.745+0000 7f33b2f0b700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.745+0000 7f33b2f0b700 10 monclient: monmap: 2024-09-16T06:55:09.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:09.751087+0000 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:09.751087+0000 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 Processor -- start 2024-09-16T06:55:09.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 -- start start 2024-09-16T06:55:09.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient: init 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-a con 0x7f33ac132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-b con 0x7f33ac06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-c con 0x7f33ac06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac06c280 con 0x7f33ac132330 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac06bfa0 con 0x7f33ac06cf50 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac09ace0 con 0x7f33ac06d480 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient(hunting): authenticate will time out at 1856.885584s 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b1708700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f33ac06d480 legacy=0x7f33ac13a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:09.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b1f09700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f33ac132330 0x7f33ac132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39144/0 (socket says 172.21.15.76:39144) 2024-09-16T06:55:09.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b1f09700 1 -- 172.21.15.76:0/728291157 learned_addr learned my addr 172.21.15.76:0/728291157 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697767379 0 0) 0x7f33ac06c280 con 0x7f33ac132330 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): my global_id is 314151 2024-09-16T06:55:09.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33980035d0 con 0x7f33ac132330 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363059274 0 0) 0x7f33ac06bfa0 con 0x7f33ac06cf50 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient(hunting): my global_id is 284437 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac06c280 con 0x7f33ac06cf50 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661332963 0 0) 0x7f33980035d0 con 0x7f33ac132330 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 >> v1:172.21.15.76:6790/0 conn(0x7f33ac06d480 legacy=0x7f33ac13a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 >> v1:172.21.15.189:6789/0 conn(0x7f33ac06cf50 legacy=0x7f33ac13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 monclient: found mon.noname-a 2024-09-16T06:55:09.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac09ae70 con 0x7f33ac132330 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33a8001610 con 0x7f33ac132330 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 5 monclient: authenticate success, global_id 314151 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: dump: 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.753 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33a80018f0 con 0x7f33ac132330 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 1 -- 172.21.15.76:0/728291157 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f33a8001ae0 con 0x7f33ac132330 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 4 set_mon_vals no callback set 2024-09-16T06:55:09.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b0f07700 10 monclient: dump: 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.755 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 10 monclient: shutdown 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.749+0000 7f33b2f0b700 1 -- 172.21.15.76:0/728291157 >> v1:172.21.15.76:6789/0 conn(0x7f33ac132330 legacy=0x7f33ac132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- 172.21.15.76:0/728291157 shutdown_connections 2024-09-16T06:55:09.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- 172.21.15.76:0/728291157 >> 172.21.15.76:0/728291157 conn(0x7f33ac0a41c0 msgr2=0x7f33ac0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- 172.21.15.76:0/728291157 shutdown_connections 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- 172.21.15.76:0/728291157 wait complete. 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient: build_initial_monmap 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient: monmap: 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:09.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:09.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 Processor -- start 2024-09-16T06:55:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- start start 2024-09-16T06:55:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient: init 2024-09-16T06:55:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-b con 0x7f33ac06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-a con 0x7f33ac06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient(hunting): picked mon.noname-c con 0x7f33ac132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac12f140 con 0x7f33ac06d480 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b270a700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f33ac132330 legacy=0x7f33ac1d2200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac130cb0 con 0x7f33ac06cf50 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ac131620 con 0x7f33ac132330 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b2f0b700 10 monclient(hunting): authenticate will time out at 1856.890947s 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b1708700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f33ac06d480 0x7f33ac1cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39158/0 (socket says 172.21.15.76:39158) 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33b1708700 1 -- 172.21.15.76:0/1462582250 learned_addr learned my addr 172.21.15.76:0/1462582250 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:09.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835810648 0 0) 0x7f33ac12f140 con 0x7f33ac06d480 2024-09-16T06:55:09.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): my global_id is 314154 2024-09-16T06:55:09.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33900035d0 con 0x7f33ac06d480 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339745553 0 0) 0x7f33ac130cb0 con 0x7f33ac06cf50 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 10 monclient(hunting): my global_id is 284440 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac12f140 con 0x7f33ac06cf50 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824850711 0 0) 0x7f33900035d0 con 0x7f33ac06d480 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.753+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33ac130cb0 con 0x7f33ac06d480 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f339c004200 con 0x7f33ac06d480 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient(hunting): dump: 2024-09-16T06:55:09.764 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336183773 0 0) 0x7f33ac130cb0 con 0x7f33ac06d480 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 >> v1:172.21.15.76:6790/0 conn(0x7f33ac132330 legacy=0x7f33ac1d2200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:55:09.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 >> v1:172.21.15.189:6789/0 conn(0x7f33ac06cf50 legacy=0x7f33ac13e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 monclient: found mon.a 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac1309d0 con 0x7f33ac06d480 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: _finish_auth 0 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 5 monclient: authenticate success, global_id 314154 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _renew_subs 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac12f310 con 0x7f33ac06d480 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _renew_subs 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ac13eab0 con 0x7f33ac06d480 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f339c0044e0 con 0x7f33ac06d480 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f339c004870 con 0x7f33ac06d480 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a3fff700 4 set_mon_vals no callback set 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 10 monclient: dump: 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:09.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f339c004a50 con 0x7f33ac06d480 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f339c043f60 con 0x7f33ac06d480 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394005100 con 0x7f33ac06d480 2024-09-16T06:55:09.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.757+0000 7f33b1f09700 10 monclient: get_auth_request con 0x7f33900361b0 auth_method 0 2024-09-16T06:55:09.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.765+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f339c0148a0 con 0x7f33ac06d480 2024-09-16T06:55:09.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.765+0000 7f33a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:09.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.765+0000 7f33a27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:09.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33b2f0b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33b2f0b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33b2f0b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3394005930 con 0x7f33ac06d480 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33a27fc700 1 -- 172.21.15.76:0/1462582250 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2732419874) 0x7f339c01ce90 con 0x7f33ac06d480 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33a27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:09.997+0000 7f33a27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:09.998 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:10.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 >> v1:172.21.15.189:6812/3098810674 conn(0x7f33900361b0 legacy=0x7f3390038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:10.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 10 monclient: shutdown 2024-09-16T06:55:10.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:10.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 >> v1:172.21.15.76:6789/0 conn(0x7f33ac06d480 legacy=0x7f33ac1cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:10.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 shutdown_connections 2024-09-16T06:55:10.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 >> 172.21.15.76:0/1462582250 conn(0x7f33ac0a41c0 msgr2=0x7f33ac09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:10.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 shutdown_connections 2024-09-16T06:55:10.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:10.001+0000 7f33b2f0b700 1 -- 172.21.15.76:0/1462582250 wait complete. 2024-09-16T06:55:13.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:13.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: get_monmap_and_config 2024-09-16T06:55:13.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: build_initial_monmap 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: monmap: 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:13.136252+0000 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:13.136252+0000 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:13.131 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 Processor -- start 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- start start 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: init 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-c con 0x7f8d7c01f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-b con 0x7f8d7c025410 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-a con 0x7f8d7c025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c01f8a0 con 0x7f8d7c025860 2024-09-16T06:55:13.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0233f0 con 0x7f8d7c01f4c0 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c020050 con 0x7f8d7c025410 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient(hunting): authenticate will time out at 1860.269570s 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d81d9b700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f8d7c025410 legacy=0x7f8d7c0bebf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d82d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8d7c025860 0x7f8d7c0c2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39174/0 (socket says 172.21.15.76:39174) 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d82d9d700 1 -- 172.21.15.76:0/3277943442 learned_addr learned my addr 172.21.15.76:0/3277943442 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457179438 0 0) 0x7f8d7c01f8a0 con 0x7f8d7c025860 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:13.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): my global_id is 314157 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c0035d0 con 0x7f8d7c025860 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975784190 0 0) 0x7f8d7c0233f0 con 0x7f8d7c01f4c0 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient(hunting): my global_id is 284443 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c01f8a0 con 0x7f8d7c01f4c0 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2963603392 0 0) 0x7f8d6c0035d0 con 0x7f8d7c025860 2024-09-16T06:55:13.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 >> v1:172.21.15.76:6790/0 conn(0x7f8d7c025410 legacy=0x7f8d7c0bebf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 >> v1:172.21.15.189:6789/0 conn(0x7f8d7c01f4c0 legacy=0x7f8d7c024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: _finish_hunting 0 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 monclient: found mon.noname-a 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c021a90 con 0x7f8d7c025860 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: _finish_auth 0 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: _check_auth_tickets 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 5 monclient: authenticate success, global_id 314157 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d84069400 con 0x7f8d7c025860 2024-09-16T06:55:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: dump: 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:13.143 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d8404ec20 con 0x7f8d7c025860 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 1 -- 172.21.15.76:0/3277943442 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d840008d0 con 0x7f8d7c025860 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 4 set_mon_vals no callback set 2024-09-16T06:55:13.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8159a700 10 monclient: dump: 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 10 monclient: shutdown 2024-09-16T06:55:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- 172.21.15.76:0/3277943442 >> v1:172.21.15.76:6789/0 conn(0x7f8d7c025860 legacy=0x7f8d7c0c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- 172.21.15.76:0/3277943442 shutdown_connections 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- 172.21.15.76:0/3277943442 >> 172.21.15.76:0/3277943442 conn(0x7f8d7c005b40 msgr2=0x7f8d7c005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- 172.21.15.76:0/3277943442 shutdown_connections 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.133+0000 7f8d8359e700 1 -- 172.21.15.76:0/3277943442 wait complete. 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: build_initial_monmap 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: monmap: 2024-09-16T06:55:13.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 Processor -- start 2024-09-16T06:55:13.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- start start 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: init 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-c con 0x7f8d7c01f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-b con 0x7f8d7c025410 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient(hunting): picked mon.noname-a con 0x7f8d7c025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8259c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f8d7c01f4c0 legacy=0x7f8d7c147ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:13.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c028780 con 0x7f8d7c025860 2024-09-16T06:55:13.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c1481e0 con 0x7f8d7c025410 2024-09-16T06:55:13.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0204a0 con 0x7f8d7c01f4c0 2024-09-16T06:55:13.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:13.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient(hunting): authenticate will time out at 1860.274020s 2024-09-16T06:55:13.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d82d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8d7c025860 0x7f8d7c156530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39184/0 (socket says 172.21.15.76:39184) 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d82d9d700 1 -- 172.21.15.76:0/2926355022 learned_addr learned my addr 172.21.15.76:0/2926355022 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857181243 0 0) 0x7f8d7c028780 con 0x7f8d7c025860 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:13.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): my global_id is 314160 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d640035d0 con 0x7f8d7c025860 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1979052882 0 0) 0x7f8d640035d0 con 0x7f8d7c025860 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7c028780 con 0x7f8d7c025860 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d840008d0 con 0x7f8d7c025860 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): dump: 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:13.151 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871889534 0 0) 0x7f8d7c1481e0 con 0x7f8d7c025410 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient(hunting): my global_id is 284446 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d640035d0 con 0x7f8d7c025410 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2542572286 0 0) 0x7f8d7c028780 con 0x7f8d7c025860 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 >> v1:172.21.15.76:6790/0 conn(0x7f8d7c01f4c0 legacy=0x7f8d7c147ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 >> v1:172.21.15.189:6789/0 conn(0x7f8d7c025410 legacy=0x7f8d7c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 monclient: found mon.a 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c12c950 con 0x7f8d7c025860 2024-09-16T06:55:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: _finish_auth 0 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 5 monclient: authenticate success, global_id 314160 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _renew_subs 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c12f430 con 0x7f8d7c025860 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _renew_subs 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c12dee0 con 0x7f8d7c025860 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d84069400 con 0x7f8d7c025860 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8d84067aa0 con 0x7f8d7c025860 2024-09-16T06:55:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d80d99700 4 set_mon_vals no callback set 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.137+0000 7f8d72ffd700 10 monclient: dump: 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:13.155 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d8359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d8359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d8359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d68005100 con 0x7f8d7c025860 2024-09-16T06:55:13.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8d84067cc0 con 0x7f8d7c025860 2024-09-16T06:55:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8d840bdd50 con 0x7f8d7c025860 2024-09-16T06:55:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.141+0000 7f8d8259c700 10 monclient: get_auth_request con 0x7f8d64036190 auth_method 0 2024-09-16T06:55:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.145+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8d8406c3e0 con 0x7f8d7c025860 2024-09-16T06:55:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.145+0000 7f8d72ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.145+0000 7f8d72ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d68005930 con 0x7f8d7c025860 2024-09-16T06:55:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d72ffd700 1 -- 172.21.15.76:0/2926355022 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4063339779) 0x7f8d8406c200 con 0x7f8d7c025860 2024-09-16T06:55:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d72ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d72ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:13.376 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8d64036190 legacy=0x7f8d64038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 10 monclient: shutdown 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 >> v1:172.21.15.76:6789/0 conn(0x7f8d7c025860 legacy=0x7f8d7c156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 shutdown_connections 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 >> 172.21.15.76:0/2926355022 conn(0x7f8d7c005b40 msgr2=0x7f8d7c01a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.377+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 shutdown_connections 2024-09-16T06:55:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:13.381+0000 7f8d8359e700 1 -- 172.21.15.76:0/2926355022 wait complete. 2024-09-16T06:55:16.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: get_monmap_and_config 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: build_initial_monmap 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: monmap: 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:16.518645+0000 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:16.518645+0000 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:16.513 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:16.514 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 1 Processor -- start 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 1 -- start start 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: init 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-a con 0x7f90a409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-c con 0x7f90a4098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:16.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-b con 0x7f90a4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409d560 con 0x7f90a409ed60 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409f140 con 0x7f90a4098be0 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409d790 con 0x7f90a4099030 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.513+0000 7f90aa9ea700 10 monclient(hunting): authenticate will time out at 1863.651962s 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.517+0000 7f90aa1e9700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f90a413ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.517+0000 7f90a99e8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f90a409ed60 0x7f90a40984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:37400/0 (socket says 172.21.15.76:37400) 2024-09-16T06:55:16.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.517+0000 7f90a99e8700 1 -- 172.21.15.76:0/3096651915 learned_addr learned my addr 172.21.15.76:0/3096651915 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:16.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:16.721+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f90a413ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:17.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:17.125+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f90a413ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:17.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:17.929+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f90a413ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.513+0000 7f909b7fe700 10 monclient(hunting): tick 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 monclient(hunting): continuing hunt 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f90a413ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.189:6789/0 conn(0x7f90a4098be0 legacy=0x7f90a41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6789/0 conn(0x7f90a409ed60 legacy=0x7f90a40984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:19.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f9088005950 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-b con 0x7f90a4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-a con 0x7f90a409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409f620 con 0x7f90a409ed60 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409d790 con 0x7f9088005950 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f90a91e7700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9088012600 con 0x7f90a4099030 2024-09-16T06:55:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.517+0000 7f909b7fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:19.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:19.721+0000 7f90a91e7700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:20.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:20.125+0000 7f90a91e7700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:20.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:20.929+0000 7f90a91e7700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800b670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): tick 2024-09-16T06:55:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:55:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 monclient(hunting): continuing hunt 2024-09-16T06:55:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:55:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800b670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.189:6789/0 conn(0x7f9088005950 legacy=0x7f9088007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6789/0 conn(0x7f90a409ed60 legacy=0x7f908800ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f9088012890 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-a con 0x7f9088008570 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): picked mon.noname-b con 0x7f90a4099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9088013810 con 0x7f9088008570 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9088012600 con 0x7f9088012890 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9088008970 con 0x7f90a4099030 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f909b7fe700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.517+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800ee30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:22.721+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800ee30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:23.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.125+0000 7f90aa1e9700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800ee30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393751694 0 0) 0x7f9088012600 con 0x7f9088012890 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 10 monclient(hunting): _negotiate 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 10 monclient(hunting): my global_id is 284449 2024-09-16T06:55:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.649+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90900035d0 con 0x7f9088012890 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186580967 0 0) 0x7f90900035d0 con 0x7f9088012890 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6790/0 conn(0x7f90a4099030 legacy=0x7f908800ee30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.76:6789/0 conn(0x7f9088008570 legacy=0x7f9088007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: _finish_hunting 0 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 monclient: found mon.noname-c 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:23.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908800c690 con 0x7f9088012890 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: _finish_auth 0 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: _check_auth_tickets 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90aa9ea700 5 monclient: authenticate success, global_id 284449 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9094001590 con 0x7f9088012890 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:23.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: dump: 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:23.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:23.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:23.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:23.657 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:23.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90aa9ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:23.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90940017b0 con 0x7f9088012890 2024-09-16T06:55:23.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:23.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90aa9ea700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 1 -- 172.21.15.76:0/3096651915 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9094001b40 con 0x7f9088012890 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90aa9ea700 4 set_mon_vals no callback set 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.653+0000 7f90a89e6700 10 monclient: dump: 2024-09-16T06:55:23.658 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:23.659 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:23.659 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.659 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.659 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:23.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:23.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:23.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:23.660 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:23.660 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:23.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 10 monclient: shutdown 2024-09-16T06:55:23.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:23.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3096651915 >> v1:172.21.15.189:6789/0 conn(0x7f9088012890 legacy=0x7f9088012c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:23.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3096651915 shutdown_connections 2024-09-16T06:55:23.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3096651915 >> 172.21.15.76:0/3096651915 conn(0x7f90a40a0030 msgr2=0x7f90a40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3096651915 shutdown_connections 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3096651915 wait complete. 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 10 monclient: build_initial_monmap 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 10 monclient: monmap: 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:23.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:23.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 Processor -- start 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 1 -- start start 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.657+0000 7f90aa9ea700 10 monclient: init 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-a con 0x7f9088008570 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-c con 0x7f9088012890 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:23.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient(hunting): picked mon.noname-b con 0x7f90a4099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a41b1510 con 0x7f9088008570 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a41b1890 con 0x7f90a4099030 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a409a540 con 0x7f9088012890 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90a91e7700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9088012890 legacy=0x7f90a41ca920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient(hunting): authenticate will time out at 1870.797001s 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90a99e8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9088008570 0x7f90a41b8600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:37430/0 (socket says 172.21.15.76:37430) 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90a99e8700 1 -- 172.21.15.76:0/3110052405 learned_addr learned my addr 172.21.15.76:0/3110052405 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297511403 0 0) 0x7f90a41b1890 con 0x7f90a4099030 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:23.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): my global_id is 284452 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908800cb50 con 0x7f90a4099030 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620647833 0 0) 0x7f908800cb50 con 0x7f90a4099030 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90a41b1890 con 0x7f90a4099030 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f909c004320 con 0x7f90a4099030 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): dump: 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.675 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:23.676 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:23.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:23.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2263111189 0 0) 0x7f90a41b1890 con 0x7f90a4099030 2024-09-16T06:55:23.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 >> v1:172.21.15.76:6790/0 conn(0x7f9088012890 legacy=0x7f90a41ca920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:55:23.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 >> v1:172.21.15.76:6789/0 conn(0x7f9088008570 legacy=0x7f90a41b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 monclient: found mon.b 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a40a14e0 con 0x7f90a4099030 2024-09-16T06:55:23.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 5 monclient: authenticate success, global_id 284452 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _renew_subs 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a4098460 con 0x7f90a4099030 2024-09-16T06:55:23.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _renew_subs 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a41c5d10 con 0x7f90a4099030 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f909c004730 con 0x7f90a4099030 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f909c004ac0 con 0x7f90a4099030 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:23.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909bfff700 4 set_mon_vals no callback set 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.661+0000 7f909a7fc700 10 monclient: dump: 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:23.681 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f90aa9ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f90aa9ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f90aa9ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c005100 con 0x7f90a4099030 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f909c004c20 con 0x7f90a4099030 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f909c043c70 con 0x7f90a4099030 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.665+0000 7f90a99e8700 10 monclient: get_auth_request con 0x7f90880396c0 auth_method 0 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.673+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f909c0143e0 con 0x7f90a4099030 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.673+0000 7f909a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:23.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.673+0000 7f909a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:23.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f90aa9ea700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f90aa9ea700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f90aa9ea700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f908c005930 con 0x7f90a4099030 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f909a7fc700 1 -- 172.21.15.76:0/3110052405 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3956930845) 0x7f909c012a80 con 0x7f90a4099030 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f909a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:23.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.905+0000 7f909a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:23.906 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 >> v1:172.21.15.189:6812/3098810674 conn(0x7f90880396c0 legacy=0x7f908803bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 10 monclient: shutdown 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 >> v1:172.21.15.189:6789/0 conn(0x7f90a4099030 legacy=0x7f90a41ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 shutdown_connections 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 >> 172.21.15.76:0/3110052405 conn(0x7f90a40a0030 msgr2=0x7f90a406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:23.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 shutdown_connections 2024-09-16T06:55:23.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:23.909+0000 7f90aa9ea700 1 -- 172.21.15.76:0/3110052405 wait complete. 2024-09-16T06:55:23.919 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:55:23.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:55:23.920 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:55:24.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 10 monclient: get_monmap_and_config 2024-09-16T06:55:24.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 10 monclient: build_initial_monmap 2024-09-16T06:55:24.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 10 monclient: monmap: 2024-09-16T06:55:24.044 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:24.034829+0000 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:24.034829+0000 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 1 Processor -- start 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 1 -- start start 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.029+0000 7f822d7f0700 10 monclient: init 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:24.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-c con 0x7f8228097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-a con 0x7f8228099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-b con 0x7f8228139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228098990 con 0x7f8228099090 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228097f40 con 0x7f8228097b60 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822809d610 con 0x7f8228139500 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient(hunting): authenticate will time out at 1871.168340s 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822cfef700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f8228139500 legacy=0x7f822813b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f821f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8228099090 0x7f822809f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51400/0 (socket says 172.21.15.76:51400) 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f821f7fe700 1 -- 172.21.15.76:0/1932050117 learned_addr learned my addr 172.21.15.76:0/1932050117 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600581328 0 0) 0x7f8228098990 con 0x7f8228099090 2024-09-16T06:55:24.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): my global_id is 314166 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82140035d0 con 0x7f8228099090 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218387591 0 0) 0x7f8228097f40 con 0x7f8228097b60 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient(hunting): my global_id is 284455 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228098990 con 0x7f8228097b60 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944311677 0 0) 0x7f82140035d0 con 0x7f8228099090 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 >> v1:172.21.15.76:6790/0 conn(0x7f8228139500 legacy=0x7f822813b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:24.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 >> v1:172.21.15.189:6789/0 conn(0x7f8228097b60 legacy=0x7f822809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 monclient: found mon.noname-a 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280986b0 con 0x7f8228099090 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 5 monclient: authenticate success, global_id 314166 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8210001610 con 0x7f8228099090 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: dump: 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82100018f0 con 0x7f8228099090 2024-09-16T06:55:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 4 set_mon_vals no callback set 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 1 -- 172.21.15.76:0/1932050117 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8210001c80 con 0x7f8228099090 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f82277fe700 10 monclient: dump: 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: shutdown 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- 172.21.15.76:0/1932050117 >> v1:172.21.15.76:6789/0 conn(0x7f8228099090 legacy=0x7f822809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- 172.21.15.76:0/1932050117 shutdown_connections 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- 172.21.15.76:0/1932050117 >> 172.21.15.76:0/1932050117 conn(0x7f82280a0030 msgr2=0x7f82280a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- 172.21.15.76:0/1932050117 shutdown_connections 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 1 -- 172.21.15.76:0/1932050117 wait complete. 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: build_initial_monmap 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.033+0000 7f822d7f0700 10 monclient: monmap: 2024-09-16T06:55:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 Processor -- start 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- start start 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: init 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:24.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-b con 0x7f8228097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-c con 0x7f8228099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient(hunting): picked mon.noname-a con 0x7f8228139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822809b0a0 con 0x7f8228139500 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f821f7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f8228099090 legacy=0x7f8228070c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822813c200 con 0x7f8228097b60 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280a2cc0 con 0x7f8228099090 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient(hunting): authenticate will time out at 1871.173140s 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822cfef700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8228139500 0x7f82281d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51408/0 (socket says 172.21.15.76:51408) 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822cfef700 1 -- 172.21.15.76:0/1854350817 learned_addr learned my addr 172.21.15.76:0/1854350817 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376378892 0 0) 0x7f822809b0a0 con 0x7f8228139500 2024-09-16T06:55:24.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): my global_id is 314169 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82080035d0 con 0x7f8228139500 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457444101 0 0) 0x7f8228098990 con 0x7f8228097b60 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): my global_id is 284458 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822809b0a0 con 0x7f8228097b60 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223627894 0 0) 0x7f82080035d0 con 0x7f8228139500 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8228098990 con 0x7f8228139500 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8220004460 con 0x7f8228139500 2024-09-16T06:55:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): dump: 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3706340622 0 0) 0x7f8228098990 con 0x7f8228139500 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6790/0 conn(0x7f8228099090 legacy=0x7f8228070c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.189:6789/0 conn(0x7f8228097b60 legacy=0x7f8228070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 monclient: found mon.a 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822809a540 con 0x7f8228139500 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 5 monclient: authenticate success, global_id 314169 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _renew_subs 2024-09-16T06:55:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8228071c40 con 0x7f8228139500 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _renew_subs 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8228071330 con 0x7f8228139500 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8220004740 con 0x7f8228139500 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8220004ad0 con 0x7f8228139500 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f8226ffd700 4 set_mon_vals no callback set 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient: dump: 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f820c006590 con 0x7f820c002830 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678603796 0 0) 0x7f820c006590 con 0x7f820c002830 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 10 monclient(hunting): my global_id is 314172 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.037+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228098990 con 0x7f820c002830 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8220004cb0 con 0x7f8228139500 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8220044120 con 0x7f8228139500 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2436166223 0 0) 0x7f8228098990 con 0x7f820c002830 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f820c006590 con 0x7f820c002830 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8218001520 con 0x7f820c002830 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 10 monclient: dump: 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.059 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f821f7fe700 10 monclient: get_auth_request con 0x7f8208036240 auth_method 0 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3790477549 0 0) 0x7f820c006590 con 0x7f820c002830 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f820c00a190 con 0x7f820c002830 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8218001a90 con 0x7f820c002830 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6789/0 conn(0x7f820c002830 legacy=0x7f820c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.041+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6789/0 conn(0x7f820c002830 legacy=0x7f820c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:24.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:24.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f820c009e50 con 0x7f820c009170 2024-09-16T06:55:24.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212172278 0 0) 0x7f820c009e50 con 0x7f820c009170 2024-09-16T06:55:24.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient(hunting): my global_id is 314175 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f820c006590 con 0x7f820c009170 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 559430022 0 0) 0x7f820c006590 con 0x7f820c009170 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f820c009e50 con 0x7f820c009170 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8218003840 con 0x7f820c009170 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient: dump: 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3381328777 0 0) 0x7f820c009e50 con 0x7f820c009170 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f820c00c340 con 0x7f820c009170 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4035 (unknown 0 0 2859771438) 0x7f82180046f0 con 0x7f820c009170 2024-09-16T06:55:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6789/0 conn(0x7f820c009170 legacy=0x7f820c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f82257fa700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6789/0 conn(0x7f820c009170 legacy=0x7f820c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 152, 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:24.065 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.066 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.067 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:24.068 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.069 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:24.070 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.071 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.073 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.075 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:24.076 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8208036240 legacy=0x7f8208038700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 10 monclient: shutdown 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 >> v1:172.21.15.76:6789/0 conn(0x7f8228139500 legacy=0x7f82281d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822cfef700 1 -- 172.21.15.76:0/1854350817 reap_dead start 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822cfef700 1 -- 172.21.15.76:0/1854350817 reap_dead start 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 shutdown_connections 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 >> 172.21.15.76:0/1854350817 conn(0x7f82280a0030 msgr2=0x7f82280a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 shutdown_connections 2024-09-16T06:55:24.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.053+0000 7f822d7f0700 1 -- 172.21.15.76:0/1854350817 wait complete. 2024-09-16T06:55:24.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:55:24.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:55:24.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 10 monclient: get_monmap_and_config 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 10 monclient: build_initial_monmap 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 10 monclient: monmap: 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:24.203003+0000 2024-09-16T06:55:24.197 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:24.203003+0000 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:24.198 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 1 Processor -- start 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 1 -- start start 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.197+0000 7f0dcdb23700 10 monclient: init 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-a con 0x7f0dc809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-b con 0x7f0dc8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-c con 0x7f0dc8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:24.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc809d560 con 0x7f0dc809ed60 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc809f140 con 0x7f0dc8099030 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc809d790 con 0x7f0dc8098be0 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient(hunting): authenticate will time out at 1871.336348s 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc7fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f0dc8098be0 legacy=0x7f0dc81374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dccb21700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0dc809ed60 0x7f0dc80984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51424/0 (socket says 172.21.15.76:51424) 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dccb21700 1 -- 172.21.15.76:0/4081286607 learned_addr learned my addr 172.21.15.76:0/4081286607 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754242352 0 0) 0x7f0dc809f140 con 0x7f0dc8099030 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): my global_id is 284461 2024-09-16T06:55:24.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db00035d0 con 0x7f0dc8099030 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984204269 0 0) 0x7f0dc809d560 con 0x7f0dc809ed60 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient(hunting): my global_id is 314178 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc809f140 con 0x7f0dc809ed60 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157452413 0 0) 0x7f0db00035d0 con 0x7f0dc8099030 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 >> v1:172.21.15.76:6790/0 conn(0x7f0dc8098be0 legacy=0x7f0dc81374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 >> v1:172.21.15.76:6789/0 conn(0x7f0dc809ed60 legacy=0x7f0dc80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 monclient: found mon.noname-c 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:55:24.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc809f620 con 0x7f0dc8099030 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 5 monclient: authenticate success, global_id 284461 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc00018c0 con 0x7f0dc8099030 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: dump: 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dc0001ba0 con 0x7f0dc8099030 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 1 -- 172.21.15.76:0/4081286607 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc0001f30 con 0x7f0dc8099030 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 4 set_mon_vals no callback set 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:24.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dc77fe700 10 monclient: dump: 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: shutdown 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- 172.21.15.76:0/4081286607 >> v1:172.21.15.189:6789/0 conn(0x7f0dc8099030 legacy=0x7f0dc813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- 172.21.15.76:0/4081286607 shutdown_connections 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- 172.21.15.76:0/4081286607 >> 172.21.15.76:0/4081286607 conn(0x7f0dc80a0030 msgr2=0x7f0dc80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- 172.21.15.76:0/4081286607 shutdown_connections 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 -- 172.21.15.76:0/4081286607 wait complete. 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: build_initial_monmap 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 10 monclient: monmap: 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:24.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:24.209 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.201+0000 7f0dcdb23700 1 Processor -- start 2024-09-16T06:55:24.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- start start 2024-09-16T06:55:24.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: init 2024-09-16T06:55:24.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-c con 0x7f0dc8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-b con 0x7f0dc8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient(hunting): picked mon.noname-a con 0x7f0dc809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80986c0 con 0x7f0dc809ed60 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80a12e0 con 0x7f0dc8099030 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc809a540 con 0x7f0dc8098be0 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient(hunting): authenticate will time out at 1871.340720s 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dccb21700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f0dc8098be0 legacy=0x7f0dc8096330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcd322700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0dc809ed60 0x7f0dc81cdea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51434/0 (socket says 172.21.15.76:51434) 2024-09-16T06:55:24.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcd322700 1 -- 172.21.15.76:0/780120030 learned_addr learned my addr 172.21.15.76:0/780120030 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950323369 0 0) 0x7f0dc809f620 con 0x7f0dc809ed60 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): my global_id is 314181 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40035d0 con 0x7f0dc809ed60 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709749727 0 0) 0x7f0dc80a12e0 con 0x7f0dc8099030 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): my global_id is 284464 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc809f620 con 0x7f0dc8099030 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817655970 0 0) 0x7f0db40035d0 con 0x7f0dc809ed60 2024-09-16T06:55:24.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dc80a12e0 con 0x7f0dc809ed60 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc00016e0 con 0x7f0dc809ed60 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): dump: 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.215 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3108890669 0 0) 0x7f0dc80a12e0 con 0x7f0dc809ed60 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.76:6790/0 conn(0x7f0dc8098be0 legacy=0x7f0dc8096330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6789/0 conn(0x7f0dc8099030 legacy=0x7f0dc81ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 monclient: found mon.a 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc8090ae0 con 0x7f0dc809ed60 2024-09-16T06:55:24.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 5 monclient: authenticate success, global_id 314181 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _renew_subs 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc808e9f0 con 0x7f0dc809ed60 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _renew_subs 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc81c5680 con 0x7f0dc809ed60 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dc0003cd0 con 0x7f0dc809ed60 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dc00014c0 con 0x7f0dc809ed60 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc6ffd700 4 set_mon_vals no callback set 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient: dump: 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.218 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0da8006590 con 0x7f0da8002830 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0dc0004120 con 0x7f0dc809ed60 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0dc0044140 con 0x7f0dc809ed60 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304216288 0 0) 0x7f0da8006590 con 0x7f0da8002830 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc7fff700 10 monclient: get_auth_request con 0x7f0db4036180 auth_method 0 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.205+0000 7f0dc57fa700 10 monclient(hunting): my global_id is 284467 2024-09-16T06:55:24.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc80a12e0 con 0x7f0da8002830 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 9542952 0 0) 0x7f0dc80a12e0 con 0x7f0da8002830 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0da8006590 con 0x7f0da8002830 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dbc001740 con 0x7f0da8002830 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 10 monclient: dump: 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.220 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 922676111 0 0) 0x7f0da8006590 con 0x7f0da8002830 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0da800a190 con 0x7f0da8002830 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0dbc0019f0 con 0x7f0da8002830 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:24.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6789/0 conn(0x7f0da8002830 legacy=0x7f0da8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.209+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6789/0 conn(0x7f0da8002830 legacy=0x7f0da8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:24.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:24.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0da8009e10 con 0x7f0da8009150 2024-09-16T06:55:24.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590481643 0 0) 0x7f0da8009e10 con 0x7f0da8009150 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient(hunting): my global_id is 284470 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8006590 con 0x7f0da8009150 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2981783171 0 0) 0x7f0da8006590 con 0x7f0da8009150 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0da8009e10 con 0x7f0da8009150 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0dbc004af0 con 0x7f0da8009150 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:24.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient: dump: 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1722709183 0 0) 0x7f0da8009e10 con 0x7f0da8009150 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0da800c340 con 0x7f0da8009150 2024-09-16T06:55:24.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4203 (unknown 0 0 740157985) 0x7f0dbc004d00 con 0x7f0da8009150 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6789/0 conn(0x7f0da8009150 legacy=0x7f0da80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dc57fa700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6789/0 conn(0x7f0da8009150 legacy=0x7f0da80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 152, 2024-09-16T06:55:24.226 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.227 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.229 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:24.230 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:24.231 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:24.232 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:24.235 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.236 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:24.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:24.238 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:24.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0db4036180 legacy=0x7f0db4038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 10 monclient: shutdown 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 >> v1:172.21.15.76:6789/0 conn(0x7f0dc809ed60 legacy=0x7f0dc81cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcd322700 1 -- 172.21.15.76:0/780120030 reap_dead start 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcd322700 1 -- 172.21.15.76:0/780120030 reap_dead start 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 shutdown_connections 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 >> 172.21.15.76:0/780120030 conn(0x7f0dc80a0030 msgr2=0x7f0dc806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 shutdown_connections 2024-09-16T06:55:24.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:24.221+0000 7f0dcdb23700 1 -- 172.21.15.76:0/780120030 wait complete. 2024-09-16T06:55:24.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:55:24.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:55:44.241 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:55:44.242 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:55:44.242 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:55:44.242 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:55:44.248 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:55:44.248 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:55:44.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:44.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: get_monmap_and_config 2024-09-16T06:55:44.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: monmap: 2024-09-16T06:55:44.357 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.357 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:44.362565+0000 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:44.362565+0000 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 1 Processor -- start 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 1 -- start start 2024-09-16T06:55:44.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: init 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-b con 0x7f507409fac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-a con 0x7f5074134000 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-c con 0x7f5074137b10 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740a11a0 con 0x7f5074134000 2024-09-16T06:55:44.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507409b890 con 0x7f5074137b10 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f5078be3700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f507409fac0 legacy=0x7f507409cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507409bf60 con 0x7f507409fac0 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.357+0000 7f507ae47700 10 monclient(hunting): authenticate will time out at 1891.495971s 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f5073fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5074134000 0x7f50741363f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42136/0 (socket says 172.21.15.76:42136) 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f5073fff700 1 -- 172.21.15.76:0/2429910219 learned_addr learned my addr 172.21.15.76:0/2429910219 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296490415 0 0) 0x7f50740a11a0 con 0x7f5074134000 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient(hunting): my global_id is 314184 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50640035d0 con 0x7f5074134000 2024-09-16T06:55:44.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124713289 0 0) 0x7f50640035d0 con 0x7f5074134000 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 >> v1:172.21.15.76:6790/0 conn(0x7f507409fac0 legacy=0x7f507409cc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 >> v1:172.21.15.189:6789/0 conn(0x7f5074137b10 legacy=0x7f5074139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 monclient: found mon.noname-a 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507409c240 con 0x7f5074134000 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5060001610 con 0x7f5074134000 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:44.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: dump: 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 5 monclient: authenticate success, global_id 314184 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5060001930 con 0x7f5074134000 2024-09-16T06:55:44.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 4 set_mon_vals no callback set 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 1 -- 172.21.15.76:0/2429910219 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5060001cc0 con 0x7f5074134000 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f50737fe700 10 monclient: dump: 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: shutdown 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- 172.21.15.76:0/2429910219 >> v1:172.21.15.76:6789/0 conn(0x7f5074134000 legacy=0x7f50741363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- 172.21.15.76:0/2429910219 shutdown_connections 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- 172.21.15.76:0/2429910219 >> 172.21.15.76:0/2429910219 conn(0x7f50740a3e50 msgr2=0x7f50740a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- 172.21.15.76:0/2429910219 shutdown_connections 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- 172.21.15.76:0/2429910219 wait complete. 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: monmap: 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:44.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 Processor -- start 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- start start 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: init 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-b con 0x7f507409fac0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-a con 0x7f5074134000 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient(hunting): picked mon.noname-c con 0x7f5074137b10 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5074072450 con 0x7f5074134000 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5074137840 con 0x7f507409fac0 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50741372f0 con 0x7f5074137b10 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f507ae47700 10 monclient(hunting): authenticate will time out at 1891.499585s 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f5073fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5074134000 0x7f50741caed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42164/0 (socket says 172.21.15.76:42164) 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.361+0000 7f5073fff700 1 -- 172.21.15.76:0/1276515179 learned_addr learned my addr 172.21.15.76:0/1276515179 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636276917 0 0) 0x7f5074072450 con 0x7f5074134000 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): my global_id is 314187 2024-09-16T06:55:44.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50580035d0 con 0x7f5074134000 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867419348 0 0) 0x7f5074137840 con 0x7f507409fac0 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): my global_id is 284476 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5074072450 con 0x7f507409fac0 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956144732 0 0) 0x7f50580035d0 con 0x7f5074134000 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5074137840 con 0x7f5074134000 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f50600015d0 con 0x7f5074134000 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:44.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): dump: 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637084325 0 0) 0x7f5074137840 con 0x7f5074134000 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 >> v1:172.21.15.76:6790/0 conn(0x7f5074137b10 legacy=0x7f50741ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 >> v1:172.21.15.189:6789/0 conn(0x7f507409fac0 legacy=0x7f50741c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 monclient: found mon.a 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507411e670 con 0x7f5074134000 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 5 monclient: authenticate success, global_id 314187 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _renew_subs 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507411ddd0 con 0x7f5074134000 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _renew_subs 2024-09-16T06:55:44.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5060004cf0 con 0x7f5074134000 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f507411ea90 con 0x7f5074134000 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5060005000 con 0x7f5074134000 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 10 monclient: dump: 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.375 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f5072ffd700 4 set_mon_vals no callback set 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f506000fa70 con 0x7f5074134000 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5060044080 con 0x7f5074134000 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f5078be3700 10 monclient: get_auth_request con 0x7f5058036220 auth_method 0 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.365+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50740678b0 con 0x7f5074134000 2024-09-16T06:55:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.373+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5060014a80 con 0x7f5074134000 2024-09-16T06:55:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.373+0000 7f50717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.373+0000 7f50717fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f507ae47700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f507ae47700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f507ae47700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5074069620 con 0x7f5074134000 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f50717fa700 1 -- 172.21.15.76:0/1276515179 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 212875668) 0x7f5060012ef0 con 0x7f5074134000 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f50717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:44.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.613+0000 7f50717fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:44.614 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5058036220 legacy=0x7f50580386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 10 monclient: shutdown 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 >> v1:172.21.15.76:6789/0 conn(0x7f5074134000 legacy=0x7f50741caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 shutdown_connections 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 >> 172.21.15.76:0/1276515179 conn(0x7f50740a3e50 msgr2=0x7f5074136a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 shutdown_connections 2024-09-16T06:55:44.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.617+0000 7f507ae47700 1 -- 172.21.15.76:0/1276515179 wait complete. 2024-09-16T06:55:44.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:55:44.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:55:44.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:55:44.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 10 monclient: get_monmap_and_config 2024-09-16T06:55:44.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 10 monclient: monmap: 2024-09-16T06:55:44.745 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.745 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:44.750946+0000 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:44.750946+0000 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:44.746 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 1 Processor -- start 2024-09-16T06:55:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 1 -- start start 2024-09-16T06:55:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.744+0000 7f14f9ce5700 10 monclient: init 2024-09-16T06:55:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-a con 0x7f14f409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-c con 0x7f14f409c460 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-b con 0x7f14f409fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f409a040 con 0x7f14f409f230 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f409ac80 con 0x7f14f409c460 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f409def0 con 0x7f14f409fd20 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient(hunting): authenticate will time out at 1891.884476s 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f14f409fd20 0x7f14f4142160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60258/0 (socket says 172.21.15.76:60258) 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f3fff700 1 -- 172.21.15.76:0/2709271548 learned_addr learned my addr 172.21.15.76:0/2709271548 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854180970 0 0) 0x7f14f409def0 con 0x7f14f409fd20 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): my global_id is 324101 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e00035d0 con 0x7f14f409fd20 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656744949 0 0) 0x7f14f409a040 con 0x7f14f409f230 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): my global_id is 314190 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f409def0 con 0x7f14f409f230 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179814601 0 0) 0x7f14e4001490 con 0x7f14f409c460 2024-09-16T06:55:44.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient(hunting): my global_id is 284479 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f409a040 con 0x7f14f409c460 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818875421 0 0) 0x7f14e00035d0 con 0x7f14f409fd20 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 >> v1:172.21.15.189:6789/0 conn(0x7f14f409c460 legacy=0x7f14f413ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 >> v1:172.21.15.76:6789/0 conn(0x7f14f409f230 legacy=0x7f14f409f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 monclient: found mon.noname-b 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:44.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f409a410 con 0x7f14f409fd20 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14ec001610 con 0x7f14f409fd20 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: dump: 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.754 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 5 monclient: authenticate success, global_id 324101 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ec0018f0 con 0x7f14f409fd20 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 1 -- 172.21.15.76:0/2709271548 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14ec001c80 con 0x7f14f409fd20 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 4 set_mon_vals no callback set 2024-09-16T06:55:44.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f27fc700 10 monclient: dump: 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: shutdown 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- 172.21.15.76:0/2709271548 >> v1:172.21.15.76:6790/0 conn(0x7f14f409fd20 legacy=0x7f14f4142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- 172.21.15.76:0/2709271548 shutdown_connections 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- 172.21.15.76:0/2709271548 >> 172.21.15.76:0/2709271548 conn(0x7f14f40a3e50 msgr2=0x7f14f40a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- 172.21.15.76:0/2709271548 shutdown_connections 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 1 -- 172.21.15.76:0/2709271548 wait complete. 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.748+0000 7f14f9ce5700 10 monclient: monmap: 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 Processor -- start 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- start start 2024-09-16T06:55:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: init 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-c con 0x7f14f409c460 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-b con 0x7f14f409f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient(hunting): picked mon.noname-a con 0x7f14f409fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f4138a30 con 0x7f14f409fd20 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f4072450 con 0x7f14f409f230 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f40a2ca0 con 0x7f14f409c460 2024-09-16T06:55:44.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient(hunting): authenticate will time out at 1891.889322s 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f14f409c460 0x7f14f4134ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60264/0 (socket says 172.21.15.76:60264) 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f37fe700 1 -- 172.21.15.76:0/3593943732 learned_addr learned my addr 172.21.15.76:0/3593943732 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f14f409fd20 0x7f14f41d6760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42180/0 (socket says 172.21.15.76:42180) 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636842978 0 0) 0x7f14f4138a30 con 0x7f14f409fd20 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): my global_id is 314193 2024-09-16T06:55:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d40035d0 con 0x7f14f409fd20 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178910962 0 0) 0x7f14f40a2ca0 con 0x7f14f409c460 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): my global_id is 324104 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f4138a30 con 0x7f14f409c460 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012595502 0 0) 0x7f14f4072450 con 0x7f14f409f230 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): my global_id is 284482 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f40a2ca0 con 0x7f14f409f230 2024-09-16T06:55:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588753292 0 0) 0x7f14d40035d0 con 0x7f14f409fd20 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14f4072450 con 0x7f14f409fd20 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14ec0015d0 con 0x7f14f409fd20 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): dump: 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 479494170 0 0) 0x7f14f4138a30 con 0x7f14f409c460 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d40035d0 con 0x7f14f409c460 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14dc001830 con 0x7f14f409c460 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient(hunting): dump: 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.768 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45978593 0 0) 0x7f14f4072450 con 0x7f14f409fd20 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6790/0 conn(0x7f14f409c460 legacy=0x7f14f4134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.189:6789/0 conn(0x7f14f409f230 legacy=0x7f14f41355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 monclient: found mon.a 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f41362d0 con 0x7f14f409fd20 2024-09-16T06:55:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 5 monclient: authenticate success, global_id 314193 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _renew_subs 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f4142b20 con 0x7f14f409fd20 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _renew_subs 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f4136a10 con 0x7f14f409fd20 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ec004b40 con 0x7f14f409fd20 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14ec004e50 con 0x7f14f409fd20 2024-09-16T06:55:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14f1ffb700 4 set_mon_vals no callback set 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.752+0000 7f14ebfff700 10 monclient: dump: 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.771 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14f9ce5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f14d8006590 con 0x7f14d8002830 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f14ec00f8c0 con 0x7f14f409fd20 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f14ec043f20 con 0x7f14f409fd20 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430090463 0 0) 0x7f14d8006590 con 0x7f14d8002830 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient(hunting): my global_id is 314196 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f4072450 con 0x7f14d8002830 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14f2ffd700 10 monclient: get_auth_request con 0x7f14d4036430 auth_method 0 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2471105055 0 0) 0x7f14f4072450 con 0x7f14d8002830 2024-09-16T06:55:44.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d8006590 con 0x7f14d8002830 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14dc003840 con 0x7f14d8002830 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient: dump: 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.773 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2097341110 0 0) 0x7f14d8006590 con 0x7f14d8002830 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14d800a190 con 0x7f14d8002830 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f14dc005870 con 0x7f14d8002830 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6789/0 conn(0x7f14d8002830 legacy=0x7f14d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.756+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6789/0 conn(0x7f14d8002830 legacy=0x7f14d8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:44.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f14d8009e30 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428824658 0 0) 0x7f14d8009e30 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient(hunting): my global_id is 314199 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8006590 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2032752562 0 0) 0x7f14d8006590 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d8009e30 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14dc003f90 con 0x7f14d8009150 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient: dump: 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3176466260 0 0) 0x7f14d8009e30 con 0x7f14d8009150 2024-09-16T06:55:44.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f14d800c340 con 0x7f14d8009150 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1982732272) 0x7f14dc004210 con 0x7f14d8009150 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6789/0 conn(0x7f14d8009150 legacy=0x7f14d80095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14ebfff700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6789/0 conn(0x7f14d8009150 legacy=0x7f14d80095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:44.778 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.779 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.780 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.781 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:44.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:44.787 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.788 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.790 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:44.790 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.189:6812/3098810674 conn(0x7f14d4036430 legacy=0x7f14d40388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 10 monclient: shutdown 2024-09-16T06:55:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 >> v1:172.21.15.76:6789/0 conn(0x7f14f409fd20 legacy=0x7f14f41d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f3fff700 1 -- 172.21.15.76:0/3593943732 reap_dead start 2024-09-16T06:55:44.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f3fff700 1 -- 172.21.15.76:0/3593943732 reap_dead start 2024-09-16T06:55:44.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 shutdown_connections 2024-09-16T06:55:44.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 >> 172.21.15.76:0/3593943732 conn(0x7f14f40a3e50 msgr2=0x7f14f409c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 shutdown_connections 2024-09-16T06:55:44.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.768+0000 7f14f9ce5700 1 -- 172.21.15.76:0/3593943732 wait complete. 2024-09-16T06:55:44.792 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:44.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:55:44.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: get_monmap_and_config 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: monmap: 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:44.917904+0000 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:44.917904+0000 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.912 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.913 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.913 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:44.913 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:44.913 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 1 Processor -- start 2024-09-16T06:55:44.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 1 -- start start 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: init 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-c con 0x7f64c00973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-a con 0x7f64c0097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-b con 0x7f64c01400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0099350 con 0x7f64c0097eb0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c009f690 con 0x7f64c00973c0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0099bf0 con 0x7f64c01400d0 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c6f23700 10 monclient(hunting): authenticate will time out at 1892.051153s 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c5720700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f64c0097eb0 0x7f64c013e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42206/0 (socket says 172.21.15.76:42206) 2024-09-16T06:55:44.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c5720700 1 -- 172.21.15.76:0/327012586 learned_addr learned my addr 172.21.15.76:0/327012586 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117707450 0 0) 0x7f64c0099350 con 0x7f64c0097eb0 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): my global_id is 314202 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac0035d0 con 0x7f64c0097eb0 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478988751 0 0) 0x7f64c0099bf0 con 0x7f64c01400d0 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 10 monclient(hunting): my global_id is 324107 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.912+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0099350 con 0x7f64c01400d0 2024-09-16T06:55:44.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513017976 0 0) 0x7f64c009f690 con 0x7f64c00973c0 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient(hunting): my global_id is 284485 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0099bf0 con 0x7f64c00973c0 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1120222764 0 0) 0x7f64ac0035d0 con 0x7f64c0097eb0 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 >> v1:172.21.15.76:6790/0 conn(0x7f64c01400d0 legacy=0x7f64c01424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 >> v1:172.21.15.189:6789/0 conn(0x7f64c00973c0 legacy=0x7f64c00977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 monclient: found mon.noname-a 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0098360 con 0x7f64c0097eb0 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64bc001610 con 0x7f64c0097eb0 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 5 monclient: authenticate success, global_id 314202 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: dump: 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.920 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64bc001830 con 0x7f64c0097eb0 2024-09-16T06:55:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 4 set_mon_vals no callback set 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 1 -- 172.21.15.76:0/327012586 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64bc001bc0 con 0x7f64c0097eb0 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c4f1f700 10 monclient: dump: 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: shutdown 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- 172.21.15.76:0/327012586 >> v1:172.21.15.76:6789/0 conn(0x7f64c0097eb0 legacy=0x7f64c013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- 172.21.15.76:0/327012586 shutdown_connections 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- 172.21.15.76:0/327012586 >> 172.21.15.76:0/327012586 conn(0x7f64c00a0030 msgr2=0x7f64c00a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- 172.21.15.76:0/327012586 shutdown_connections 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- 172.21.15.76:0/327012586 wait complete. 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: build_initial_monmap 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: monmap: 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 Processor -- start 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- start start 2024-09-16T06:55:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: init 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-b con 0x7f64c00973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-a con 0x7f64c0097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient(hunting): picked mon.noname-c con 0x7f64c01400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0072660 con 0x7f64c0097eb0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0071960 con 0x7f64c00973c0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c01cf070 con 0x7f64c01400d0 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c6f23700 10 monclient(hunting): authenticate will time out at 1892.055760s 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c5720700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f64c0097eb0 0x7f64c01ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42218/0 (socket says 172.21.15.76:42218) 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.916+0000 7f64c5720700 1 -- 172.21.15.76:0/1874171438 learned_addr learned my addr 172.21.15.76:0/1874171438 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576019832 0 0) 0x7f64c0072660 con 0x7f64c0097eb0 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): my global_id is 314205 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a40035d0 con 0x7f64c0097eb0 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121439985 0 0) 0x7f64c01cf070 con 0x7f64c01400d0 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): my global_id is 324110 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0072660 con 0x7f64c01400d0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462934759 0 0) 0x7f64c0071960 con 0x7f64c00973c0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): my global_id is 284488 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c01cf070 con 0x7f64c00973c0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610617364 0 0) 0x7f64a40035d0 con 0x7f64c0097eb0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c0071960 con 0x7f64c0097eb0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64bc001510 con 0x7f64c0097eb0 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): dump: 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847966215 0 0) 0x7f64c0072660 con 0x7f64c01400d0 2024-09-16T06:55:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a40035d0 con 0x7f64c01400d0 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64b8001c30 con 0x7f64c01400d0 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): dump: 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.934 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 148308241 0 0) 0x7f64c0071960 con 0x7f64c0097eb0 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6790/0 conn(0x7f64c01400d0 legacy=0x7f64c01d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.189:6789/0 conn(0x7f64c00973c0 legacy=0x7f64c01ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 monclient: found mon.a 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c013ecb0 con 0x7f64c0097eb0 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: _finish_auth 0 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 5 monclient: authenticate success, global_id 314205 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 10 monclient: _renew_subs 2024-09-16T06:55:44.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c013e040 con 0x7f64c0097eb0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 10 monclient: _renew_subs 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c006c410 con 0x7f64c0097eb0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64bc004b40 con 0x7f64c0097eb0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64bc004ed0 con 0x7f64c0097eb0 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b7fff700 4 set_mon_vals no callback set 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: dump: 2024-09-16T06:55:44.936 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f64a8006590 con 0x7f64a8002830 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f64bc005030 con 0x7f64c0097eb0 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f64bc043f80 con 0x7f64c0097eb0 2024-09-16T06:55:44.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951970447 0 0) 0x7f64a8006590 con 0x7f64a8002830 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient(hunting): my global_id is 324113 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c0071960 con 0x7f64a8002830 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64c6722700 10 monclient: get_auth_request con 0x7f64a4036170 auth_method 0 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 379995960 0 0) 0x7f64c0071960 con 0x7f64a8002830 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a8006590 con 0x7f64a8002830 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64b0001d50 con 0x7f64a8002830 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 10 monclient: dump: 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.938 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2358855984 0 0) 0x7f64a8006590 con 0x7f64a8002830 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.920+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64a800a190 con 0x7f64a8002830 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.924+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f64b00059a0 con 0x7f64a8002830 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.924+0000 7f64b67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.924+0000 7f64b67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:44.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.924+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6790/0 conn(0x7f64a8002830 legacy=0x7f64a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.924+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6790/0 conn(0x7f64a8002830 legacy=0x7f64a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:44.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:44.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f64a8009e30 con 0x7f64a8009150 2024-09-16T06:55:44.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642653389 0 0) 0x7f64a8009e30 con 0x7f64a8009150 2024-09-16T06:55:44.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:55:44.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient(hunting): my global_id is 324116 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8006590 con 0x7f64a8009150 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1627430204 0 0) 0x7f64a8006590 con 0x7f64a8009150 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a8009e30 con 0x7f64a8009150 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f64b0003ec0 con 0x7f64a8009150 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient: dump: 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:44.942 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3084886035 0 0) 0x7f64a8009e30 con 0x7f64a8009150 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f64a800c340 con 0x7f64a8009150 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 1757189580) 0x7f64b0005bb0 con 0x7f64a8009150 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6790/0 conn(0x7f64a8009150 legacy=0x7f64a80095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64b67fc700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6790/0 conn(0x7f64a8009150 legacy=0x7f64a80095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:44.944 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.945 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.946 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:44.947 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:44.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:44.949 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:44.950 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:44.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:44.955 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:44.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.189:6812/3098810674 conn(0x7f64a4036170 legacy=0x7f64a4038630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 10 monclient: shutdown 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 >> v1:172.21.15.76:6789/0 conn(0x7f64c0097eb0 legacy=0x7f64c01ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6722700 1 -- 172.21.15.76:0/1874171438 reap_dead start 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6722700 1 -- 172.21.15.76:0/1874171438 reap_dead start 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 shutdown_connections 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 >> 172.21.15.76:0/1874171438 conn(0x7f64c00a0030 msgr2=0x7f64c009c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 shutdown_connections 2024-09-16T06:55:44.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:44.932+0000 7f64c6f23700 1 -- 172.21.15.76:0/1874171438 wait complete. 2024-09-16T06:55:44.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:44.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:55:44.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: get_monmap_and_config 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: build_initial_monmap 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: monmap: 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:45.080962+0000 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:45.080962+0000 2024-09-16T06:55:45.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:45.076 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.076 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:45.076 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:45.076 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:45.076 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 1 Processor -- start 2024-09-16T06:55:45.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 1 -- start start 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: init 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient(hunting): picked mon.noname-b con 0x7fa78c0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient(hunting): picked mon.noname-c con 0x7fa78c097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient(hunting): picked mon.noname-a con 0x7fa78c1400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c099350 con 0x7fa78c1400d0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c09f690 con 0x7fa78c097eb0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c099bf0 con 0x7fa78c0973c0 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient(hunting): authenticate will time out at 1892.214389s 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa78c0973c0 0x7fa78c0977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60284/0 (socket says 172.21.15.76:60284) 2024-09-16T06:55:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa790e5f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa78c1400d0 0x7fa78c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42222/0 (socket says 172.21.15.76:42222) 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78bfff700 1 -- 172.21.15.76:0/1387844724 learned_addr learned my addr 172.21.15.76:0/1387844724 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264618026 0 0) 0x7fa78c099bf0 con 0x7fa78c0973c0 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): my global_id is 324119 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7740035d0 con 0x7fa78c0973c0 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375615941 0 0) 0x7fa78c099350 con 0x7fa78c1400d0 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): my global_id is 314208 2024-09-16T06:55:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c099bf0 con 0x7fa78c1400d0 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950996549 0 0) 0x7fa78c09f690 con 0x7fa78c097eb0 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient(hunting): my global_id is 284491 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c099350 con 0x7fa78c097eb0 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3267416881 0 0) 0x7fa7740035d0 con 0x7fa78c0973c0 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 >> v1:172.21.15.189:6789/0 conn(0x7fa78c097eb0 legacy=0x7fa78c13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 >> v1:172.21.15.76:6789/0 conn(0x7fa78c1400d0 legacy=0x7fa78c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 monclient: found mon.noname-b 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c098360 con 0x7fa78c0973c0 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: _finish_auth 0 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa780001610 con 0x7fa78c0973c0 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: dump: 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.084 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 5 monclient: authenticate success, global_id 324119 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7800018f0 con 0x7fa78c0973c0 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 1 -- 172.21.15.76:0/1387844724 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa780001c80 con 0x7fa78c0973c0 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:45.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa791660700 4 set_mon_vals no callback set 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.076+0000 7fa78affd700 10 monclient: dump: 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: shutdown 2024-09-16T06:55:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/1387844724 >> v1:172.21.15.76:6790/0 conn(0x7fa78c0973c0 legacy=0x7fa78c0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/1387844724 shutdown_connections 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/1387844724 >> 172.21.15.76:0/1387844724 conn(0x7fa78c0a0030 msgr2=0x7fa78c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/1387844724 shutdown_connections 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/1387844724 wait complete. 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: build_initial_monmap 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: monmap: 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 Processor -- start 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- start start 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: init 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient(hunting): picked mon.noname-b con 0x7fa78c0973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient(hunting): picked mon.noname-a con 0x7fa78c097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient(hunting): picked mon.noname-c con 0x7fa78c1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c072660 con 0x7fa78c097eb0 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c09a540 con 0x7fa78c0973c0 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c13fb70 con 0x7fa78c1400d0 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient(hunting): authenticate will time out at 1892.218560s 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa78b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa78c097eb0 0x7fa78c1d3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42236/0 (socket says 172.21.15.76:42236) 2024-09-16T06:55:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa78b7fe700 1 -- 172.21.15.76:0/2697717177 learned_addr learned my addr 172.21.15.76:0/2697717177 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88111333 0 0) 0x7fa78c13fb70 con 0x7fa78c1400d0 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): my global_id is 324122 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7780035d0 con 0x7fa78c1400d0 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036911142 0 0) 0x7fa78c072660 con 0x7fa78c097eb0 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): my global_id is 314211 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c13fb70 con 0x7fa78c097eb0 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976875353 0 0) 0x7fa78c09a540 con 0x7fa78c0973c0 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): my global_id is 284494 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c072660 con 0x7fa78c0973c0 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658070860 0 0) 0x7fa78c13fb70 con 0x7fa78c097eb0 2024-09-16T06:55:45.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa78c09a540 con 0x7fa78c097eb0 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa77c001cf0 con 0x7fa78c097eb0 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): dump: 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666449523 0 0) 0x7fa7780035d0 con 0x7fa78c1400d0 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa78c13fb70 con 0x7fa78c1400d0 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa784001830 con 0x7fa78c1400d0 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient(hunting): dump: 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259910508 0 0) 0x7fa78c13fb70 con 0x7fa78c1400d0 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6789/0 conn(0x7fa78c0973c0 legacy=0x7fa78c1ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.76:6789/0 conn(0x7fa78c097eb0 legacy=0x7fa78c1d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 monclient: found mon.c 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c06c6f0 con 0x7fa78c1400d0 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient: _finish_auth 0 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa788ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 5 monclient: authenticate success, global_id 324122 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: _renew_subs 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.080+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c13ea10 con 0x7fa78c1400d0 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa791660700 10 monclient: _renew_subs 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa791660700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c06d0b0 con 0x7fa78c1400d0 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa784001d00 con 0x7fa78c1400d0 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa784004a30 con 0x7fa78c1400d0 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa789ffb700 4 set_mon_vals no callback set 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: dump: 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.100 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa78400f4a0 con 0x7fa78c1400d0 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa784043a70 con 0x7fa78c1400d0 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa791660700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:45.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa76c006590 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa78bfff700 10 monclient: get_auth_request con 0x7fa778036260 auth_method 0 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201090277 0 0) 0x7fa76c006590 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient(hunting): my global_id is 284497 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c13fb70 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 740154823 0 0) 0x7fa78c13fb70 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76c006590 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa77c003c40 con 0x7fa76c002830 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 10 monclient: dump: 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3298982582 0 0) 0x7fa76c006590 con 0x7fa76c002830 2024-09-16T06:55:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.084+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa76c00a190 con 0x7fa76c002830 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.088+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa77c003f90 con 0x7fa76c002830 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.088+0000 7fa788ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.088+0000 7fa788ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.088+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6789/0 conn(0x7fa76c002830 legacy=0x7fa76c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.088+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6789/0 conn(0x7fa76c002830 legacy=0x7fa76c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.092+0000 7fa791660700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.092+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa76c009e30 con 0x7fa76c009150 2024-09-16T06:55:45.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184094502 0 0) 0x7fa76c009e30 con 0x7fa76c009150 2024-09-16T06:55:45.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient(hunting): my global_id is 284500 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c006590 con 0x7fa76c009150 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2139362844 0 0) 0x7fa76c006590 con 0x7fa76c009150 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa76c009e30 con 0x7fa76c009150 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa77c004510 con 0x7fa76c009150 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient: dump: 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.106 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 491407620 0 0) 0x7fa76c009e30 con 0x7fa76c009150 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa76c00c340 con 0x7fa76c009150 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 2735063730) 0x7fa77c0047f0 con 0x7fa76c009150 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6789/0 conn(0x7fa76c009150 legacy=0x7fa76c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa788ff9700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6789/0 conn(0x7fa76c009150 legacy=0x7fa76c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:45.109 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:45.110 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:45.111 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:45.112 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:45.114 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:45.115 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.116 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.117 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.118 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:45.119 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa778036260 legacy=0x7fa778038720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 10 monclient: shutdown 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 >> v1:172.21.15.76:6790/0 conn(0x7fa78c1400d0 legacy=0x7fa78c1d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa790e5f700 1 -- 172.21.15.76:0/2697717177 reap_dead start 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa790e5f700 1 -- 172.21.15.76:0/2697717177 reap_dead start 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 shutdown_connections 2024-09-16T06:55:45.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 >> 172.21.15.76:0/2697717177 conn(0x7fa78c0a0030 msgr2=0x7fa78c096220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:45.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 shutdown_connections 2024-09-16T06:55:45.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.096+0000 7fa791660700 1 -- 172.21.15.76:0/2697717177 wait complete. 2024-09-16T06:55:45.122 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:45.122 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:55:45.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:55:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: get_monmap_and_config 2024-09-16T06:55:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: monmap: 2024-09-16T06:55:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:45.239 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:45.244816+0000 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:45.244816+0000 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:45.240 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:45.241 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:45.241 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 1 Processor -- start 2024-09-16T06:55:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 1 -- start start 2024-09-16T06:55:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: init 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0d180973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0d18097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0d181400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d18099350 con 0x7f0d180973c0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1809f690 con 0x7f0d18097eb0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d18099bf0 con 0x7f0d181400d0 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient(hunting): authenticate will time out at 1892.378166s 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d177fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0d181400d0 0x7f0d181424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60300/0 (socket says 172.21.15.76:60300) 2024-09-16T06:55:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d16ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0d180973c0 0x7f0d180977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42248/0 (socket says 172.21.15.76:42248) 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d177fe700 1 -- 172.21.15.76:0/238108032 learned_addr learned my addr 172.21.15.76:0/238108032 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025072625 0 0) 0x7f0d18099350 con 0x7f0d180973c0 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): my global_id is 314214 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d000035d0 con 0x7f0d180973c0 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492114059 0 0) 0x7f0d18099bf0 con 0x7f0d181400d0 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): my global_id is 324125 2024-09-16T06:55:45.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d18099350 con 0x7f0d181400d0 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491167997 0 0) 0x7f0d1809f690 con 0x7f0d18097eb0 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient(hunting): my global_id is 284503 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d18099bf0 con 0x7f0d18097eb0 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1159663874 0 0) 0x7f0d000035d0 con 0x7f0d180973c0 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 >> v1:172.21.15.76:6790/0 conn(0x7f0d181400d0 legacy=0x7f0d181424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 >> v1:172.21.15.189:6789/0 conn(0x7f0d18097eb0 legacy=0x7f0d1813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 monclient: found mon.noname-a 2024-09-16T06:55:45.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18098360 con 0x7f0d180973c0 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: _finish_auth 0 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c001610 con 0x7f0d180973c0 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 5 monclient: authenticate success, global_id 314214 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: dump: 2024-09-16T06:55:45.261 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d0c001870 con 0x7f0d180973c0 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:45.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 1 -- 172.21.15.76:0/238108032 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c001c40 con 0x7f0d180973c0 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d17fff700 4 set_mon_vals no callback set 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.240+0000 7f0d15ffb700 10 monclient: dump: 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.263 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: shutdown 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- 172.21.15.76:0/238108032 >> v1:172.21.15.76:6789/0 conn(0x7f0d180973c0 legacy=0x7f0d180977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- 172.21.15.76:0/238108032 shutdown_connections 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- 172.21.15.76:0/238108032 >> 172.21.15.76:0/238108032 conn(0x7f0d180a0030 msgr2=0x7f0d180a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- 172.21.15.76:0/238108032 shutdown_connections 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- 172.21.15.76:0/238108032 wait complete. 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: build_initial_monmap 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: monmap: 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:45.264 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 Processor -- start 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- start start 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: init 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0d180973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0d18097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0d181400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d18072660 con 0x7f0d18097eb0 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1809e4a0 con 0x7f0d181400d0 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1809a540 con 0x7f0d180973c0 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d17fff700 10 monclient(hunting): authenticate will time out at 1892.383717s 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d16ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0d180973c0 0x7f0d1813c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60304/0 (socket says 172.21.15.76:60304) 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d167fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0d18097eb0 0x7f0d1813ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42260/0 (socket says 172.21.15.76:42260) 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.244+0000 7f0d16ffd700 1 -- 172.21.15.76:0/1557730178 learned_addr learned my addr 172.21.15.76:0/1557730178 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721097893 0 0) 0x7f0d1809a540 con 0x7f0d180973c0 2024-09-16T06:55:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): my global_id is 324128 2024-09-16T06:55:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d040035d0 con 0x7f0d180973c0 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221173022 0 0) 0x7f0d18072660 con 0x7f0d18097eb0 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): my global_id is 314217 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1809a540 con 0x7f0d18097eb0 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608322415 0 0) 0x7f0d1809e4a0 con 0x7f0d181400d0 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): my global_id is 284506 2024-09-16T06:55:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d18072660 con 0x7f0d181400d0 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839636489 0 0) 0x7f0d040035d0 con 0x7f0d180973c0 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d1809e4a0 con 0x7f0d180973c0 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c001bc0 con 0x7f0d180973c0 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1618340890 0 0) 0x7f0d1809a540 con 0x7f0d18097eb0 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d040035d0 con 0x7f0d18097eb0 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d08001c30 con 0x7f0d18097eb0 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient(hunting): dump: 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1579972779 0 0) 0x7f0d18072660 con 0x7f0d181400d0 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d1809a540 con 0x7f0d181400d0 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 695679933 0 0) 0x7f0d1809e4a0 con 0x7f0d180973c0 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 >> v1:172.21.15.189:6789/0 conn(0x7f0d181400d0 legacy=0x7f0d1813d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 >> v1:172.21.15.76:6789/0 conn(0x7f0d18097eb0 legacy=0x7f0d1813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 monclient: found mon.c 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1813ec20 con 0x7f0d180973c0 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 5 monclient: authenticate success, global_id 324128 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _renew_subs 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1806c410 con 0x7f0d180973c0 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _renew_subs 2024-09-16T06:55:45.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1813df40 con 0x7f0d180973c0 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d0c004c90 con 0x7f0d180973c0 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0d0c004fa0 con 0x7f0d180973c0 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d157fa700 4 set_mon_vals no callback set 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 10 monclient: dump: 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:45.273 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8005100 con 0x7f0d180973c0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0d0c00fa90 con 0x7f0d180973c0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0d0c044000 con 0x7f0d180973c0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.248+0000 7f0d167fc700 10 monclient: get_auth_request con 0x7f0d04036410 auth_method 0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.256+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0d0c014770 con 0x7f0d180973c0 2024-09-16T06:55:45.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.256+0000 7f0cff7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:45.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.256+0000 7f0cff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:45.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0d17fff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:55:45.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0d17fff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:55:45.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0d17fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:45.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0cf8005bc0 con 0x7f0d180973c0 2024-09-16T06:55:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0cff7fe700 1 -- 172.21.15.76:0/1557730178 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0d0c014770 con 0x7f0d180973c0 2024-09-16T06:55:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0cff7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:55:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f0cff7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=13,logm=43} crc {auth=2288565323,config=1097430493,health=3925450473,kv=1001081567,logm=2103096850}) 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=10,logm=49} crc {auth=830378157,config=1097430493,health=437598485,kv=2199210827,logm=3674178698}) 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.492+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=13,logm=43} crc {auth=2288565323,config=1097430493,health=3925450473,kv=1001081567,logm=2103096850}) 2024-09-16T06:55:45.494 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=12,kv=13,logm=42} crc {auth=1702446180,config=1097430493,health=4121957689,kv=1001081567,logm=538009837}) 2024-09-16T06:55:45.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0d04036410 legacy=0x7f0d040388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 10 monclient: shutdown 2024-09-16T06:55:45.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:45.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 >> v1:172.21.15.76:6790/0 conn(0x7f0d180973c0 legacy=0x7f0d1813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:45.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 shutdown_connections 2024-09-16T06:55:45.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 >> 172.21.15.76:0/1557730178 conn(0x7f0d180a0030 msgr2=0x7f0d1809c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:45.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 shutdown_connections 2024-09-16T06:55:45.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f0d17fff700 1 -- 172.21.15.76:0/1557730178 wait complete. 2024-09-16T06:55:45.498 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140615370}) 2024-09-16T06:55:45.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2017698284}) 2024-09-16T06:55:45.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140615370}) 2024-09-16T06:55:45.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.496+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554015722}) 2024-09-16T06:55:45.500 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.500 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509234415}) 2024-09-16T06:55:45.501 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737523173}) 2024-09-16T06:55:45.501 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.501 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509234415}) 2024-09-16T06:55:45.501 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.500+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503251828}) 2024-09-16T06:55:45.503 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.503 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903838792}) 2024-09-16T06:55:45.503 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2657036205}) 2024-09-16T06:55:45.503 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.503 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903838792}) 2024-09-16T06:55:45.504 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434036369}) 2024-09-16T06:55:45.506 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.506 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794513750}) 2024-09-16T06:55:45.506 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333190164}) 2024-09-16T06:55:45.506 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.507 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794513750}) 2024-09-16T06:55:45.507 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.504+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968843361}) 2024-09-16T06:55:45.508 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.508 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465450595}) 2024-09-16T06:55:45.508 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181812876}) 2024-09-16T06:55:45.509 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.509 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465450595}) 2024-09-16T06:55:45.509 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3275890152}) 2024-09-16T06:55:45.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:55:45.525 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.525 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662577865}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077667301}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662577865}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169157804}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339354979}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523426457}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339354979}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.508+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649172120}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709004820}) 2024-09-16T06:55:45.526 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216245425}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709004820}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909470195}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676337819}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311455820}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676337819}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1566405118}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245159231}) 2024-09-16T06:55:45.527 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501969955}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245159231}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=348987851}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616766248}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3793265207}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616766248}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.512+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835780406}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613246305}) 2024-09-16T06:55:45.528 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960530592}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613246305}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114459685}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4127277518}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941544037}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4127277518}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688849762}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237820667}) 2024-09-16T06:55:45.529 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187210626}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237820667}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441350790}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668717704}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765047872}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668717704}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.516+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318050927}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679596569}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967185098}) 2024-09-16T06:55:45.530 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679596569}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522692493}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251032896}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416385889}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251032896}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1453685172}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303550158}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041978160}) 2024-09-16T06:55:45.531 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303550158}) 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1541249576}) 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199266079}) 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041090968}) 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199266079}) 2024-09-16T06:55:45.532 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.520+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876469774}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774850875}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873664421}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774850875}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573733326}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738263261}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307039050}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738263261}) 2024-09-16T06:55:45.533 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854106196}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275310987}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300235227}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275310987}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713759050}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181413638}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=24914483}) 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.534 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181413638}) 2024-09-16T06:55:45.535 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.524+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879562424}) 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751676327}) 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737343154}) 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751676327}) 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3856655476}) 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.542 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765589775}) 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974680977}) 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765589775}) 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2415236070}) 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098834893}) 2024-09-16T06:55:45.543 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840974385}) 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098834893}) 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504400157}) 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742641652}) 2024-09-16T06:55:45.544 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514765636}) 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742641652}) 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.528+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3242788270}) 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042236650}) 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378735160}) 2024-09-16T06:55:45.545 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042236650}) 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921238836}) 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387968903}) 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288402619}) 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387968903}) 2024-09-16T06:55:45.546 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009127739}) 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1603563966}) 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729586707}) 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1603563966}) 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363234436}) 2024-09-16T06:55:45.547 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251809439}) 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840088085}) 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251809439}) 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.532+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889851471}) 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699248643}) 2024-09-16T06:55:45.548 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859093235}) 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699248643}) 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950648652}) 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389058373}) 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294088809}) 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.549 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389058373}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065897241}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856507}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945134945}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856507}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556399609}) 2024-09-16T06:55:45.550 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989158291}) 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381291516}) 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989158291}) 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.536+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877399212}) 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886260579}) 2024-09-16T06:55:45.551 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076408326}) 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886260579}) 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335469280}) 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781467466}) 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278265378}) 2024-09-16T06:55:45.552 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781467466}) 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032866415}) 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284839623}) 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543314254}) 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284839623}) 2024-09-16T06:55:45.553 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541829103}) 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717965172}) 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283744476}) 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717965172}) 2024-09-16T06:55:45.554 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.540+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053363995}) 2024-09-16T06:55:45.566 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653064344}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037825152}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653064344}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902151013}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360397745}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534144786}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360397745}) 2024-09-16T06:55:45.567 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3067867212}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907141968}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319825585}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907141968}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.544+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424582544}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850446551}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347011519}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850446551}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722310612}) 2024-09-16T06:55:45.568 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739803399}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737199642}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739803399}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555346007}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263009458}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277287314}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263009458}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445265992}) 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.569 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368488608}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893091562}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368488608}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.548+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185988948}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978541905}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435880023}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978541905}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741095661}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373913459}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659718903}) 2024-09-16T06:55:45.570 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373913459}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077462567}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979679168}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109373226}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979679168}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589352817}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745820907}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326457037}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745820907}) 2024-09-16T06:55:45.571 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.552+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3771036942}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970359231}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625453150}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970359231}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650678066}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507761683}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169420509}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507761683}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768510343}) 2024-09-16T06:55:45.572 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867526294}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299509612}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867526294}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317852825}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185352334}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966400279}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185352334}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.556+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1060661403}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491646508}) 2024-09-16T06:55:45.573 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895649456}) 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491646508}) 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1048688503}) 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449135516}) 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266362073}) 2024-09-16T06:55:45.574 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449135516}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117524308}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852205697}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222656452}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852205697}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1055806803}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666335238}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2684482489}) 2024-09-16T06:55:45.575 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666335238}) 2024-09-16T06:55:45.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.560+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1711979130}) 2024-09-16T06:55:45.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371958060}) 2024-09-16T06:55:45.576 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836970096}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371958060}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065148492}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197405871}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187371147}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197405871}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421996040}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=249717640}) 2024-09-16T06:55:45.577 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169640344}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=249717640}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849179466}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206723318}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981179984}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206723318}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.564+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1134198117}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890095569}) 2024-09-16T06:55:45.578 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501902042}) 2024-09-16T06:55:45.579 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.579 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890095569}) 2024-09-16T06:55:45.579 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505489730}) 2024-09-16T06:55:45.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954984343}) 2024-09-16T06:55:45.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434999839}) 2024-09-16T06:55:45.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.587 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954984343}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804803185}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155262645}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104558725}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155262645}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081041971}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395278698}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156241488}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395278698}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.568+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381979388}) 2024-09-16T06:55:45.588 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763386975}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421812647}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763386975}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805097048}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713686321}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733353989}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713686321}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846863846}) 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.589 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622541733}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709409672}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622541733}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496602536}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235823956}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933132983}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235823956}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.572+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3820333135}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411206240}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725468704}) 2024-09-16T06:55:45.590 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411206240}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2813534170}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533339788}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=227166906}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533339788}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=711560950}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587661520}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406193715}) 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.591 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587661520}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.576+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596563097}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465087968}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273902611}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2465087968}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760509114}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399855253}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177390691}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399855253}) 2024-09-16T06:55:45.592 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3124463204}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532910935}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516101066}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532910935}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.580+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931216685}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820926020}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340635807}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820926020}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260877032}) 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.593 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989823844}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242719278}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989823844}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513437259}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339956149}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782793788}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339956149}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283357541}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980855690}) 2024-09-16T06:55:45.594 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182870358}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980855690}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.584+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3914717183}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474448312}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=968024977}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474448312}) 2024-09-16T06:55:45.595 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773145148}) 2024-09-16T06:55:45.599 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.599 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205671929}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917726577}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205671929}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285020101}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368321850}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167192661}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368321850}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.588+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591634320}) 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.600 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490292109}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530083310}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490292109}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499663395}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4238493813}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448733895}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4238493813}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082290869}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828267112}) 2024-09-16T06:55:45.601 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364332355}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828267112}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.592+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589702710}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516137550}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672371379}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516137550}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143433883}) 2024-09-16T06:55:45.602 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1146646155}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984837438}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1146646155}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96610811}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595219600}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177409297}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595219600}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598325615}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386213605}) 2024-09-16T06:55:45.603 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2209484323}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386213605}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.596+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131494563}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2331313011}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131042576}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2331313011}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299821440}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2117052213}) 2024-09-16T06:55:45.604 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902667903}) 2024-09-16T06:55:45.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2117052213}) 2024-09-16T06:55:45.605 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956702899}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225222550}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802939001}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225222550}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589053371}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426273178}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517213976}) 2024-09-16T06:55:45.608 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426273178}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.600+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554754479}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271882957}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887649522}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271882957}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786315597}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540338419}) 2024-09-16T06:55:45.609 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505483479}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540338419}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747811109}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335781985}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854186362}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335781985}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966655444}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1063645587}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235116831}) 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.610 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1063645587}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.604+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3224477385}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710563077}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156449185}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710563077}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203153031}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556927315}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293461965}) 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.611 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556927315}) 2024-09-16T06:55:45.612 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309952817}) 2024-09-16T06:55:45.613 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122603386}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200746874}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122603386}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3147131691}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241608631}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451638643}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241608631}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.608+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249703675}) 2024-09-16T06:55:45.614 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506092469}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095562849}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506092469}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480247428}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_snap=2,osdmap=3} crc {logm=955840742,mdsmap=3585761405,mgr=2759932744,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2521674775,mon_config_key=3740968640,monmap=1887442722,osd_metadata=2101314684,osd_snap=2033388227,osdmap=743383572}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1905624301,mdsmap=1531151339,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3480303105,mon_config_key=3219578969,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_snap=2,osdmap=3} crc {logm=955840742,mdsmap=3585761405,mgr=2759932744,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2521674775,mon_config_key=3740968640,monmap=1887442722,osd_metadata=2101314684,osd_snap=2033388227,osdmap=743383572}) 2024-09-16T06:55:45.615 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.612+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=2} crc {logm=3185204252,mdsmap=1531151339,mgr=3454917385,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3480303105,mon_config_key=6416393,monmap=968649221,osd_metadata=3528436810,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1702415139}) 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=695582453}) 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=49} crc {osdmap=1342869772}) 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=695582453}) 2024-09-16T06:55:45.620 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:55:45.620+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=52} crc {osdmap=981923}) 2024-09-16T06:55:46.512 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:55:46.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.624+0000 7f39a6057700 10 monclient: get_monmap_and_config 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.624+0000 7f39a6057700 10 monclient: build_initial_monmap 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.624+0000 7f39a6057700 10 monclient: monmap: 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:46.631024+0000 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:46.631024+0000 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 Processor -- start 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- start start 2024-09-16T06:55:46.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient: init 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-c con 0x7f39a0131910 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-b con 0x7f39a01347f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-a con 0x7f39a0138300 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a0132400 con 0x7f39a0138300 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a009fcd0 con 0x7f39a0131910 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a009b300 con 0x7f39a01347f0 2024-09-16T06:55:46.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient(hunting): authenticate will time out at 1893.765564s 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a4854700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f39a01347f0 0x7f39a0136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60322/0 (socket says 172.21.15.76:60322) 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a4854700 1 -- 172.21.15.76:0/1053592151 learned_addr learned my addr 172.21.15.76:0/1053592151 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488111140 0 0) 0x7f39a009b300 con 0x7f39a01347f0 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): my global_id is 324131 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c0034f0 con 0x7f39a01347f0 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641441206 0 0) 0x7f39a0132400 con 0x7f39a0138300 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): my global_id is 314220 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a009b300 con 0x7f39a0138300 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113721374 0 0) 0x7f39a009fcd0 con 0x7f39a0131910 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient(hunting): my global_id is 284509 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a0132400 con 0x7f39a0131910 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658129874 0 0) 0x7f398c0034f0 con 0x7f39a01347f0 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 >> v1:172.21.15.189:6789/0 conn(0x7f39a0131910 legacy=0x7f39a0131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 >> v1:172.21.15.76:6789/0 conn(0x7f39a0138300 legacy=0x7f39a013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: _finish_hunting 0 2024-09-16T06:55:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 monclient: found mon.noname-b 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a009b7e0 con 0x7f39a01347f0 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: _finish_auth 0 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: _check_auth_tickets 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 5 monclient: authenticate success, global_id 324131 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3990001610 con 0x7f39a01347f0 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: dump: 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:46.633 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39900018f0 con 0x7f39a01347f0 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:46.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 1 -- 172.21.15.76:0/1053592151 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3990001c80 con 0x7f39a01347f0 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 4 set_mon_vals no callback set 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f3997fff700 10 monclient: dump: 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:46.635 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 10 monclient: shutdown 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- 172.21.15.76:0/1053592151 >> v1:172.21.15.76:6790/0 conn(0x7f39a01347f0 legacy=0x7f39a0136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- 172.21.15.76:0/1053592151 shutdown_connections 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- 172.21.15.76:0/1053592151 >> 172.21.15.76:0/1053592151 conn(0x7f39a00a4430 msgr2=0x7f39a00a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- 172.21.15.76:0/1053592151 shutdown_connections 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.628+0000 7f39a6057700 1 -- 172.21.15.76:0/1053592151 wait complete. 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: build_initial_monmap 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: monmap: 2024-09-16T06:55:46.636 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:46.637 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 Processor -- start 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- start start 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: init 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-c con 0x7f39a0131910 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-b con 0x7f39a01347f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient(hunting): picked mon.noname-a con 0x7f39a0138300 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a012d420 con 0x7f39a0138300 2024-09-16T06:55:46.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a0137e70 con 0x7f39a01347f0 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a011b270 con 0x7f39a0131910 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient(hunting): authenticate will time out at 1893.769692s 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a5055700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f39a0131910 0x7f39a011a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60324/0 (socket says 172.21.15.76:60324) 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a5856700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f39a0138300 0x7f39a01d2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42280/0 (socket says 172.21.15.76:42280) 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a5055700 1 -- 172.21.15.76:0/2552657384 learned_addr learned my addr 172.21.15.76:0/2552657384 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112130355 0 0) 0x7f39a012d420 con 0x7f39a0138300 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): my global_id is 314223 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39840035d0 con 0x7f39a0138300 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723217827 0 0) 0x7f39a011b270 con 0x7f39a0131910 2024-09-16T06:55:46.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): my global_id is 324134 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a012d420 con 0x7f39a0131910 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031931998 0 0) 0x7f39a0137e70 con 0x7f39a01347f0 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): my global_id is 284512 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a011b270 con 0x7f39a01347f0 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024715557 0 0) 0x7f39840035d0 con 0x7f39a0138300 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a0137e70 con 0x7f39a0138300 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3998001c30 con 0x7f39a0138300 2024-09-16T06:55:46.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): dump: 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:46.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031910047 0 0) 0x7f39a012d420 con 0x7f39a0131910 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39840035d0 con 0x7f39a0131910 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f399c0018f0 con 0x7f39a0131910 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient(hunting): dump: 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:46.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 668679079 0 0) 0x7f39a011b270 con 0x7f39a01347f0 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a012d420 con 0x7f39a01347f0 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1859716089 0 0) 0x7f39a0137e70 con 0x7f39a0138300 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 >> v1:172.21.15.76:6790/0 conn(0x7f39a0131910 legacy=0x7f39a011a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 >> v1:172.21.15.189:6789/0 conn(0x7f39a01347f0 legacy=0x7f39a011ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 monclient: found mon.a 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a009b0f0 con 0x7f39a0138300 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: _finish_auth 0 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:55:46.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 5 monclient: authenticate success, global_id 314223 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _renew_subs 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a011c6d0 con 0x7f39a0138300 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _renew_subs 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a0071790 con 0x7f39a0138300 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3998003a40 con 0x7f39a0138300 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3998001360 con 0x7f39a0138300 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f39977fe700 4 set_mon_vals no callback set 2024-09-16T06:55:46.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.632+0000 7f3995ffb700 10 monclient: dump: 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f39a6057700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f39a6057700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f39a6057700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:46.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3980005100 con 0x7f39a0138300 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3998003dd0 con 0x7f39a0138300 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3998044070 con 0x7f39a0138300 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.636+0000 7f39a5055700 10 monclient: get_auth_request con 0x7f3984036480 auth_method 0 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.644+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f39980147e0 con 0x7f39a0138300 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.644+0000 7f3995ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:46.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.644+0000 7f3995ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:46.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:55:46.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:46.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:46.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3980005930 con 0x7f39a0138300 2024-09-16T06:55:46.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f3995ffb700 1 -- 172.21.15.76:0/2552657384 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2025779483) 0x7f3998012e80 con 0x7f39a0138300 2024-09-16T06:55:46.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f3995ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:55:46.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f3995ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3984036480 legacy=0x7f3984038940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 10 monclient: shutdown 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.880+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 >> v1:172.21.15.76:6789/0 conn(0x7f39a0138300 legacy=0x7f39a01d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.884+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 shutdown_connections 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.884+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 >> 172.21.15.76:0/2552657384 conn(0x7f39a00a4430 msgr2=0x7f39a009d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.884+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 shutdown_connections 2024-09-16T06:55:46.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:46.884+0000 7f39a6057700 1 -- 172.21.15.76:0/2552657384 wait complete. 2024-09-16T06:55:46.894 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:55:46.894 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:55:46.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:55:46.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: get_monmap_and_config 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: monmap: 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:47.012928+0000 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:47.012928+0000 2024-09-16T06:55:47.007 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.008 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.008 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.008 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:47.008 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:47.008 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 1 Processor -- start 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 1 -- start start 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: init 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient(hunting): picked mon.noname-c con 0x7f8120026f40 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient(hunting): picked mon.noname-a con 0x7f8120007740 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient(hunting): picked mon.noname-b con 0x7f81200c1d40 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120023780 con 0x7f8120007740 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120027320 con 0x7f8120026f40 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120007d40 con 0x7f81200c1d40 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient(hunting): authenticate will time out at 1894.146320s 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f8125d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8120007740 0x7f81200c0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42282/0 (socket says 172.21.15.76:42282) 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f8125d9b700 1 -- 172.21.15.76:0/374972381 learned_addr learned my addr 172.21.15.76:0/374972381 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122516134 0 0) 0x7f8120023780 con 0x7f8120007740 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): my global_id is 314226 2024-09-16T06:55:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81100035d0 con 0x7f8120007740 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266647684 0 0) 0x7f8120007d40 con 0x7f81200c1d40 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient(hunting): my global_id is 324137 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8120023780 con 0x7f81200c1d40 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343075970 0 0) 0x7f81100035d0 con 0x7f8120007740 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 >> v1:172.21.15.76:6790/0 conn(0x7f81200c1d40 legacy=0x7f81200c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 >> v1:172.21.15.189:6789/0 conn(0x7f8120026f40 legacy=0x7f8120024cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 monclient: found mon.noname-a 2024-09-16T06:55:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200234a0 con 0x7f8120007740 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8118001610 con 0x7f8120007740 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: dump: 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 5 monclient: authenticate success, global_id 314226 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8118001900 con 0x7f8120007740 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 1 -- 172.21.15.76:0/374972381 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8118001d10 con 0x7f8120007740 2024-09-16T06:55:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812759e700 4 set_mon_vals no callback set 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.008+0000 7f812559a700 10 monclient: dump: 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.023 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: shutdown 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/374972381 >> v1:172.21.15.76:6789/0 conn(0x7f8120007740 legacy=0x7f81200c0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/374972381 shutdown_connections 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/374972381 >> 172.21.15.76:0/374972381 conn(0x7f8120005b40 msgr2=0x7f8120005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/374972381 shutdown_connections 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/374972381 wait complete. 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: monmap: 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 Processor -- start 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- start start 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: init 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient(hunting): picked mon.noname-c con 0x7f8120007740 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient(hunting): picked mon.noname-a con 0x7f8120026f40 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient(hunting): picked mon.noname-b con 0x7f81200c1d40 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200bec10 con 0x7f8120026f40 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200bd1d0 con 0x7f81200c1d40 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200c1ae0 con 0x7f8120007740 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient(hunting): authenticate will time out at 1894.150479s 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8125d9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8120026f40 0x7f81200bcac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42292/0 (socket says 172.21.15.76:42292) 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8125d9b700 1 -- 172.21.15.76:0/2536756400 learned_addr learned my addr 172.21.15.76:0/2536756400 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812659c700 1 --1- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6790/0 conn(0x7f8120007740 0x7f81200bc3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60338/0 (socket says 172.21.15.76:60338) 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086844644 0 0) 0x7f81200bec10 con 0x7f8120026f40 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): my global_id is 314229 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81080035d0 con 0x7f8120026f40 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682618277 0 0) 0x7f81200c1ae0 con 0x7f8120007740 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): my global_id is 324140 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81200bec10 con 0x7f8120007740 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474270734 0 0) 0x7f81200bd1d0 con 0x7f81200c1d40 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): my global_id is 284518 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81200c1ae0 con 0x7f81200c1d40 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2116306234 0 0) 0x7f81200bec10 con 0x7f8120007740 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81200bd1d0 con 0x7f8120007740 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f811c0018b0 con 0x7f8120007740 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): dump: 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3919760695 0 0) 0x7f81080035d0 con 0x7f8120026f40 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81200bec10 con 0x7f8120026f40 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81180015d0 con 0x7f8120026f40 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient(hunting): dump: 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3856658026 0 0) 0x7f81200bd1d0 con 0x7f8120007740 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.189:6789/0 conn(0x7f81200c1d40 legacy=0x7f8120158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6789/0 conn(0x7f8120026f40 legacy=0x7f81200bcac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 monclient: found mon.c 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812002aae0 con 0x7f8120007740 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f8116ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 5 monclient: authenticate success, global_id 324140 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: _renew_subs 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.012+0000 7f812759e700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81200226b0 con 0x7f8120007740 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f812759e700 10 monclient: _renew_subs 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f812759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f812759e700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81200bf1d0 con 0x7f8120007740 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f811c001c40 con 0x7f8120007740 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f811c003b10 con 0x7f8120007740 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8124d99700 4 set_mon_vals no callback set 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: dump: 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f812759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f812759e700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f810c006590 con 0x7f810c002830 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388390119 0 0) 0x7f810c006590 con 0x7f810c002830 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient(hunting): my global_id is 314232 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81200bd1d0 con 0x7f810c002830 2024-09-16T06:55:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3903271493 0 0) 0x7f81200bd1d0 con 0x7f810c002830 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f810c006590 con 0x7f810c002830 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f811c003d30 con 0x7f8120007740 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8118004c20 con 0x7f810c002830 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: dump: 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f811c043da0 con 0x7f8120007740 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4056835537 0 0) 0x7f810c006590 con 0x7f810c002830 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f810c00a190 con 0x7f810c002830 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8126d9d700 10 monclient: get_auth_request con 0x7f81080365a0 auth_method 0 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8118004eb0 con 0x7f810c002830 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6789/0 conn(0x7f810c002830 legacy=0x7f810c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.016+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6789/0 conn(0x7f810c002830 legacy=0x7f810c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f812759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f812759e700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f810c009e50 con 0x7f810c009110 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268680931 0 0) 0x7f810c009e50 con 0x7f810c009110 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient(hunting): my global_id is 314235 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810c006590 con 0x7f810c009110 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3988246248 0 0) 0x7f810c006590 con 0x7f810c009110 2024-09-16T06:55:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f810c009e50 con 0x7f810c009110 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f81180054d0 con 0x7f810c009110 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient: dump: 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 781253042 0 0) 0x7f810c009e50 con 0x7f810c009110 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f810c00c340 con 0x7f810c009110 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 2462095577) 0x7f81180056f0 con 0x7f810c009110 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6789/0 conn(0x7f810c009110 legacy=0x7f810c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8116ffd700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6789/0 conn(0x7f810c009110 legacy=0x7f810c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:47.040 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.041 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.042 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:47.043 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.044 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.045 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.047 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:47.048 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.050 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:47.051 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.189:6812/3098810674 conn(0x7f81080365a0 legacy=0x7f8108038a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 10 monclient: shutdown 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 >> v1:172.21.15.76:6790/0 conn(0x7f8120007740 legacy=0x7f81200bc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8126d9d700 1 -- 172.21.15.76:0/2536756400 reap_dead start 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8126d9d700 1 -- 172.21.15.76:0/2536756400 reap_dead start 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 shutdown_connections 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 >> 172.21.15.76:0/2536756400 conn(0x7f8120005b40 msgr2=0x7f81200c4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 shutdown_connections 2024-09-16T06:55:47.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.028+0000 7f8114ff9700 1 -- 172.21.15.76:0/2536756400 wait complete. 2024-09-16T06:55:47.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:47.054 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:55:47.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:55:47.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: get_monmap_and_config 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: monmap: 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:47.172851+0000 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:47.172851+0000 2024-09-16T06:55:47.168 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:47.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:47.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 1 Processor -- start 2024-09-16T06:55:47.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 1 -- start start 2024-09-16T06:55:47.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: init 2024-09-16T06:55:47.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:55:47.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient(hunting): picked mon.noname-c con 0x7f20580a20c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient(hunting): picked mon.noname-b con 0x7f20580a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient(hunting): picked mon.noname-a con 0x7f205813b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205809b3e0 con 0x7f205813b6b0 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580a3450 con 0x7f20580a20c0 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205809b1a0 con 0x7f20580a24a0 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205debc700 10 monclient(hunting): authenticate will time out at 1894.306925s 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f2057fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f20580a24a0 0x7f20580a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60352/0 (socket says 172.21.15.76:60352) 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f205d6bb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f205813b6b0 0x7f205813daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42312/0 (socket says 172.21.15.76:42312) 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f2057fff700 1 -- 172.21.15.76:0/3677737658 learned_addr learned my addr 172.21.15.76:0/3677737658 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52719434 0 0) 0x7f205809b1a0 con 0x7f20580a24a0 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): my global_id is 324143 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20440035d0 con 0x7f20580a24a0 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161576762 0 0) 0x7f205809b3e0 con 0x7f205813b6b0 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): my global_id is 314238 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205809b1a0 con 0x7f205813b6b0 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213691681 0 0) 0x7f20580a3450 con 0x7f20580a20c0 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 10 monclient(hunting): my global_id is 284521 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205809b3e0 con 0x7f20580a20c0 2024-09-16T06:55:47.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046875987 0 0) 0x7f20440035d0 con 0x7f20580a24a0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 >> v1:172.21.15.189:6789/0 conn(0x7f20580a20c0 legacy=0x7f205806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.168+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 >> v1:172.21.15.76:6789/0 conn(0x7f205813b6b0 legacy=0x7f205813daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 1 monclient: found mon.noname-b 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205806bd30 con 0x7f20580a24a0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2048001610 con 0x7f20580a24a0 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 5 monclient: authenticate success, global_id 324143 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:47.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: dump: 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:47.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2048001830 con 0x7f20580a24a0 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 1 -- 172.21.15.76:0/3677737658 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2048001b80 con 0x7f20580a24a0 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f20577fe700 10 monclient: dump: 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.185 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 4 set_mon_vals no callback set 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 10 monclient: shutdown 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- 172.21.15.76:0/3677737658 >> v1:172.21.15.76:6790/0 conn(0x7f20580a24a0 legacy=0x7f20580a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- 172.21.15.76:0/3677737658 shutdown_connections 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- 172.21.15.76:0/3677737658 >> 172.21.15.76:0/3677737658 conn(0x7f20580a41c0 msgr2=0x7f20580a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- 172.21.15.76:0/3677737658 shutdown_connections 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- 172.21.15.76:0/3677737658 wait complete. 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 10 monclient: monmap: 2024-09-16T06:55:47.186 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 Processor -- start 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 1 -- start start 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.172+0000 7f205debc700 10 monclient: init 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:55:47.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient(hunting): picked mon.noname-a con 0x7f20580a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient(hunting): picked mon.noname-c con 0x7f20580a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient(hunting): picked mon.noname-b con 0x7f205813b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2058137720 con 0x7f20580a20c0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2058139760 con 0x7f205813b6b0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20581374a0 con 0x7f20580a24a0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient(hunting): authenticate will time out at 1894.312768s 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205ceba700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f20580a20c0 0x7f2058136680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42320/0 (socket says 172.21.15.76:42320) 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205ceba700 1 -- 172.21.15.76:0/3244174077 learned_addr learned my addr 172.21.15.76:0/3244174077 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856515724 0 0) 0x7f2058137720 con 0x7f20580a20c0 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): my global_id is 314241 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20380035d0 con 0x7f20580a20c0 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917347298 0 0) 0x7f20581374a0 con 0x7f20580a24a0 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): my global_id is 324146 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2058137720 con 0x7f20580a24a0 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970534417 0 0) 0x7f2058139760 con 0x7f205813b6b0 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): my global_id is 284524 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20581374a0 con 0x7f205813b6b0 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2344991336 0 0) 0x7f20380035d0 con 0x7f20580a20c0 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2058139760 con 0x7f20580a20c0 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2040001990 con 0x7f20580a20c0 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): dump: 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.190 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 250763503 0 0) 0x7f2058137720 con 0x7f20580a24a0 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20380035d0 con 0x7f20580a24a0 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2048001370 con 0x7f20580a24a0 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient(hunting): dump: 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.191 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 780034684 0 0) 0x7f2058139760 con 0x7f20580a20c0 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6790/0 conn(0x7f20580a24a0 legacy=0x7f2058136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.189:6789/0 conn(0x7f205813b6b0 legacy=0x7f20581d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 monclient: found mon.a 2024-09-16T06:55:47.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20581391a0 con 0x7f20580a20c0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 5 monclient: authenticate success, global_id 314241 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _renew_subs 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2058139480 con 0x7f20580a20c0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _renew_subs 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20581384d0 con 0x7f20580a20c0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2040001c70 con 0x7f20580a20c0 2024-09-16T06:55:47.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2040003b10 con 0x7f20580a20c0 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 10 monclient: dump: 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.194 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f2056ffd700 4 set_mon_vals no callback set 2024-09-16T06:55:47.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f205813eb50 con 0x7f205809faf0 2024-09-16T06:55:47.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.176+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2040003c70 con 0x7f20580a20c0 2024-09-16T06:55:47.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2040043ef0 con 0x7f20580a20c0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008719183 0 0) 0x7f205813eb50 con 0x7f205809faf0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient(hunting): my global_id is 324149 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2058139760 con 0x7f205809faf0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f205d6bb700 10 monclient: get_auth_request con 0x7f2038036680 auth_method 0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2521352311 0 0) 0x7f2058139760 con 0x7f205809faf0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f205813eb50 con 0x7f205809faf0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2048004a30 con 0x7f205809faf0 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient: dump: 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 564131834 0 0) 0x7f205813eb50 con 0x7f205809faf0 2024-09-16T06:55:47.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f205813f1c0 con 0x7f205809faf0 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2048006bd0 con 0x7f205809faf0 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6790/0 conn(0x7f205809faf0 legacy=0x7f205809fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.180+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6790/0 conn(0x7f205809faf0 legacy=0x7f205809fed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f205debc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f20580a09f0 con 0x7f20580a05e0 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366643603 0 0) 0x7f20580a09f0 con 0x7f20580a05e0 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 10 monclient(hunting): my global_id is 324152 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.188+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205813eb50 con 0x7f20580a05e0 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2083152988 0 0) 0x7f205813eb50 con 0x7f20580a05e0 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20580a09f0 con 0x7f20580a05e0 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f20480050e0 con 0x7f20580a05e0 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 10 monclient: dump: 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.201 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1018363155 0 0) 0x7f20580a09f0 con 0x7f20580a05e0 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f20580672a0 con 0x7f20580a05e0 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 4087897717) 0x7f20480064f0 con 0x7f20580a05e0 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6790/0 conn(0x7f20580a05e0 legacy=0x7f205802c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f20557fa700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6790/0 conn(0x7f20580a05e0 legacy=0x7f205802c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:47.203 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.204 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.205 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:47.206 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.207 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:47.208 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.209 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.213 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:47.214 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.214 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.214 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:47.214 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:47.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2038036680 legacy=0x7f2038038b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 10 monclient: shutdown 2024-09-16T06:55:47.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205d6bb700 1 -- 172.21.15.76:0/3244174077 reap_dead start 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 >> v1:172.21.15.76:6789/0 conn(0x7f20580a20c0 legacy=0x7f2058136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 shutdown_connections 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 >> 172.21.15.76:0/3244174077 conn(0x7f20580a41c0 msgr2=0x7f205813aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 shutdown_connections 2024-09-16T06:55:47.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.192+0000 7f205debc700 1 -- 172.21.15.76:0/3244174077 wait complete. 2024-09-16T06:55:47.216 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:47.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:55:47.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: get_monmap_and_config 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: monmap: 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:55:47.341648+0000 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:55:47.341648+0000 2024-09-16T06:55:47.336 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.337 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.337 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.337 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:55:47.337 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:55:47.337 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 1 Processor -- start 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 1 -- start start 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: init 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient(hunting): picked mon.noname-a con 0x7f38980a20c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient(hunting): picked mon.noname-c con 0x7f38980a24a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient(hunting): picked mon.noname-b con 0x7f389813b6b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389809b3e0 con 0x7f38980a20c0 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980a3450 con 0x7f38980a24a0 2024-09-16T06:55:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389809b1a0 con 0x7f389813b6b0 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389dada700 10 monclient(hunting): authenticate will time out at 1894.475497s 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389d2d9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f389813b6b0 0x7f389813daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60370/0 (socket says 172.21.15.76:60370) 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389cad8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f38980a20c0 0x7f389806cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42330/0 (socket says 172.21.15.76:42330) 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.336+0000 7f389d2d9700 1 -- 172.21.15.76:0/156621719 learned_addr learned my addr 172.21.15.76:0/156621719 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083168599 0 0) 0x7f389809b1a0 con 0x7f389813b6b0 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): my global_id is 324155 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38780035d0 con 0x7f389813b6b0 2024-09-16T06:55:47.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321328 0 0) 0x7f389809b3e0 con 0x7f38980a20c0 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): my global_id is 314244 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389809b1a0 con 0x7f38980a20c0 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242707856 0 0) 0x7f38980a3450 con 0x7f38980a24a0 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient(hunting): my global_id is 284527 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389809b3e0 con 0x7f38980a24a0 2024-09-16T06:55:47.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659720342 0 0) 0x7f38780035d0 con 0x7f389813b6b0 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 >> v1:172.21.15.189:6789/0 conn(0x7f38980a24a0 legacy=0x7f38980a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 >> v1:172.21.15.76:6789/0 conn(0x7f38980a20c0 legacy=0x7f389806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 monclient: found mon.noname-b 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389806bd30 con 0x7f389813b6b0 2024-09-16T06:55:47.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 5 monclient: authenticate success, global_id 324155 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3888001610 con 0x7f389813b6b0 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: dump: 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3888001880 con 0x7f389813b6b0 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 1 -- 172.21.15.76:0/156621719 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3888001c50 con 0x7f389813b6b0 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f38977fe700 10 monclient: dump: 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.346 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 10 monclient: get_monmap_and_config success 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 4 set_mon_vals no callback set 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 10 monclient: shutdown 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 1 -- 172.21.15.76:0/156621719 >> v1:172.21.15.76:6790/0 conn(0x7f389813b6b0 legacy=0x7f389813daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 1 -- 172.21.15.76:0/156621719 shutdown_connections 2024-09-16T06:55:47.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 1 -- 172.21.15.76:0/156621719 >> 172.21.15.76:0/156621719 conn(0x7f38980a41c0 msgr2=0x7f38980a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 1 -- 172.21.15.76:0/156621719 shutdown_connections 2024-09-16T06:55:47.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 1 -- 172.21.15.76:0/156621719 wait complete. 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 10 monclient: build_initial_monmap 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.340+0000 7f389dada700 10 monclient: monmap: 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:55:47.354 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 Processor -- start 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- start start 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: init 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _reopen_session rank -1 2024-09-16T06:55:47.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient(hunting): picked mon.noname-b con 0x7f38980a20c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient(hunting): picked mon.noname-a con 0x7f38980a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient(hunting): picked mon.noname-c con 0x7f389813b6b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980a13c0 con 0x7f38980a24a0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389809e280 con 0x7f38980a20c0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38981374a0 con 0x7f389813b6b0 2024-09-16T06:55:47.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient(hunting): _renew_subs 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient(hunting): authenticate will time out at 1894.481867s 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389d2d9700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f389813b6b0 0x7f38981d2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60380/0 (socket says 172.21.15.76:60380) 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f3897fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f38980a24a0 0x7f3898136d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42340/0 (socket says 172.21.15.76:42340) 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389d2d9700 1 -- 172.21.15.76:0/2011170661 learned_addr learned my addr 172.21.15.76:0/2011170661 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113120292 0 0) 0x7f38981374a0 con 0x7f389813b6b0 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): my global_id is 324158 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387c0035d0 con 0x7f389813b6b0 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422465443 0 0) 0x7f389809b3e0 con 0x7f38980a24a0 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): my global_id is 314247 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38981374a0 con 0x7f38980a24a0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526960417 0 0) 0x7f389809e280 con 0x7f38980a20c0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): my global_id is 284530 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389809b3e0 con 0x7f38980a20c0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138527942 0 0) 0x7f387c0035d0 con 0x7f389813b6b0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f389809e280 con 0x7f389813b6b0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3888001b90 con 0x7f389813b6b0 2024-09-16T06:55:47.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): dump: 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:55:47.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616184332 0 0) 0x7f38981374a0 con 0x7f38980a24a0 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f387c0035d0 con 0x7f38980a24a0 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38800016b0 con 0x7f38980a24a0 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): dump: 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159789748 0 0) 0x7f389809b3e0 con 0x7f38980a20c0 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38981374a0 con 0x7f38980a20c0 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3890001cd0 con 0x7f38980a20c0 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient(hunting): dump: 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.361 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879200216 0 0) 0x7f389809e280 con 0x7f389813b6b0 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6789/0 conn(0x7f38980a20c0 legacy=0x7f3898136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.76:6789/0 conn(0x7f38980a24a0 legacy=0x7f3898136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: _finish_hunting 0 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 monclient: found mon.c 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389813ebe0 con 0x7f389813b6b0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: _finish_auth 0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: _check_auth_tickets 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 5 monclient: authenticate success, global_id 324158 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _renew_subs 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389809e8b0 con 0x7f389813b6b0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _renew_subs 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898137840 con 0x7f389813b6b0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3888001200 con 0x7f389813b6b0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3888004ca0 con 0x7f389813b6b0 2024-09-16T06:55:47.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f3896ffd700 4 set_mon_vals no callback set 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.344+0000 7f38957fa700 10 monclient: dump: 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f388800f710 con 0x7f389813b6b0 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f389dada700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3870006590 con 0x7f3870002830 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3888043fa0 con 0x7f389813b6b0 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902952061 0 0) 0x7f3870006590 con 0x7f3870002830 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f3897fff700 10 monclient: get_auth_request con 0x7f387c036560 auth_method 0 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient(hunting): my global_id is 284533 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389809e280 con 0x7f3870002830 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1353142163 0 0) 0x7f389809e280 con 0x7f3870002830 2024-09-16T06:55:47.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3870006590 con 0x7f3870002830 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3890004db0 con 0x7f3870002830 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient: dump: 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 404395855 0 0) 0x7f3870006590 con 0x7f3870002830 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f387000a190 con 0x7f3870002830 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f38900050b0 con 0x7f3870002830 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6789/0 conn(0x7f3870002830 legacy=0x7f3870004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.348+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6789/0 conn(0x7f3870002830 legacy=0x7f3870004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:55:47.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3870009e30 con 0x7f3870009150 2024-09-16T06:55:47.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953284104 0 0) 0x7f3870009e30 con 0x7f3870009150 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient(hunting): _negotiate 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient(hunting): my global_id is 284536 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3870006590 con 0x7f3870009150 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2337818532 0 0) 0x7f3870006590 con 0x7f3870009150 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3870009e30 con 0x7f3870009150 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f38900055b0 con 0x7f3870009150 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient: dump: 2024-09-16T06:55:47.369 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2293642916 0 0) 0x7f3870009e30 con 0x7f3870009150 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f387000c340 con 0x7f3870009150 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 4139580672) 0x7f38900078f0 con 0x7f3870009150 2024-09-16T06:55:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6789/0 conn(0x7f3870009150 legacy=0x7f38700095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f38957fa700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6789/0 conn(0x7f3870009150 legacy=0x7f38700095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 156, 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:55:47.371 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.372 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.373 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:55:47.374 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:55:47.375 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:55:47.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:55:47.377 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.378 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:55:47.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:55:47.380 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.381 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:55:47.382 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:55:47.383 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:55:47.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.189:6812/3098810674 conn(0x7f387c036560 legacy=0x7f387c038a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 10 monclient: shutdown 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 >> v1:172.21.15.76:6790/0 conn(0x7f389813b6b0 legacy=0x7f38981d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389d2d9700 1 -- 172.21.15.76:0/2011170661 reap_dead start 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389d2d9700 1 -- 172.21.15.76:0/2011170661 reap_dead start 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 shutdown_connections 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 >> 172.21.15.76:0/2011170661 conn(0x7f38980a41c0 msgr2=0x7f389813ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 shutdown_connections 2024-09-16T06:55:47.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:55:47.360+0000 7f389dada700 1 -- 172.21.15.76:0/2011170661 wait complete. 2024-09-16T06:55:47.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:55:47.386 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T06:55:47.386 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T06:55:47.386 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T06:55:47.386 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:55:47.386 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T06:55:47.386 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:55:47.420 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T06:55:47.421 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:55:47.421 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:55:47.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:56:07.422 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:56:07.423 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:56:07.423 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:56:07.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:56:07.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: get_monmap_and_config 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: build_initial_monmap 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: monmap: 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:07.546050+0000 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:07.546050+0000 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:07.596 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:07.597 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 1 Processor -- start 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 1 -- start start 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: init 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-c con 0x7fca4c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-b con 0x7fca4c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:07.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-a con 0x7fca4c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09d560 con 0x7fca4c099030 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09f140 con 0x7fca4c09ed60 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09d790 con 0x7fca4c098be0 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca5273c700 10 monclient(hunting): authenticate will time out at 1914.679384s 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca51f3b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca4c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca50f39700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 0x7fca4c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42824/0 (socket says 172.21.15.76:42824) 2024-09-16T06:56:07.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.540+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 learned_addr learned my addr 172.21.15.76:0/2498194695 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:07.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:07.744+0000 7fca51f3b700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca4c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:08.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:08.148+0000 7fca51f3b700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca4c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:08.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:08.952+0000 7fca51f3b700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca4c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:09.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:09.364+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 legacy=0x7fca4c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T06:56:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:09.364+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 legacy=0x7fca4c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:56:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:09.364+0000 7fca50f39700 1 --1- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 0x7fca4c1374e0 :-1 s=OPENED pgs=1751 cs=1 l=1).handle_message read tag failed 2024-09-16T06:56:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:09.364+0000 7fca50f39700 1 --1- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 0x7fca4c1374e0 :-1 s=OPENED pgs=1751 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:56:09.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:09.364+0000 7fca43fff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6790/0 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): tick 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 monclient(hunting): continuing hunt 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 legacy=0x7fca4c1374e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.189:6789/0 conn(0x7fca4c09ed60 legacy=0x7fca4c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca4c13ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): picked mon.noname-c con 0x7fca30005880 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): picked mon.noname-a con 0x7fca4c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:10.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): picked mon.noname-b con 0x7fca4c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09f620 con 0x7fca4c099030 2024-09-16T06:56:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09d560 con 0x7fca30005880 2024-09-16T06:56:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 1 -- 172.21.15.76:0/2498194695 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca30012550 con 0x7fca4c098be0 2024-09-16T06:56:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca42ffd700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.540+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca3000b5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:10.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:10.744+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca3000b5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:11.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:11.148+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca3000b5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:11.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:11.952+0000 7fca50f39700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca3000b5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:12.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471557457 0 0) 0x7fca4c09d560 con 0x7fca30005880 2024-09-16T06:56:12.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient(hunting): my global_id is 284539 2024-09-16T06:56:12.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca380035d0 con 0x7fca30005880 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009008095 0 0) 0x7fca380035d0 con 0x7fca30005880 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6790/0 conn(0x7fca4c098be0 legacy=0x7fca3000ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.76:6789/0 conn(0x7fca4c099030 legacy=0x7fca3000b5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: _finish_hunting 0 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 monclient: found mon.noname-c 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30013760 con 0x7fca30005880 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: _finish_auth 0 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: _check_auth_tickets 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca5273c700 5 monclient: authenticate success, global_id 284539 2024-09-16T06:56:12.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca3c001590 con 0x7fca30005880 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca5273c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca43fff700 10 monclient: dump: 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.468+0000 7fca5273c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:12.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca3c001830 con 0x7fca30005880 2024-09-16T06:56:12.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:12.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:12.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 1 -- 172.21.15.76:0/2498194695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca3c001bc0 con 0x7fca30005880 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 4 set_mon_vals no callback set 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca43fff700 10 monclient: dump: 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.473 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 10 monclient: shutdown 2024-09-16T06:56:12.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:12.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 1 -- 172.21.15.76:0/2498194695 >> v1:172.21.15.189:6789/0 conn(0x7fca30005880 legacy=0x7fca30007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 1 -- 172.21.15.76:0/2498194695 shutdown_connections 2024-09-16T06:56:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 1 -- 172.21.15.76:0/2498194695 >> 172.21.15.76:0/2498194695 conn(0x7fca4c0a0030 msgr2=0x7fca4c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 1 -- 172.21.15.76:0/2498194695 shutdown_connections 2024-09-16T06:56:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.472+0000 7fca5273c700 1 -- 172.21.15.76:0/2498194695 wait complete. 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient: build_initial_monmap 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient: monmap: 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:12.475 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 1 Processor -- start 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 1 -- start start 2024-09-16T06:56:12.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient: init 2024-09-16T06:56:12.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:12.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:56:12.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-a con 0x7fca30005880 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-b con 0x7fca4c098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient(hunting): picked mon.noname-c con 0x7fca4c099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09a540 con 0x7fca30005880 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5173a700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fca30005880 legacy=0x7fca4c0964b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c090980 con 0x7fca4c098be0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c0a2d50 con 0x7fca4c099030 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca5273c700 10 monclient(hunting): authenticate will time out at 1919.614530s 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca50f39700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fca4c098be0 0x7fca4c1ca8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.76:37990/0 (socket says 172.21.15.76:37990) 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca50f39700 1 -- 172.21.15.76:0/4005796907 learned_addr learned my addr 172.21.15.76:0/4005796907 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca51f3b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fca4c099030 0x7fca4c1ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42876/0 (socket says 172.21.15.76:42876) 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177534847 0 0) 0x7fca4c090980 con 0x7fca4c098be0 2024-09-16T06:56:12.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): my global_id is 284542 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30007480 con 0x7fca4c098be0 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684714679 0 0) 0x7fca30007480 con 0x7fca4c098be0 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca4c090980 con 0x7fca4c098be0 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca48001c10 con 0x7fca4c098be0 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): dump: 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.481 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.476+0000 7fca41ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 14728480 0 0) 0x7fca4c090980 con 0x7fca4c098be0 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 >> v1:172.21.15.76:6790/0 conn(0x7fca4c099030 legacy=0x7fca4c1ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 >> v1:172.21.15.76:6789/0 conn(0x7fca30005880 legacy=0x7fca4c0964b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:56:12.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 monclient: found mon.b 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c090f40 con 0x7fca4c098be0 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: _finish_auth 0 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 5 monclient: authenticate success, global_id 284542 2024-09-16T06:56:12.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _renew_subs 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c0906a0 con 0x7fca4c098be0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _renew_subs 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c08edd0 con 0x7fca4c098be0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca48004c10 con 0x7fca4c098be0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fca48004fa0 con 0x7fca4c098be0 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 10 monclient: dump: 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca437fe700 4 set_mon_vals no callback set 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fca48005100 con 0x7fca4c098be0 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fca480442e0 con 0x7fca4c098be0 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:12.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c005100 con 0x7fca4c098be0 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.480+0000 7fca5173a700 10 monclient: get_auth_request con 0x7fca30038c80 auth_method 0 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.484+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fca48014c20 con 0x7fca4c098be0 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.484+0000 7fca41ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:12.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.484+0000 7fca41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca5273c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:56:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca5273c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca5273c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:56:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fca2c005930 con 0x7fca4c098be0 2024-09-16T06:56:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca41ffb700 1 -- 172.21.15.76:0/4005796907 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 21751465) 0x7fca4801d210 con 0x7fca4c098be0 2024-09-16T06:56:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca41ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.716+0000 7fca41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:12.717 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":158,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:56:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 >> v1:172.21.15.189:6812/3098810674 conn(0x7fca30038c80 legacy=0x7fca3003b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 10 monclient: shutdown 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 >> v1:172.21.15.189:6789/0 conn(0x7fca4c098be0 legacy=0x7fca4c1ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 shutdown_connections 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 >> 172.21.15.76:0/4005796907 conn(0x7fca4c0a0030 msgr2=0x7fca4c06dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 shutdown_connections 2024-09-16T06:56:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.720+0000 7fca5273c700 1 -- 172.21.15.76:0/4005796907 wait complete. 2024-09-16T06:56:12.731 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:56:12.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T06:56:12.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.844+0000 7fad78e96700 10 monclient: get_monmap_and_config 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.844+0000 7fad78e96700 10 monclient: build_initial_monmap 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.844+0000 7fad78e96700 10 monclient: monmap: 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:12.851424+0000 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:12.851424+0000 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 Processor -- start 2024-09-16T06:56:12.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- start start 2024-09-16T06:56:12.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient: init 2024-09-16T06:56:12.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-a con 0x7fad7409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-b con 0x7fad74098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-c con 0x7fad74099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7409d560 con 0x7fad7409ed60 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7409f140 con 0x7fad74099030 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad737fe700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fad7409ed60 legacy=0x7fad740984d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7409d790 con 0x7fad74098be0 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient(hunting): authenticate will time out at 1919.985319s 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad72ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fad74098be0 0x7fad741374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42890/0 (socket says 172.21.15.76:42890) 2024-09-16T06:56:12.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad72ffd700 1 -- 172.21.15.76:0/2014513467 learned_addr learned my addr 172.21.15.76:0/2014513467 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331338717 0 0) 0x7fad7409d790 con 0x7fad74098be0 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): my global_id is 324161 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c0057e0 con 0x7fad74098be0 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922921572 0 0) 0x7fad7409f140 con 0x7fad74099030 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient(hunting): my global_id is 284545 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad7409d790 con 0x7fad74099030 2024-09-16T06:56:12.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 551381357 0 0) 0x7fad5c0057e0 con 0x7fad74098be0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 >> v1:172.21.15.189:6789/0 conn(0x7fad74099030 legacy=0x7fad7413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 >> v1:172.21.15.76:6789/0 conn(0x7fad7409ed60 legacy=0x7fad740984d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: _finish_hunting 0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 monclient: found mon.noname-b 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7409f620 con 0x7fad74098be0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: _finish_auth 0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: _check_auth_tickets 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad64001990 con 0x7fad74098be0 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 5 monclient: authenticate success, global_id 324161 2024-09-16T06:56:12.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: dump: 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad64001bb0 con 0x7fad74098be0 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 4 set_mon_vals no callback set 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 1 -- 172.21.15.76:0/2014513467 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad64001f40 con 0x7fad74098be0 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad727fc700 10 monclient: dump: 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 10 monclient: shutdown 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- 172.21.15.76:0/2014513467 >> v1:172.21.15.76:6790/0 conn(0x7fad74098be0 legacy=0x7fad741374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- 172.21.15.76:0/2014513467 shutdown_connections 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- 172.21.15.76:0/2014513467 >> 172.21.15.76:0/2014513467 conn(0x7fad740a0030 msgr2=0x7fad740a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- 172.21.15.76:0/2014513467 shutdown_connections 2024-09-16T06:56:12.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.848+0000 7fad78e96700 1 -- 172.21.15.76:0/2014513467 wait complete. 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: build_initial_monmap 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: monmap: 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:12.860 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 Processor -- start 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- start start 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: init 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-b con 0x7fad74098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-a con 0x7fad74099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient(hunting): picked mon.noname-c con 0x7fad7409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7409a540 con 0x7fad74099030 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740a1370 con 0x7fad74098be0 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740987d0 con 0x7fad7409ed60 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient(hunting): authenticate will time out at 1919.990054s 2024-09-16T06:56:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad72ffd700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7fad74099030 legacy=0x7fad741ca9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad73fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fad7409ed60 0x7fad741ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42904/0 (socket says 172.21.15.76:42904) 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad73fff700 1 -- 172.21.15.76:0/1594968781 learned_addr learned my addr 172.21.15.76:0/1594968781 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035730370 0 0) 0x7fad740987d0 con 0x7fad7409ed60 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): my global_id is 324164 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600035d0 con 0x7fad7409ed60 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323148117 0 0) 0x7fad740a1370 con 0x7fad74098be0 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): my global_id is 284548 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad740987d0 con 0x7fad74098be0 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2336013724 0 0) 0x7fad600035d0 con 0x7fad7409ed60 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad740a1370 con 0x7fad7409ed60 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad6c001980 con 0x7fad7409ed60 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): dump: 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.863 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3488642602 0 0) 0x7fad740987d0 con 0x7fad74098be0 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad600035d0 con 0x7fad74098be0 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3343455431 0 0) 0x7fad740a1370 con 0x7fad7409ed60 2024-09-16T06:56:12.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.189:6789/0 conn(0x7fad74098be0 legacy=0x7fad74096550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6789/0 conn(0x7fad74099030 legacy=0x7fad741ca9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 monclient: found mon.c 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7408f7e0 con 0x7fad7409ed60 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: _finish_auth 0 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 5 monclient: authenticate success, global_id 324164 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _renew_subs 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad74090d00 con 0x7fad7409ed60 2024-09-16T06:56:12.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _renew_subs 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad74096c60 con 0x7fad7409ed60 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad6c001d30 con 0x7fad7409ed60 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad6c003b90 con 0x7fad7409ed60 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad5bfff700 10 monclient: dump: 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.866 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.852+0000 7fad71ffb700 4 set_mon_vals no callback set 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad78e96700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad54006590 con 0x7fad54002830 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fad6c003d70 con 0x7fad7409ed60 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fad6c043f60 con 0x7fad7409ed60 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253687452 0 0) 0x7fad54006590 con 0x7fad54002830 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient(hunting): my global_id is 324167 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad740a1370 con 0x7fad54002830 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad72ffd700 10 monclient: get_auth_request con 0x7fad60036260 auth_method 0 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 46449591 0 0) 0x7fad740a1370 con 0x7fad54002830 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad54006590 con 0x7fad54002830 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad68004780 con 0x7fad54002830 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient: dump: 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.868 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 985909567 0 0) 0x7fad54006590 con 0x7fad54002830 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad5400a190 con 0x7fad54002830 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fad68006ab0 con 0x7fad54002830 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:12.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6790/0 conn(0x7fad54002830 legacy=0x7fad54004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.856+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6790/0 conn(0x7fad54002830 legacy=0x7fad54004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad54009e30 con 0x7fad54009150 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149551770 0 0) 0x7fad54009e30 con 0x7fad54009150 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient(hunting): my global_id is 324170 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad54006590 con 0x7fad54009150 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1872401672 0 0) 0x7fad54006590 con 0x7fad54009150 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad54009e30 con 0x7fad54009150 2024-09-16T06:56:12.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fad68004ec0 con 0x7fad54009150 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient: dump: 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:12.872 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4098177270 0 0) 0x7fad54009e30 con 0x7fad54009150 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fad5400c340 con 0x7fad54009150 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 1845618153) 0x7fad68006300 con 0x7fad54009150 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6790/0 conn(0x7fad54009150 legacy=0x7fad540095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad5bfff700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6790/0 conn(0x7fad54009150 legacy=0x7fad540095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:12.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 158, 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:12.874 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:12.875 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:12.876 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:12.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:12.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:12.882 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:12.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:12.884 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.189:6812/3098810674 conn(0x7fad60036260 legacy=0x7fad60038720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 10 monclient: shutdown 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 >> v1:172.21.15.76:6790/0 conn(0x7fad7409ed60 legacy=0x7fad741ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad73fff700 1 -- 172.21.15.76:0/1594968781 reap_dead start 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 shutdown_connections 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 >> 172.21.15.76:0/1594968781 conn(0x7fad740a0030 msgr2=0x7fad7406dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 shutdown_connections 2024-09-16T06:56:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:12.868+0000 7fad78e96700 1 -- 172.21.15.76:0/1594968781 wait complete. 2024-09-16T06:56:12.886 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:56:12.887 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:56:12.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:56:13.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: get_monmap_and_config 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: build_initial_monmap 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: monmap: 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:13.009363+0000 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:13.009363+0000 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:13.004 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.005 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:13.005 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:13.005 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:13.005 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 1 Processor -- start 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 1 -- start start 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: init 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient(hunting): picked mon.noname-b con 0x7f485809ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient(hunting): picked mon.noname-a con 0x7f4858098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient(hunting): picked mon.noname-c con 0x7f4858099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485809d560 con 0x7f4858098be0 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485809f140 con 0x7f4858099030 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485809d790 con 0x7f485809ed60 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485759e700 10 monclient(hunting): authenticate will time out at 1920.143130s 2024-09-16T06:56:13.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f4855d9b700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f4858098be0 legacy=0x7f48581374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f485809ed60 0x7f48580984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42940/0 (socket says 172.21.15.76:42940) 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485659c700 1 -- 172.21.15.76:0/1903078611 learned_addr learned my addr 172.21.15.76:0/1903078611 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334180160 0 0) 0x7f485809d790 con 0x7f485809ed60 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 10 monclient(hunting): _negotiate 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 10 monclient(hunting): my global_id is 324173 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.004+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48400035d0 con 0x7f485809ed60 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923754223 0 0) 0x7f48400035d0 con 0x7f485809ed60 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 >> v1:172.21.15.189:6789/0 conn(0x7f4858099030 legacy=0x7f485813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 >> v1:172.21.15.76:6789/0 conn(0x7f4858098be0 legacy=0x7f48581374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: _finish_hunting 0 2024-09-16T06:56:13.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 monclient: found mon.noname-b 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485809f620 con 0x7f485809ed60 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: _finish_auth 0 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: _check_auth_tickets 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 5 monclient: authenticate success, global_id 324173 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4848001610 con 0x7f485809ed60 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: dump: 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.019 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48480018b0 con 0x7f485809ed60 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:13.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 4 set_mon_vals no callback set 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 1 -- 172.21.15.76:0/1903078611 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4848001c80 con 0x7f485809ed60 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485559a700 10 monclient: dump: 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.021 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: shutdown 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1903078611 >> v1:172.21.15.76:6790/0 conn(0x7f485809ed60 legacy=0x7f48580984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1903078611 shutdown_connections 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1903078611 >> 172.21.15.76:0/1903078611 conn(0x7f48580a0030 msgr2=0x7f48580a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1903078611 shutdown_connections 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1903078611 wait complete. 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: build_initial_monmap 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: monmap: 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:13.022 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 Processor -- start 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- start start 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: init 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:13.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient(hunting): picked mon.noname-b con 0x7f4858098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient(hunting): picked mon.noname-c con 0x7f4858099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient(hunting): picked mon.noname-a con 0x7f485809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4858071c90 con 0x7f485809ed60 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4858098740 con 0x7f4858098be0 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485809a540 con 0x7f4858099030 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient(hunting): authenticate will time out at 1920.146515s 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4856d9d700 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f485809ed60 legacy=0x7f48581d20c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6789/0 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4855d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4858099030 0x7f48581ce990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:42942/0 (socket says 172.21.15.76:42942) 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4855d9b700 1 -- 172.21.15.76:0/1107931184 learned_addr learned my addr 172.21.15.76:0/1107931184 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400262259 0 0) 0x7f485809a540 con 0x7f4858099030 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:13.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): my global_id is 324176 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48380035d0 con 0x7f4858099030 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713895238 0 0) 0x7f4858098740 con 0x7f4858098be0 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): my global_id is 284554 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485809a540 con 0x7f4858098be0 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224101202 0 0) 0x7f48380035d0 con 0x7f4858099030 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4858098740 con 0x7f4858099030 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f484c004200 con 0x7f4858099030 2024-09-16T06:56:13.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): dump: 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:13.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1256103979 0 0) 0x7f485809a540 con 0x7f4858098be0 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48380035d0 con 0x7f4858098be0 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4848001510 con 0x7f4858098be0 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient(hunting): dump: 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.027 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2462120896 0 0) 0x7f4858098740 con 0x7f4858099030 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6789/0 conn(0x7f4858098be0 legacy=0x7f4858096680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.76:6789/0 conn(0x7f485809ed60 legacy=0x7f48581d20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 monclient: found mon.c 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485808ee10 con 0x7f4858099030 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient: _finish_auth 0 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f4846ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:13.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 5 monclient: authenticate success, global_id 324176 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _renew_subs 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485808f0f0 con 0x7f4858099030 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _renew_subs 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.008+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4858096d90 con 0x7f4858099030 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f484c004470 con 0x7f4858099030 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4854d99700 4 set_mon_vals no callback set 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f484c004800 con 0x7f4858099030 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:13.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: dump: 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f485759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f483c006590 con 0x7f483c002830 2024-09-16T06:56:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f484c004a60 con 0x7f4858099030 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f484c043d50 con 0x7f4858099030 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584139297 0 0) 0x7f483c006590 con 0x7f483c002830 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient(hunting): my global_id is 284557 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4858098740 con 0x7f483c002830 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4856d9d700 10 monclient: get_auth_request con 0x7f48380362a0 auth_method 0 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2383132179 0 0) 0x7f4858098740 con 0x7f483c002830 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f483c006590 con 0x7f483c002830 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4848004d80 con 0x7f483c002830 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: dump: 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3571323674 0 0) 0x7f483c006590 con 0x7f483c002830 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f483c00a190 con 0x7f483c002830 2024-09-16T06:56:13.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f4848007480 con 0x7f483c002830 2024-09-16T06:56:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6789/0 conn(0x7f483c002830 legacy=0x7f483c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.012+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6789/0 conn(0x7f483c002830 legacy=0x7f483c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f485759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f483c009e30 con 0x7f483c009150 2024-09-16T06:56:13.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148268440 0 0) 0x7f483c009e30 con 0x7f483c009150 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 10 monclient(hunting): my global_id is 284560 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.020+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c006590 con 0x7f483c009150 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1711119019 0 0) 0x7f483c006590 con 0x7f483c009150 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f483c009e30 con 0x7f483c009150 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4848005400 con 0x7f483c009150 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 10 monclient: dump: 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:13.035 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1781114855 0 0) 0x7f483c009e30 con 0x7f483c009150 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f483c00c340 con 0x7f483c009150 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 2159906765) 0x7f48480075f0 con 0x7f483c009150 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:13.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6789/0 conn(0x7f483c009150 legacy=0x7f483c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4846ffd700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6789/0 conn(0x7f483c009150 legacy=0x7f483c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 158, 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:13.037 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:13.038 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:13.039 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:13.040 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:13.041 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.042 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:13.044 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:13.045 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:13.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:13.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:13.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:13.049 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:13.049 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.189:6812/3098810674 conn(0x7f48380362a0 legacy=0x7f4838038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 10 monclient: shutdown 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 >> v1:172.21.15.76:6790/0 conn(0x7f4858099030 legacy=0x7f48581ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4856d9d700 1 -- 172.21.15.76:0/1107931184 reap_dead start 2024-09-16T06:56:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f4856d9d700 1 -- 172.21.15.76:0/1107931184 reap_dead start 2024-09-16T06:56:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 shutdown_connections 2024-09-16T06:56:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 >> 172.21.15.76:0/1107931184 conn(0x7f48580a0030 msgr2=0x7f48580972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 shutdown_connections 2024-09-16T06:56:13.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:13.024+0000 7f485759e700 1 -- 172.21.15.76:0/1107931184 wait complete. 2024-09-16T06:56:13.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:56:13.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:56:33.052 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T06:56:33.053 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T06:56:33.053 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T06:56:33.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T06:56:33.057 INFO:tasks.ceph.mon.a:Started 2024-09-16T06:56:33.058 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T06:56:33.058 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T06:56:33.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:56:53.059 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T06:56:53.059 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T06:56:53.059 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:56:53.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:56:53.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: get_monmap_and_config 2024-09-16T06:56:53.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: monmap: 2024-09-16T06:56:53.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:53.185243+0000 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:53.185243+0000 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:53.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 1 Processor -- start 2024-09-16T06:56:53.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 1 -- start start 2024-09-16T06:56:53.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: init 2024-09-16T06:56:53.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:56:53.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-a con 0x7f029409f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-c con 0x7f029409b6a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-b con 0x7f0294138030 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029409f610 con 0x7f029409f230 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029409c220 con 0x7f029409b6a0 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029409a880 con 0x7f0294138030 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f029aa96700 10 monclient(hunting): authenticate will time out at 1960.319649s 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f0299033700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0294138030 0x7f029413a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53122/0 (socket says 172.21.15.76:53122) 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f0299033700 1 -- 172.21.15.76:0/1081521458 learned_addr learned my addr 172.21.15.76:0/1081521458 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.182+0000 7f0298832700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f029409f230 0x7f0294099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51478/0 (socket says 172.21.15.76:51478) 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285644219 0 0) 0x7f029409a880 con 0x7f0294138030 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): my global_id is 324179 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02840035d0 con 0x7f0294138030 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585669287 0 0) 0x7f029409f610 con 0x7f029409f230 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): my global_id is 334098 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029409a880 con 0x7f029409f230 2024-09-16T06:56:53.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842080924 0 0) 0x7f029409c220 con 0x7f029409b6a0 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient(hunting): my global_id is 284563 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029409f610 con 0x7f029409b6a0 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255946209 0 0) 0x7f029409a880 con 0x7f029409f230 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 >> v1:172.21.15.76:6790/0 conn(0x7f0294138030 legacy=0x7f029413a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 >> v1:172.21.15.189:6789/0 conn(0x7f029409b6a0 legacy=0x7f029409a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 monclient: found mon.noname-a 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02940a0100 con 0x7f029409f230 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 5 monclient: authenticate success, global_id 334098 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0288001570 con 0x7f029409f230 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: dump: 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.189 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0288001890 con 0x7f029409f230 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 1 -- 172.21.15.76:0/1081521458 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0288001c20 con 0x7f029409f230 2024-09-16T06:56:53.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 4 set_mon_vals no callback set 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f02937fe700 10 monclient: dump: 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 10 monclient: shutdown 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- 172.21.15.76:0/1081521458 >> v1:172.21.15.76:6789/0 conn(0x7f029409f230 legacy=0x7f0294099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- 172.21.15.76:0/1081521458 shutdown_connections 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- 172.21.15.76:0/1081521458 >> 172.21.15.76:0/1081521458 conn(0x7f02940a3e50 msgr2=0x7f02940a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- 172.21.15.76:0/1081521458 shutdown_connections 2024-09-16T06:56:53.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- 172.21.15.76:0/1081521458 wait complete. 2024-09-16T06:56:53.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 10 monclient: monmap: 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 Processor -- start 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 1 -- start start 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.186+0000 7f029aa96700 10 monclient: init 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-a con 0x7f029409b6a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-b con 0x7f029409f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient(hunting): picked mon.noname-c con 0x7f0294138030 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029411fd30 con 0x7f029409b6a0 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02940a2440 con 0x7f029409f230 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029411dcb0 con 0x7f0294138030 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient(hunting): authenticate will time out at 1960.324301s 2024-09-16T06:56:53.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f0299033700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0294138030 0x7f02941d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53126/0 (socket says 172.21.15.76:53126) 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f0299033700 1 -- 172.21.15.76:0/2527328147 learned_addr learned my addr 172.21.15.76:0/2527328147 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944454697 0 0) 0x7f029411dcb0 con 0x7f0294138030 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): my global_id is 324182 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02780035d0 con 0x7f0294138030 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924940432 0 0) 0x7f029411fd30 con 0x7f029409b6a0 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): my global_id is 334101 2024-09-16T06:56:53.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029411dcb0 con 0x7f029409b6a0 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366642432 0 0) 0x7f02940a2440 con 0x7f029409f230 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): my global_id is 284566 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029411fd30 con 0x7f029409f230 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091423366 0 0) 0x7f029411dcb0 con 0x7f029409b6a0 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02940a2440 con 0x7f029409b6a0 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0288001530 con 0x7f029409b6a0 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): dump: 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.201 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026642098 0 0) 0x7f02780035d0 con 0x7f0294138030 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f029411dcb0 con 0x7f0294138030 2024-09-16T06:56:53.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f028c001d70 con 0x7f0294138030 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient(hunting): dump: 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.203 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1237647354 0 0) 0x7f02940a2440 con 0x7f029409b6a0 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 >> v1:172.21.15.76:6790/0 conn(0x7f0294138030 legacy=0x7f02941d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 >> v1:172.21.15.189:6789/0 conn(0x7f029409f230 legacy=0x7f02941cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 monclient: found mon.a 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02940a2990 con 0x7f029409b6a0 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 5 monclient: authenticate success, global_id 334101 2024-09-16T06:56:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _renew_subs 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0294072450 con 0x7f029409b6a0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _renew_subs 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0294133d50 con 0x7f029409b6a0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0288001190 con 0x7f029409b6a0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0288004d60 con 0x7f029409b6a0 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02927fc700 4 set_mon_vals no callback set 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 10 monclient: dump: 2024-09-16T06:56:53.205 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0288004ec0 con 0x7f029409b6a0 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0288043f90 con 0x7f029409b6a0 2024-09-16T06:56:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f0293fff700 10 monclient: get_auth_request con 0x7f02780361e0 auth_method 0 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.190+0000 7f029aa96700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.194+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0274005100 con 0x7f029409b6a0 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.198+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0288014760 con 0x7f029409b6a0 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.198+0000 7f02917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.198+0000 7f02917fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:53.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f029aa96700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:56:53.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f029aa96700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:53.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f029aa96700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0274005930 con 0x7f029409b6a0 2024-09-16T06:56:53.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f02917fa700 1 -- 172.21.15.76:0/2527328147 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 580850641) 0x7f0288012e00 con 0x7f029409b6a0 2024-09-16T06:56:53.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f02917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:53.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.430+0000 7f02917fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:53.430 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 >> v1:172.21.15.189:6812/3098810674 conn(0x7f02780361e0 legacy=0x7f02780386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 10 monclient: shutdown 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 >> v1:172.21.15.76:6789/0 conn(0x7f029409b6a0 legacy=0x7f0294133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 shutdown_connections 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 >> 172.21.15.76:0/2527328147 conn(0x7f02940a3e50 msgr2=0x7f029409e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 shutdown_connections 2024-09-16T06:56:53.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.434+0000 7f029aa96700 1 -- 172.21.15.76:0/2527328147 wait complete. 2024-09-16T06:56:53.448 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:56:53.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:56:53.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: get_monmap_and_config 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: monmap: 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:53.569006+0000 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:53.569006+0000 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.563 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.564 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.564 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:53.564 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:53.564 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 1 Processor -- start 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 1 -- start start 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: init 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient(hunting): picked mon.noname-c con 0x7f525009f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient(hunting): picked mon.noname-a con 0x7f525009d520 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient(hunting): picked mon.noname-b con 0x7f525009fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525009b520 con 0x7f525009d520 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5250071f30 con 0x7f525009f230 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525009b0e0 con 0x7f525009fd20 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient(hunting): authenticate will time out at 1960.702353s 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f52555b7700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f525009fd20 0x7f525013ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53132/0 (socket says 172.21.15.76:53132) 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f525009d520 0x7f5250137440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51488/0 (socket says 172.21.15.76:51488) 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f52555b7700 1 -- 172.21.15.76:0/2987704667 learned_addr learned my addr 172.21.15.76:0/2987704667 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095567837 0 0) 0x7f525009b0e0 con 0x7f525009fd20 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): my global_id is 324185 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c0035d0 con 0x7f525009fd20 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024627733 0 0) 0x7f525009b520 con 0x7f525009d520 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient(hunting): my global_id is 334104 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525009b0e0 con 0x7f525009d520 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586297958 0 0) 0x7f525009b0e0 con 0x7f525009d520 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 >> v1:172.21.15.76:6790/0 conn(0x7f525009fd20 legacy=0x7f525013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 >> v1:172.21.15.189:6789/0 conn(0x7f525009f230 legacy=0x7f525009f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 monclient: found mon.noname-a 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525009c7d0 con 0x7f525009d520 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 5 monclient: authenticate success, global_id 334104 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5248001590 con 0x7f525009d520 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: dump: 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.574 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5248001740 con 0x7f525009d520 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 1 -- 172.21.15.76:0/2987704667 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5248001b10 con 0x7f525009d520 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f525701a700 4 set_mon_vals no callback set 2024-09-16T06:56:53.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.566+0000 7f524f7fe700 10 monclient: dump: 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: shutdown 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/2987704667 >> v1:172.21.15.76:6789/0 conn(0x7f525009d520 legacy=0x7f5250137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/2987704667 shutdown_connections 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/2987704667 >> 172.21.15.76:0/2987704667 conn(0x7f52500a3e50 msgr2=0x7f52500a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/2987704667 shutdown_connections 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/2987704667 wait complete. 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: monmap: 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:53.577 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 Processor -- start 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- start start 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: init 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient(hunting): picked mon.noname-c con 0x7f525009f230 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient(hunting): picked mon.noname-b con 0x7f525009fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient(hunting): picked mon.noname-a con 0x7f52501397d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500a2a10 con 0x7f52501397d0 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5250072450 con 0x7f525009fd20 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525013af50 con 0x7f525009f230 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient(hunting): authenticate will time out at 1960.706404s 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f5254db6700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f525009f230 0x7f52501389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53148/0 (socket says 172.21.15.76:53148) 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f5254db6700 1 -- 172.21.15.76:0/91769009 learned_addr learned my addr 172.21.15.76:0/91769009 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268976199 0 0) 0x7f525013af50 con 0x7f525009f230 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): my global_id is 324188 2024-09-16T06:56:53.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52300035d0 con 0x7f525009f230 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445871170 0 0) 0x7f52500a2a10 con 0x7f52501397d0 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): my global_id is 334107 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525013af50 con 0x7f52501397d0 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056813381 0 0) 0x7f5250072450 con 0x7f525009fd20 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): my global_id is 284572 2024-09-16T06:56:53.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52500a2a10 con 0x7f525009fd20 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1068670603 0 0) 0x7f525013af50 con 0x7f52501397d0 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5250072450 con 0x7f52501397d0 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5240001470 con 0x7f52501397d0 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): dump: 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.581 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189414087 0 0) 0x7f52300035d0 con 0x7f525009f230 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f525013af50 con 0x7f525009f230 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5238001930 con 0x7f525009f230 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient(hunting): dump: 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.582 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 493056484 0 0) 0x7f5250072450 con 0x7f52501397d0 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6790/0 conn(0x7f525009f230 legacy=0x7f52501389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.189:6789/0 conn(0x7f525009fd20 legacy=0x7f52501390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 monclient: found mon.a 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525011aa90 con 0x7f52501397d0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f524d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 5 monclient: authenticate success, global_id 334107 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _renew_subs 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525011b330 con 0x7f52501397d0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _renew_subs 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.570+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5250119e70 con 0x7f52501397d0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5240004c10 con 0x7f52501397d0 2024-09-16T06:56:53.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5240004f20 con 0x7f52501397d0 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524e7fc700 4 set_mon_vals no callback set 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: dump: 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f525701a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5234006590 con 0x7f5234002830 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f524000f990 con 0x7f52501397d0 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5240044080 con 0x7f52501397d0 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708949738 0 0) 0x7f5234006590 con 0x7f5234002830 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient(hunting): my global_id is 334110 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250072450 con 0x7f5234002830 2024-09-16T06:56:53.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524ffff700 10 monclient: get_auth_request con 0x7f5230036660 auth_method 0 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2660177841 0 0) 0x7f5250072450 con 0x7f5234002830 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5234006590 con 0x7f5234002830 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5238003860 con 0x7f5234002830 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: dump: 2024-09-16T06:56:53.589 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2696694559 0 0) 0x7f5234006590 con 0x7f5234002830 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f523400a190 con 0x7f5234002830 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5238005870 con 0x7f5234002830 2024-09-16T06:56:53.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6789/0 conn(0x7f5234002830 legacy=0x7f5234004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.574+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6789/0 conn(0x7f5234002830 legacy=0x7f5234004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5234009e30 con 0x7f5234009150 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492559856 0 0) 0x7f5234009e30 con 0x7f5234009150 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient(hunting): my global_id is 334113 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5234006590 con 0x7f5234009150 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1979684527 0 0) 0x7f5234006590 con 0x7f5234009150 2024-09-16T06:56:53.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5234009e30 con 0x7f5234009150 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5238003ed0 con 0x7f5234009150 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient: dump: 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.592 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3425321296 0 0) 0x7f5234009e30 con 0x7f5234009150 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f523400c340 con 0x7f5234009150 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3716053967) 0x7f5238004190 con 0x7f5234009150 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6789/0 conn(0x7f5234009150 legacy=0x7f52340095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f524d7fa700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6789/0 conn(0x7f5234009150 legacy=0x7f52340095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:53.593 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:53.594 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.595 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.596 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:53.597 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:53.598 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.599 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.600 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.601 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:53.602 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.603 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.604 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:53.605 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5230036660 legacy=0x7f5230038b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 10 monclient: shutdown 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 >> v1:172.21.15.76:6789/0 conn(0x7f52501397d0 legacy=0x7f52501d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f52555b7700 1 -- 172.21.15.76:0/91769009 reap_dead start 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f52555b7700 1 -- 172.21.15.76:0/91769009 reap_dead start 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 shutdown_connections 2024-09-16T06:56:53.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 >> 172.21.15.76:0/91769009 conn(0x7f52500a3e50 msgr2=0x7f52500a2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 shutdown_connections 2024-09-16T06:56:53.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.586+0000 7f525701a700 1 -- 172.21.15.76:0/91769009 wait complete. 2024-09-16T06:56:53.608 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:53.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:56:53.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:56:53.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: get_monmap_and_config 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: monmap: 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:53.732621+0000 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:53.732621+0000 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:53.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 1 Processor -- start 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 1 -- start start 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: init 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-b con 0x7f8f8c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-a con 0x7f8f8c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-c con 0x7f8f8c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c06c280 con 0x7f8f8c06d480 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c06bfa0 con 0x7f8f8c06cf50 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c09ace0 con 0x7f8f8c132330 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient(hunting): authenticate will time out at 1960.866237s 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f914c4700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8f8c132330 0x7f8f8c132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53160/0 (socket says 172.21.15.76:53160) 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f914c4700 1 -- 172.21.15.76:0/1400729202 learned_addr learned my addr 172.21.15.76:0/1400729202 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304529394 0 0) 0x7f8f8c06c280 con 0x7f8f8c06d480 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): my global_id is 334116 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f780035d0 con 0x7f8f8c06d480 2024-09-16T06:56:53.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665230292 0 0) 0x7f8f8c09ace0 con 0x7f8f8c132330 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): my global_id is 324191 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c06c280 con 0x7f8f8c132330 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703464624 0 0) 0x7f8f8c06bfa0 con 0x7f8f8c06cf50 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient(hunting): my global_id is 284575 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c09ace0 con 0x7f8f8c06cf50 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859230520 0 0) 0x7f8f780035d0 con 0x7f8f8c06d480 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c132330 legacy=0x7f8f8c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 >> v1:172.21.15.189:6789/0 conn(0x7f8f8c06cf50 legacy=0x7f8f8c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 monclient: found mon.noname-a 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c09ae70 con 0x7f8f8c06d480 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 5 monclient: authenticate success, global_id 334116 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f74001610 con 0x7f8f8c06d480 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: dump: 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.735 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f740018f0 con 0x7f8f8c06d480 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 1 -- 172.21.15.76:0/1400729202 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f74001b20 con 0x7f8f8c06d480 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f924c6700 4 set_mon_vals no callback set 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.730+0000 7f8f8bfff700 10 monclient: dump: 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 10 monclient: shutdown 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- 172.21.15.76:0/1400729202 >> v1:172.21.15.76:6789/0 conn(0x7f8f8c06d480 legacy=0x7f8f8c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- 172.21.15.76:0/1400729202 shutdown_connections 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- 172.21.15.76:0/1400729202 >> 172.21.15.76:0/1400729202 conn(0x7f8f8c0a41c0 msgr2=0x7f8f8c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- 172.21.15.76:0/1400729202 shutdown_connections 2024-09-16T06:56:53.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- 172.21.15.76:0/1400729202 wait complete. 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 10 monclient: monmap: 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:53.743 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 Processor -- start 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 1 -- start start 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.734+0000 7f8f924c6700 10 monclient: init 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-b con 0x7f8f8c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-a con 0x7f8f8c06d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient(hunting): picked mon.noname-c con 0x7f8f8c132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c130650 con 0x7f8f8c06d480 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c09e8b0 con 0x7f8f8c06cf50 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c12df20 con 0x7f8f8c132330 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient(hunting): authenticate will time out at 1960.872853s 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f91cc5700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8f8c132330 0x7f8f8c1d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53164/0 (socket says 172.21.15.76:53164) 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f91cc5700 1 -- 172.21.15.76:0/348606592 learned_addr learned my addr 172.21.15.76:0/348606592 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633058791 0 0) 0x7f8f8c12df20 con 0x7f8f8c132330 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): my global_id is 324194 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c0035d0 con 0x7f8f8c132330 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397150216 0 0) 0x7f8f8c130650 con 0x7f8f8c06d480 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): my global_id is 334119 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c12df20 con 0x7f8f8c06d480 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370712504 0 0) 0x7f8f8c09e8b0 con 0x7f8f8c06cf50 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): my global_id is 284578 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c130650 con 0x7f8f8c06cf50 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072869785 0 0) 0x7f8f6c0035d0 con 0x7f8f8c132330 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f8c09e8b0 con 0x7f8f8c132330 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f84001c50 con 0x7f8f8c132330 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): dump: 2024-09-16T06:56:53.746 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748331190 0 0) 0x7f8f8c12df20 con 0x7f8f8c06d480 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f6c0035d0 con 0x7f8f8c06d480 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f74001000 con 0x7f8f8c06d480 2024-09-16T06:56:53.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient(hunting): dump: 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399254796 0 0) 0x7f8f8c130650 con 0x7f8f8c06cf50 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f8c12df20 con 0x7f8f8c06cf50 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1822597958 0 0) 0x7f8f8c09e8b0 con 0x7f8f8c132330 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.189:6789/0 conn(0x7f8f8c06cf50 legacy=0x7f8f8c12d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6789/0 conn(0x7f8f8c06d480 legacy=0x7f8f8c12d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 monclient: found mon.c 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c131140 con 0x7f8f8c132330 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 5 monclient: authenticate success, global_id 324194 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _renew_subs 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c131650 con 0x7f8f8c132330 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _renew_subs 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c13eb60 con 0x7f8f8c132330 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f84001fe0 con 0x7f8f8c132330 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f8affd700 4 set_mon_vals no callback set 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f84003ed0 con 0x7f8f8c132330 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 10 monclient: dump: 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8f840040b0 con 0x7f8f8c132330 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.738+0000 7f8f924c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8f84044290 con 0x7f8f8c132330 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f8c130800 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f914c4700 10 monclient: get_auth_request con 0x7f8f6c036500 auth_method 0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461902696 0 0) 0x7f8f6c0035d0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient(hunting): my global_id is 324197 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c09e8b0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1611577252 0 0) 0x7f8f8c09e8b0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f6c0035d0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f74003910 con 0x7f8f8c09fad0 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient: dump: 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2572434579 0 0) 0x7f8f6c0035d0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f8c0a3450 con 0x7f8f8c09fad0 2024-09-16T06:56:53.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8f740056e0 con 0x7f8f8c09fad0 2024-09-16T06:56:53.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:53.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c09fad0 legacy=0x7f8f8c09feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.742+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c09fad0 legacy=0x7f8f8c09feb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8f8c130800 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550397038 0 0) 0x7f8f74003e70 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient(hunting): my global_id is 324200 2024-09-16T06:56:53.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c0035d0 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2602378317 0 0) 0x7f8f6c0035d0 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f74003e70 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f74004270 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient: dump: 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.756 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 508380228 0 0) 0x7f8f74003e70 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f8c12ebc0 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 950589659) 0x7f8f74005850 con 0x7f8f8c0a05c0 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c0a05c0 legacy=0x7f8f8c02c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f89ffb700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c0a05c0 legacy=0x7f8f8c02c6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.757 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:53.758 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:53.759 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:53.760 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:53.761 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.762 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.763 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.765 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.766 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:56:53.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.768 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:53.769 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8f6c036500 legacy=0x7f8f6c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 10 monclient: shutdown 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 >> v1:172.21.15.76:6790/0 conn(0x7f8f8c132330 legacy=0x7f8f8c1d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f91cc5700 1 -- 172.21.15.76:0/348606592 reap_dead start 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f91cc5700 1 -- 172.21.15.76:0/348606592 reap_dead start 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 shutdown_connections 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 >> 172.21.15.76:0/348606592 conn(0x7f8f8c0a41c0 msgr2=0x7f8f8c09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 shutdown_connections 2024-09-16T06:56:53.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.750+0000 7f8f924c6700 1 -- 172.21.15.76:0/348606592 wait complete. 2024-09-16T06:56:53.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:53.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:56:53.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 10 monclient: get_monmap_and_config 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 10 monclient: monmap: 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:53.895116+0000 2024-09-16T06:56:53.889 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:53.895116+0000 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:53.890 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 1 Processor -- start 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 1 -- start start 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.890+0000 7f010eec9700 10 monclient: init 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-a con 0x7f01080973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-c con 0x7f0108097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-b con 0x7f01081400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0108099350 con 0x7f01080973c0 2024-09-16T06:56:53.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010809f690 con 0x7f0108097eb0 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0108099bf0 con 0x7f01081400d0 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient(hunting): authenticate will time out at 1961.028624s 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010dec7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f01080973c0 0x7f01080977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47838/0 (socket says 172.21.15.76:47838) 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010dec7700 1 -- 172.21.15.76:0/3598096277 learned_addr learned my addr 172.21.15.76:0/3598096277 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475830891 0 0) 0x7f0108099350 con 0x7f01080973c0 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): my global_id is 334122 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f40035d0 con 0x7f01080973c0 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266633946 0 0) 0x7f0108099bf0 con 0x7f01081400d0 2024-09-16T06:56:53.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient(hunting): my global_id is 324203 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0108099350 con 0x7f01081400d0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275056150 0 0) 0x7f00f40035d0 con 0x7f01080973c0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 >> v1:172.21.15.76:6790/0 conn(0x7f01081400d0 legacy=0x7f01081424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 >> v1:172.21.15.189:6789/0 conn(0x7f0108097eb0 legacy=0x7f010813e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 monclient: found mon.noname-a 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0108098360 con 0x7f01080973c0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 5 monclient: authenticate success, global_id 334122 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0100001990 con 0x7f01080973c0 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: dump: 2024-09-16T06:56:53.905 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:53.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0100001bb0 con 0x7f01080973c0 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 1 -- 172.21.15.76:0/3598096277 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0100001f40 con 0x7f01080973c0 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 4 set_mon_vals no callback set 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010cec5700 10 monclient: dump: 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.907 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: shutdown 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- 172.21.15.76:0/3598096277 >> v1:172.21.15.76:6789/0 conn(0x7f01080973c0 legacy=0x7f01080977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- 172.21.15.76:0/3598096277 shutdown_connections 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- 172.21.15.76:0/3598096277 >> 172.21.15.76:0/3598096277 conn(0x7f01080a0030 msgr2=0x7f01080a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- 172.21.15.76:0/3598096277 shutdown_connections 2024-09-16T06:56:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 1 -- 172.21.15.76:0/3598096277 wait complete. 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: build_initial_monmap 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.894+0000 7f010eec9700 10 monclient: monmap: 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 Processor -- start 2024-09-16T06:56:53.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- start start 2024-09-16T06:56:53.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: init 2024-09-16T06:56:53.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:53.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-c con 0x7f01080973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-a con 0x7f0108097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient(hunting): picked mon.noname-b con 0x7f01081400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0108072660 con 0x7f0108097eb0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010806c410 con 0x7f01081400d0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010813df30 con 0x7f01080973c0 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient(hunting): authenticate will time out at 1961.032802s 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010d6c6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0108097eb0 0x7f010813c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47848/0 (socket says 172.21.15.76:47848) 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010d6c6700 1 -- 172.21.15.76:0/247564718 learned_addr learned my addr 172.21.15.76:0/247564718 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:53.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721513254 0 0) 0x7f0108072660 con 0x7f0108097eb0 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): my global_id is 334125 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ec0035d0 con 0x7f0108097eb0 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178609010 0 0) 0x7f010813df30 con 0x7f01080973c0 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): my global_id is 324206 2024-09-16T06:56:53.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0108072660 con 0x7f01080973c0 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114692577 0 0) 0x7f00ec0035d0 con 0x7f0108097eb0 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f010813df30 con 0x7f0108097eb0 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00fc001960 con 0x7f0108097eb0 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): dump: 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.914 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178204369 0 0) 0x7f0108072660 con 0x7f01080973c0 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00ec0035d0 con 0x7f01080973c0 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576093490 0 0) 0x7f010806c410 con 0x7f01081400d0 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): my global_id is 284584 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0108072660 con 0x7f01081400d0 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0100001950 con 0x7f01080973c0 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient(hunting): dump: 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.916 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3395024835 0 0) 0x7f010813df30 con 0x7f0108097eb0 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.76:6790/0 conn(0x7f01080973c0 legacy=0x7f010813c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6789/0 conn(0x7f01081400d0 legacy=0x7f010813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 monclient: found mon.a 2024-09-16T06:56:53.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010813f170 con 0x7f0108097eb0 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: _finish_auth 0 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 5 monclient: authenticate success, global_id 334125 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _renew_subs 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01080a3100 con 0x7f0108097eb0 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _renew_subs 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f010813f7c0 con 0x7f0108097eb0 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00fc0036c0 con 0x7f0108097eb0 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f00fc0039d0 con 0x7f0108097eb0 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fbfff700 4 set_mon_vals no callback set 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 10 monclient: dump: 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.919 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f00fc003b30 con 0x7f0108097eb0 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.898+0000 7f010eec9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00f0006590 con 0x7f00f0002830 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f00fc043d20 con 0x7f0108097eb0 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f010dec7700 10 monclient: get_auth_request con 0x7f00ec0364a0 auth_method 0 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331572615 0 0) 0x7f00f0006590 con 0x7f00f0002830 2024-09-16T06:56:53.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient(hunting): my global_id is 284587 2024-09-16T06:56:53.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010813df30 con 0x7f00f0002830 2024-09-16T06:56:53.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4276987080 0 0) 0x7f010813df30 con 0x7f00f0002830 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f0006590 con 0x7f00f0002830 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f01040016c0 con 0x7f00f0002830 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.902+0000 7f00fa7fc700 10 monclient: dump: 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.922 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.923 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.923 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 875932938 0 0) 0x7f00f0006590 con 0x7f00f0002830 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00f000a190 con 0x7f00f0002830 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f0104001ab0 con 0x7f00f0002830 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6789/0 conn(0x7f00f0002830 legacy=0x7f00f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.906+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6789/0 conn(0x7f00f0002830 legacy=0x7f00f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.914+0000 7f010eec9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.914+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f00f0009e50 con 0x7f00f0009170 2024-09-16T06:56:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780613703 0 0) 0x7f00f0009e50 con 0x7f00f0009170 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient(hunting): my global_id is 284590 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f0006590 con 0x7f00f0009170 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1319827418 0 0) 0x7f00f0006590 con 0x7f00f0009170 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f0009e50 con 0x7f00f0009170 2024-09-16T06:56:53.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f01040038b0 con 0x7f00f0009170 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient: dump: 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:53.927 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4040230647 0 0) 0x7f00f0009e50 con 0x7f00f0009170 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f00f000c340 con 0x7f00f0009170 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3973765116) 0x7f0104005a90 con 0x7f00f0009170 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6789/0 conn(0x7f00f0009170 legacy=0x7f00f0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f00fa7fc700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6789/0 conn(0x7f00f0009170 legacy=0x7f00f0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:53.929 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:53.929 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:53.930 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:56:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:53.931 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.932 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.933 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.935 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:53.936 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:53.937 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:53.938 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:53.939 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:53.940 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:53.941 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.942 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:53.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.942 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:53.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.942 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:53.943 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:53.943 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:53.944 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.944 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.944 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.946 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:53.946 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:53.946 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:53.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.948 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.949 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.949 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:53.949 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:53.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:53.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.951 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:53.952 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:53.952 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:53.952 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:53.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.953 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.954 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:53.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.954 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:53.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.955 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.955 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.955 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:56:53.955 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.955 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:53.956 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:56:53.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:53.956 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:53.956 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:53.956 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:53.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:53.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:53.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:53.957 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:53.957 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:53.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.189:6812/3098810674 conn(0x7f00ec0364a0 legacy=0x7f00ec038960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 10 monclient: shutdown 2024-09-16T06:56:53.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:53.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 >> v1:172.21.15.76:6789/0 conn(0x7f0108097eb0 legacy=0x7f010813c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010e6c8700 1 -- 172.21.15.76:0/247564718 reap_dead start 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010e6c8700 1 -- 172.21.15.76:0/247564718 reap_dead start 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 shutdown_connections 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 >> 172.21.15.76:0/247564718 conn(0x7f01080a0030 msgr2=0x7f010809cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 shutdown_connections 2024-09-16T06:56:53.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:53.918+0000 7f010eec9700 1 -- 172.21.15.76:0/247564718 wait complete. 2024-09-16T06:56:53.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:53.962 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:56:53.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: get_monmap_and_config 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: build_initial_monmap 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: monmap: 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:54.085025+0000 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:54.085025+0000 2024-09-16T06:56:54.079 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:54.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:54.080 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 1 Processor -- start 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 1 -- start start 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: init 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:54.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-c con 0x7f08dc0973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-a con 0x7f08dc097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-b con 0x7f08dc1400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc099350 con 0x7f08dc097eb0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc09f690 con 0x7f08dc0973c0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc099bf0 con 0x7f08dc1400d0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient(hunting): authenticate will time out at 1961.218417s 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08db7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f08dc097eb0 0x7f08dc13e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47864/0 (socket says 172.21.15.76:47864) 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08db7fe700 1 -- 172.21.15.76:0/331326953 learned_addr learned my addr 172.21.15.76:0/331326953 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190164566 0 0) 0x7f08dc099bf0 con 0x7f08dc1400d0 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): my global_id is 324209 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c40035d0 con 0x7f08dc1400d0 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180520666 0 0) 0x7f08dc099350 con 0x7f08dc097eb0 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): my global_id is 334128 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc099bf0 con 0x7f08dc097eb0 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002887350 0 0) 0x7f08dc09f690 con 0x7f08dc0973c0 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient(hunting): my global_id is 284593 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc099350 con 0x7f08dc0973c0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004425774 0 0) 0x7f08c40035d0 con 0x7f08dc1400d0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 >> v1:172.21.15.189:6789/0 conn(0x7f08dc0973c0 legacy=0x7f08dc0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 >> v1:172.21.15.76:6789/0 conn(0x7f08dc097eb0 legacy=0x7f08dc13e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: _finish_hunting 0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 monclient: found mon.noname-b 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc098360 con 0x7f08dc1400d0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: _finish_auth 0 2024-09-16T06:56:54.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: _check_auth_tickets 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08d4001990 con 0x7f08dc1400d0 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: dump: 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 5 monclient: authenticate success, global_id 324209 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d4001c70 con 0x7f08dc1400d0 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 1 -- 172.21.15.76:0/331326953 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08d4002000 con 0x7f08dc1400d0 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08e15ce700 4 set_mon_vals no callback set 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.082+0000 7f08daffd700 10 monclient: dump: 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: shutdown 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:54.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- 172.21.15.76:0/331326953 >> v1:172.21.15.76:6790/0 conn(0x7f08dc1400d0 legacy=0x7f08dc1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- 172.21.15.76:0/331326953 shutdown_connections 2024-09-16T06:56:54.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- 172.21.15.76:0/331326953 >> 172.21.15.76:0/331326953 conn(0x7f08dc0a0030 msgr2=0x7f08dc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:54.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- 172.21.15.76:0/331326953 shutdown_connections 2024-09-16T06:56:54.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- 172.21.15.76:0/331326953 wait complete. 2024-09-16T06:56:54.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: build_initial_monmap 2024-09-16T06:56:54.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: monmap: 2024-09-16T06:56:54.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 Processor -- start 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- start start 2024-09-16T06:56:54.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: init 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-c con 0x7f08dc0973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-b con 0x7f08dc097eb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient(hunting): picked mon.noname-a con 0x7f08dc1400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc072660 con 0x7f08dc1400d0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc13fb70 con 0x7f08dc097eb0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08dc06c910 con 0x7f08dc0973c0 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08e15ce700 10 monclient(hunting): authenticate will time out at 1961.223876s 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08dbfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f08dc0973c0 0x7f08dc1ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43464/0 (socket says 172.21.15.76:43464) 2024-09-16T06:56:54.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.086+0000 7f08dbfff700 1 -- 172.21.15.76:0/219750416 learned_addr learned my addr 172.21.15.76:0/219750416 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367667383 0 0) 0x7f08dc06c910 con 0x7f08dc0973c0 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): my global_id is 324212 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c80035d0 con 0x7f08dc0973c0 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670152110 0 0) 0x7f08dc072660 con 0x7f08dc1400d0 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): my global_id is 334131 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08dc06c910 con 0x7f08dc1400d0 2024-09-16T06:56:54.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3268371371 0 0) 0x7f08c80035d0 con 0x7f08dc0973c0 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08dc072660 con 0x7f08dc0973c0 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08d0001970 con 0x7f08dc0973c0 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): dump: 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:54.097 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57192441 0 0) 0x7f08dc13fb70 con 0x7f08dc097eb0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): my global_id is 284596 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c80035d0 con 0x7f08dc097eb0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 616675630 0 0) 0x7f08dc06c910 con 0x7f08dc1400d0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08dc13fb70 con 0x7f08dc1400d0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08d4001950 con 0x7f08dc1400d0 2024-09-16T06:56:54.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient(hunting): dump: 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1968825086 0 0) 0x7f08dc072660 con 0x7f08dc0973c0 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 >> v1:172.21.15.189:6789/0 conn(0x7f08dc097eb0 legacy=0x7f08dc1d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 >> v1:172.21.15.76:6789/0 conn(0x7f08dc1400d0 legacy=0x7f08dc1d6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 monclient: found mon.c 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc09a540 con 0x7f08dc0973c0 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: _finish_auth 0 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 5 monclient: authenticate success, global_id 324212 2024-09-16T06:56:54.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _renew_subs 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08dc13eec0 con 0x7f08dc0973c0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _renew_subs 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08dc13e2f0 con 0x7f08dc0973c0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d0001ce0 con 0x7f08dc0973c0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f08d0003b10 con 0x7f08dc0973c0 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08da7fc700 4 set_mon_vals no callback set 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 10 monclient: dump: 2024-09-16T06:56:54.101 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f08d0003cf0 con 0x7f08dc0973c0 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:54.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc005100 con 0x7f08dc0973c0 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f08d0043f90 con 0x7f08dc0973c0 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.090+0000 7f08db7fe700 10 monclient: get_auth_request con 0x7f08c80364b0 auth_method 0 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.098+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f08d00148d0 con 0x7f08dc0973c0 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.098+0000 7f08d8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:54.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.098+0000 7f08d8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:54.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08e15ce700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:56:54.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08e15ce700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:56:54.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08e15ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:54.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f08bc005bc0 con 0x7f08dc0973c0 2024-09-16T06:56:54.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08d8ff9700 1 -- 172.21.15.76:0/219750416 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f08d0012d40 con 0x7f08dc0973c0 2024-09-16T06:56:54.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08d8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:56:54.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f08d8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:54.337 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.338 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=768178776,config=1097430493,health=139399798,kv=283846835,logm=3826399842}) 2024-09-16T06:56:54.338 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=10,kv=13,logm=49} crc {auth=3009356955,config=1228700967,health=2810073241,kv=1001081567,logm=1483233395}) 2024-09-16T06:56:54.338 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.338 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=768178776,config=1097430493,health=139399798,kv=283846835,logm=3826399842}) 2024-09-16T06:56:54.338 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.338+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=13,logm=48} crc {auth=286786434,config=1097430493,health=139399798,kv=1001081567,logm=1638715154}) 2024-09-16T06:56:54.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 >> v1:172.21.15.189:6812/3098810674 conn(0x7f08c80364b0 legacy=0x7f08c8038970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 10 monclient: shutdown 2024-09-16T06:56:54.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:54.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 >> v1:172.21.15.76:6790/0 conn(0x7f08dc0973c0 legacy=0x7f08dc1ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:54.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 shutdown_connections 2024-09-16T06:56:54.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 >> 172.21.15.76:0/219750416 conn(0x7f08dc0a0030 msgr2=0x7f08dc095810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:54.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 shutdown_connections 2024-09-16T06:56:54.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f08e15ce700 1 -- 172.21.15.76:0/219750416 wait complete. 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156589959}) 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223489145}) 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156589959}) 2024-09-16T06:56:54.341 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.342+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467473130}) 2024-09-16T06:56:54.343 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619232678}) 2024-09-16T06:56:54.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115932259}) 2024-09-16T06:56:54.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619232678}) 2024-09-16T06:56:54.344 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.346+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524647074}) 2024-09-16T06:56:54.346 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.346 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731471066}) 2024-09-16T06:56:54.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028556987}) 2024-09-16T06:56:54.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731471066}) 2024-09-16T06:56:54.347 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.350+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1661362810}) 2024-09-16T06:56:54.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222006736}) 2024-09-16T06:56:54.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262217624}) 2024-09-16T06:56:54.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.350 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222006736}) 2024-09-16T06:56:54.351 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461753389}) 2024-09-16T06:56:54.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:56:54.352 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.352 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234680533}) 2024-09-16T06:56:54.352 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3729144351}) 2024-09-16T06:56:54.352 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234680533}) 2024-09-16T06:56:54.353 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3366560727}) 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.354+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159770280}) 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205800449}) 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159770280}) 2024-09-16T06:56:54.354 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3582299978}) 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=786271179}) 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1880399471}) 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=786271179}) 2024-09-16T06:56:54.355 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367793166}) 2024-09-16T06:56:54.356 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.356 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470249274}) 2024-09-16T06:56:54.356 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023923552}) 2024-09-16T06:56:54.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470249274}) 2024-09-16T06:56:54.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488994644}) 2024-09-16T06:56:54.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.357 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680933749}) 2024-09-16T06:56:54.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869287316}) 2024-09-16T06:56:54.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680933749}) 2024-09-16T06:56:54.358 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.358+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245912669}) 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739390558}) 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942772143}) 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739390558}) 2024-09-16T06:56:54.359 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778599666}) 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071925316}) 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555257591}) 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071925316}) 2024-09-16T06:56:54.360 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404671565}) 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079581449}) 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871096378}) 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079581449}) 2024-09-16T06:56:54.361 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506827135}) 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743652192}) 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862226139}) 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743652192}) 2024-09-16T06:56:54.362 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.362+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822183856}) 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250631136}) 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432411978}) 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250631136}) 2024-09-16T06:56:54.363 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503597723}) 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932793941}) 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=923774170}) 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932793941}) 2024-09-16T06:56:54.364 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=518924156}) 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235613871}) 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398774910}) 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235613871}) 2024-09-16T06:56:54.365 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.366+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1154322009}) 2024-09-16T06:56:54.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511800770}) 2024-09-16T06:56:54.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3958561612}) 2024-09-16T06:56:54.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.366 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511800770}) 2024-09-16T06:56:54.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199800708}) 2024-09-16T06:56:54.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550768298}) 2024-09-16T06:56:54.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918400960}) 2024-09-16T06:56:54.367 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550768298}) 2024-09-16T06:56:54.368 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136378785}) 2024-09-16T06:56:54.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367679190}) 2024-09-16T06:56:54.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319164982}) 2024-09-16T06:56:54.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.369 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367679190}) 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.370+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=719475339}) 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318864363}) 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327388417}) 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.370 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318864363}) 2024-09-16T06:56:54.371 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311015561}) 2024-09-16T06:56:54.371 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633261852}) 2024-09-16T06:56:54.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809497755}) 2024-09-16T06:56:54.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633261852}) 2024-09-16T06:56:54.372 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2433030574}) 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646637702}) 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526980283}) 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646637702}) 2024-09-16T06:56:54.373 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.374+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287644327}) 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800574767}) 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302623844}) 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800574767}) 2024-09-16T06:56:54.374 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545197937}) 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451676531}) 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762385240}) 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451676531}) 2024-09-16T06:56:54.375 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754434036}) 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910692332}) 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675533524}) 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910692332}) 2024-09-16T06:56:54.376 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=458386232}) 2024-09-16T06:56:54.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978109941}) 2024-09-16T06:56:54.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86324736}) 2024-09-16T06:56:54.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.377 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2978109941}) 2024-09-16T06:56:54.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.378+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909519237}) 2024-09-16T06:56:54.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184393273}) 2024-09-16T06:56:54.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=858405662}) 2024-09-16T06:56:54.378 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184393273}) 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1876945214}) 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399242007}) 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189928316}) 2024-09-16T06:56:54.379 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399242007}) 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837154}) 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95084396}) 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221620113}) 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.380 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95084396}) 2024-09-16T06:56:54.381 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471496748}) 2024-09-16T06:56:54.381 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.381 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208184670}) 2024-09-16T06:56:54.381 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344642137}) 2024-09-16T06:56:54.381 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.382 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208184670}) 2024-09-16T06:56:54.382 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.382+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163634206}) 2024-09-16T06:56:54.382 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.382 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955140915}) 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700184496}) 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955140915}) 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515676587}) 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.383 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907512999}) 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37059569}) 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907512999}) 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467370998}) 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255486324}) 2024-09-16T06:56:54.384 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314522839}) 2024-09-16T06:56:54.385 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.385 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255486324}) 2024-09-16T06:56:54.385 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.386+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878019296}) 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163327025}) 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772352123}) 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163327025}) 2024-09-16T06:56:54.386 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450000807}) 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042710694}) 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324648066}) 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042710694}) 2024-09-16T06:56:54.387 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547326014}) 2024-09-16T06:56:54.388 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.388 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146736709}) 2024-09-16T06:56:54.388 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011522055}) 2024-09-16T06:56:54.388 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.389 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146736709}) 2024-09-16T06:56:54.389 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.390+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26868951}) 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31147478}) 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4073997003}) 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31147478}) 2024-09-16T06:56:54.390 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819348650}) 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625002437}) 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503321465}) 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625002437}) 2024-09-16T06:56:54.392 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668629685}) 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1723395537}) 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224894456}) 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1723395537}) 2024-09-16T06:56:54.393 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.394+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045547529}) 2024-09-16T06:56:54.394 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.395 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706171871}) 2024-09-16T06:56:54.395 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279739356}) 2024-09-16T06:56:54.395 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.395 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706171871}) 2024-09-16T06:56:54.395 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=791442404}) 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315394990}) 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801947750}) 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315394990}) 2024-09-16T06:56:54.396 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50236351}) 2024-09-16T06:56:54.397 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.397 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994102379}) 2024-09-16T06:56:54.398 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837758443}) 2024-09-16T06:56:54.398 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.398 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994102379}) 2024-09-16T06:56:54.398 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.398+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244266717}) 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2186747457}) 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997489352}) 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2186747457}) 2024-09-16T06:56:54.399 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777074386}) 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437106369}) 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617014897}) 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437106369}) 2024-09-16T06:56:54.400 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572997744}) 2024-09-16T06:56:54.401 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.401 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958523970}) 2024-09-16T06:56:54.401 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173167214}) 2024-09-16T06:56:54.401 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.401 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958523970}) 2024-09-16T06:56:54.402 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.402+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409273722}) 2024-09-16T06:56:54.402 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.402 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061698641}) 2024-09-16T06:56:54.402 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117278630}) 2024-09-16T06:56:54.403 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.403 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061698641}) 2024-09-16T06:56:54.403 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629733150}) 2024-09-16T06:56:54.403 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.404 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=721716057}) 2024-09-16T06:56:54.404 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980028842}) 2024-09-16T06:56:54.404 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.404 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=721716057}) 2024-09-16T06:56:54.404 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2933289185}) 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983866390}) 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179453923}) 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983866390}) 2024-09-16T06:56:54.405 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.406+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019647972}) 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=314134458}) 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138419437}) 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=314134458}) 2024-09-16T06:56:54.406 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324777858}) 2024-09-16T06:56:54.407 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.407 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952256002}) 2024-09-16T06:56:54.407 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951427136}) 2024-09-16T06:56:54.408 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.408 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952256002}) 2024-09-16T06:56:54.408 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918212890}) 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3438860526}) 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074637315}) 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3438860526}) 2024-09-16T06:56:54.409 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.410+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468113303}) 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330086346}) 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601315363}) 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330086346}) 2024-09-16T06:56:54.410 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608378999}) 2024-09-16T06:56:54.411 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.411 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=451323169}) 2024-09-16T06:56:54.412 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750041534}) 2024-09-16T06:56:54.412 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.412 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=451323169}) 2024-09-16T06:56:54.412 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896818171}) 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252786570}) 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140023121}) 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252786570}) 2024-09-16T06:56:54.413 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.414+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028134770}) 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774643402}) 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2924372976}) 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774643402}) 2024-09-16T06:56:54.414 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091402052}) 2024-09-16T06:56:54.415 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.415 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37601309}) 2024-09-16T06:56:54.416 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169675524}) 2024-09-16T06:56:54.416 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.416 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37601309}) 2024-09-16T06:56:54.416 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134742042}) 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4116241628}) 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170638558}) 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4116241628}) 2024-09-16T06:56:54.417 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.418+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179095809}) 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261362486}) 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3700545809}) 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261362486}) 2024-09-16T06:56:54.418 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040944556}) 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045693352}) 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941891347}) 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045693352}) 2024-09-16T06:56:54.419 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783354292}) 2024-09-16T06:56:54.420 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.420 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213049403}) 2024-09-16T06:56:54.420 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=996651188}) 2024-09-16T06:56:54.420 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.421 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213049403}) 2024-09-16T06:56:54.421 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3252049890}) 2024-09-16T06:56:54.421 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.421 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615265636}) 2024-09-16T06:56:54.422 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419116516}) 2024-09-16T06:56:54.422 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.422 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615265636}) 2024-09-16T06:56:54.422 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.422+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948740309}) 2024-09-16T06:56:54.422 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.423 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019592041}) 2024-09-16T06:56:54.423 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319748571}) 2024-09-16T06:56:54.423 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.423 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019592041}) 2024-09-16T06:56:54.423 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102232182}) 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247755271}) 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488061178}) 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247755271}) 2024-09-16T06:56:54.424 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072716983}) 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692179977}) 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524746950}) 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692179977}) 2024-09-16T06:56:54.425 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.426+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4184489372}) 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042981875}) 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159029860}) 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042981875}) 2024-09-16T06:56:54.426 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2934401484}) 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871778795}) 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032313351}) 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871778795}) 2024-09-16T06:56:54.427 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1853069664}) 2024-09-16T06:56:54.428 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.428 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648408320}) 2024-09-16T06:56:54.428 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2707387686}) 2024-09-16T06:56:54.429 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.429 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648408320}) 2024-09-16T06:56:54.429 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591132520}) 2024-09-16T06:56:54.429 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.430+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.430 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411847028}) 2024-09-16T06:56:54.430 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911022555}) 2024-09-16T06:56:54.430 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.430 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411847028}) 2024-09-16T06:56:54.430 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359085106}) 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136513422}) 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652167817}) 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136513422}) 2024-09-16T06:56:54.431 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766927447}) 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765702497}) 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51329151}) 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1765702497}) 2024-09-16T06:56:54.432 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693825307}) 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423187952}) 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683849422}) 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423187952}) 2024-09-16T06:56:54.433 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.434+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3856857996}) 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539488946}) 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2502768735}) 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539488946}) 2024-09-16T06:56:54.434 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181867106}) 2024-09-16T06:56:54.435 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.435 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628987787}) 2024-09-16T06:56:54.435 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251891290}) 2024-09-16T06:56:54.435 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.435 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628987787}) 2024-09-16T06:56:54.436 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2784531680}) 2024-09-16T06:56:54.436 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.436 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042406212}) 2024-09-16T06:56:54.436 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185994918}) 2024-09-16T06:56:54.437 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.437 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042406212}) 2024-09-16T06:56:54.437 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597936659}) 2024-09-16T06:56:54.437 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.437 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380671185}) 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087796710}) 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380671185}) 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.438+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47973088}) 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.438 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462257534}) 2024-09-16T06:56:54.439 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=261822057}) 2024-09-16T06:56:54.439 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.439 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462257534}) 2024-09-16T06:56:54.439 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289669943}) 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807622296}) 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3097731550}) 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807622296}) 2024-09-16T06:56:54.440 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264218242}) 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067466177}) 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465162384}) 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067466177}) 2024-09-16T06:56:54.441 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.442+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421946107}) 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3323504146}) 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2062288402}) 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3323504146}) 2024-09-16T06:56:54.442 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892422539}) 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993803479}) 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1376004748}) 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993803479}) 2024-09-16T06:56:54.443 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657926061}) 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462614728}) 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2689358396}) 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462614728}) 2024-09-16T06:56:54.444 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444652864}) 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399148636}) 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987453235}) 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399148636}) 2024-09-16T06:56:54.445 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.446+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090096847}) 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352294245}) 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814618153}) 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352294245}) 2024-09-16T06:56:54.446 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553011749}) 2024-09-16T06:56:54.447 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.447 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657544143}) 2024-09-16T06:56:54.447 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131922339}) 2024-09-16T06:56:54.447 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.447 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657544143}) 2024-09-16T06:56:54.448 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747901047}) 2024-09-16T06:56:54.448 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.448 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219941557}) 2024-09-16T06:56:54.448 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810523291}) 2024-09-16T06:56:54.449 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.449 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219941557}) 2024-09-16T06:56:54.449 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209013163}) 2024-09-16T06:56:54.449 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002709794}) 2024-09-16T06:56:54.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382337663}) 2024-09-16T06:56:54.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002709794}) 2024-09-16T06:56:54.450 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.450+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296015275}) 2024-09-16T06:56:54.451 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.451 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612247790}) 2024-09-16T06:56:54.451 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560951901}) 2024-09-16T06:56:54.451 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.451 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612247790}) 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960169392}) 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046851871}) 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003093255}) 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046851871}) 2024-09-16T06:56:54.452 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801378989}) 2024-09-16T06:56:54.453 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.453 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239679330}) 2024-09-16T06:56:54.453 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566258305}) 2024-09-16T06:56:54.453 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239679330}) 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277159635}) 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573461089}) 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158542567}) 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573461089}) 2024-09-16T06:56:54.454 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.454+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445080675}) 2024-09-16T06:56:54.455 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.455 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375965328}) 2024-09-16T06:56:54.455 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166682852}) 2024-09-16T06:56:54.455 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.455 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375965328}) 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062172432}) 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1156995570}) 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621785184}) 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1156995570}) 2024-09-16T06:56:54.456 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=290887618}) 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078449275}) 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004650041}) 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078449275}) 2024-09-16T06:56:54.457 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.458+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413961825}) 2024-09-16T06:56:54.458 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.458 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328677020}) 2024-09-16T06:56:54.459 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=292176105}) 2024-09-16T06:56:54.459 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.459 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328677020}) 2024-09-16T06:56:54.459 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305286753}) 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200361553}) 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448805020}) 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200361553}) 2024-09-16T06:56:54.460 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3702728180}) 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182046980}) 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.462+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1031881339}) 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182046980}) 2024-09-16T06:56:54.462 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755535677}) 2024-09-16T06:56:54.463 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.464 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=123213123}) 2024-09-16T06:56:54.464 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409255771}) 2024-09-16T06:56:54.464 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.464 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=123213123}) 2024-09-16T06:56:54.464 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606576159}) 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800851151}) 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=261519159}) 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800851151}) 2024-09-16T06:56:54.465 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.466+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369996753}) 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731321534}) 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265760159}) 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731321534}) 2024-09-16T06:56:54.467 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247208898}) 2024-09-16T06:56:54.468 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.468 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103702132}) 2024-09-16T06:56:54.469 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517461720}) 2024-09-16T06:56:54.469 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.469 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103702132}) 2024-09-16T06:56:54.469 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.470+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116223889}) 2024-09-16T06:56:54.470 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.471 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797827631}) 2024-09-16T06:56:54.471 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721048342}) 2024-09-16T06:56:54.471 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.471 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797827631}) 2024-09-16T06:56:54.471 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4017403194}) 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145087869}) 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087363232}) 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145087869}) 2024-09-16T06:56:54.473 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.474+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094481301}) 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211132173}) 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2047114785}) 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211132173}) 2024-09-16T06:56:54.475 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750016273}) 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900156776}) 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1048731723}) 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900156776}) 2024-09-16T06:56:54.477 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.478+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25349222}) 2024-09-16T06:56:54.479 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.479 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590436800}) 2024-09-16T06:56:54.479 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868080002}) 2024-09-16T06:56:54.479 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.480 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590436800}) 2024-09-16T06:56:54.480 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895179657}) 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102889194}) 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829647446}) 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102889194}) 2024-09-16T06:56:54.481 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.482+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255769113}) 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2011528675}) 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802732442}) 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2011528675}) 2024-09-16T06:56:54.483 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272701508}) 2024-09-16T06:56:54.484 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.485 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352484934}) 2024-09-16T06:56:54.485 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020809213}) 2024-09-16T06:56:54.485 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.485 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352484934}) 2024-09-16T06:56:54.485 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.486+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649076639}) 2024-09-16T06:56:54.486 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.486 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258296710}) 2024-09-16T06:56:54.486 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787440940}) 2024-09-16T06:56:54.486 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.486 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258296710}) 2024-09-16T06:56:54.487 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143550486}) 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100056278}) 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148320232}) 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100056278}) 2024-09-16T06:56:54.488 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.490+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2175375206}) 2024-09-16T06:56:54.490 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.491 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19} crc {logm=2839644405,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4082470593,mon_config_key=1714515979}) 2024-09-16T06:56:54.491 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=2,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=4} crc {logm=2846852888,mdsmap=3140514009,mgr=278123252,mgr_metadata=1347778423,mgrstat=3012501084,mon_config_key=3333735536,monmap=1887442722,osd_metadata=2356802856}) 2024-09-16T06:56:54.491 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.491 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19} crc {logm=2839644405,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=4082470593,mon_config_key=1714515979}) 2024-09-16T06:56:54.491 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.494+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19} crc {logm=1023204103,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2177261256,mon_config_key=1791344815}) 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {monmap=1887442722,osd_metadata=2526275005,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1383571158}) 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {monmap=1820574642,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1004655786}) 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {monmap=1887442722,osd_metadata=2526275005,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1383571158}) 2024-09-16T06:56:54.499 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:56:54.502+0000 7f136743d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=553062916,osdmap=2996441903}) 2024-09-16T06:56:55.352 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:56:55.353 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:56:55.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: get_monmap_and_config 2024-09-16T06:56:55.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: build_initial_monmap 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: monmap: 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:55.478390+0000 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:55.478390+0000 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:55.473 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:55.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 1 Processor -- start 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 1 -- start start 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: init 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0e70097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0e70099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0e70139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:55.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e70098990 con 0x7f0e70099090 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e70097f40 con 0x7f0e70097b60 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7009d610 con 0x7f0e70139500 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6ffff700 10 monclient(hunting): authenticate will time out at 1962.611662s 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6e7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0e70099090 0x7f0e7009f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47882/0 (socket says 172.21.15.76:47882) 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0e70139500 0x7f0e7013b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43478/0 (socket says 172.21.15.76:43478) 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.474+0000 7f0e6e7fc700 1 -- 172.21.15.76:0/773707134 learned_addr learned my addr 172.21.15.76:0/773707134 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994148750 0 0) 0x7f0e70098990 con 0x7f0e70099090 2024-09-16T06:56:55.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 334134 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e580035d0 con 0x7f0e70099090 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744365471 0 0) 0x7f0e7009d610 con 0x7f0e70139500 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 324215 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e70098990 con 0x7f0e70139500 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658314978 0 0) 0x7f0e70097f40 con 0x7f0e70097b60 2024-09-16T06:56:55.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 284599 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7009d610 con 0x7f0e70097b60 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3587623106 0 0) 0x7f0e580035d0 con 0x7f0e70099090 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 >> v1:172.21.15.76:6790/0 conn(0x7f0e70139500 legacy=0x7f0e7013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 >> v1:172.21.15.189:6789/0 conn(0x7f0e70097b60 legacy=0x7f0e7009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 monclient: found mon.noname-a 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:55.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e700986b0 con 0x7f0e70099090 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: _finish_auth 0 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e60001610 con 0x7f0e70099090 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: dump: 2024-09-16T06:56:55.481 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 5 monclient: authenticate success, global_id 334134 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e600018f0 con 0x7f0e70099090 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 1 -- 172.21.15.76:0/773707134 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e60001c80 con 0x7f0e70099090 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 4 set_mon_vals no callback set 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6dffb700 10 monclient: dump: 2024-09-16T06:56:55.483 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 10 monclient: shutdown 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 1 -- 172.21.15.76:0/773707134 >> v1:172.21.15.76:6789/0 conn(0x7f0e70099090 legacy=0x7f0e7009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 1 -- 172.21.15.76:0/773707134 shutdown_connections 2024-09-16T06:56:55.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 1 -- 172.21.15.76:0/773707134 >> 172.21.15.76:0/773707134 conn(0x7f0e700a0030 msgr2=0x7f0e700a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:55.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 1 -- 172.21.15.76:0/773707134 shutdown_connections 2024-09-16T06:56:55.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 1 -- 172.21.15.76:0/773707134 wait complete. 2024-09-16T06:56:55.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.478+0000 7f0e6ffff700 10 monclient: build_initial_monmap 2024-09-16T06:56:55.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: monmap: 2024-09-16T06:56:55.491 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:55.491 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:55.491 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 Processor -- start 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 -- start start 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: init 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0e70097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0e70099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0e70139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e700711d0 con 0x7f0e70139500 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7009a5b0 con 0x7f0e70097b60 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7006d140 con 0x7f0e70099090 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient(hunting): authenticate will time out at 1962.618310s 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0e70139500 0x7f0e70137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47896/0 (socket says 172.21.15.76:47896) 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6f7fe700 1 -- 172.21.15.76:0/1582768078 learned_addr learned my addr 172.21.15.76:0/1582768078 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803950036 0 0) 0x7f0e70098990 con 0x7f0e70139500 2024-09-16T06:56:55.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): my global_id is 334137 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c0035d0 con 0x7f0e70139500 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976098175 0 0) 0x7f0e7006d140 con 0x7f0e70099090 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): my global_id is 324218 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e70098990 con 0x7f0e70099090 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951066561 0 0) 0x7f0e7009d610 con 0x7f0e70097b60 2024-09-16T06:56:55.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): my global_id is 284602 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7006d140 con 0x7f0e70097b60 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613874492 0 0) 0x7f0e5c0035d0 con 0x7f0e70139500 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7009d610 con 0x7f0e70139500 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e68001a90 con 0x7f0e70139500 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): dump: 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.495 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856413006 0 0) 0x7f0e70098990 con 0x7f0e70099090 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e5c0035d0 con 0x7f0e70099090 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e600015d0 con 0x7f0e70099090 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:55.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient(hunting): dump: 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1266506077 0 0) 0x7f0e7009d610 con 0x7f0e70139500 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 >> v1:172.21.15.76:6790/0 conn(0x7f0e70099090 legacy=0x7f0e70137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 >> v1:172.21.15.189:6789/0 conn(0x7f0e70097b60 legacy=0x7f0e70136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient: _finish_hunting 0 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 monclient: found mon.a 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7006bed0 con 0x7f0e70139500 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient: _finish_auth 0 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 10 monclient: _check_auth_tickets 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 5 monclient: authenticate success, global_id 334137 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: _renew_subs 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.482+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e70138c40 con 0x7f0e70139500 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 10 monclient: _renew_subs 2024-09-16T06:56:55.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e70138770 con 0x7f0e70139500 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e68001fe0 con 0x7f0e70139500 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e68004db0 con 0x7f0e70139500 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6d7fa700 4 set_mon_vals no callback set 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 10 monclient: dump: 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.500 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e50005100 con 0x7f0e70139500 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0e6800f820 con 0x7f0e70139500 2024-09-16T06:56:55.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0e68043ea0 con 0x7f0e70139500 2024-09-16T06:56:55.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.486+0000 7f0e6effd700 10 monclient: get_auth_request con 0x7f0e5c036390 auth_method 0 2024-09-16T06:56:55.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.494+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0e68010d20 con 0x7f0e70139500 2024-09-16T06:56:55.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.494+0000 7f0e577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:55.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.494+0000 7f0e577fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:55.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.730+0000 7f0e6ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:56:55.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.730+0000 7f0e6ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:55.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.730+0000 7f0e6ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.730+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0e50005930 con 0x7f0e70139500 2024-09-16T06:56:55.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e577fe700 1 -- 172.21.15.76:0/1582768078 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1452399454) 0x7f0e68012d40 con 0x7f0e70139500 2024-09-16T06:56:55.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:55.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e577fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:55.730 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0e5c036390 legacy=0x7f0e5c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 10 monclient: shutdown 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 >> v1:172.21.15.76:6789/0 conn(0x7f0e70139500 legacy=0x7f0e70137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 shutdown_connections 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 >> 172.21.15.76:0/1582768078 conn(0x7f0e700a0030 msgr2=0x7f0e70132750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:55.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 shutdown_connections 2024-09-16T06:56:55.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.734+0000 7f0e6ffff700 1 -- 172.21.15.76:0/1582768078 wait complete. 2024-09-16T06:56:55.745 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:56:55.745 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:56:55.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:56:55.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:56:55.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: get_monmap_and_config 2024-09-16T06:56:55.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: build_initial_monmap 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: monmap: 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:55.866543+0000 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:55.866543+0000 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:55.861 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 1 Processor -- start 2024-09-16T06:56:55.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 1 -- start start 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: init 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-b con 0x7fa114097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-c con 0x7fa114099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-a con 0x7fa114139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa114098990 con 0x7fa114139500 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa114097f40 con 0x7fa114099090 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11409d610 con 0x7fa114097b60 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11a45e700 10 monclient(hunting): authenticate will time out at 1962.999775s 2024-09-16T06:56:55.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11945c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa114097b60 0x7fa11409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43494/0 (socket says 172.21.15.76:43494) 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa119c5d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa114139500 0x7fa11413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47900/0 (socket says 172.21.15.76:47900) 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.862+0000 7fa11945c700 1 -- 172.21.15.76:0/4233828510 learned_addr learned my addr 172.21.15.76:0/4233828510 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237197879 0 0) 0x7fa114098990 con 0x7fa114139500 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): my global_id is 334140 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1000035d0 con 0x7fa114139500 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056631144 0 0) 0x7fa11409d610 con 0x7fa114097b60 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): my global_id is 324221 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa114098990 con 0x7fa114097b60 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752596679 0 0) 0x7fa114097f40 con 0x7fa114099090 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient(hunting): my global_id is 284605 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11409d610 con 0x7fa114099090 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848808291 0 0) 0x7fa1000035d0 con 0x7fa114139500 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 >> v1:172.21.15.76:6790/0 conn(0x7fa114097b60 legacy=0x7fa11409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 >> v1:172.21.15.189:6789/0 conn(0x7fa114099090 legacy=0x7fa11409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 monclient: found mon.noname-a 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:55.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140986b0 con 0x7fa114139500 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: _finish_auth 0 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa10c001990 con 0x7fa114139500 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: dump: 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.871 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 5 monclient: authenticate success, global_id 334140 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa10c001c70 con 0x7fa114139500 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 1 -- 172.21.15.76:0/4233828510 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa10c002000 con 0x7fa114139500 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 4 set_mon_vals no callback set 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa10bfff700 10 monclient: dump: 2024-09-16T06:56:55.872 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: shutdown 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- 172.21.15.76:0/4233828510 >> v1:172.21.15.76:6789/0 conn(0x7fa114139500 legacy=0x7fa11413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- 172.21.15.76:0/4233828510 shutdown_connections 2024-09-16T06:56:55.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- 172.21.15.76:0/4233828510 >> 172.21.15.76:0/4233828510 conn(0x7fa1140a0030 msgr2=0x7fa1140a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- 172.21.15.76:0/4233828510 shutdown_connections 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- 172.21.15.76:0/4233828510 wait complete. 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: build_initial_monmap 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: monmap: 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 Processor -- start 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- start start 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: init 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-b con 0x7fa114097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-c con 0x7fa114099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient(hunting): picked mon.noname-a con 0x7fa114139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11406dd00 con 0x7fa114139500 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11409a540 con 0x7fa114097b60 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11406fe70 con 0x7fa114099090 2024-09-16T06:56:55.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa11a45e700 10 monclient(hunting): authenticate will time out at 1963.003861s 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa118c5b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fa114099090 0x7fa11406d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43510/0 (socket says 172.21.15.76:43510) 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.866+0000 7fa118c5b700 1 -- 172.21.15.76:0/274513282 learned_addr learned my addr 172.21.15.76:0/274513282 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089279934 0 0) 0x7fa11406fe70 con 0x7fa114099090 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): my global_id is 324224 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80035d0 con 0x7fa114099090 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442798970 0 0) 0x7fa11406dd00 con 0x7fa114139500 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): my global_id is 334143 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11406fe70 con 0x7fa114139500 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2101659174 0 0) 0x7fa0f80035d0 con 0x7fa114099090 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa11406dd00 con 0x7fa114099090 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa110001990 con 0x7fa114099090 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): dump: 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.877 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322201089 0 0) 0x7fa11409a540 con 0x7fa114097b60 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): my global_id is 284608 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80035d0 con 0x7fa114097b60 2024-09-16T06:56:55.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2121699469 0 0) 0x7fa11406dd00 con 0x7fa114099090 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.189:6789/0 conn(0x7fa114097b60 legacy=0x7fa11406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6789/0 conn(0x7fa114139500 legacy=0x7fa1141cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 monclient: found mon.c 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114070150 con 0x7fa114099090 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: _finish_auth 0 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 5 monclient: authenticate success, global_id 324224 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 10 monclient: _renew_subs 2024-09-16T06:56:55.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa114070ea0 con 0x7fa114099090 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 10 monclient: _renew_subs 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11406fb90 con 0x7fa114099090 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa110001d40 con 0x7fa114099090 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa110003b90 con 0x7fa114099090 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa10b7fe700 4 set_mon_vals no callback set 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: dump: 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.880 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa110003d70 con 0x7fa114099090 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa0f4006590 con 0x7fa0f4002830 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa110043e70 con 0x7fa114099090 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa11945c700 10 monclient: get_auth_request con 0x7fa0f8036340 auth_method 0 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220390541 0 0) 0x7fa0f4006590 con 0x7fa0f4002830 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient(hunting): my global_id is 334146 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11406dd00 con 0x7fa0f4002830 2024-09-16T06:56:55.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1477303498 0 0) 0x7fa11406dd00 con 0x7fa0f4002830 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f4006590 con 0x7fa0f4002830 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa10c0017b0 con 0x7fa0f4002830 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 10 monclient: dump: 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.885 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 367379987 0 0) 0x7fa0f4006590 con 0x7fa0f4002830 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.870+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0f400a190 con 0x7fa0f4002830 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.874+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fa10c001530 con 0x7fa0f4002830 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.874+0000 7fa109ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.874+0000 7fa109ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.874+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6789/0 conn(0x7fa0f4002830 legacy=0x7fa0f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.874+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6789/0 conn(0x7fa0f4002830 legacy=0x7fa0f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa11a45e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa0f4009e50 con 0x7fa0f4009170 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622651846 0 0) 0x7fa0f4009e50 con 0x7fa0f4009170 2024-09-16T06:56:55.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient(hunting): my global_id is 334149 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f4006590 con 0x7fa0f4009170 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3705691842 0 0) 0x7fa0f4006590 con 0x7fa0f4009170 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f4009e50 con 0x7fa0f4009170 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa10c0053d0 con 0x7fa0f4009170 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient: dump: 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.887 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2794811040 0 0) 0x7fa0f4009e50 con 0x7fa0f4009170 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa0f400c340 con 0x7fa0f4009170 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 2157749420) 0x7fa10c006fe0 con 0x7fa0f4009170 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6789/0 conn(0x7fa0f4009170 legacy=0x7fa0f4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa109ffb700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6789/0 conn(0x7fa0f4009170 legacy=0x7fa0f4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:55.901 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:55.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa0f8036340 legacy=0x7fa0f8038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.882+0000 7fa11a45e700 10 monclient: shutdown 2024-09-16T06:56:55.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:55.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 >> v1:172.21.15.76:6790/0 conn(0x7fa114099090 legacy=0x7fa11406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa119c5d700 1 -- 172.21.15.76:0/274513282 reap_dead start 2024-09-16T06:56:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 shutdown_connections 2024-09-16T06:56:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 >> 172.21.15.76:0/274513282 conn(0x7fa1140a0030 msgr2=0x7fa1140a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 shutdown_connections 2024-09-16T06:56:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:55.886+0000 7fa11a45e700 1 -- 172.21.15.76:0/274513282 wait complete. 2024-09-16T06:56:55.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:55.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:56:55.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:56:56.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.018+0000 7f13e8e04700 10 monclient: get_monmap_and_config 2024-09-16T06:56:56.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.018+0000 7f13e8e04700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.018+0000 7f13e8e04700 10 monclient: monmap: 2024-09-16T06:56:56.017 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:56.023292+0000 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:56.023292+0000 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:56.018 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 Processor -- start 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- start start 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient: init 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-c con 0x7f13e40a20c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-b con 0x7f13e40a24a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-a con 0x7f13e413b6b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e409b3e0 con 0x7f13e413b6b0 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e40a3450 con 0x7f13e40a20c0 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e409b1a0 con 0x7f13e40a24a0 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient(hunting): authenticate will time out at 1963.156830s 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f13e413b6b0 0x7f13e413daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47924/0 (socket says 172.21.15.76:47924) 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e3fff700 1 -- 172.21.15.76:0/3347663742 learned_addr learned my addr 172.21.15.76:0/3347663742 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13daffd700 1 --1- 172.21.15.76:0/3347663742 >> v1:172.21.15.76:6790/0 conn(0x7f13e40a24a0 0x7f13e40a28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43516/0 (socket says 172.21.15.76:43516) 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971545416 0 0) 0x7f13e409b3e0 con 0x7f13e413b6b0 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): my global_id is 334152 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c40035d0 con 0x7f13e413b6b0 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114386855 0 0) 0x7f13e409b1a0 con 0x7f13e40a24a0 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): my global_id is 324227 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e409b3e0 con 0x7f13e40a24a0 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909075951 0 0) 0x7f13e40a3450 con 0x7f13e40a20c0 2024-09-16T06:56:56.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient(hunting): my global_id is 284611 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e409b1a0 con 0x7f13e40a20c0 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597483652 0 0) 0x7f13c40035d0 con 0x7f13e413b6b0 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 >> v1:172.21.15.76:6790/0 conn(0x7f13e40a24a0 legacy=0x7f13e40a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 >> v1:172.21.15.189:6789/0 conn(0x7f13e40a20c0 legacy=0x7f13e406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e406bd30 con 0x7f13e413b6b0 2024-09-16T06:56:56.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 5 monclient: authenticate success, global_id 334152 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc001990 con 0x7f13e413b6b0 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: dump: 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.026 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13dc001c00 con 0x7f13e413b6b0 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 4 set_mon_vals no callback set 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 1 -- 172.21.15.76:0/3347663742 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13dc001f90 con 0x7f13e413b6b0 2024-09-16T06:56:56.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e2ffd700 10 monclient: dump: 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.028 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 10 monclient: shutdown 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- 172.21.15.76:0/3347663742 >> v1:172.21.15.76:6789/0 conn(0x7f13e413b6b0 legacy=0x7f13e413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- 172.21.15.76:0/3347663742 shutdown_connections 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- 172.21.15.76:0/3347663742 >> 172.21.15.76:0/3347663742 conn(0x7f13e40a41c0 msgr2=0x7f13e40a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- 172.21.15.76:0/3347663742 shutdown_connections 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.022+0000 7f13e8e04700 1 -- 172.21.15.76:0/3347663742 wait complete. 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: monmap: 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:56.030 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 Processor -- start 2024-09-16T06:56:56.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- start start 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: init 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-c con 0x7f13e40a20c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-a con 0x7f13e40a24a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient(hunting): picked mon.noname-b con 0x7f13e413b6b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e41c72a0 con 0x7f13e40a24a0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e4138810 con 0x7f13e413b6b0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e40a7250 con 0x7f13e40a20c0 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient(hunting): authenticate will time out at 1963.161736s 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13e40a20c0 0x7f13e41c6480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43522/0 (socket says 172.21.15.76:43522) 2024-09-16T06:56:56.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e37fe700 1 -- 172.21.15.76:0/2173088309 learned_addr learned my addr 172.21.15.76:0/2173088309 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739731195 0 0) 0x7f13e41c72a0 con 0x7f13e40a24a0 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): my global_id is 334155 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c80035d0 con 0x7f13e40a24a0 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897514418 0 0) 0x7f13e40a7250 con 0x7f13e40a20c0 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): my global_id is 324230 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e41c72a0 con 0x7f13e40a20c0 2024-09-16T06:56:56.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210474152 0 0) 0x7f13e4138810 con 0x7f13e413b6b0 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): my global_id is 284614 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e40a7250 con 0x7f13e413b6b0 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963383917 0 0) 0x7f13c80035d0 con 0x7f13e40a24a0 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e4138810 con 0x7f13e40a24a0 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13cc0018b0 con 0x7f13e40a24a0 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): dump: 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.041 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094013582 0 0) 0x7f13e41c72a0 con 0x7f13e40a20c0 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13c80035d0 con 0x7f13e40a20c0 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13d40018f0 con 0x7f13e40a20c0 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient(hunting): dump: 2024-09-16T06:56:56.042 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3318129567 0 0) 0x7f13e4138810 con 0x7f13e40a24a0 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6790/0 conn(0x7f13e40a20c0 legacy=0x7f13e41c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.189:6789/0 conn(0x7f13e413b6b0 legacy=0x7f13e41d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 monclient: found mon.a 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e409ed50 con 0x7f13e40a24a0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 5 monclient: authenticate success, global_id 334155 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _renew_subs 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e413a2d0 con 0x7f13e40a24a0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _renew_subs 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e4139600 con 0x7f13e40a24a0 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13cc004890 con 0x7f13e40a24a0 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13cc004ba0 con 0x7f13e40a24a0 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e0ff9700 10 monclient: dump: 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.045 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.026+0000 7f13e27fc700 4 set_mon_vals no callback set 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e8e04700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13cc00f610 con 0x7f13e40a24a0 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13bc006590 con 0x7f13bc002830 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f13cc043e80 con 0x7f13e40a24a0 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970191859 0 0) 0x7f13bc006590 con 0x7f13bc002830 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient(hunting): my global_id is 324233 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e4138810 con 0x7f13bc002830 2024-09-16T06:56:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e3fff700 10 monclient: get_auth_request con 0x7f13c80364c0 auth_method 0 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4225926717 0 0) 0x7f13e4138810 con 0x7f13bc002830 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc006590 con 0x7f13bc002830 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13d4003860 con 0x7f13bc002830 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient: dump: 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2961240446 0 0) 0x7f13bc006590 con 0x7f13bc002830 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13bc00a190 con 0x7f13bc002830 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f13d4005870 con 0x7f13bc002830 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6790/0 conn(0x7f13bc002830 legacy=0x7f13bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.030+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6790/0 conn(0x7f13bc002830 legacy=0x7f13bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13bc009e30 con 0x7f13bc009150 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754760388 0 0) 0x7f13bc009e30 con 0x7f13bc009150 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient(hunting): my global_id is 324236 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc006590 con 0x7f13bc009150 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3275343487 0 0) 0x7f13bc006590 con 0x7f13bc009150 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc009e30 con 0x7f13bc009150 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13d4003ed0 con 0x7f13bc009150 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient: dump: 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1026582167 0 0) 0x7f13bc009e30 con 0x7f13bc009150 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13bc00c340 con 0x7f13bc009150 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 4044744145) 0x7f13d40041d0 con 0x7f13bc009150 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6790/0 conn(0x7f13bc009150 legacy=0x7f13bc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e0ff9700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6790/0 conn(0x7f13bc009150 legacy=0x7f13bc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:56.054 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.055 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:56.056 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.057 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:56.058 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.059 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.060 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.061 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:56.062 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13c80364c0 legacy=0x7f13c8038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 10 monclient: shutdown 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 >> v1:172.21.15.76:6789/0 conn(0x7f13e40a24a0 legacy=0x7f13e41c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e3fff700 1 -- 172.21.15.76:0/2173088309 reap_dead start 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e3fff700 1 -- 172.21.15.76:0/2173088309 reap_dead start 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 shutdown_connections 2024-09-16T06:56:56.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.038+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 >> 172.21.15.76:0/2173088309 conn(0x7f13e40a41c0 msgr2=0x7f13e40a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.042+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 shutdown_connections 2024-09-16T06:56:56.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.042+0000 7f13e8e04700 1 -- 172.21.15.76:0/2173088309 wait complete. 2024-09-16T06:56:56.065 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:56.065 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:56:56.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 10 monclient: get_monmap_and_config 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 10 monclient: monmap: 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:56.184759+0000 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:56.184759+0000 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 1 Processor -- start 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 1 -- start start 2024-09-16T06:56:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.182+0000 7fd18188b700 10 monclient: init 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-c con 0x7fd17c0a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-a con 0x7fd17c0a11a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-b con 0x7fd17c071da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0a4c20 con 0x7fd17c0a11a0 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c09c120 con 0x7fd17c0a4840 2024-09-16T06:56:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0a1620 con 0x7fd17c071da0 2024-09-16T06:56:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18188b700 10 monclient(hunting): authenticate will time out at 1963.320781s 2024-09-16T06:56:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd17bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd17c0a11a0 0x7fd17c09ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47946/0 (socket says 172.21.15.76:47946) 2024-09-16T06:56:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd18108a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd17c071da0 0x7fd17c13a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:43550/0 (socket says 172.21.15.76:43550) 2024-09-16T06:56:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.186+0000 7fd17bfff700 1 -- 172.21.15.76:0/1514244838 learned_addr learned my addr 172.21.15.76:0/1514244838 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609051382 0 0) 0x7fd17c0a4c20 con 0x7fd17c0a11a0 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): my global_id is 334158 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c0035d0 con 0x7fd17c0a11a0 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225080081 0 0) 0x7fd17c0a1620 con 0x7fd17c071da0 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient(hunting): my global_id is 324239 2024-09-16T06:56:56.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c0a4c20 con 0x7fd17c071da0 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828049094 0 0) 0x7fd16c0035d0 con 0x7fd17c0a11a0 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 >> v1:172.21.15.76:6790/0 conn(0x7fd17c071da0 legacy=0x7fd17c13a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 >> v1:172.21.15.189:6789/0 conn(0x7fd17c0a4840 legacy=0x7fd17c09b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0a0670 con 0x7fd17c0a11a0 2024-09-16T06:56:56.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 5 monclient: authenticate success, global_id 334158 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd168001610 con 0x7fd17c0a11a0 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: dump: 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.197 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd168001870 con 0x7fd17c0a11a0 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 1 -- 172.21.15.76:0/1514244838 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd168001c40 con 0x7fd17c0a11a0 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 4 set_mon_vals no callback set 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd17b7fe700 10 monclient: dump: 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.199 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: shutdown 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- 172.21.15.76:0/1514244838 >> v1:172.21.15.76:6789/0 conn(0x7fd17c0a11a0 legacy=0x7fd17c09ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- 172.21.15.76:0/1514244838 shutdown_connections 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- 172.21.15.76:0/1514244838 >> 172.21.15.76:0/1514244838 conn(0x7fd17c0a4430 msgr2=0x7fd17c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- 172.21.15.76:0/1514244838 shutdown_connections 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- 172.21.15.76:0/1514244838 wait complete. 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: monmap: 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 Processor -- start 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- start start 2024-09-16T06:56:56.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: init 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-a con 0x7fd17c071da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-c con 0x7fd17c0a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient(hunting): picked mon.noname-b con 0x7fd17c1d01b0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c070fb0 con 0x7fd17c071da0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c126230 con 0x7fd17c1d01b0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c11fcd0 con 0x7fd17c0a4840 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd18188b700 10 monclient(hunting): authenticate will time out at 1963.327290s 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd180889700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd17c071da0 0x7fd17c11eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47960/0 (socket says 172.21.15.76:47960) 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd180889700 1 -- 172.21.15.76:0/2500503089 learned_addr learned my addr 172.21.15.76:0/2500503089 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303176499 0 0) 0x7fd17c070fb0 con 0x7fd17c071da0 2024-09-16T06:56:56.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): my global_id is 334161 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1600035d0 con 0x7fd17c071da0 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586135326 0 0) 0x7fd17c11fcd0 con 0x7fd17c0a4840 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 10 monclient(hunting): my global_id is 324242 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.190+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c070fb0 con 0x7fd17c0a4840 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255681620 0 0) 0x7fd1600035d0 con 0x7fd17c071da0 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd17c11fcd0 con 0x7fd17c071da0 2024-09-16T06:56:56.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd170001910 con 0x7fd17c071da0 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): dump: 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.204 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173048653 0 0) 0x7fd17c126230 con 0x7fd17c1d01b0 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): my global_id is 284617 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1600035d0 con 0x7fd17c1d01b0 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1708812791 0 0) 0x7fd17c070fb0 con 0x7fd17c0a4840 2024-09-16T06:56:56.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd17c126230 con 0x7fd17c0a4840 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd168001b80 con 0x7fd17c0a4840 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): dump: 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.206 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2368424356 0 0) 0x7fd17c11fcd0 con 0x7fd17c071da0 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.76:6790/0 conn(0x7fd17c0a4840 legacy=0x7fd17c11f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6789/0 conn(0x7fd17c1d01b0 legacy=0x7fd17c1d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 monclient: found mon.a 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0a3870 con 0x7fd17c071da0 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 5 monclient: authenticate success, global_id 334161 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 10 monclient: _renew_subs 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c120400 con 0x7fd17c071da0 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 10 monclient: _renew_subs 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c09f500 con 0x7fd17c071da0 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd170001c60 con 0x7fd17c071da0 2024-09-16T06:56:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd170003b10 con 0x7fd17c071da0 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd17a7fc700 4 set_mon_vals no callback set 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: dump: 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.209 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd170003c70 con 0x7fd17c071da0 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd15c006590 con 0x7fd15c002830 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd170043e10 con 0x7fd17c071da0 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd18108a700 10 monclient: get_auth_request con 0x7fd160036490 auth_method 0 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756430126 0 0) 0x7fd15c006590 con 0x7fd15c002830 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient(hunting): my global_id is 284620 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17c11fcd0 con 0x7fd15c002830 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3087976316 0 0) 0x7fd17c11fcd0 con 0x7fd15c002830 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd15c006590 con 0x7fd15c002830 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd168004d80 con 0x7fd15c002830 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 10 monclient: dump: 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2252478428 0 0) 0x7fd15c006590 con 0x7fd15c002830 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.194+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd15c00a190 con 0x7fd15c002830 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.198+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fd168005020 con 0x7fd15c002830 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.198+0000 7fd1797fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.198+0000 7fd1797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.198+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6789/0 conn(0x7fd15c002830 legacy=0x7fd15c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.198+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6789/0 conn(0x7fd15c002830 legacy=0x7fd15c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd18188b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd15c009e30 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772607293 0 0) 0x7fd15c009e30 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient(hunting): my global_id is 284623 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c006590 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3549642608 0 0) 0x7fd15c006590 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd15c009e30 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd1680055b0 con 0x7fd15c009150 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:56:56.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.206+0000 7fd1797fa700 10 monclient: dump: 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 232734085 0 0) 0x7fd15c009e30 con 0x7fd15c009150 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd15c00c340 con 0x7fd15c009150 2024-09-16T06:56:56.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 2978640700) 0x7fd168006880 con 0x7fd15c009150 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6789/0 conn(0x7fd15c009150 legacy=0x7fd15c0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd1797fa700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6789/0 conn(0x7fd15c009150 legacy=0x7fd15c0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:56:56.217 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 162, 2024-09-16T06:56:56.218 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:56:56.218 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:56:56.218 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:56:56.218 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:56:56.218 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.219 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:56:56.219 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:56.219 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:56:56.219 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:56:56.219 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.220 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.220 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.220 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.220 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.220 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.221 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.221 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.221 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.221 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.221 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.222 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:56:56.222 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:56:56.222 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.222 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.222 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.223 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.223 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.223 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.223 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.223 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.224 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.224 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.224 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.224 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:56:56.225 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:56.226 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:56:56.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:56:56.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:56:56.226 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:56:56.226 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:56:56.227 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:56:56.228 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:56:56.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:56:56.228 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:56:56.228 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:56:56.228 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:56:56.229 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:56:56.229 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:56:56.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:56:56.229 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:56:56.229 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:56:56.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.231 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.233 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:56:56.234 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:56:56.235 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.236 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:56:56.237 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:56:56.238 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd160036490 legacy=0x7fd160038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 10 monclient: shutdown 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 >> v1:172.21.15.76:6789/0 conn(0x7fd17c071da0 legacy=0x7fd17c11eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18108a700 1 -- 172.21.15.76:0/2500503089 reap_dead start 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18108a700 1 -- 172.21.15.76:0/2500503089 reap_dead start 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 shutdown_connections 2024-09-16T06:56:56.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 >> 172.21.15.76:0/2500503089 conn(0x7fd17c0a4430 msgr2=0x7fd17c06f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 shutdown_connections 2024-09-16T06:56:56.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.210+0000 7fd18188b700 1 -- 172.21.15.76:0/2500503089 wait complete. 2024-09-16T06:56:56.240 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:56:56.240 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:56:56.240 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:56:56.241 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:56:56.241 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:56:56.241 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:56:56.241 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:56:56.286 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:56:56.286 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:56:56.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: monmap: 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:56.408171+0000 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:56.408171+0000 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:56.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 Processor -- start 2024-09-16T06:56:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- start start 2024-09-16T06:56:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: init 2024-09-16T06:56:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fd9600a04e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9600a0fd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fd9600a1420 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006c400 con 0x7fd9600a0fd0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006c9a0 con 0x7fd9600a1420 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96006c1c0 con 0x7fd9600a04e0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient(hunting): authenticate will time out at 1963.541543s 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95e59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fd9600a04e0 legacy=0x7fd9600a08c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95dd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd9600a0fd0 0x7fd96013a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47986/0 (socket says 172.21.15.76:47986) 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95dd9b700 1 -- 172.21.15.76:0/2422297364 learned_addr learned my addr 172.21.15.76:0/2422297364 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55242013 0 0) 0x7fd96006c400 con 0x7fd9600a0fd0 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient(hunting): my global_id is 334164 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9480035d0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2759690914 0 0) 0x7fd9480035d0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 >> v1:172.21.15.76:6790/0 conn(0x7fd9600a04e0 legacy=0x7fd9600a08c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 >> v1:172.21.15.189:6789/0 conn(0x7fd9600a1420 legacy=0x7fd96013dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 monclient: found mon.noname-a 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96009f8f0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 5 monclient: authenticate success, global_id 334164 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd950001610 con 0x7fd9600a0fd0 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: dump: 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.409 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd950001930 con 0x7fd9600a0fd0 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 4 set_mon_vals no callback set 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 1 -- 172.21.15.76:0/2422297364 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd950001cc0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95d59a700 10 monclient: dump: 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 10 monclient: shutdown 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- 172.21.15.76:0/2422297364 >> v1:172.21.15.76:6789/0 conn(0x7fd9600a0fd0 legacy=0x7fd96013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- 172.21.15.76:0/2422297364 shutdown_connections 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- 172.21.15.76:0/2422297364 >> 172.21.15.76:0/2422297364 conn(0x7fd9600a5870 msgr2=0x7fd9600a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- 172.21.15.76:0/2422297364 shutdown_connections 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.406+0000 7fd95f59e700 1 -- 172.21.15.76:0/2422297364 wait complete. 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: build_initial_monmap 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: monmap: 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:56.412 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:56.413 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:56.413 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 Processor -- start 2024-09-16T06:56:56.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- start start 2024-09-16T06:56:56.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: init 2024-09-16T06:56:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:56.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fd9600a04e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9600a0fd0 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fd9600a1420 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd960098f00 con 0x7fd9600a0fd0 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96013db50 con 0x7fd9600a04e0 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96009d050 con 0x7fd9600a1420 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:56.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95ed9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fd9600a1420 legacy=0x7fd9601d2380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient(hunting): authenticate will time out at 1963.545414s 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95dd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd9600a0fd0 0x7fd96013a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47996/0 (socket says 172.21.15.76:47996) 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95dd9b700 1 -- 172.21.15.76:0/4184629748 learned_addr learned my addr 172.21.15.76:0/4184629748 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950496824 0 0) 0x7fd960098f00 con 0x7fd9600a0fd0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): my global_id is 334167 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9400035d0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116147185 0 0) 0x7fd9400035d0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd960098f00 con 0x7fd9600a0fd0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd9500015d0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): dump: 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346905285 0 0) 0x7fd96013db50 con 0x7fd9600a04e0 2024-09-16T06:56:56.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _negotiate 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient(hunting): my global_id is 284629 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9400035d0 con 0x7fd9600a04e0 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64961430 0 0) 0x7fd960098f00 con 0x7fd9600a0fd0 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 >> v1:172.21.15.76:6790/0 conn(0x7fd9600a1420 legacy=0x7fd9601d2380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 >> v1:172.21.15.189:6789/0 conn(0x7fd9600a04e0 legacy=0x7fd960139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: _finish_hunting 0 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 monclient: found mon.a 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96009d5a0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: _finish_auth 0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: _check_auth_tickets 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 5 monclient: authenticate success, global_id 334167 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _renew_subs 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96013aa00 con 0x7fd9600a0fd0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _renew_subs 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96013d2b0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd950004d70 con 0x7fd9600a0fd0 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:56.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd950005100 con 0x7fd9600a0fd0 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95cd99700 4 set_mon_vals no callback set 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd94effd700 10 monclient: dump: 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.410+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd944005100 con 0x7fd9600a0fd0 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.414+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd9500052e0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.414+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd9500441e0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.414+0000 7fd95e59c700 10 monclient: get_auth_request con 0x7fd940036180 auth_method 0 2024-09-16T06:56:56.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.418+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd950001150 con 0x7fd9600a0fd0 2024-09-16T06:56:56.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.418+0000 7fd94effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:56.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.418+0000 7fd94effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd944005930 con 0x7fd9600a0fd0 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd94effd700 1 -- 172.21.15.76:0/4184629748 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3999302753) 0x7fd950012ff0 con 0x7fd9600a0fd0 2024-09-16T06:56:56.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd94effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:56:56.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd94effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:56:56.652 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:56:56.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd940036180 legacy=0x7fd940038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 10 monclient: shutdown 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 >> v1:172.21.15.76:6789/0 conn(0x7fd9600a0fd0 legacy=0x7fd96013a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 shutdown_connections 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.654+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 >> 172.21.15.76:0/4184629748 conn(0x7fd9600a5870 msgr2=0x7fd96009b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.658+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 shutdown_connections 2024-09-16T06:56:56.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:56.658+0000 7fd95f59e700 1 -- 172.21.15.76:0/4184629748 wait complete. 2024-09-16T06:56:59.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:56:59.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: get_monmap_and_config 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: build_initial_monmap 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: monmap: 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:56:59.792673+0000 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:56:59.792673+0000 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:56:59.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:56:59.788 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 1 Processor -- start 2024-09-16T06:56:59.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 1 -- start start 2024-09-16T06:56:59.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: init 2024-09-16T06:56:59.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-c con 0x7f6b84097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-b con 0x7f6b84099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-a con 0x7f6b84139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b84098990 con 0x7f6b84139500 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b84097f40 con 0x7f6b84097b60 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8409d610 con 0x7f6b84099090 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient(hunting): authenticate will time out at 1966.926498s 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b7bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6b84099090 legacy=0x7f6b8409f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b88e6f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6b84139500 0x7f6b8413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48000/0 (socket says 172.21.15.76:48000) 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b88e6f700 1 -- 172.21.15.76:0/2319137707 learned_addr learned my addr 172.21.15.76:0/2319137707 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013441824 0 0) 0x7f6b84098990 con 0x7f6b84139500 2024-09-16T06:56:59.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): my global_id is 334170 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b700035d0 con 0x7f6b84139500 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925497292 0 0) 0x7f6b84097f40 con 0x7f6b84097b60 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _negotiate 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient(hunting): my global_id is 284632 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b84098990 con 0x7f6b84097b60 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039044313 0 0) 0x7f6b700035d0 con 0x7f6b84139500 2024-09-16T06:56:59.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 >> v1:172.21.15.76:6790/0 conn(0x7f6b84099090 legacy=0x7f6b8409f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 >> v1:172.21.15.189:6789/0 conn(0x7f6b84097b60 legacy=0x7f6b8409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: _finish_hunting 0 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 monclient: found mon.noname-a 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840986b0 con 0x7f6b84139500 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: _finish_auth 0 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: _check_auth_tickets 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 5 monclient: authenticate success, global_id 334170 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b7c001990 con 0x7f6b84139500 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: dump: 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:56:59.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b7c001c70 con 0x7f6b84139500 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 1 -- 172.21.15.76:0/2319137707 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b7c002000 con 0x7f6b84139500 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 10 monclient: get_monmap_and_config success 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b89670700 4 set_mon_vals no callback set 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.790+0000 7f6b837fe700 10 monclient: dump: 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: shutdown 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/2319137707 >> v1:172.21.15.76:6789/0 conn(0x7f6b84139500 legacy=0x7f6b8413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/2319137707 shutdown_connections 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/2319137707 >> 172.21.15.76:0/2319137707 conn(0x7f6b840a0030 msgr2=0x7f6b840a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/2319137707 shutdown_connections 2024-09-16T06:56:59.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/2319137707 wait complete. 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: build_initial_monmap 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: monmap: 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:56:59.800 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 Processor -- start 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- start start 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: init 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _reopen_session rank -1 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:56:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-a con 0x7f6b84097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-c con 0x7f6b84099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient(hunting): picked mon.noname-b con 0x7f6b84139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840710c0 con 0x7f6b84097b60 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8413c190 con 0x7f6b84139500 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840a2b90 con 0x7f6b84099090 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient(hunting): _renew_subs 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient(hunting): authenticate will time out at 1966.930097s 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b7bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6b84099090 legacy=0x7f6b840709b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b83fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6b84097b60 0x7f6b840702a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48012/0 (socket says 172.21.15.76:48012) 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b83fff700 1 -- 172.21.15.76:0/4033063234 learned_addr learned my addr 172.21.15.76:0/4033063234 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:56:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130194702 0 0) 0x7f6b840710c0 con 0x7f6b84097b60 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): my global_id is 334173 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b640035d0 con 0x7f6b84097b60 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2006442992 0 0) 0x7f6b640035d0 con 0x7f6b84097b60 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b840710c0 con 0x7f6b84097b60 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b74001910 con 0x7f6b84097b60 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): dump: 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289184449 0 0) 0x7f6b8413c190 con 0x7f6b84139500 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _negotiate 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:56:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient(hunting): my global_id is 284635 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b640035d0 con 0x7f6b84139500 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3743574439 0 0) 0x7f6b840710c0 con 0x7f6b84097b60 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 >> v1:172.21.15.76:6790/0 conn(0x7f6b84099090 legacy=0x7f6b840709b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 >> v1:172.21.15.189:6789/0 conn(0x7f6b84139500 legacy=0x7f6b841d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: _finish_hunting 0 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 monclient: found mon.a 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8409a540 con 0x7f6b84097b60 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: _finish_auth 0 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: _check_auth_tickets 2024-09-16T06:56:59.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 5 monclient: authenticate success, global_id 334173 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _renew_subs 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8409bf60 con 0x7f6b84097b60 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _renew_subs 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b74001ce0 con 0x7f6b84097b60 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b84071a30 con 0x7f6b84097b60 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6b74003b90 con 0x7f6b84097b60 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b827fc700 4 set_mon_vals no callback set 2024-09-16T06:56:59.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.794+0000 7f6b817fa700 10 monclient: dump: 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:56:59.807 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6b74003d70 con 0x7f6b84097b60 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b89670700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b89670700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b89670700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8409fad0 con 0x7f6b84097b60 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6b74044000 con 0x7f6b84097b60 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.798+0000 7f6b7bfff700 10 monclient: get_auth_request con 0x7f6b64036200 auth_method 0 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.802+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f6b74014710 con 0x7f6b84097b60 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.802+0000 7f6b817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:56:59.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:56:59.802+0000 7f6b817fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:00.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b89670700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:57:00.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b89670700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:00.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b89670700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:00.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6b840672a0 con 0x7f6b84097b60 2024-09-16T06:57:00.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b817fa700 1 -- 172.21.15.76:0/4033063234 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 3196865088) 0x7f6b74014530 con 0x7f6b84097b60 2024-09-16T06:57:00.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b817fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:00.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.034+0000 7f6b817fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:00.033 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6b64036200 legacy=0x7f6b640386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 10 monclient: shutdown 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 >> v1:172.21.15.76:6789/0 conn(0x7f6b84097b60 legacy=0x7f6b840702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 shutdown_connections 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 >> 172.21.15.76:0/4033063234 conn(0x7f6b840a0030 msgr2=0x7f6b8413ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 shutdown_connections 2024-09-16T06:57:00.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:00.038+0000 7f6b89670700 1 -- 172.21.15.76:0/4033063234 wait complete. 2024-09-16T06:57:03.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:57:03.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 10 monclient: get_monmap_and_config 2024-09-16T06:57:03.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 10 monclient: build_initial_monmap 2024-09-16T06:57:03.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 10 monclient: monmap: 2024-09-16T06:57:03.169 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:03.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:03.174988+0000 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:03.174988+0000 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 1 Processor -- start 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 1 -- start start 2024-09-16T06:57:03.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.170+0000 7f44cd9bd700 10 monclient: init 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-a con 0x7f44c809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-b con 0x7f44c8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-c con 0x7f44c8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c809d560 con 0x7f44c809ed60 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c809f140 con 0x7f44c8099030 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c809d790 con 0x7f44c8098be0 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cd9bd700 10 monclient(hunting): authenticate will time out at 1970.308215s 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44c7fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cc9bb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f44c809ed60 0x7f44c80984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48028/0 (socket says 172.21.15.76:48028) 2024-09-16T06:57:03.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.174+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 learned_addr learned my addr 172.21.15.76:0/1822916458 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:03.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.378+0000 7f44c7fff700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:03.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:03.782+0000 7f44c7fff700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:04.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:04.586+0000 7f44c7fff700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): tick 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 monclient(hunting): continuing hunt 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.189:6789/0 conn(0x7f44c8099030 legacy=0x7f44c813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6789/0 conn(0x7f44c809ed60 legacy=0x7f44c80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:06.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-b con 0x7f44c8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-a con 0x7f44b4006ed0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-c con 0x7f44c8099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c809f620 con 0x7f44b4006ed0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c809d790 con 0x7f44c8099030 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4010420 con 0x7f44c8098be0 2024-09-16T06:57:06.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.174+0000 7f44c67fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:06.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.378+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:06.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:06.782+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:07.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:07.586+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:09.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): tick 2024-09-16T06:57:09.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:57:09.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 monclient(hunting): continuing hunt 2024-09-16T06:57:09.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:57:09.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.189:6789/0 conn(0x7f44c8099030 legacy=0x7f44b400cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6789/0 conn(0x7f44b4006ed0 legacy=0x7f44b4009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-b con 0x7f44c8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-c con 0x7f44b4010d80 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): picked mon.noname-a con 0x7f44b400d380 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4011630 con 0x7f44b400d380 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b4010420 con 0x7f44b4010d80 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4010650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b400d780 con 0x7f44c8098be0 2024-09-16T06:57:09.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.174+0000 7f44c67fc700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:09.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.378+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4010650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:09.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:09.782+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4010650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:10.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:10.586+0000 7f44cc9bb700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4010650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845838816 0 0) 0x7f44b4011630 con 0x7f44b400d380 2024-09-16T06:57:11.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient(hunting): my global_id is 334176 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b00035d0 con 0x7f44b400d380 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 768757691 0 0) 0x7f44b00035d0 con 0x7f44b400d380 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44b4010650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.189:6789/0 conn(0x7f44b4010d80 legacy=0x7f44b4005790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 monclient: found mon.noname-a 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40092d0 con 0x7f44b400d380 2024-09-16T06:57:11.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44c0001510 con 0x7f44b400d380 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 5 monclient: authenticate success, global_id 334176 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: dump: 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.218 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c0001760 con 0x7f44b400d380 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 1 -- 172.21.15.76:0/1822916458 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44c0001b30 con 0x7f44b400d380 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44c77fe700 10 monclient: dump: 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:11.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 4 set_mon_vals no callback set 2024-09-16T06:57:11.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.218+0000 7f44cd9bd700 10 monclient: shutdown 2024-09-16T06:57:11.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- 172.21.15.76:0/1822916458 >> v1:172.21.15.76:6789/0 conn(0x7f44b400d380 legacy=0x7f44b400cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- 172.21.15.76:0/1822916458 shutdown_connections 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- 172.21.15.76:0/1822916458 >> 172.21.15.76:0/1822916458 conn(0x7f44c80a0030 msgr2=0x7f44c80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- 172.21.15.76:0/1822916458 shutdown_connections 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- 172.21.15.76:0/1822916458 wait complete. 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient: build_initial_monmap 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient: monmap: 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:11.223 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 Processor -- start 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 1 -- start start 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient: init 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.222+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-b con 0x7f44b400d380 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-a con 0x7f44b4010d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 10 monclient(hunting): picked mon.noname-c con 0x7f44c8098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd1bc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81ce320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c81b17e0 con 0x7f44b4010d80 2024-09-16T06:57:11.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c81b0310 con 0x7f44b400d380 2024-09-16T06:57:11.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44c7fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f44b4010d80 0x7f44c81afc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42372/0 (socket says 172.21.15.76:42372) 2024-09-16T06:57:11.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44c7fff700 1 -- 172.21.15.76:0/2202599964 learned_addr learned my addr 172.21.15.76:0/2202599964 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:11.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c80a1500 con 0x7f44c8098be0 2024-09-16T06:57:11.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:11.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.226+0000 7f44cd9bd700 10 monclient(hunting): authenticate will time out at 1978.360529s 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569294682 0 0) 0x7f44c81b0310 con 0x7f44b400d380 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 10 monclient(hunting): my global_id is 284638 2024-09-16T06:57:11.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4011630 con 0x7f44b400d380 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.230+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287548155 0 0) 0x7f44b4011630 con 0x7f44b400d380 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44c81b0310 con 0x7f44b400d380 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44b8001bf0 con 0x7f44b400d380 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): dump: 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.231 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495685853 0 0) 0x7f44c81b17e0 con 0x7f44b4010d80 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient(hunting): my global_id is 334179 2024-09-16T06:57:11.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4011630 con 0x7f44b4010d80 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3428436507 0 0) 0x7f44c81b0310 con 0x7f44b400d380 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 >> v1:172.21.15.76:6790/0 conn(0x7f44c8098be0 legacy=0x7f44c81ce320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 >> v1:172.21.15.76:6789/0 conn(0x7f44b4010d80 legacy=0x7f44c81afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 monclient: found mon.b 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8098920 con 0x7f44b400d380 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 5 monclient: authenticate success, global_id 284638 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _renew_subs 2024-09-16T06:57:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c809a540 con 0x7f44b400d380 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _renew_subs 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c81b3110 con 0x7f44b400d380 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44b8004c10 con 0x7f44b400d380 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f44b8004f20 con 0x7f44b400d380 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c6ffd700 4 set_mon_vals no callback set 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 10 monclient: dump: 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.235 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f44b8005100 con 0x7f44b400d380 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8005100 con 0x7f44b400d380 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f44b8044160 con 0x7f44b400d380 2024-09-16T06:57:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.234+0000 7f44c7fff700 10 monclient: get_auth_request con 0x7f44b40397c0 auth_method 0 2024-09-16T06:57:11.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.242+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f44b8014860 con 0x7f44b400d380 2024-09-16T06:57:11.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.242+0000 7f44c57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.242+0000 7f44c57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:11.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.474+0000 7f44cd9bd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:57:11.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.474+0000 7f44cd9bd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:11.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.474+0000 7f44cd9bd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:57:11.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.474+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f44a8005930 con 0x7f44b400d380 2024-09-16T06:57:11.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44c57fa700 1 -- 172.21.15.76:0/2202599964 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 407188266) 0x7f44b8012f00 con 0x7f44b400d380 2024-09-16T06:57:11.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44c57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:11.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44c57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:11.475 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":166,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:57:11.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 >> v1:172.21.15.189:6812/3098810674 conn(0x7f44b40397c0 legacy=0x7f44b403bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 10 monclient: shutdown 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 >> v1:172.21.15.189:6789/0 conn(0x7f44b400d380 legacy=0x7f44c81af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 shutdown_connections 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 >> 172.21.15.76:0/2202599964 conn(0x7f44c80a0030 msgr2=0x7f44c806ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 shutdown_connections 2024-09-16T06:57:11.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.478+0000 7f44cd9bd700 1 -- 172.21.15.76:0/2202599964 wait complete. 2024-09-16T06:57:11.489 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:57:11.489 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:57:11.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: build_initial_monmap 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: monmap: 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:11.608958+0000 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:11.608958+0000 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:11.603 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.604 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:11.604 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:11.604 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:11.604 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 1 Processor -- start 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 1 -- start start 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: init 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9a10097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:11.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9a10099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9a10139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10098990 con 0x7f9a10097b60 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10097f40 con 0x7f9a10139500 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1009d610 con 0x7f9a10099090 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient(hunting): authenticate will time out at 1978.742124s 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a07fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9a10099090 legacy=0x7f9a1009f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0e59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9a10097b60 0x7f9a1009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42374/0 (socket says 172.21.15.76:42374) 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0e59c700 1 -- 172.21.15.76:0/2821444932 learned_addr learned my addr 172.21.15.76:0/2821444932 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413077192 0 0) 0x7f9a10098990 con 0x7f9a10097b60 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): my global_id is 334182 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99fc0035d0 con 0x7f9a10097b60 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578048082 0 0) 0x7f9a10097f40 con 0x7f9a10139500 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient(hunting): my global_id is 284641 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10098990 con 0x7f9a10139500 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211210940 0 0) 0x7f99fc0035d0 con 0x7f9a10097b60 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 >> v1:172.21.15.76:6790/0 conn(0x7f9a10099090 legacy=0x7f9a1009f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 >> v1:172.21.15.189:6789/0 conn(0x7f9a10139500 legacy=0x7f9a1013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 monclient: found mon.noname-a 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a100986b0 con 0x7f9a10097b60 2024-09-16T06:57:11.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 5 monclient: authenticate success, global_id 334182 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a00001610 con 0x7f9a10097b60 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: dump: 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.611 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a000018f0 con 0x7f9a10097b60 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 1 -- 172.21.15.76:0/2821444932 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a00001c80 con 0x7f9a10097b60 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0f59e700 4 set_mon_vals no callback set 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.606+0000 7f9a0dd9b700 10 monclient: dump: 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.613 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: shutdown 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/2821444932 >> v1:172.21.15.76:6789/0 conn(0x7f9a10097b60 legacy=0x7f9a1009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/2821444932 shutdown_connections 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/2821444932 >> 172.21.15.76:0/2821444932 conn(0x7f9a100a0030 msgr2=0x7f9a100a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/2821444932 shutdown_connections 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/2821444932 wait complete. 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: build_initial_monmap 2024-09-16T06:57:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: monmap: 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 Processor -- start 2024-09-16T06:57:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- start start 2024-09-16T06:57:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: init 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9a10097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9a10099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9a10139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10136a60 con 0x7f9a10139500 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1009a540 con 0x7f9a10099090 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0e59c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f9a10097b60 legacy=0x7f9a10135c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10138100 con 0x7f9a10097b60 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient(hunting): authenticate will time out at 1978.745944s 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9a10139500 0x7f9a101d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42380/0 (socket says 172.21.15.76:42380) 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0ed9d700 1 -- 172.21.15.76:0/785488718 learned_addr learned my addr 172.21.15.76:0/785488718 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850776164 0 0) 0x7f9a10136a60 con 0x7f9a10139500 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): my global_id is 334185 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f00035d0 con 0x7f9a10139500 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942430322 0 0) 0x7f9a1009a540 con 0x7f9a10099090 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): my global_id is 284644 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10136a60 con 0x7f9a10099090 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515910104 0 0) 0x7f99f00035d0 con 0x7f9a10139500 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a1009a540 con 0x7f9a10139500 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a08001c30 con 0x7f9a10139500 2024-09-16T06:57:11.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): dump: 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1457390516 0 0) 0x7f9a1009a540 con 0x7f9a10139500 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6790/0 conn(0x7f9a10097b60 legacy=0x7f9a10135c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.189:6789/0 conn(0x7f9a10099090 legacy=0x7f9a10136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 monclient: found mon.a 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a101383e0 con 0x7f9a10139500 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 5 monclient: authenticate success, global_id 334185 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _renew_subs 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a101386c0 con 0x7f9a10139500 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _renew_subs 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1013c820 con 0x7f9a10139500 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a08003a40 con 0x7f9a10139500 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a08003dd0 con 0x7f9a10139500 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a0d59a700 4 set_mon_vals no callback set 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.610+0000 7f9a06ffd700 10 monclient: dump: 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.626 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a0f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99f4006590 con 0x7f99f4002830 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774966459 0 0) 0x7f99f4006590 con 0x7f99f4002830 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient(hunting): my global_id is 334188 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a1009a540 con 0x7f99f4002830 2024-09-16T06:57:11.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2464062749 0 0) 0x7f9a1009a540 con 0x7f99f4002830 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f4006590 con 0x7f99f4002830 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a00001620 con 0x7f99f4002830 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient: dump: 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.628 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 621503591 0 0) 0x7f99f4006590 con 0x7f99f4002830 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99f400a190 con 0x7f99f4002830 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9a08003fb0 con 0x7f9a10139500 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f9a08044170 con 0x7f9a10139500 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f9a000049f0 con 0x7f99f4002830 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:11.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6789/0 conn(0x7f99f4002830 legacy=0x7f99f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6789/0 conn(0x7f99f4002830 legacy=0x7f99f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:11.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.614+0000 7f9a07fff700 10 monclient: get_auth_request con 0x7f99f0036730 auth_method 0 2024-09-16T06:57:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a0f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f99f400c4e0 con 0x7f99f4009150 2024-09-16T06:57:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253645442 0 0) 0x7f99f400c4e0 con 0x7f99f4009150 2024-09-16T06:57:11.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient(hunting): my global_id is 334191 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4006590 con 0x7f99f4009150 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1681578352 0 0) 0x7f99f4006590 con 0x7f99f4009150 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f400c4e0 con 0x7f99f4009150 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9a00004fd0 con 0x7f99f4009150 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient: dump: 2024-09-16T06:57:11.632 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4110130098 0 0) 0x7f99f400c4e0 con 0x7f99f4009150 2024-09-16T06:57:11.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99f4009cf0 con 0x7f99f4009150 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4035 (unknown 0 0 4040280273) 0x7f9a000052b0 con 0x7f99f4009150 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.622+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6789/0 conn(0x7f99f4009150 legacy=0x7f99f40095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a06ffd700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6789/0 conn(0x7f99f4009150 legacy=0x7f99f40095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:11.634 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 166, 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:11.635 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.636 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.637 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:11.638 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.639 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.641 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.189:6812/3098810674 conn(0x7f99f0036730 legacy=0x7f99f0038bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 10 monclient: shutdown 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 >> v1:172.21.15.76:6789/0 conn(0x7f9a10139500 legacy=0x7f9a101d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0ed9d700 1 -- 172.21.15.76:0/785488718 reap_dead start 2024-09-16T06:57:11.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0ed9d700 1 -- 172.21.15.76:0/785488718 reap_dead start 2024-09-16T06:57:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 shutdown_connections 2024-09-16T06:57:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 >> 172.21.15.76:0/785488718 conn(0x7f9a100a0030 msgr2=0x7f9a10134900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 shutdown_connections 2024-09-16T06:57:11.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.626+0000 7f9a0f59e700 1 -- 172.21.15.76:0/785488718 wait complete. 2024-09-16T06:57:11.650 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:57:11.650 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:57:11.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: build_initial_monmap 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: monmap: 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:11.773893+0000 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:11.773893+0000 2024-09-16T06:57:11.768 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:11.769 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.769 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:11.769 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:11.769 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:11.769 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 1 Processor -- start 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 1 -- start start 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: init 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f228c097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f228c099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f228c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c098990 con 0x7f228c139500 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c097f40 con 0x7f228c097b60 2024-09-16T06:57:11.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c09d610 con 0x7f228c099090 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228bfff700 10 monclient(hunting): authenticate will time out at 1978.907171s 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f22827fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f228c099090 legacy=0x7f228c09f470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f228c139500 0x7f228c13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42398/0 (socket says 172.21.15.76:42398) 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.770+0000 7f228b7fe700 1 -- 172.21.15.76:0/4234304558 learned_addr learned my addr 172.21.15.76:0/4234304558 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922840260 0 0) 0x7f228c098990 con 0x7f228c139500 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): my global_id is 334194 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22780035d0 con 0x7f228c139500 2024-09-16T06:57:11.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208633177 0 0) 0x7f228c097f40 con 0x7f228c097b60 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient(hunting): my global_id is 284647 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c098990 con 0x7f228c097b60 2024-09-16T06:57:11.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101893333 0 0) 0x7f22780035d0 con 0x7f228c139500 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 >> v1:172.21.15.76:6790/0 conn(0x7f228c099090 legacy=0x7f228c09f470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 >> v1:172.21.15.189:6789/0 conn(0x7f228c097b60 legacy=0x7f228c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0986b0 con 0x7f228c139500 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2284001990 con 0x7f228c139500 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 5 monclient: authenticate success, global_id 334194 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: dump: 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2284001c70 con 0x7f228c139500 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 1 -- 172.21.15.76:0/4234304558 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2284002000 con 0x7f228c139500 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 4 set_mon_vals no callback set 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228a7fc700 10 monclient: dump: 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: shutdown 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- 172.21.15.76:0/4234304558 >> v1:172.21.15.76:6789/0 conn(0x7f228c139500 legacy=0x7f228c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- 172.21.15.76:0/4234304558 shutdown_connections 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- 172.21.15.76:0/4234304558 >> 172.21.15.76:0/4234304558 conn(0x7f228c0a0030 msgr2=0x7f228c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- 172.21.15.76:0/4234304558 shutdown_connections 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- 172.21.15.76:0/4234304558 wait complete. 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: build_initial_monmap 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: monmap: 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:11.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 Processor -- start 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- start start 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: init 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f228c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f228c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f228c139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c08f450 con 0x7f228c139500 2024-09-16T06:57:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0a2b50 con 0x7f228c099090 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c09c2f0 con 0x7f228c097b60 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228bfff700 10 monclient(hunting): authenticate will time out at 1978.911057s 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228affd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f228c097b60 legacy=0x7f228c0960e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f228c139500 0x7f228c1cddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:42400/0 (socket says 172.21.15.76:42400) 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f228b7fe700 1 -- 172.21.15.76:0/3859170863 learned_addr learned my addr 172.21.15.76:0/3859170863 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037112850 0 0) 0x7f228c08f450 con 0x7f228c139500 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): my global_id is 334197 2024-09-16T06:57:11.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c0035d0 con 0x7f228c139500 2024-09-16T06:57:11.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288642079 0 0) 0x7f228c0a2b50 con 0x7f228c099090 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 10 monclient(hunting): my global_id is 284650 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.774+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c08f450 con 0x7f228c099090 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208920874 0 0) 0x7f226c0035d0 con 0x7f228c139500 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c0a2b50 con 0x7f228c139500 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2284001950 con 0x7f228c139500 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): dump: 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.786 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261561531 0 0) 0x7f228c08f450 con 0x7f228c099090 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f226c0035d0 con 0x7f228c099090 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2115571523 0 0) 0x7f228c0a2b50 con 0x7f228c139500 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.76:6790/0 conn(0x7f228c097b60 legacy=0x7f228c0960e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:57:11.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6789/0 conn(0x7f228c099090 legacy=0x7f228c1ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: _finish_hunting 0 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 monclient: found mon.a 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c090890 con 0x7f228c139500 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: _finish_auth 0 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: _check_auth_tickets 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 5 monclient: authenticate success, global_id 334197 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 10 monclient: _renew_subs 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c08e810 con 0x7f228c139500 2024-09-16T06:57:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 10 monclient: _renew_subs 2024-09-16T06:57:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:11.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c090b70 con 0x7f228c139500 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f228c09d360 con 0x7f228c0967f0 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22840050a0 con 0x7f228c139500 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2289ffb700 4 set_mon_vals no callback set 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f22840053b0 con 0x7f228c139500 2024-09-16T06:57:11.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: dump: 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2284005510 con 0x7f228c139500 2024-09-16T06:57:11.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2284044300 con 0x7f228c139500 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511426148 0 0) 0x7f228c09d360 con 0x7f228c0967f0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient(hunting): my global_id is 284653 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0a2b50 con 0x7f228c0967f0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f22827fc700 10 monclient: get_auth_request con 0x7f226c036280 auth_method 0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4209875046 0 0) 0x7f228c0a2b50 con 0x7f228c0967f0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c09d360 con 0x7f228c0967f0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2274001740 con 0x7f228c0967f0 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:11.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 10 monclient: dump: 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2310860656 0 0) 0x7f228c09d360 con 0x7f228c0967f0 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.778+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f228c08f220 con 0x7f228c0967f0 2024-09-16T06:57:11.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.782+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2274001be0 con 0x7f228c0967f0 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.782+0000 7f2283fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.782+0000 7f2283fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.782+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6789/0 conn(0x7f228c0967f0 legacy=0x7f228c06eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.782+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6789/0 conn(0x7f228c0967f0 legacy=0x7f228c06eeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f228bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f228c0672a0 con 0x7f228c02c6b0 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717044511 0 0) 0x7f228c0672a0 con 0x7f228c02c6b0 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient(hunting): my global_id is 284656 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c09d360 con 0x7f228c02c6b0 2024-09-16T06:57:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1022594509 0 0) 0x7f228c09d360 con 0x7f228c02c6b0 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c0672a0 con 0x7f228c02c6b0 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2274003ad0 con 0x7f228c02c6b0 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 10 monclient: dump: 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:11.797 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2853641545 0 0) 0x7f228c0672a0 con 0x7f228c02c6b0 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.790+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f228c09fb80 con 0x7f228c02c6b0 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4037 (unknown 0 0 373973696) 0x7f2274005870 con 0x7f228c02c6b0 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f2283fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f2283fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:11.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6789/0 conn(0x7f228c02c6b0 legacy=0x7f228c06f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f2283fff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6789/0 conn(0x7f228c02c6b0 legacy=0x7f228c06f5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 166, 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:57:11.799 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.800 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:11.802 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:11.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:11.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:11.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.189:6812/3098810674 conn(0x7f226c036280 legacy=0x7f226c038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 10 monclient: shutdown 2024-09-16T06:57:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 >> v1:172.21.15.76:6789/0 conn(0x7f228c139500 legacy=0x7f228c1cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228b7fe700 1 -- 172.21.15.76:0/3859170863 reap_dead start 2024-09-16T06:57:11.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228b7fe700 1 -- 172.21.15.76:0/3859170863 reap_dead start 2024-09-16T06:57:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 shutdown_connections 2024-09-16T06:57:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 >> 172.21.15.76:0/3859170863 conn(0x7f228c0a0030 msgr2=0x7f228c0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 shutdown_connections 2024-09-16T06:57:11.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:11.794+0000 7f228bfff700 1 -- 172.21.15.76:0/3859170863 wait complete. 2024-09-16T06:57:11.815 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:57:11.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:57:31.816 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:57:31.817 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:57:31.817 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:57:31.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:57:31.820 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:57:31.820 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:57:31.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 10 monclient: get_monmap_and_config 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 10 monclient: build_initial_monmap 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 10 monclient: monmap: 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:31.930941+0000 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:31.930941+0000 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:31.926 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 1 Processor -- start 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 1 -- start start 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.925+0000 7fefa2517700 10 monclient: init 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-c con 0x7fef9c09c8e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-b con 0x7fef9c09f410 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-a con 0x7fef9c09d3d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09ed10 con 0x7fef9c09d3d0 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09dc40 con 0x7fef9c09c8e0 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0a1c20 con 0x7fef9c09f410 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient(hunting): authenticate will time out at 1999.064266s 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa0d14700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fef9c09f410 legacy=0x7fef9c138560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa1d16700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fef9c09d3d0 0x7fef9c13bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59030/0 (socket says 172.21.15.76:59030) 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa1d16700 1 -- 172.21.15.76:0/4130537180 learned_addr learned my addr 172.21.15.76:0/4130537180 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375483232 0 0) 0x7fef9c09ed10 con 0x7fef9c09d3d0 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient(hunting): my global_id is 334200 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef880035d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000547955 0 0) 0x7fef880035d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 >> v1:172.21.15.76:6790/0 conn(0x7fef9c09f410 legacy=0x7fef9c138560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 >> v1:172.21.15.189:6789/0 conn(0x7fef9c09c8e0 legacy=0x7fef9c09ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: _finish_hunting 0 2024-09-16T06:57:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 monclient: found mon.noname-a 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c09c3e0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: _finish_auth 0 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: _check_auth_tickets 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 5 monclient: authenticate success, global_id 334200 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef94001990 con 0x7fef9c09d3d0 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: dump: 2024-09-16T06:57:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef94001cb0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 4 set_mon_vals no callback set 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 1 -- 172.21.15.76:0/4130537180 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef94002080 con 0x7fef9c09d3d0 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fef93fff700 10 monclient: dump: 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:31.935 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: shutdown 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- 172.21.15.76:0/4130537180 >> v1:172.21.15.76:6789/0 conn(0x7fef9c09d3d0 legacy=0x7fef9c13bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- 172.21.15.76:0/4130537180 shutdown_connections 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- 172.21.15.76:0/4130537180 >> 172.21.15.76:0/4130537180 conn(0x7fef9c0a2300 msgr2=0x7fef9c0a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- 172.21.15.76:0/4130537180 shutdown_connections 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 1 -- 172.21.15.76:0/4130537180 wait complete. 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: build_initial_monmap 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.929+0000 7fefa2517700 10 monclient: monmap: 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:31.936 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 Processor -- start 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- start start 2024-09-16T06:57:31.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: init 2024-09-16T06:57:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:57:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-c con 0x7fef9c09c8e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-a con 0x7fef9c09d3d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:31.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient(hunting): picked mon.noname-b con 0x7fef9c09f410 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c09c5d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c094380 con 0x7fef9c09f410 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c094a70 con 0x7fef9c09c8e0 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient(hunting): authenticate will time out at 1999.069260s 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa1515700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef9c09c8e0 0x7fef9c098860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46622/0 (socket says 172.21.15.76:46622) 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa1515700 1 -- 172.21.15.76:0/2198261122 learned_addr learned my addr 172.21.15.76:0/2198261122 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765600988 0 0) 0x7fef9c09c5d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): my global_id is 334203 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef800035d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865266802 0 0) 0x7fef9c094380 con 0x7fef9c09f410 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): my global_id is 284662 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c09c5d0 con 0x7fef9c09f410 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057222611 0 0) 0x7fef800035d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c094380 con 0x7fef9c09d3d0 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef8c004140 con 0x7fef9c09d3d0 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:31.943 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1616980419 0 0) 0x7fef9c094380 con 0x7fef9c09d3d0 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 >> v1:172.21.15.76:6790/0 conn(0x7fef9c09c8e0 legacy=0x7fef9c098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 >> v1:172.21.15.189:6789/0 conn(0x7fef9c09f410 legacy=0x7fef9c1d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 monclient: found mon.a 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c094d50 con 0x7fef9c09d3d0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: _finish_auth 0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 5 monclient: authenticate success, global_id 334203 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _renew_subs 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c06d650 con 0x7fef9c09d3d0 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _renew_subs 2024-09-16T06:57:31.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c093b80 con 0x7fef9c09d3d0 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef8c004360 con 0x7fef9c09d3d0 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef8c0046f0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef937fe700 4 set_mon_vals no callback set 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fef91ffb700 10 monclient: dump: 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:31.946 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.933+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c005100 con 0x7fef9c09d3d0 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.937+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fef8c0048d0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.937+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fef8c043da0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.937+0000 7fefa1515700 10 monclient: get_auth_request con 0x7fef80036200 auth_method 0 2024-09-16T06:57:31.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.941+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fef8c0146e0 con 0x7fef9c09d3d0 2024-09-16T06:57:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.941+0000 7fef91ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:31.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:31.941+0000 7fef91ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:32.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fef7c005930 con 0x7fef9c09d3d0 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fef91ffb700 1 -- 172.21.15.76:0/2198261122 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1939615625) 0x7fef8c012b50 con 0x7fef9c09d3d0 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fef91ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fef91ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:32.175 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:57:32.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 >> v1:172.21.15.189:6812/3098810674 conn(0x7fef80036200 legacy=0x7fef800386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 10 monclient: shutdown 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 >> v1:172.21.15.76:6789/0 conn(0x7fef9c09d3d0 legacy=0x7fef9c1ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 shutdown_connections 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 >> 172.21.15.76:0/2198261122 conn(0x7fef9c0a2300 msgr2=0x7fef9c070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 shutdown_connections 2024-09-16T06:57:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.177+0000 7fefa2517700 1 -- 172.21.15.76:0/2198261122 wait complete. 2024-09-16T06:57:32.190 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:57:32.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:57:32.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:57:32.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: get_monmap_and_config 2024-09-16T06:57:32.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: monmap: 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:32.309249+0000 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:32.309249+0000 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:32.304 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 1 Processor -- start 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 1 -- start start 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: init 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient(hunting): picked mon.noname-a con 0x7f144c09b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient(hunting): picked mon.noname-c con 0x7f144c0964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient(hunting): picked mon.noname-b con 0x7f144c138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c098560 con 0x7f144c09b1c0 2024-09-16T06:57:32.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c099ec0 con 0x7f144c0964f0 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c097030 con 0x7f144c138870 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451847700 10 monclient(hunting): authenticate will time out at 1999.442824s 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451046700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f144c138870 0x7f144c13ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46632/0 (socket says 172.21.15.76:46632) 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1450845700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f144c09b1c0 0x7f144c09ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59056/0 (socket says 172.21.15.76:59056) 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f1451046700 1 -- 172.21.15.76:0/3411574803 learned_addr learned my addr 172.21.15.76:0/3411574803 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591546027 0 0) 0x7f144c098560 con 0x7f144c09b1c0 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): my global_id is 334206 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14340035d0 con 0x7f144c09b1c0 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842670647 0 0) 0x7f144c099ec0 con 0x7f144c0964f0 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): my global_id is 284665 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c098560 con 0x7f144c0964f0 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885379526 0 0) 0x7f144c097030 con 0x7f144c138870 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 10 monclient(hunting): my global_id is 344099 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c099ec0 con 0x7f144c138870 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.305+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030388549 0 0) 0x7f144c099ec0 con 0x7f144c138870 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 >> v1:172.21.15.189:6789/0 conn(0x7f144c0964f0 legacy=0x7f144c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 >> v1:172.21.15.76:6789/0 conn(0x7f144c09b1c0 legacy=0x7f144c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c09b5a0 con 0x7f144c138870 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 5 monclient: authenticate success, global_id 344099 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1444001530 con 0x7f144c138870 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: dump: 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14440017a0 con 0x7f144c138870 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 4 set_mon_vals no callback set 2024-09-16T06:57:32.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 1 -- 172.21.15.76:0/3411574803 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1444001bb0 con 0x7f144c138870 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f144b7fe700 10 monclient: dump: 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.315 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: shutdown 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/3411574803 >> v1:172.21.15.76:6790/0 conn(0x7f144c138870 legacy=0x7f144c13ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/3411574803 shutdown_connections 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/3411574803 >> 172.21.15.76:0/3411574803 conn(0x7f144c0a0030 msgr2=0x7f144c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/3411574803 shutdown_connections 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/3411574803 wait complete. 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: monmap: 2024-09-16T06:57:32.316 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 Processor -- start 2024-09-16T06:57:32.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- start start 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: init 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient(hunting): picked mon.noname-b con 0x7f144c0964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient(hunting): picked mon.noname-a con 0x7f144c09b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient(hunting): picked mon.noname-c con 0x7f144c138870 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c08f9a0 con 0x7f144c09b1c0 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c09c040 con 0x7f144c0964f0 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144c09a540 con 0x7f144c138870 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient(hunting): authenticate will time out at 1999.446539s 2024-09-16T06:57:32.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451046700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f144c138870 0x7f144c08d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46634/0 (socket says 172.21.15.76:46634) 2024-09-16T06:57:32.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451046700 1 -- 172.21.15.76:0/4083683835 learned_addr learned my addr 172.21.15.76:0/4083683835 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522506694 0 0) 0x7f144c08f9a0 con 0x7f144c09b1c0 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): my global_id is 334209 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14380035d0 con 0x7f144c09b1c0 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155475148 0 0) 0x7f144c09a540 con 0x7f144c138870 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): my global_id is 344102 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c08f9a0 con 0x7f144c138870 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948754176 0 0) 0x7f144c09c040 con 0x7f144c0964f0 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): my global_id is 284668 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c09a540 con 0x7f144c0964f0 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271943939 0 0) 0x7f14380035d0 con 0x7f144c09b1c0 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f144c09c040 con 0x7f144c09b1c0 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f143c001c30 con 0x7f144c09b1c0 2024-09-16T06:57:32.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): dump: 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.327 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216266835 0 0) 0x7f144c08f9a0 con 0x7f144c138870 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14380035d0 con 0x7f144c138870 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1444001b70 con 0x7f144c138870 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient(hunting): dump: 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.328 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 291886230 0 0) 0x7f144c09c040 con 0x7f144c09b1c0 2024-09-16T06:57:32.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6790/0 conn(0x7f144c138870 legacy=0x7f144c08d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.189:6789/0 conn(0x7f144c0964f0 legacy=0x7f144c08c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 monclient: found mon.a 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0a2f70 con 0x7f144c09b1c0 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f14497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 5 monclient: authenticate success, global_id 334209 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _renew_subs 2024-09-16T06:57:32.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c08e1a0 con 0x7f144c09b1c0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _renew_subs 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.309+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c1cec20 con 0x7f144c09b1c0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f143c003a20 con 0x7f144c09b1c0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f143c003d30 con 0x7f144c09b1c0 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: dump: 2024-09-16T06:57:32.331 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f144affd700 4 set_mon_vals no callback set 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f1451847700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f143c00e7a0 con 0x7f144c09b1c0 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f142c006590 con 0x7f142c002830 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f143c043020 con 0x7f144c09b1c0 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332330867 0 0) 0x7f142c006590 con 0x7f142c002830 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient(hunting): my global_id is 334212 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c09c040 con 0x7f142c002830 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f1451046700 10 monclient: get_auth_request con 0x7f1438036430 auth_method 0 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2576710903 0 0) 0x7f144c09c040 con 0x7f142c002830 2024-09-16T06:57:32.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f142c006590 con 0x7f142c002830 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f14400016e0 con 0x7f142c002830 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: dump: 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.334 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3239811060 0 0) 0x7f142c006590 con 0x7f142c002830 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f142c00a190 con 0x7f142c002830 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f1440001ad0 con 0x7f142c002830 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6789/0 conn(0x7f142c002830 legacy=0x7f142c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.313+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6789/0 conn(0x7f142c002830 legacy=0x7f142c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f1451847700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:32.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f142c009e50 con 0x7f142c009170 2024-09-16T06:57:32.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420664297 0 0) 0x7f142c009e50 con 0x7f142c009170 2024-09-16T06:57:32.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient(hunting): my global_id is 334215 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142c006590 con 0x7f142c009170 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 989972064 0 0) 0x7f142c006590 con 0x7f142c009170 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f142c009e50 con 0x7f142c009170 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1440004a30 con 0x7f142c009170 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 10 monclient: dump: 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.338 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.339 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2285010967 0 0) 0x7f142c009e50 con 0x7f142c009170 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.325+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f142c00c340 con 0x7f142c009170 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1292278419) 0x7f1440005fe0 con 0x7f142c009170 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f14497fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f14497fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6789/0 conn(0x7f142c009170 legacy=0x7f142c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f14497fa700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6789/0 conn(0x7f142c009170 legacy=0x7f142c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.340 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:57:32.341 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.342 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.343 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:32.344 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:32.345 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.348 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.350 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.351 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.352 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:57:32.353 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:32.354 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1438036430 legacy=0x7f14380388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 10 monclient: shutdown 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 >> v1:172.21.15.76:6789/0 conn(0x7f144c09b1c0 legacy=0x7f144c08c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451046700 1 -- 172.21.15.76:0/4083683835 reap_dead start 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 shutdown_connections 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 >> 172.21.15.76:0/4083683835 conn(0x7f144c0a0030 msgr2=0x7f144c0a1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 shutdown_connections 2024-09-16T06:57:32.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.329+0000 7f1451847700 1 -- 172.21.15.76:0/4083683835 wait complete. 2024-09-16T06:57:32.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:32.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:57:32.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: get_monmap_and_config 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: monmap: 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:32.481670+0000 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:32.481670+0000 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.476 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:32.477 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:32.477 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 1 Processor -- start 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 1 -- start start 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: init 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-a con 0x7f13f801f4c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-b con 0x7f13f8025410 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-c con 0x7f13f8025860 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f801f8a0 con 0x7f13f801f4c0 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f80233f0 con 0x7f13f8025860 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f8020050 con 0x7f13f8025410 2024-09-16T06:57:32.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14064ff700 10 monclient(hunting): authenticate will time out at 1999.614978s 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f1404cfc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13f8025410 0x7f13f80bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46640/0 (socket says 172.21.15.76:46640) 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f14054fd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f13f801f4c0 0x7f13f8024d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59088/0 (socket says 172.21.15.76:59088) 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f1404cfc700 1 -- 172.21.15.76:0/2738701837 learned_addr learned my addr 172.21.15.76:0/2738701837 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426337789 0 0) 0x7f13f8020050 con 0x7f13f8025410 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): my global_id is 344105 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec0035d0 con 0x7f13f8025410 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717270688 0 0) 0x7f13f80233f0 con 0x7f13f8025860 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 10 monclient(hunting): my global_id is 284671 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f8020050 con 0x7f13f8025860 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.477+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303892154 0 0) 0x7f13f801f8a0 con 0x7f13f801f4c0 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient(hunting): my global_id is 334218 2024-09-16T06:57:32.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f80233f0 con 0x7f13f801f4c0 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1535084321 0 0) 0x7f13ec0035d0 con 0x7f13f8025410 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 >> v1:172.21.15.189:6789/0 conn(0x7f13f8025860 legacy=0x7f13f80c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 >> v1:172.21.15.76:6789/0 conn(0x7f13f801f4c0 legacy=0x7f13f8024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 monclient: found mon.noname-b 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8021a90 con 0x7f13f8025410 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 5 monclient: authenticate success, global_id 344105 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13fc001610 con 0x7f13f8025410 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: dump: 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.492 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13fc001880 con 0x7f13f8025410 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 1 -- 172.21.15.76:0/2738701837 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13fc001c10 con 0x7f13f8025410 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 4 set_mon_vals no callback set 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f13f7fff700 10 monclient: dump: 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.494 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: shutdown 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- 172.21.15.76:0/2738701837 >> v1:172.21.15.76:6790/0 conn(0x7f13f8025410 legacy=0x7f13f80bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- 172.21.15.76:0/2738701837 shutdown_connections 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- 172.21.15.76:0/2738701837 >> 172.21.15.76:0/2738701837 conn(0x7f13f8005b40 msgr2=0x7f13f8005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- 172.21.15.76:0/2738701837 shutdown_connections 2024-09-16T06:57:32.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- 172.21.15.76:0/2738701837 wait complete. 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: monmap: 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 Processor -- start 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- start start 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: init 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-c con 0x7f13f801f4c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-b con 0x7f13f8025410 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 10 monclient(hunting): picked mon.noname-a con 0x7f13f8025860 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f812db80 con 0x7f13f8025860 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14054fd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13f801f4c0 0x7f13f8140b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46646/0 (socket says 172.21.15.76:46646) 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14054fd700 1 -- 172.21.15.76:0/1306603224 learned_addr learned my addr 172.21.15.76:0/1306603224 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.481+0000 7f14064ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f801be40 con 0x7f13f8025410 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f812f4e0 con 0x7f13f801f4c0 2024-09-16T06:57:32.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient(hunting): authenticate will time out at 1999.620124s 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429460959 0 0) 0x7f13f812db80 con 0x7f13f8025860 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): my global_id is 334221 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e40035d0 con 0x7f13f8025860 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777932998 0 0) 0x7f13f812f4e0 con 0x7f13f801f4c0 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): my global_id is 344108 2024-09-16T06:57:32.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f812db80 con 0x7f13f801f4c0 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23586226 0 0) 0x7f13f801be40 con 0x7f13f8025410 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): my global_id is 284674 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f812f4e0 con 0x7f13f8025410 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284988292 0 0) 0x7f13e40035d0 con 0x7f13f8025860 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13f801be40 con 0x7f13f8025860 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1400069400 con 0x7f13f8025860 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:32.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681065760 0 0) 0x7f13f812db80 con 0x7f13f801f4c0 2024-09-16T06:57:32.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e40035d0 con 0x7f13f801f4c0 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13f00018b0 con 0x7f13f801f4c0 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3174414115 0 0) 0x7f13f801be40 con 0x7f13f8025860 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6790/0 conn(0x7f13f801f4c0 legacy=0x7f13f8140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.189:6789/0 conn(0x7f13f8025410 legacy=0x7f13f8152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 monclient: found mon.a 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f812d180 con 0x7f13f8025860 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 5 monclient: authenticate success, global_id 334221 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient: _renew_subs 2024-09-16T06:57:32.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f80192a0 con 0x7f13f8025860 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient: _renew_subs 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f812ec40 con 0x7f13f8025860 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f140004ec20 con 0x7f13f8025860 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1400066de0 con 0x7f13f8025860 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f77fe700 4 set_mon_vals no callback set 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: dump: 2024-09-16T06:57:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13e0006590 con 0x7f13e0002830 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380238486 0 0) 0x7f13e0006590 con 0x7f13e0002830 2024-09-16T06:57:32.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient(hunting): my global_id is 344111 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f801be40 con 0x7f13e0002830 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1400067aa0 con 0x7f13f8025860 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f14000bc840 con 0x7f13f8025860 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4169330748 0 0) 0x7f13f801be40 con 0x7f13e0002830 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f1404cfc700 10 monclient: get_auth_request con 0x7f13e4036420 auth_method 0 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e0006590 con 0x7f13e0002830 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13f0004a30 con 0x7f13e0002830 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.485+0000 7f13f5ffb700 10 monclient: dump: 2024-09-16T06:57:32.505 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3599748722 0 0) 0x7f13e0006590 con 0x7f13e0002830 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13e000a190 con 0x7f13e0002830 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f13f0007480 con 0x7f13e0002830 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6790/0 conn(0x7f13e0002830 legacy=0x7f13e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.489+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6790/0 conn(0x7f13e0002830 legacy=0x7f13e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f14064ff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f13e0009e50 con 0x7f13e0009170 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966275069 0 0) 0x7f13e0009e50 con 0x7f13e0009170 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 10 monclient(hunting): my global_id is 344114 2024-09-16T06:57:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e0006590 con 0x7f13e0009170 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1708031449 0 0) 0x7f13e0006590 con 0x7f13e0009170 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.497+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e0009e50 con 0x7f13e0009170 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13f0005410 con 0x7f13e0009170 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 10 monclient: dump: 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.509 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 179809076 0 0) 0x7f13e0009e50 con 0x7f13e0009170 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13e000c340 con 0x7f13e0009170 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 2961020011) 0x7f13f00075f0 con 0x7f13e0009170 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6790/0 conn(0x7f13e0009170 legacy=0x7f13e0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f13f5ffb700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6790/0 conn(0x7f13e0009170 legacy=0x7f13e0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.510 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:57:32.511 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.512 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.513 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:32.514 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.515 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.516 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.517 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.518 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:32.519 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:32.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.521 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:32.522 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.189:6812/3098810674 conn(0x7f13e4036420 legacy=0x7f13e40388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 10 monclient: shutdown 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 >> v1:172.21.15.76:6789/0 conn(0x7f13f8025860 legacy=0x7f13f8156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f1405cfe700 1 -- 172.21.15.76:0/1306603224 reap_dead start 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f1405cfe700 1 -- 172.21.15.76:0/1306603224 reap_dead start 2024-09-16T06:57:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 shutdown_connections 2024-09-16T06:57:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 >> 172.21.15.76:0/1306603224 conn(0x7f13f8005b40 msgr2=0x7f13f801a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 shutdown_connections 2024-09-16T06:57:32.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.501+0000 7f14064ff700 1 -- 172.21.15.76:0/1306603224 wait complete. 2024-09-16T06:57:32.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:32.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:57:32.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: get_monmap_and_config 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: monmap: 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:32.649332+0000 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:32.649332+0000 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.644 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:32.645 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 1 Processor -- start 2024-09-16T06:57:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 1 -- start start 2024-09-16T06:57:32.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: init 2024-09-16T06:57:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:57:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-a con 0x7fe72c09ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-c con 0x7fe72c0709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-b con 0x7fe72c070de0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c09f140 con 0x7fe72c09ed60 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c072530 con 0x7fe72c0709d0 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c097f00 con 0x7fe72c070de0 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient(hunting): authenticate will time out at 1999.782518s 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73135d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe72c09ed60 0x7fe72c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59108/0 (socket says 172.21.15.76:59108) 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe731b5e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe72c070de0 0x7fe72c13de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46672/0 (socket says 172.21.15.76:46672) 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73135d700 1 -- 172.21.15.76:0/628099786 learned_addr learned my addr 172.21.15.76:0/628099786 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521707636 0 0) 0x7fe72c097f00 con 0x7fe72c070de0 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): my global_id is 344117 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7180035d0 con 0x7fe72c070de0 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928721068 0 0) 0x7fe72c09f140 con 0x7fe72c09ed60 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient(hunting): my global_id is 334224 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe72c097f00 con 0x7fe72c09ed60 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 93090661 0 0) 0x7fe72c097f00 con 0x7fe72c09ed60 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 >> v1:172.21.15.76:6790/0 conn(0x7fe72c070de0 legacy=0x7fe72c13de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 >> v1:172.21.15.189:6789/0 conn(0x7fe72c0709d0 legacy=0x7fe72c13a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 monclient: found mon.noname-a 2024-09-16T06:57:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c09a320 con 0x7fe72c09ed60 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7280015f0 con 0x7fe72c09ed60 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 5 monclient: authenticate success, global_id 334224 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: dump: 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.651 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe728001910 con 0x7fe72c09ed60 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe73235f700 4 set_mon_vals no callback set 2024-09-16T06:57:32.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 1 -- 172.21.15.76:0/628099786 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe728001ca0 con 0x7fe72c09ed60 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.645+0000 7fe71ffff700 10 monclient: dump: 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.653 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 10 monclient: shutdown 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- 172.21.15.76:0/628099786 >> v1:172.21.15.76:6789/0 conn(0x7fe72c09ed60 legacy=0x7fe72c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- 172.21.15.76:0/628099786 shutdown_connections 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- 172.21.15.76:0/628099786 >> 172.21.15.76:0/628099786 conn(0x7fe72c0a0030 msgr2=0x7fe72c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- 172.21.15.76:0/628099786 shutdown_connections 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- 172.21.15.76:0/628099786 wait complete. 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 10 monclient: monmap: 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:32.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 Processor -- start 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 1 -- start start 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.649+0000 7fe73235f700 10 monclient: init 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-b con 0x7fe72c0709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-a con 0x7fe72c070de0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient(hunting): picked mon.noname-c con 0x7fe72c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c09a540 con 0x7fe72c070de0 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c0951e0 con 0x7fe72c0709d0 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c093390 con 0x7fe72c09ed60 2024-09-16T06:57:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient(hunting): authenticate will time out at 1999.788816s 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe731b5e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fe72c09ed60 0x7fe72c1d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46680/0 (socket says 172.21.15.76:46680) 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe730b5c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe72c070de0 0x7fe72c1cec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59122/0 (socket says 172.21.15.76:59122) 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe731b5e700 1 -- 172.21.15.76:0/3564717154 learned_addr learned my addr 172.21.15.76:0/3564717154 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281579991 0 0) 0x7fe72c093390 con 0x7fe72c09ed60 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): my global_id is 344120 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7100035d0 con 0x7fe72c09ed60 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304240276 0 0) 0x7fe72c09a540 con 0x7fe72c070de0 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): my global_id is 334227 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe72c093390 con 0x7fe72c070de0 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335313305 0 0) 0x7fe72c0951e0 con 0x7fe72c0709d0 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): my global_id is 284680 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe72c09a540 con 0x7fe72c0709d0 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094205841 0 0) 0x7fe7100035d0 con 0x7fe72c09ed60 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe72c0951e0 con 0x7fe72c09ed60 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe724001c50 con 0x7fe72c09ed60 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): dump: 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1665030527 0 0) 0x7fe72c093390 con 0x7fe72c070de0 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7100035d0 con 0x7fe72c070de0 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe7200016a0 con 0x7fe72c070de0 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): dump: 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980072885 0 0) 0x7fe72c09a540 con 0x7fe72c0709d0 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe72c093390 con 0x7fe72c0709d0 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe728001c20 con 0x7fe72c0709d0 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient(hunting): dump: 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2044602880 0 0) 0x7fe7100035d0 con 0x7fe72c070de0 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.76:6790/0 conn(0x7fe72c09ed60 legacy=0x7fe72c1d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6789/0 conn(0x7fe72c0709d0 legacy=0x7fe72c096710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 monclient: found mon.a 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0a27b0 con 0x7fe72c070de0 2024-09-16T06:57:32.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 5 monclient: authenticate success, global_id 334227 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _renew_subs 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c0a2260 con 0x7fe72c070de0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _renew_subs 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c092e80 con 0x7fe72c070de0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe720001c10 con 0x7fe72c070de0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe720001d80 con 0x7fe72c070de0 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71f7fe700 4 set_mon_vals no callback set 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe71dffb700 10 monclient: dump: 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.653+0000 7fe73235f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe70c006590 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fe720003840 con 0x7fe72c070de0 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fe720043bf0 con 0x7fe72c070de0 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261808369 0 0) 0x7fe70c006590 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient(hunting): my global_id is 284683 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7100035d0 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe731b5e700 10 monclient: get_auth_request con 0x7fe710036530 auth_method 0 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 455621794 0 0) 0x7fe7100035d0 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c006590 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe728004be0 con 0x7fe70c002830 2024-09-16T06:57:32.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient: dump: 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1950223918 0 0) 0x7fe70c006590 con 0x7fe70c002830 2024-09-16T06:57:32.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe70c00a190 con 0x7fe70c002830 2024-09-16T06:57:32.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fe728004e80 con 0x7fe70c002830 2024-09-16T06:57:32.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:32.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6789/0 conn(0x7fe70c002830 legacy=0x7fe70c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.657+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6789/0 conn(0x7fe70c002830 legacy=0x7fe70c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:32.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe70c009e10 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768579294 0 0) 0x7fe70c009e10 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient(hunting): my global_id is 284686 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c006590 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1300156465 0 0) 0x7fe70c006590 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c009e10 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fe728006ea0 con 0x7fe70c009150 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient: dump: 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.674 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4074494109 0 0) 0x7fe70c009e10 con 0x7fe70c009150 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe70c00c340 con 0x7fe70c009150 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3973563457) 0x7fe7280056b0 con 0x7fe70c009150 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:32.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6789/0 conn(0x7fe70c009150 legacy=0x7fe70c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe71dffb700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6789/0 conn(0x7fe70c009150 legacy=0x7fe70c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:32.676 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.677 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.678 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:32.680 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:32.681 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.682 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:32.684 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.686 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:32.687 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.189:6812/3098810674 conn(0x7fe710036530 legacy=0x7fe7100389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 10 monclient: shutdown 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 >> v1:172.21.15.76:6789/0 conn(0x7fe72c070de0 legacy=0x7fe72c1cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe731b5e700 1 -- 172.21.15.76:0/3564717154 reap_dead start 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe731b5e700 1 -- 172.21.15.76:0/3564717154 reap_dead start 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 shutdown_connections 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 >> 172.21.15.76:0/3564717154 conn(0x7fe72c0a0030 msgr2=0x7fe72c08aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 shutdown_connections 2024-09-16T06:57:32.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.669+0000 7fe73235f700 1 -- 172.21.15.76:0/3564717154 wait complete. 2024-09-16T06:57:32.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:32.690 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:57:32.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:57:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.809+0000 7f3b371d5700 10 monclient: get_monmap_and_config 2024-09-16T06:57:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.809+0000 7f3b371d5700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.809+0000 7f3b371d5700 10 monclient: monmap: 2024-09-16T06:57:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:32.814968+0000 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:32.814968+0000 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:32.810 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 Processor -- start 2024-09-16T06:57:32.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- start start 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient: init 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-b con 0x7f3b30132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-c con 0x7f3b3006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-a con 0x7f3b3006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3006c280 con 0x7f3b3006cf50 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3006bfa0 con 0x7f3b3006d480 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3009ace0 con 0x7f3b30132330 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient(hunting): authenticate will time out at 1999.948992s 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b361d3700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3b30132330 0x7f3b30132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46692/0 (socket says 172.21.15.76:46692) 2024-09-16T06:57:32.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b361d3700 1 -- 172.21.15.76:0/2522814385 learned_addr learned my addr 172.21.15.76:0/2522814385 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739140190 0 0) 0x7f3b3009ace0 con 0x7f3b30132330 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): my global_id is 344123 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b1c0035d0 con 0x7f3b30132330 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356510200 0 0) 0x7f3b3006c280 con 0x7f3b3006cf50 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): my global_id is 334230 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3009ace0 con 0x7f3b3006cf50 2024-09-16T06:57:32.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080071269 0 0) 0x7f3b3006bfa0 con 0x7f3b3006d480 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient(hunting): my global_id is 284689 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3006c280 con 0x7f3b3006d480 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817512369 0 0) 0x7f3b1c0035d0 con 0x7f3b30132330 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 >> v1:172.21.15.189:6789/0 conn(0x7f3b3006d480 legacy=0x7f3b3013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 >> v1:172.21.15.76:6789/0 conn(0x7f3b3006cf50 legacy=0x7f3b3013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 monclient: found mon.noname-b 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3009ae70 con 0x7f3b30132330 2024-09-16T06:57:32.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 5 monclient: authenticate success, global_id 344123 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b2c001610 con 0x7f3b30132330 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: dump: 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.819 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b2c001830 con 0x7f3b30132330 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 4 set_mon_vals no callback set 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 1 -- 172.21.15.76:0/2522814385 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b2c001a20 con 0x7f3b30132330 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b351d1700 10 monclient: dump: 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 10 monclient: shutdown 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:32.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- 172.21.15.76:0/2522814385 >> v1:172.21.15.76:6790/0 conn(0x7f3b30132330 legacy=0x7f3b30132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- 172.21.15.76:0/2522814385 shutdown_connections 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- 172.21.15.76:0/2522814385 >> 172.21.15.76:0/2522814385 conn(0x7f3b300a41c0 msgr2=0x7f3b300a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- 172.21.15.76:0/2522814385 shutdown_connections 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.813+0000 7f3b371d5700 1 -- 172.21.15.76:0/2522814385 wait complete. 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: build_initial_monmap 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: monmap: 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:32.822 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 1 Processor -- start 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 1 -- start start 2024-09-16T06:57:32.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: init 2024-09-16T06:57:32.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-a con 0x7f3b3006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-c con 0x7f3b3006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient(hunting): picked mon.noname-b con 0x7f3b30132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3009e8b0 con 0x7f3b3006cf50 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b301ca6e0 con 0x7f3b30132330 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300a7050 con 0x7f3b3006d480 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient(hunting): authenticate will time out at 1999.954415s 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b359d2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3b3006d480 0x7f3b301c9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:46696/0 (socket says 172.21.15.76:46696) 2024-09-16T06:57:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b359d2700 1 -- 172.21.15.76:0/4138098667 learned_addr learned my addr 172.21.15.76:0/4138098667 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b361d3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3b3006cf50 0x7f3b301c98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:59136/0 (socket says 172.21.15.76:59136) 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569620210 0 0) 0x7f3b300a7050 con 0x7f3b3006d480 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): my global_id is 344126 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b140035d0 con 0x7f3b3006d480 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430861903 0 0) 0x7f3b3009e8b0 con 0x7f3b3006cf50 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): my global_id is 334233 2024-09-16T06:57:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b300a7050 con 0x7f3b3006cf50 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518649950 0 0) 0x7f3b301ca6e0 con 0x7f3b30132330 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): my global_id is 284692 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3009e8b0 con 0x7f3b30132330 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975166766 0 0) 0x7f3b140035d0 con 0x7f3b3006d480 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b301ca6e0 con 0x7f3b3006d480 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20001990 con 0x7f3b3006d480 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): dump: 2024-09-16T06:57:32.830 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1224507101 0 0) 0x7f3b300a7050 con 0x7f3b3006cf50 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b140035d0 con 0x7f3b3006cf50 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b2c000fe0 con 0x7f3b3006cf50 2024-09-16T06:57:32.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient(hunting): dump: 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2455298631 0 0) 0x7f3b301ca6e0 con 0x7f3b3006d480 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 >> v1:172.21.15.189:6789/0 conn(0x7f3b30132330 legacy=0x7f3b301d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 >> v1:172.21.15.76:6789/0 conn(0x7f3b3006cf50 legacy=0x7f3b301c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 monclient: found mon.c 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300a6b20 con 0x7f3b3006d480 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient: _finish_auth 0 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 5 monclient: authenticate success, global_id 344126 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.817+0000 7f3b371d5700 10 monclient: _renew_subs 2024-09-16T06:57:32.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3012ea90 con 0x7f3b3006d480 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 monclient: _renew_subs 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3012dec0 con 0x7f3b3006d480 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b20001d40 con 0x7f3b3006d480 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b20003b10 con 0x7f3b3006d480 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b349d0700 4 set_mon_vals no callback set 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 10 monclient: dump: 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:32.834 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:32.835 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b18005100 con 0x7f3b3006d480 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3b20003c70 con 0x7f3b3006d480 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3b20043f20 con 0x7f3b3006d480 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.821+0000 7f3b361d3700 10 monclient: get_auth_request con 0x7f3b140364c0 auth_method 0 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.829+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3b20014860 con 0x7f3b3006d480 2024-09-16T06:57:32.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.829+0000 7f3b26ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:32.829+0000 7f3b26ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:33.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.061+0000 7f3b371d5700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:57:33.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.061+0000 7f3b371d5700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:57:33.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.061+0000 7f3b371d5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:33.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.061+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3b18005bc0 con 0x7f3b3006d480 2024-09-16T06:57:33.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.065+0000 7f3b26ffd700 1 -- 172.21.15.76:0/4138098667 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f3b20012cd0 con 0x7f3b3006d480 2024-09-16T06:57:33.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.065+0000 7f3b26ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:57:33.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.065+0000 7f3b26ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:33.064 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=3024309123,config=1097430493,health=427299957,kv=1228748077,logm=1001290270}) 2024-09-16T06:57:33.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=11,logm=49} crc {auth=1466764840,config=1228700967,health=427299957,kv=2218890894,logm=412841333}) 2024-09-16T06:57:33.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=3024309123,config=1097430493,health=427299957,kv=1228748077,logm=1001290270}) 2024-09-16T06:57:33.065 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.065+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=2956265681,config=1097430493,health=4277071928,kv=4108833078,logm=550033426}) 2024-09-16T06:57:33.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130423947}) 2024-09-16T06:57:33.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738161800}) 2024-09-16T06:57:33.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130423947}) 2024-09-16T06:57:33.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.069+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=795252146}) 2024-09-16T06:57:33.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3b140364c0 legacy=0x7f3b14038980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:33.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 10 monclient: shutdown 2024-09-16T06:57:33.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:33.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 >> v1:172.21.15.76:6790/0 conn(0x7f3b3006d480 legacy=0x7f3b301c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:33.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 shutdown_connections 2024-09-16T06:57:33.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 >> 172.21.15.76:0/4138098667 conn(0x7f3b300a41c0 msgr2=0x7f3b3009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:33.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 shutdown_connections 2024-09-16T06:57:33.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:33.069+0000 7f3b371d5700 1 -- 172.21.15.76:0/4138098667 wait complete. 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103227241}) 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233885760}) 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103227241}) 2024-09-16T06:57:33.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769968589}) 2024-09-16T06:57:33.073 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699059129}) 2024-09-16T06:57:33.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738236450}) 2024-09-16T06:57:33.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699059129}) 2024-09-16T06:57:33.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.073+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285328961}) 2024-09-16T06:57:33.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114586515}) 2024-09-16T06:57:33.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197319645}) 2024-09-16T06:57:33.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114586515}) 2024-09-16T06:57:33.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1129128781}) 2024-09-16T06:57:33.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487824576}) 2024-09-16T06:57:33.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944079474}) 2024-09-16T06:57:33.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487824576}) 2024-09-16T06:57:33.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.077+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979138592}) 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256668617}) 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193132408}) 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256668617}) 2024-09-16T06:57:33.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435574023}) 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162743331}) 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688075601}) 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162743331}) 2024-09-16T06:57:33.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294623483}) 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135670435}) 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988015285}) 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2135670435}) 2024-09-16T06:57:33.081 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024307522}) 2024-09-16T06:57:33.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:57:33.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424060474}) 2024-09-16T06:57:33.082 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726127298}) 2024-09-16T06:57:33.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.081+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424060474}) 2024-09-16T06:57:33.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532502871}) 2024-09-16T06:57:33.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.083 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375151041}) 2024-09-16T06:57:33.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87102779}) 2024-09-16T06:57:33.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375151041}) 2024-09-16T06:57:33.084 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3236736293}) 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287580426}) 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122526828}) 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287580426}) 2024-09-16T06:57:33.085 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384511233}) 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264294773}) 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633265115}) 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264294773}) 2024-09-16T06:57:33.086 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.085+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420699710}) 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187993756}) 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582041713}) 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187993756}) 2024-09-16T06:57:33.087 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131586580}) 2024-09-16T06:57:33.088 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293605712}) 2024-09-16T06:57:33.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057768228}) 2024-09-16T06:57:33.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293605712}) 2024-09-16T06:57:33.089 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713177338}) 2024-09-16T06:57:33.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310887057}) 2024-09-16T06:57:33.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634340830}) 2024-09-16T06:57:33.090 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310887057}) 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.089+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302826756}) 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132304603}) 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746057198}) 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132304603}) 2024-09-16T06:57:33.091 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274285713}) 2024-09-16T06:57:33.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.092 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611966236}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830057684}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611966236}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=573164230}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930882874}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637372119}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930882874}) 2024-09-16T06:57:33.093 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612742370}) 2024-09-16T06:57:33.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822013118}) 2024-09-16T06:57:33.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861699075}) 2024-09-16T06:57:33.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822013118}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.093+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084035386}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=500332755}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211172511}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=500332755}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239970006}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251086995}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453146826}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251086995}) 2024-09-16T06:57:33.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2403107730}) 2024-09-16T06:57:33.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520068531}) 2024-09-16T06:57:33.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883586080}) 2024-09-16T06:57:33.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520068531}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2637033755}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343977510}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474695888}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343977510}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.097+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582495502}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320123342}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892684570}) 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320123342}) 2024-09-16T06:57:33.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3828187991}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093345064}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072330318}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093345064}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244565012}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163394983}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992479533}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163394983}) 2024-09-16T06:57:33.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133206901}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444961705}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564785947}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444961705}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.101+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762595490}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045837373}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341229189}) 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045837373}) 2024-09-16T06:57:33.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221270564}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465573344}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=638729555}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465573344}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934093065}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697449502}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1007250609}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697449502}) 2024-09-16T06:57:33.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766032152}) 2024-09-16T06:57:33.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318475081}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161559440}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318475081}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.105+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4016977847}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515074531}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669345356}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515074531}) 2024-09-16T06:57:33.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510794952}) 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379827772}) 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561027288}) 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379827772}) 2024-09-16T06:57:33.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578590632}) 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940471340}) 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279246118}) 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940471340}) 2024-09-16T06:57:33.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2848995968}) 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380855085}) 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953815907}) 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380855085}) 2024-09-16T06:57:33.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.109+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877727245}) 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146933292}) 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901675633}) 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146933292}) 2024-09-16T06:57:33.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878180646}) 2024-09-16T06:57:33.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775547271}) 2024-09-16T06:57:33.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4173989204}) 2024-09-16T06:57:33.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775547271}) 2024-09-16T06:57:33.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050605551}) 2024-09-16T06:57:33.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288685463}) 2024-09-16T06:57:33.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078929821}) 2024-09-16T06:57:33.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288685463}) 2024-09-16T06:57:33.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.113+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574012672}) 2024-09-16T06:57:33.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.114 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405404443}) 2024-09-16T06:57:33.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528930469}) 2024-09-16T06:57:33.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405404443}) 2024-09-16T06:57:33.115 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3219418649}) 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575832593}) 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221775765}) 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575832593}) 2024-09-16T06:57:33.116 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105294982}) 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412970090}) 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692297698}) 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412970090}) 2024-09-16T06:57:33.117 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.117+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3687631211}) 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175090524}) 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602374987}) 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175090524}) 2024-09-16T06:57:33.118 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979324582}) 2024-09-16T06:57:33.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907866631}) 2024-09-16T06:57:33.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451789018}) 2024-09-16T06:57:33.119 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907866631}) 2024-09-16T06:57:33.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818493931}) 2024-09-16T06:57:33.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457973803}) 2024-09-16T06:57:33.120 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957448905}) 2024-09-16T06:57:33.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457973803}) 2024-09-16T06:57:33.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048534536}) 2024-09-16T06:57:33.121 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352200330}) 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941514118}) 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352200330}) 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.121+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422660179}) 2024-09-16T06:57:33.122 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933321660}) 2024-09-16T06:57:33.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302942244}) 2024-09-16T06:57:33.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933321660}) 2024-09-16T06:57:33.123 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4207458447}) 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590593538}) 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387859155}) 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590593538}) 2024-09-16T06:57:33.124 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936013661}) 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2051306983}) 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207388008}) 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2051306983}) 2024-09-16T06:57:33.125 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.125+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874350263}) 2024-09-16T06:57:33.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166826059}) 2024-09-16T06:57:33.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718155148}) 2024-09-16T06:57:33.126 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166826059}) 2024-09-16T06:57:33.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2765987682}) 2024-09-16T06:57:33.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.127 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888926063}) 2024-09-16T06:57:33.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731594742}) 2024-09-16T06:57:33.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888926063}) 2024-09-16T06:57:33.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454370120}) 2024-09-16T06:57:33.128 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885622693}) 2024-09-16T06:57:33.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923832020}) 2024-09-16T06:57:33.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885622693}) 2024-09-16T06:57:33.129 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864837212}) 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403245741}) 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4182240259}) 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403245741}) 2024-09-16T06:57:33.130 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.129+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=889575245}) 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130342974}) 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979905735}) 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130342974}) 2024-09-16T06:57:33.131 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924822812}) 2024-09-16T06:57:33.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640767331}) 2024-09-16T06:57:33.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988003731}) 2024-09-16T06:57:33.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.132 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640767331}) 2024-09-16T06:57:33.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=742843324}) 2024-09-16T06:57:33.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.133 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483602723}) 2024-09-16T06:57:33.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352200470}) 2024-09-16T06:57:33.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483602723}) 2024-09-16T06:57:33.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.133+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3756155341}) 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556010118}) 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861690507}) 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556010118}) 2024-09-16T06:57:33.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779817760}) 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201985369}) 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861944618}) 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201985369}) 2024-09-16T06:57:33.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1514266550}) 2024-09-16T06:57:33.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=534274661}) 2024-09-16T06:57:33.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995934629}) 2024-09-16T06:57:33.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=534274661}) 2024-09-16T06:57:33.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.137+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760663350}) 2024-09-16T06:57:33.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511866264}) 2024-09-16T06:57:33.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839791150}) 2024-09-16T06:57:33.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511866264}) 2024-09-16T06:57:33.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=292871839}) 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659754160}) 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188403955}) 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659754160}) 2024-09-16T06:57:33.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300469153}) 2024-09-16T06:57:33.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057970533}) 2024-09-16T06:57:33.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553431994}) 2024-09-16T06:57:33.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057970533}) 2024-09-16T06:57:33.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.141+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401502806}) 2024-09-16T06:57:33.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566911655}) 2024-09-16T06:57:33.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952004809}) 2024-09-16T06:57:33.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566911655}) 2024-09-16T06:57:33.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155796903}) 2024-09-16T06:57:33.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837723367}) 2024-09-16T06:57:33.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770692049}) 2024-09-16T06:57:33.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837723367}) 2024-09-16T06:57:33.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=739009691}) 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788222960}) 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734943815}) 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788222960}) 2024-09-16T06:57:33.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841428475}) 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055803717}) 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.145+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712605800}) 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055803717}) 2024-09-16T06:57:33.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630204484}) 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2815444482}) 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678816873}) 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2815444482}) 2024-09-16T06:57:33.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744986160}) 2024-09-16T06:57:33.148 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007985482}) 2024-09-16T06:57:33.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103561331}) 2024-09-16T06:57:33.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007985482}) 2024-09-16T06:57:33.149 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2307467279}) 2024-09-16T06:57:33.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719800253}) 2024-09-16T06:57:33.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767812132}) 2024-09-16T06:57:33.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.150 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719800253}) 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.149+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931528725}) 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939440426}) 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291342578}) 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.151 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939440426}) 2024-09-16T06:57:33.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049137150}) 2024-09-16T06:57:33.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156299597}) 2024-09-16T06:57:33.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730232499}) 2024-09-16T06:57:33.152 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156299597}) 2024-09-16T06:57:33.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696617}) 2024-09-16T06:57:33.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125561651}) 2024-09-16T06:57:33.153 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104296309}) 2024-09-16T06:57:33.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125561651}) 2024-09-16T06:57:33.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.153+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068954244}) 2024-09-16T06:57:33.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.154 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934699929}) 2024-09-16T06:57:33.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955758914}) 2024-09-16T06:57:33.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934699929}) 2024-09-16T06:57:33.155 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193861587}) 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312282300}) 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901155582}) 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312282300}) 2024-09-16T06:57:33.156 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425161757}) 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798999546}) 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456190564}) 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798999546}) 2024-09-16T06:57:33.157 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138379183}) 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.157+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256308272}) 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477647408}) 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256308272}) 2024-09-16T06:57:33.158 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2185987007}) 2024-09-16T06:57:33.159 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3880966592}) 2024-09-16T06:57:33.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239362731}) 2024-09-16T06:57:33.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3880966592}) 2024-09-16T06:57:33.160 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532097138}) 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641548529}) 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784282522}) 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641548529}) 2024-09-16T06:57:33.161 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830731701}) 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831360796}) 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106976630}) 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831360796}) 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.161+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=788169374}) 2024-09-16T06:57:33.163 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.164 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683150336}) 2024-09-16T06:57:33.164 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=894738282}) 2024-09-16T06:57:33.164 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.164 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683150336}) 2024-09-16T06:57:33.164 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356342380}) 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885209571}) 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410976301}) 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885209571}) 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950034982}) 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.165 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12957462}) 2024-09-16T06:57:33.166 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52231165}) 2024-09-16T06:57:33.166 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.166 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12957462}) 2024-09-16T06:57:33.166 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.165+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537527708}) 2024-09-16T06:57:33.166 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.167 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493006297}) 2024-09-16T06:57:33.167 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094367031}) 2024-09-16T06:57:33.167 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.167 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493006297}) 2024-09-16T06:57:33.167 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881276823}) 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529199486}) 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128910669}) 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529199486}) 2024-09-16T06:57:33.168 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705437131}) 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177851072}) 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050627231}) 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177851072}) 2024-09-16T06:57:33.169 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697813455}) 2024-09-16T06:57:33.171 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.171 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013044201}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763914622}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.169+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013044201}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078452833}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3242466045}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2915527589}) 2024-09-16T06:57:33.172 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3242466045}) 2024-09-16T06:57:33.173 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337257481}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357680208}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217218809}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357680208}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044357978}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605781012}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918439637}) 2024-09-16T06:57:33.174 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605781012}) 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.173+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4073396386}) 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430751310}) 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081198158}) 2024-09-16T06:57:33.175 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.176 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430751310}) 2024-09-16T06:57:33.176 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215707789}) 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501696885}) 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154232590}) 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501696885}) 2024-09-16T06:57:33.177 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640751675}) 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928092532}) 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280260654}) 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928092532}) 2024-09-16T06:57:33.178 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.177+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519238158}) 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=854356496}) 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458628653}) 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=854356496}) 2024-09-16T06:57:33.179 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2959808744}) 2024-09-16T06:57:33.180 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.180 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837483847}) 2024-09-16T06:57:33.180 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261575639}) 2024-09-16T06:57:33.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837483847}) 2024-09-16T06:57:33.181 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457463464}) 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600904326}) 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948087286}) 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600904326}) 2024-09-16T06:57:33.182 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.181+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=293615397}) 2024-09-16T06:57:33.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.183 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729629126}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474289646}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729629126}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=600080362}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662790009}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180005172}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662790009}) 2024-09-16T06:57:33.184 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985717614}) 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953039287}) 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706018682}) 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953039287}) 2024-09-16T06:57:33.185 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.185+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430948471}) 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337919977}) 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969332055}) 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337919977}) 2024-09-16T06:57:33.186 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541014397}) 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964499659}) 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2662198346}) 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964499659}) 2024-09-16T06:57:33.187 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195699653}) 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698368712}) 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888292993}) 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2698368712}) 2024-09-16T06:57:33.188 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748195180}) 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2790949892}) 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558444959}) 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2790949892}) 2024-09-16T06:57:33.189 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.189+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259310217}) 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796771665}) 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584084316}) 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796771665}) 2024-09-16T06:57:33.190 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082673675}) 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2059420959}) 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165136704}) 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2059420959}) 2024-09-16T06:57:33.191 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3041868338}) 2024-09-16T06:57:33.192 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.192 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229953602}) 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566538415}) 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229953602}) 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3660983263}) 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391601419}) 2024-09-16T06:57:33.193 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473215061}) 2024-09-16T06:57:33.194 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.194 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391601419}) 2024-09-16T06:57:33.194 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.193+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=994684923}) 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382730403}) 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118288010}) 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382730403}) 2024-09-16T06:57:33.195 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580454978}) 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403474182}) 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882394725}) 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403474182}) 2024-09-16T06:57:33.196 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140638464}) 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132475686}) 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489222882}) 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132475686}) 2024-09-16T06:57:33.197 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731127911}) 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207311369}) 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108990387}) 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207311369}) 2024-09-16T06:57:33.198 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.197+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715263487}) 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253564930}) 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446073852}) 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253564930}) 2024-09-16T06:57:33.199 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3887685494}) 2024-09-16T06:57:33.200 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.200 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904192514}) 2024-09-16T06:57:33.200 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2358752773}) 2024-09-16T06:57:33.200 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.200 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904192514}) 2024-09-16T06:57:33.201 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484191644}) 2024-09-16T06:57:33.201 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.201 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635055353}) 2024-09-16T06:57:33.201 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988360614}) 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635055353}) 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.201+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1331676464}) 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484736042}) 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222159733}) 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.202 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484736042}) 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301254964}) 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020040}) 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145924407}) 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.203 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020040}) 2024-09-16T06:57:33.204 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=515225390}) 2024-09-16T06:57:33.204 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.204 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621358403}) 2024-09-16T06:57:33.205 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288737025}) 2024-09-16T06:57:33.205 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.205 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621358403}) 2024-09-16T06:57:33.205 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2459293832}) 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711215341}) 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388297550}) 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711215341}) 2024-09-16T06:57:33.208 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.205+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355737747}) 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2128077278}) 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374217434}) 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2128077278}) 2024-09-16T06:57:33.209 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3249110282}) 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=591560322,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2441972308,mon_config_key=4279594037,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2585511408}) 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=11} crc {logm=4132416086,mdsmap=2826658659,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2691451012,mon_config_key=2387200789,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=1780345972}) 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=7} crc {logm=591560322,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2441972308,mon_config_key=4279594037,monmap=1887442722,osd_metadata=3457585069,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2585511408}) 2024-09-16T06:57:33.210 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.209+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=8} crc {logm=1818076773,mdsmap=1531151339,mgr=4125395841,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2441972308,mon_config_key=4279594037,monmap=1887442722,osd_metadata=304085106,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2687085487}) 2024-09-16T06:57:33.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=3329411923}) 2024-09-16T06:57:33.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=3064509429}) 2024-09-16T06:57:33.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:57:33.215 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=3329411923}) 2024-09-16T06:57:33.216 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:57:33.217+0000 7faaebb2e700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=1315013543}) 2024-09-16T06:57:34.083 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:57:34.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 10 monclient: get_monmap_and_config 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 10 monclient: monmap: 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:34.189619+0000 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:34.189619+0000 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:34.185 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 1 Processor -- start 2024-09-16T06:57:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 1 -- start start 2024-09-16T06:57:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.185+0000 7f3a6a4aa700 10 monclient: init 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-c con 0x7f3a64097470 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-a con 0x7f3a64097850 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-b con 0x7f3a64097c90 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6409d760 con 0x7f3a64097850 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640a0ab0 con 0x7f3a64097470 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6409d110 con 0x7f3a64097c90 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a68ca7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a64097850 0x7f3a64136ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33624/0 (socket says 172.21.15.76:33624) 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a68ca7700 1 -- 172.21.15.76:0/3845769318 learned_addr learned my addr 172.21.15.76:0/3845769318 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient(hunting): authenticate will time out at 2001.324948s 2024-09-16T06:57:34.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719638896 0 0) 0x7f3a6409d760 con 0x7f3a64097850 2024-09-16T06:57:34.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): my global_id is 334236 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a500035d0 con 0x7f3a64097850 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123922215 0 0) 0x7f3a6409d110 con 0x7f3a64097c90 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): my global_id is 344129 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6409d760 con 0x7f3a64097c90 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338515019 0 0) 0x7f3a640a0ab0 con 0x7f3a64097470 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient(hunting): my global_id is 284695 2024-09-16T06:57:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6409d110 con 0x7f3a64097470 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583395585 0 0) 0x7f3a500035d0 con 0x7f3a64097850 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 >> v1:172.21.15.76:6790/0 conn(0x7f3a64097c90 legacy=0x7f3a6413a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 >> v1:172.21.15.189:6789/0 conn(0x7f3a64097470 legacy=0x7f3a6409e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 monclient: found mon.noname-a 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6409d3f0 con 0x7f3a64097850 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a54001610 con 0x7f3a64097850 2024-09-16T06:57:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: dump: 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 5 monclient: authenticate success, global_id 334236 2024-09-16T06:57:34.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a540018f0 con 0x7f3a64097850 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 1 -- 172.21.15.76:0/3845769318 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a54001c80 con 0x7f3a64097850 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 4 set_mon_vals no callback set 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a5bfff700 10 monclient: dump: 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.194 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: shutdown 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/3845769318 >> v1:172.21.15.76:6789/0 conn(0x7f3a64097850 legacy=0x7f3a64136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/3845769318 shutdown_connections 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/3845769318 >> 172.21.15.76:0/3845769318 conn(0x7f3a640a1120 msgr2=0x7f3a640a1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/3845769318 shutdown_connections 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/3845769318 wait complete. 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.189+0000 7f3a6a4aa700 10 monclient: monmap: 2024-09-16T06:57:34.195 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 Processor -- start 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- start start 2024-09-16T06:57:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: init 2024-09-16T06:57:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-a con 0x7f3a64097470 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-b con 0x7f3a64097850 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient(hunting): picked mon.noname-c con 0x7f3a64097c90 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a64071f70 con 0x7f3a64097470 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640a3b00 con 0x7f3a64097850 2024-09-16T06:57:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6409b630 con 0x7f3a64097c90 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient(hunting): authenticate will time out at 2001.329129s 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a694a8700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a64097470 0x7f3a641c8020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33630/0 (socket says 172.21.15.76:33630) 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a694a8700 1 -- 172.21.15.76:0/748069996 learned_addr learned my addr 172.21.15.76:0/748069996 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331136888 0 0) 0x7f3a64071f70 con 0x7f3a64097470 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): my global_id is 334239 2024-09-16T06:57:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a480035d0 con 0x7f3a64097470 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361280585 0 0) 0x7f3a6409b630 con 0x7f3a64097c90 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): my global_id is 344132 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a64071f70 con 0x7f3a64097c90 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088944194 0 0) 0x7f3a640a3b00 con 0x7f3a64097850 2024-09-16T06:57:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): my global_id is 284698 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6409b630 con 0x7f3a64097850 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692011155 0 0) 0x7f3a64071f70 con 0x7f3a64097c90 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a640a3b00 con 0x7f3a64097c90 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a5c001c70 con 0x7f3a64097c90 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:57:34.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4142632649 0 0) 0x7f3a480035d0 con 0x7f3a64097470 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a64071f70 con 0x7f3a64097470 2024-09-16T06:57:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a60001990 con 0x7f3a64097470 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884481060 0 0) 0x7f3a6409b630 con 0x7f3a64097850 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a480035d0 con 0x7f3a64097850 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a540015d0 con 0x7f3a64097850 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient(hunting): dump: 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.209 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1056525012 0 0) 0x7f3a640a3b00 con 0x7f3a64097c90 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 >> v1:172.21.15.189:6789/0 conn(0x7f3a64097850 legacy=0x7f3a641ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 >> v1:172.21.15.76:6789/0 conn(0x7f3a64097470 legacy=0x7f3a641c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 monclient: found mon.c 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6406da60 con 0x7f3a64097c90 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 5 monclient: authenticate success, global_id 344132 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _renew_subs 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6406dd40 con 0x7f3a64097c90 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _renew_subs 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a640a3f50 con 0x7f3a64097c90 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a5c003a20 con 0x7f3a64097c90 2024-09-16T06:57:34.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3a5c003d30 con 0x7f3a64097c90 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a5affd700 4 set_mon_vals no callback set 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a59ffb700 10 monclient: dump: 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.193+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a44005100 con 0x7f3a64097c90 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.197+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3a5c00e7a0 con 0x7f3a64097c90 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.197+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3a5c042d30 con 0x7f3a64097c90 2024-09-16T06:57:34.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.197+0000 7f3a694a8700 10 monclient: get_auth_request con 0x7f3a4803a950 auth_method 0 2024-09-16T06:57:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.201+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3a5c0135d0 con 0x7f3a64097c90 2024-09-16T06:57:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.201+0000 7f3a59ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.201+0000 7f3a59ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:34.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a6a4aa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:57:34.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a6a4aa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:34.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a6a4aa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a44005930 con 0x7f3a64097c90 2024-09-16T06:57:34.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a59ffb700 1 -- 172.21.15.76:0/748069996 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 127067910) 0x7f3a5c011c70 con 0x7f3a64097c90 2024-09-16T06:57:34.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a59ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:57:34.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.437+0000 7f3a59ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:34.437 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:57:34.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3a4803a950 legacy=0x7f3a4803ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 10 monclient: shutdown 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 >> v1:172.21.15.76:6790/0 conn(0x7f3a64097c90 legacy=0x7f3a641d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 shutdown_connections 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 >> 172.21.15.76:0/748069996 conn(0x7f3a640a1120 msgr2=0x7f3a6409aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 shutdown_connections 2024-09-16T06:57:34.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.441+0000 7f3a6a4aa700 1 -- 172.21.15.76:0/748069996 wait complete. 2024-09-16T06:57:34.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:57:34.450 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:57:34.451 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:57:34.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:57:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: get_monmap_and_config 2024-09-16T06:57:34.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: monmap: 2024-09-16T06:57:34.615 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:34.568552+0000 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:34.568552+0000 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 Processor -- start 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- start start 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: init 2024-09-16T06:57:34.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-b con 0x7f2fc8070ac0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-c con 0x7f2fc8070ea0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-a con 0x7f2fc813b980 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8072610 con 0x7f2fc813b980 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc809f560 con 0x7f2fc8070ea0 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8072330 con 0x7f2fc8070ac0 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient(hunting): authenticate will time out at 2001.701749s 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fce15e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f2fc8070ac0 0x7f2fc8097bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60324/0 (socket says 172.21.15.76:60324) 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fce95f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2fc813b980 0x7f2fc813dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33644/0 (socket says 172.21.15.76:33644) 2024-09-16T06:57:34.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fce15e700 1 -- 172.21.15.76:0/2707412291 learned_addr learned my addr 172.21.15.76:0/2707412291 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934166209 0 0) 0x7f2fc8072610 con 0x7f2fc813b980 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): my global_id is 334242 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb40035d0 con 0x7f2fc813b980 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633309234 0 0) 0x7f2fc8072330 con 0x7f2fc8070ac0 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): my global_id is 344135 2024-09-16T06:57:34.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc8072610 con 0x7f2fc8070ac0 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430326711 0 0) 0x7f2fc809f560 con 0x7f2fc8070ea0 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient(hunting): my global_id is 284701 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc8072330 con 0x7f2fc8070ea0 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438108219 0 0) 0x7f2fb40035d0 con 0x7f2fc813b980 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 >> v1:172.21.15.76:6790/0 conn(0x7f2fc8070ac0 legacy=0x7f2fc8097bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 >> v1:172.21.15.189:6789/0 conn(0x7f2fc8070ea0 legacy=0x7f2fc80712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 monclient: found mon.noname-a 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80974f0 con 0x7f2fc813b980 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 5 monclient: authenticate success, global_id 334242 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fc0001990 con 0x7f2fc813b980 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: dump: 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.620 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc0001c70 con 0x7f2fc813b980 2024-09-16T06:57:34.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 4 set_mon_vals no callback set 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 1 -- 172.21.15.76:0/2707412291 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fc0002000 con 0x7f2fc813b980 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcd15c700 10 monclient: dump: 2024-09-16T06:57:34.622 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 10 monclient: shutdown 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- 172.21.15.76:0/2707412291 >> v1:172.21.15.76:6789/0 conn(0x7f2fc813b980 legacy=0x7f2fc813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- 172.21.15.76:0/2707412291 shutdown_connections 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- 172.21.15.76:0/2707412291 >> 172.21.15.76:0/2707412291 conn(0x7f2fc80a0030 msgr2=0x7f2fc80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- 172.21.15.76:0/2707412291 shutdown_connections 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.565+0000 7f2fcf160700 1 -- 172.21.15.76:0/2707412291 wait complete. 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: monmap: 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.624 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 Processor -- start 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- start start 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: init 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-a con 0x7f2fc8070ac0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-b con 0x7f2fc8070ea0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient(hunting): picked mon.noname-c con 0x7f2fc813b980 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc81394d0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8098c00 con 0x7f2fc8070ea0 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc809a540 con 0x7f2fc813b980 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient(hunting): authenticate will time out at 2001.705353s 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fce15e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2fc8070ac0 0x7f2fc8136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33656/0 (socket says 172.21.15.76:33656) 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fce15e700 1 -- 172.21.15.76:0/1602640218 learned_addr learned my addr 172.21.15.76:0/1602640218 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205864657 0 0) 0x7f2fc81394d0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): my global_id is 334245 2024-09-16T06:57:34.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac0035d0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236745392 0 0) 0x7f2fc809a540 con 0x7f2fc813b980 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): my global_id is 344138 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc81394d0 con 0x7f2fc813b980 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362783976 0 0) 0x7f2fac0035d0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc809a540 con 0x7f2fc8070ac0 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fc40018b0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:34.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): dump: 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.628 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434054445 0 0) 0x7f2fc8098c00 con 0x7f2fc8070ea0 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): my global_id is 284704 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac0035d0 con 0x7f2fc8070ea0 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81347048 0 0) 0x7f2fc81394d0 con 0x7f2fc813b980 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc8098c00 con 0x7f2fc813b980 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fc0001950 con 0x7f2fc813b980 2024-09-16T06:57:34.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient(hunting): dump: 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.630 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.631 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.631 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.631 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2159115154 0 0) 0x7f2fc809a540 con 0x7f2fc8070ac0 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6790/0 conn(0x7f2fc813b980 legacy=0x7f2fc81d2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.189:6789/0 conn(0x7f2fc8070ea0 legacy=0x7f2fc8137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 monclient: found mon.a 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8137770 con 0x7f2fc8070ac0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 5 monclient: authenticate success, global_id 334245 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _renew_subs 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8135330 con 0x7f2fc8070ac0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _renew_subs 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc8139d70 con 0x7f2fc8070ac0 2024-09-16T06:57:34.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc4004890 con 0x7f2fc8070ac0 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fc4004ba0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fbeffd700 10 monclient: dump: 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.634 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.569+0000 7f2fcc95b700 4 set_mon_vals no callback set 2024-09-16T06:57:34.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fcf160700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2fb0006590 con 0x7f2fb0002830 2024-09-16T06:57:34.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f2fc400f610 con 0x7f2fc8070ac0 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f2fc4043ce0 con 0x7f2fc8070ac0 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357878208 0 0) 0x7f2fb0006590 con 0x7f2fb0002830 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient(hunting): my global_id is 334248 2024-09-16T06:57:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc809a540 con 0x7f2fb0002830 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fce95f700 10 monclient: get_auth_request con 0x7f2fac036450 auth_method 0 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 981147707 0 0) 0x7f2fc809a540 con 0x7f2fb0002830 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb0006590 con 0x7f2fb0002830 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fb8001780 con 0x7f2fb0002830 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient: dump: 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.642 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1472182271 0 0) 0x7f2fb0006590 con 0x7f2fb0002830 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fb000a190 con 0x7f2fb0002830 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f2fb8001bd0 con 0x7f2fb0002830 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6789/0 conn(0x7f2fb0002830 legacy=0x7f2fb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.573+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6789/0 conn(0x7f2fb0002830 legacy=0x7f2fb0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2fb0009e50 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108026090 0 0) 0x7f2fb0009e50 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient(hunting): my global_id is 334251 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0006590 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3559381754 0 0) 0x7f2fb0006590 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb0009e50 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f2fb8003a30 con 0x7f2fb0009170 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient: dump: 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3048410539 0 0) 0x7f2fb0009e50 con 0x7f2fb0009170 2024-09-16T06:57:34.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2fb000c340 con 0x7f2fb0009170 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3627913148) 0x7f2fb8003cc0 con 0x7f2fb0009170 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6789/0 conn(0x7f2fb0009170 legacy=0x7f2fb0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fbeffd700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6789/0 conn(0x7f2fb0009170 legacy=0x7f2fb0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:57:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:34.660 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.189:6812/3098810674 conn(0x7f2fac036450 legacy=0x7f2fac038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 10 monclient: shutdown 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 >> v1:172.21.15.76:6789/0 conn(0x7f2fc8070ac0 legacy=0x7f2fc8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fce95f700 1 -- 172.21.15.76:0/1602640218 reap_dead start 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fce95f700 1 -- 172.21.15.76:0/1602640218 reap_dead start 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 shutdown_connections 2024-09-16T06:57:34.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 >> 172.21.15.76:0/1602640218 conn(0x7f2fc80a0030 msgr2=0x7f2fc809a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 shutdown_connections 2024-09-16T06:57:34.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.585+0000 7f2fcf160700 1 -- 172.21.15.76:0/1602640218 wait complete. 2024-09-16T06:57:34.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:34.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:57:34.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:57:34.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.781+0000 7faa45e07700 10 monclient: get_monmap_and_config 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.781+0000 7faa45e07700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.781+0000 7faa45e07700 10 monclient: monmap: 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:34.787140+0000 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:34.787140+0000 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.782 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:34.783 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:34.783 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 Processor -- start 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- start start 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient: init 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-b con 0x7faa40132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-c con 0x7faa4006d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-a con 0x7faa4006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4006c280 con 0x7faa4006cf50 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4006bfa0 con 0x7faa4006d480 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4009ace0 con 0x7faa40132330 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient(hunting): authenticate will time out at 2001.920980s 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45606700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faa4006cf50 0x7faa4013da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33688/0 (socket says 172.21.15.76:33688) 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa44e05700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7faa40132330 0x7faa40132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60352/0 (socket says 172.21.15.76:60352) 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45606700 1 -- 172.21.15.76:0/242808405 learned_addr learned my addr 172.21.15.76:0/242808405 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367636686 0 0) 0x7faa4009ace0 con 0x7faa40132330 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): my global_id is 344141 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa280035d0 con 0x7faa40132330 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771959821 0 0) 0x7faa4006c280 con 0x7faa4006cf50 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): my global_id is 334254 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4009ace0 con 0x7faa4006cf50 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139380885 0 0) 0x7faa4006bfa0 con 0x7faa4006d480 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient(hunting): my global_id is 284707 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa4006c280 con 0x7faa4006d480 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658833763 0 0) 0x7faa280035d0 con 0x7faa40132330 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 >> v1:172.21.15.189:6789/0 conn(0x7faa4006d480 legacy=0x7faa4013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 >> v1:172.21.15.76:6789/0 conn(0x7faa4006cf50 legacy=0x7faa4013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4009ae70 con 0x7faa40132330 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa30001610 con 0x7faa40132330 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: dump: 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 5 monclient: authenticate success, global_id 344141 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa300018f0 con 0x7faa40132330 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 1 -- 172.21.15.76:0/242808405 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa30001ae0 con 0x7faa40132330 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 4 set_mon_vals no callback set 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa3f7fe700 10 monclient: dump: 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 10 monclient: shutdown 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- 172.21.15.76:0/242808405 >> v1:172.21.15.76:6790/0 conn(0x7faa40132330 legacy=0x7faa40132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- 172.21.15.76:0/242808405 shutdown_connections 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- 172.21.15.76:0/242808405 >> 172.21.15.76:0/242808405 conn(0x7faa400a41c0 msgr2=0x7faa400a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- 172.21.15.76:0/242808405 shutdown_connections 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.785+0000 7faa45e07700 1 -- 172.21.15.76:0/242808405 wait complete. 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: monmap: 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 Processor -- start 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- start start 2024-09-16T06:57:34.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: init 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-a con 0x7faa4006cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-c con 0x7faa4006d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient(hunting): picked mon.noname-b con 0x7faa40132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40131770 con 0x7faa4006cf50 2024-09-16T06:57:34.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40136190 con 0x7faa40132330 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40131260 con 0x7faa4006d480 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient(hunting): authenticate will time out at 2001.925747s 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa44e05700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faa4006cf50 0x7faa40135370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33696/0 (socket says 172.21.15.76:33696) 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa44e05700 1 -- 172.21.15.76:0/3345898035 learned_addr learned my addr 172.21.15.76:0/3345898035 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3ffff700 1 --1- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa4006d480 0x7faa40135a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60356/0 (socket says 172.21.15.76:60356) 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430615307 0 0) 0x7faa40131260 con 0x7faa4006d480 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): my global_id is 344144 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa2c0035d0 con 0x7faa4006d480 2024-09-16T06:57:34.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003227304 0 0) 0x7faa40131770 con 0x7faa4006cf50 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): my global_id is 334257 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40131260 con 0x7faa4006cf50 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199187450 0 0) 0x7faa40136190 con 0x7faa40132330 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): my global_id is 284710 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40131770 con 0x7faa40132330 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484722526 0 0) 0x7faa2c0035d0 con 0x7faa4006d480 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa40136190 con 0x7faa4006d480 2024-09-16T06:57:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa340018d0 con 0x7faa4006d480 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.802 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41205656 0 0) 0x7faa40131260 con 0x7faa4006cf50 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa2c0035d0 con 0x7faa4006cf50 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa30004890 con 0x7faa4006cf50 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.803 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441796341 0 0) 0x7faa40131770 con 0x7faa40132330 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa40131260 con 0x7faa40132330 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa38001b70 con 0x7faa40132330 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.804 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1354313221 0 0) 0x7faa40136190 con 0x7faa4006d480 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.189:6789/0 conn(0x7faa40132330 legacy=0x7faa401d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6789/0 conn(0x7faa4006cf50 legacy=0x7faa40135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 monclient: found mon.c 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4009e8b0 con 0x7faa4006d480 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 5 monclient: authenticate success, global_id 344144 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _renew_subs 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa40136f70 con 0x7faa4006d480 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _renew_subs 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:57:34.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401392d0 con 0x7faa4006d480 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa34001c60 con 0x7faa4006d480 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa34003b10 con 0x7faa4006d480 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3effd700 4 set_mon_vals no callback set 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.789+0000 7faa3d7fa700 10 monclient: dump: 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.807 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa45e07700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa20006590 con 0x7faa20002830 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7faa34003c70 con 0x7faa4006d480 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7faa34043d80 con 0x7faa4006d480 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586874984 0 0) 0x7faa20006590 con 0x7faa20002830 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient(hunting): my global_id is 344147 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40136190 con 0x7faa20002830 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa45606700 10 monclient: get_auth_request con 0x7faa2c036500 auth_method 0 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1256899753 0 0) 0x7faa40136190 con 0x7faa20002830 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa20006590 con 0x7faa20002830 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa30005030 con 0x7faa20002830 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient: dump: 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1112262758 0 0) 0x7faa20006590 con 0x7faa20002830 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa2000a190 con 0x7faa20002830 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7faa300072b0 con 0x7faa20002830 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa20002830 legacy=0x7faa20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.793+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa20002830 legacy=0x7faa20004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa20009e30 con 0x7faa20009150 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320227351 0 0) 0x7faa20009e30 con 0x7faa20009150 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient(hunting): my global_id is 344150 2024-09-16T06:57:34.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20006590 con 0x7faa20009150 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 136168065 0 0) 0x7faa20006590 con 0x7faa20009150 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa20009e30 con 0x7faa20009150 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7faa30005660 con 0x7faa20009150 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient: dump: 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.813 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3389626332 0 0) 0x7faa20009e30 con 0x7faa20009150 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa2000c340 con 0x7faa20009150 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 2012182519) 0x7faa300069a0 con 0x7faa20009150 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa20009150 legacy=0x7faa200095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa3d7fa700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa20009150 legacy=0x7faa200095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.814 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:34.825 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:34.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.189:6812/3098810674 conn(0x7faa2c036500 legacy=0x7faa2c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 10 monclient: shutdown 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 >> v1:172.21.15.76:6790/0 conn(0x7faa4006d480 legacy=0x7faa40135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45606700 1 -- 172.21.15.76:0/3345898035 reap_dead start 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 shutdown_connections 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 >> 172.21.15.76:0/3345898035 conn(0x7faa400a41c0 msgr2=0x7faa400a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 shutdown_connections 2024-09-16T06:57:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.801+0000 7faa45e07700 1 -- 172.21.15.76:0/3345898035 wait complete. 2024-09-16T06:57:34.827 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:34.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:57:34.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 10 monclient: get_monmap_and_config 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 10 monclient: monmap: 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:34.954825+0000 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:34.954825+0000 2024-09-16T06:57:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:34.950 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 1 Processor -- start 2024-09-16T06:57:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 1 -- start start 2024-09-16T06:57:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.949+0000 7fefa1c6f700 10 monclient: init 2024-09-16T06:57:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-a con 0x7fef9401f4c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-b con 0x7fef94025410 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-c con 0x7fef94025860 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9401f8a0 con 0x7fef9401f4c0 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef940233f0 con 0x7fef94025860 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef94020050 con 0x7fef94025410 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient(hunting): authenticate will time out at 2002.088220s 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fef94025410 0x7fef940bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:60380/0 (socket says 172.21.15.76:60380) 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9bfff700 1 -- 172.21.15.76:0/162981743 learned_addr learned my addr 172.21.15.76:0/162981743 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397122002 0 0) 0x7fef9401f8a0 con 0x7fef9401f4c0 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): my global_id is 334260 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef8c0035d0 con 0x7fef9401f4c0 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961028766 0 0) 0x7fef94020050 con 0x7fef94025410 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): my global_id is 344153 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9401f8a0 con 0x7fef94025410 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17193258 0 0) 0x7fef940233f0 con 0x7fef94025860 2024-09-16T06:57:34.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient(hunting): my global_id is 284713 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94020050 con 0x7fef94025860 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3915764519 0 0) 0x7fef8c0035d0 con 0x7fef9401f4c0 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 >> v1:172.21.15.76:6790/0 conn(0x7fef94025410 legacy=0x7fef940bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 >> v1:172.21.15.189:6789/0 conn(0x7fef94025860 legacy=0x7fef940c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94021a90 con 0x7fef9401f4c0 2024-09-16T06:57:34.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 5 monclient: authenticate success, global_id 334260 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef88001610 con 0x7fef9401f4c0 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: dump: 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.957 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef88001880 con 0x7fef9401f4c0 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient: get_monmap_and_config success 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 4 set_mon_vals no callback set 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 1 -- 172.21.15.76:0/162981743 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef88001c10 con 0x7fef9401f4c0 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fef9b7fe700 10 monclient: dump: 2024-09-16T06:57:34.958 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 10 monclient: shutdown 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- 172.21.15.76:0/162981743 >> v1:172.21.15.76:6789/0 conn(0x7fef9401f4c0 legacy=0x7fef94024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- 172.21.15.76:0/162981743 shutdown_connections 2024-09-16T06:57:34.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- 172.21.15.76:0/162981743 >> 172.21.15.76:0/162981743 conn(0x7fef94005b40 msgr2=0x7fef94005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- 172.21.15.76:0/162981743 shutdown_connections 2024-09-16T06:57:34.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.953+0000 7fefa1c6f700 1 -- 172.21.15.76:0/162981743 wait complete. 2024-09-16T06:57:34.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: build_initial_monmap 2024-09-16T06:57:34.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: monmap: 2024-09-16T06:57:34.964 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:34.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 Processor -- start 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- start start 2024-09-16T06:57:34.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: init 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-b con 0x7fef9401f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-a con 0x7fef94025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient(hunting): picked mon.noname-c con 0x7fef94025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef94028de0 con 0x7fef94025410 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef9bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fef94025410 0x7fef94153000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33728/0 (socket says 172.21.15.76:33728) 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef9bfff700 1 -- 172.21.15.76:0/1879842511 learned_addr learned my addr 172.21.15.76:0/1879842511 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef941483e0 con 0x7fef9401f4c0 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef940204a0 con 0x7fef94025860 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient(hunting): authenticate will time out at 2002.094863s 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273407982 0 0) 0x7fef94028de0 con 0x7fef94025410 2024-09-16T06:57:34.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): my global_id is 334263 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef800035d0 con 0x7fef94025410 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713545535 0 0) 0x7fef940204a0 con 0x7fef94025860 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): my global_id is 344156 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94028de0 con 0x7fef94025860 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588848194 0 0) 0x7fef800035d0 con 0x7fef94025410 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef940204a0 con 0x7fef94025410 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef90001850 con 0x7fef94025410 2024-09-16T06:57:34.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): dump: 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:57:34.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813444859 0 0) 0x7fef941483e0 con 0x7fef9401f4c0 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient(hunting): my global_id is 284716 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef800035d0 con 0x7fef9401f4c0 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1199354541 0 0) 0x7fef940204a0 con 0x7fef94025410 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.76:6790/0 conn(0x7fef94025860 legacy=0x7fef94156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6789/0 conn(0x7fef9401f4c0 legacy=0x7fef94147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient: _finish_hunting 0 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 monclient: found mon.a 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9412cb50 con 0x7fef94025410 2024-09-16T06:57:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient: _finish_auth 0 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 10 monclient: _check_auth_tickets 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fef997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 5 monclient: authenticate success, global_id 334263 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _renew_subs 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9412f630 con 0x7fef94025410 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _renew_subs 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.957+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9412e0e0 con 0x7fef94025410 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef90004940 con 0x7fef94025410 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef90004c50 con 0x7fef94025410 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef9affd700 4 set_mon_vals no callback set 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: dump: 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fef90004db0 con 0x7fef94025410 2024-09-16T06:57:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fefa1c6f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef7c006590 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fef90043f20 con 0x7fef94025410 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fefa146e700 10 monclient: get_auth_request con 0x7fef80036340 auth_method 0 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490472462 0 0) 0x7fef7c006590 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient(hunting): my global_id is 284719 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef940204a0 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1312722952 0 0) 0x7fef940204a0 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef7c006590 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef88001430 con 0x7fef7c002830 2024-09-16T06:57:34.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: dump: 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1177975442 0 0) 0x7fef7c006590 con 0x7fef7c002830 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef7c00a190 con 0x7fef7c002830 2024-09-16T06:57:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fef88001680 con 0x7fef7c002830 2024-09-16T06:57:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:57:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:57:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6789/0 conn(0x7fef7c002830 legacy=0x7fef7c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.961+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6789/0 conn(0x7fef7c002830 legacy=0x7fef7c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fefa1c6f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fef7c009e50 con 0x7fef7c009170 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638057009 0 0) 0x7fef7c009e50 con 0x7fef7c009170 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient(hunting): _negotiate 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient(hunting): my global_id is 284722 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef7c006590 con 0x7fef7c009170 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2571763549 0 0) 0x7fef7c006590 con 0x7fef7c009170 2024-09-16T06:57:34.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef7c009e50 con 0x7fef7c009170 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fef88003a30 con 0x7fef7c009170 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.969+0000 7fef997fa700 10 monclient: dump: 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:57:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3477954800 0 0) 0x7fef7c009e50 con 0x7fef7c009170 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fef7c00c340 con 0x7fef7c009170 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 2045563246) 0x7fef88004fe0 con 0x7fef7c009170 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6789/0 conn(0x7fef7c009170 legacy=0x7fef7c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fef997fa700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6789/0 conn(0x7fef7c009170 legacy=0x7fef7c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 170, 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:57:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:57:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:57:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:57:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:57:34.988 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.189:6812/3098810674 conn(0x7fef80036340 legacy=0x7fef80038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 10 monclient: shutdown 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 >> v1:172.21.15.76:6789/0 conn(0x7fef94025410 legacy=0x7fef94153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa146e700 1 -- 172.21.15.76:0/1879842511 reap_dead start 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 shutdown_connections 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 >> 172.21.15.76:0/1879842511 conn(0x7fef94005b40 msgr2=0x7fef9401a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 shutdown_connections 2024-09-16T06:57:34.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:34.973+0000 7fefa1c6f700 1 -- 172.21.15.76:0/1879842511 wait complete. 2024-09-16T06:57:34.991 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:57:34.991 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:57:34.991 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T06:57:34.991 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:57:34.991 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:57:34.991 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:57:34.991 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:57:35.046 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:57:35.046 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T06:57:35.046 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T06:57:35.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:57:53.497 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=110 2024-09-16T06:57:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T06:57:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:57:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:+ export CEPH_ARGS= 2024-09-16T06:57:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:+ CEPH_ARGS= 2024-09-16T06:57:53.498 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth del client.bar 2024-09-16T06:57:53.600 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.594+0000 7efeed798700 10 monclient: get_monmap_and_config 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.594+0000 7efeed798700 10 monclient: build_initial_monmap 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.594+0000 7efeed798700 10 monclient: monmap: 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:57:53.600281+0000 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:57:53.600281+0000 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:53.601 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:57:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 1 Processor -- start 2024-09-16T06:57:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 1 -- start start 2024-09-16T06:57:53.602 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient: init 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient(hunting): picked mon.noname-b con 0x7efee8135780 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient(hunting): picked mon.noname-c con 0x7efee8074340 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient(hunting): picked mon.noname-a con 0x7efee813bd00 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee809a7b0 con 0x7efee813bd00 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a5080 con 0x7efee8074340 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80985c0 con 0x7efee8135780 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:53.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efeed798700 10 monclient(hunting): authenticate will time out at 2026.114586s 2024-09-16T06:57:53.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efee6ffd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efee8135b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:53.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efee67fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efee8074340 0x7efee813a5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:41358/0 (socket says 172.21.15.189:41358) 2024-09-16T06:57:53.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.598+0000 7efee67fc700 1 -- 172.21.15.189:0/2461782312 learned_addr learned my addr 172.21.15.189:0/2461782312 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:57:53.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:53.802+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efee8135b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:54.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:54.206+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efee8135b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:55.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:55.010+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efee8135b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:55.047 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T06:57:55.048 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T06:57:55.048 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:57:55.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: get_monmap_and_config 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: build_initial_monmap 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: monmap: 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:57:55.169166+0000 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:57:55.169166+0000 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:57:55.164 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 1 Processor -- start 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 1 -- start start 2024-09-16T06:57:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: init 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: _reopen_session rank -1 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient(hunting): picked mon.noname-c con 0x7fa4700973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient(hunting): picked mon.noname-b con 0x7fa470097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient(hunting): picked mon.noname-a con 0x7fa4701400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470099350 con 0x7fa4701400d0 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47009f690 con 0x7fa4700973c0 2024-09-16T06:57:55.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470099bf0 con 0x7fa470097eb0 2024-09-16T06:57:55.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:55.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa474889700 10 monclient(hunting): authenticate will time out at 2022.302457s 2024-09-16T06:57:55.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa467fff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa47013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:55.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa46f59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa4701400d0 0x7fa4701424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51306/0 (socket says 172.21.15.76:51306) 2024-09-16T06:57:55.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.164+0000 7fa46f59e700 1 -- 172.21.15.76:0/2372123849 learned_addr learned my addr 172.21.15.76:0/2372123849 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:57:55.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.368+0000 7fa467fff700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa47013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:55.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:55.772+0000 7fa467fff700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa47013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:56.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:56.576+0000 7fa467fff700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa47013e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): tick 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 monclient(hunting): continuing hunt 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efee8135b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efee8074340 legacy=0x7efee813a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6789/0 conn(0x7efee813bd00 legacy=0x7efee813e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-c con 0x7efed4003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:56.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-b con 0x7efee8135780 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-a con 0x7efee813bd00 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80982e0 con 0x7efee813bd00 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80985c0 con 0x7efed4003740 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4010450 con 0x7efee8135780 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee67fc700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:56.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.598+0000 7efee4ff9700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:56.804 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:56.802+0000 7efee67fc700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:57.209 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:57.206+0000 7efee67fc700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:58.013 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.010+0000 7efee67fc700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:58.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.068+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 legacy=0x7fa4700977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T06:57:58.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.068+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 legacy=0x7fa4700977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:57:58.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.068+0000 7fa46ed9d700 1 --1- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 0x7fa4700977a0 :-1 s=OPENED pgs=1427 cs=1 l=1).handle_message read tag failed 2024-09-16T06:57:58.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.068+0000 7fa46ed9d700 1 --1- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 0x7fa4700977a0 :-1 s=OPENED pgs=1427 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:57:58.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.068+0000 7fa46e59c700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T06:57:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.066+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 legacy=0x7efed4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:57:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.066+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 legacy=0x7efed4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:57:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.066+0000 7efee6ffd700 1 --1- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 0x7efed4005b90 :-1 s=OPENED pgs=1432 cs=1 l=1).handle_message read tag failed 2024-09-16T06:57:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.066+0000 7efee6ffd700 1 --1- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 0x7efed4005b90 :-1 s=OPENED pgs=1432 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:57:58.070 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:58.066+0000 7efee5ffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-09-16T06:57:58.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): tick 2024-09-16T06:57:58.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 monclient(hunting): continuing hunt 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa47013e9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 legacy=0x7fa4700977a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6789/0 conn(0x7fa4701400d0 legacy=0x7fa4701424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): picked mon.noname-b con 0x7fa470097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): picked mon.noname-c con 0x7fa4700973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): picked mon.noname-a con 0x7fa4701400d0 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470098360 con 0x7fa4701400d0 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa4500056f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:58.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470099bf0 con 0x7fa4700973c0 2024-09-16T06:57:58.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 1 -- 172.21.15.76:0/2372123849 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45000fe40 con 0x7fa470097eb0 2024-09-16T06:57:58.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.164+0000 7fa46d59a700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:58.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.368+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa4500056f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:58.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:58.772+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa4500056f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:59.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:57:59.576+0000 7fa46ed9d700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa4500056f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:59.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): tick 2024-09-16T06:57:59.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:57:59.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 monclient(hunting): continuing hunt 2024-09-16T06:57:59.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 legacy=0x7efed4005b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6789/0 conn(0x7efee813bd00 legacy=0x7efed400cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-b con 0x7efee8135780 addr v1:172.21.15.76:6790/0 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-c con 0x7efed4003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): picked mon.noname-a con 0x7efed4010d80 addr v1:172.21.15.76:6789/0 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4011660 con 0x7efed4010d80 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4010450 con 0x7efed4003740 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed4011390 con 0x7efee8135780 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee4ff9700 10 monclient(hunting): _renew_subs 2024-09-16T06:57:59.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.602+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4010610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:57:59.809 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:57:59.806+0000 7efee6ffd700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4010610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498184515 0 0) 0x7fa470098360 con 0x7fa4701400d0 2024-09-16T06:58:00.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 10 monclient(hunting): my global_id is 334266 2024-09-16T06:58:00.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.076+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c0035d0 con 0x7fa4701400d0 2024-09-16T06:58:00.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685148882 0 0) 0x7efed4011660 con 0x7efed4010d80 2024-09-16T06:58:00.077 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 10 monclient(hunting): my global_id is 334269 2024-09-16T06:58:00.078 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.070+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed00035d0 con 0x7efed4010d80 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096069708 0 0) 0x7fa45c0035d0 con 0x7fa4701400d0 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6790/0 conn(0x7fa470097eb0 legacy=0x7fa4500056f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 legacy=0x7fa450008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 monclient: found mon.noname-a 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450011050 con 0x7fa4701400d0 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa474889700 5 monclient: authenticate success, global_id 334266 2024-09-16T06:58:00.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa474889700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa468001910 con 0x7fa4701400d0 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa46e59c700 10 monclient: dump: 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.080 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.081 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.080+0000 7fa474889700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631159862 0 0) 0x7efed00035d0 con 0x7efed4010d80 2024-09-16T06:58:00.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6790/0 conn(0x7efee8135780 legacy=0x7efed4010610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:00.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.189:6789/0 conn(0x7efed4003740 legacy=0x7efed4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.081 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 monclient: found mon.noname-a 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed400cbe0 con 0x7efed4010d80 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efee0001890 con 0x7efed4010d80 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efee5ffb700 10 monclient: dump: 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.082 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efeed798700 5 monclient: authenticate success, global_id 334269 2024-09-16T06:58:00.083 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.074+0000 7efeed798700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa468001b40 con 0x7fa4701400d0 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 1 -- 172.21.15.76:0/2372123849 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa468001ed0 con 0x7fa4701400d0 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 4 set_mon_vals no callback set 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa46e59c700 10 monclient: dump: 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efee0001ae0 con 0x7efed4010d80 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 1 -- 172.21.15.189:0/2461782312 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efee0001eb0 con 0x7efed4010d80 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efeed798700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efeed798700 4 set_mon_vals no callback set 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.082+0000 7efee5ffb700 10 monclient: dump: 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.089 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.090 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.090 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 10 monclient: shutdown 2024-09-16T06:58:00.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- 172.21.15.76:0/2372123849 >> v1:172.21.15.76:6789/0 conn(0x7fa4701400d0 legacy=0x7fa45000c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- 172.21.15.76:0/2372123849 shutdown_connections 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- 172.21.15.76:0/2372123849 >> 172.21.15.76:0/2372123849 conn(0x7fa4700a0030 msgr2=0x7fa4700a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- 172.21.15.76:0/2372123849 shutdown_connections 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- 172.21.15.76:0/2372123849 wait complete. 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 10 monclient: monmap: 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:00.097 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.098 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:00.098 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:00.098 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 Processor -- start 2024-09-16T06:58:00.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 1 -- start start 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.088+0000 7fa474889700 10 monclient: init 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient(hunting): picked mon.noname-b con 0x7fa4700973c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient(hunting): picked mon.noname-a con 0x7fa470097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient(hunting): picked mon.noname-c con 0x7fa4701400d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470072660 con 0x7fa470097eb0 2024-09-16T06:58:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47013e800 con 0x7fa4700973c0 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47006c6e0 con 0x7fa4701400d0 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient(hunting): authenticate will time out at 2027.228681s 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa46f59e700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fa4701400d0 legacy=0x7fa4701d6bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa467fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa470097eb0 0x7fa4701d3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51360/0 (socket says 172.21.15.76:51360) 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa467fff700 1 -- 172.21.15.76:0/1929532976 learned_addr learned my addr 172.21.15.76:0/1929532976 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438484196 0 0) 0x7fa470072660 con 0x7fa470097eb0 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): my global_id is 334272 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa450009970 con 0x7fa470097eb0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481063330 0 0) 0x7fa47013e800 con 0x7fa4700973c0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): my global_id is 284728 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470072660 con 0x7fa4700973c0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448503805 0 0) 0x7fa450009970 con 0x7fa470097eb0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa47013e800 con 0x7fa470097eb0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa460004290 con 0x7fa470097eb0 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): dump: 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1435010140 0 0) 0x7fa47013e800 con 0x7fa470097eb0 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 >> v1:172.21.15.76:6790/0 conn(0x7fa4701400d0 legacy=0x7fa4701d6bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:00.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 >> v1:172.21.15.189:6789/0 conn(0x7fa4700973c0 legacy=0x7fa4701ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 monclient: found mon.a 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47006c9c0 con 0x7fa470097eb0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 5 monclient: authenticate success, global_id 334272 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _renew_subs 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47013fe30 con 0x7fa470097eb0 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _renew_subs 2024-09-16T06:58:00.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47006c400 con 0x7fa470097eb0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa460004760 con 0x7fa470097eb0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47013ebd0 con 0x7fa470097eb0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fa460004a70 con 0x7fa470097eb0 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa4677fe700 10 monclient: dump: 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.092+0000 7fa46dd9b700 4 set_mon_vals no callback set 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.096+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fa460004c10 con 0x7fa470097eb0 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.096+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fa460043dc0 con 0x7fa470097eb0 2024-09-16T06:58:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.096+0000 7fa46ed9d700 10 monclient: get_auth_request con 0x7fa450036350 auth_method 0 2024-09-16T06:58:00.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.100+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fa460012bd0 con 0x7fa470097eb0 2024-09-16T06:58:00.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.100+0000 7fa4677fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.100+0000 7fa4677fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: shutdown 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- 172.21.15.189:0/2461782312 >> v1:172.21.15.76:6789/0 conn(0x7efed4010d80 legacy=0x7efed4009450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- 172.21.15.189:0/2461782312 shutdown_connections 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- 172.21.15.189:0/2461782312 >> 172.21.15.189:0/2461782312 conn(0x7efee80a4560 msgr2=0x7efee80a4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- 172.21.15.189:0/2461782312 shutdown_connections 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- 172.21.15.189:0/2461782312 wait complete. 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: monmap: 2024-09-16T06:58:00.113 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 Processor -- start 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- start start 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: init 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.114 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient(hunting): picked mon.noname-a con 0x7efed4003740 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient(hunting): picked mon.noname-c con 0x7efed4010d80 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient(hunting): picked mon.noname-b con 0x7efee8135780 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80749c0 con 0x7efed4003740 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee809b330 con 0x7efee8135780 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee813ab50 con 0x7efed4010d80 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efeed798700 10 monclient(hunting): authenticate will time out at 2032.602967s 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efee67fc700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7efed4010d80 legacy=0x7efee81cf6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efee6ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efed4003740 0x7efee81330c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.189:39040/0 (socket says 172.21.15.189:39040) 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efee6ffd700 1 -- 172.21.15.189:0/4187822987 learned_addr learned my addr 172.21.15.189:0/4187822987 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:00.115 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375245067 0 0) 0x7efee80749c0 con 0x7efed4003740 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): my global_id is 334275 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed00035d0 con 0x7efed4003740 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595760667 0 0) 0x7efee809b330 con 0x7efee8135780 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 10 monclient(hunting): my global_id is 284731 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.086+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80749c0 con 0x7efee8135780 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1130185265 0 0) 0x7efed00035d0 con 0x7efed4003740 2024-09-16T06:58:00.116 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee809b330 con 0x7efed4003740 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951048452 0 0) 0x7efee80749c0 con 0x7efee8135780 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efed00035d0 con 0x7efee8135780 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efed8004380 con 0x7efed4003740 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): dump: 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:00.117 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efee0001df0 con 0x7efee8135780 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient(hunting): dump: 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.118 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2931339093 0 0) 0x7efee809b330 con 0x7efed4003740 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 >> v1:172.21.15.76:6790/0 conn(0x7efed4010d80 legacy=0x7efee81cf6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 >> v1:172.21.15.189:6789/0 conn(0x7efee8135780 legacy=0x7efee81d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 monclient: found mon.a 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.119 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee812a280 con 0x7efed4003740 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 5 monclient: authenticate success, global_id 334275 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _renew_subs 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee812a560 con 0x7efed4003740 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _renew_subs 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efee813b2a0 con 0x7efed4003740 2024-09-16T06:58:00.120 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efed8004800 con 0x7efed4003740 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efed8004b90 con 0x7efed4003740 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efee57fa700 4 set_mon_vals no callback set 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 10 monclient: dump: 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:00.121 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efed8004d70 con 0x7efed4003740 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8005100 con 0x7efed4003740 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7efed8043e50 con 0x7efed4003740 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.090+0000 7efee67fc700 10 monclient: get_auth_request con 0x7efed0036420 auth_method 0 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.102+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7efed80147f0 con 0x7efed4003740 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.102+0000 7efecf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.122 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.102+0000 7efecf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:00.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.336+0000 7fa474889700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:00.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:00.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa4700672a0 con 0x7fa470097eb0 2024-09-16T06:58:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa4677fe700 1 -- 172.21.15.76:0/1929532976 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 1675050670) 0x7fa46001ccf0 con 0x7fa470097eb0 2024-09-16T06:58:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa4677fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa4677fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:00.340 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":172,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:00.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 >> v1:172.21.15.189:6812/3098810674 conn(0x7fa450036350 legacy=0x7fa450038810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 10 monclient: shutdown 2024-09-16T06:58:00.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 >> v1:172.21.15.76:6789/0 conn(0x7fa470097eb0 legacy=0x7fa4701d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 shutdown_connections 2024-09-16T06:58:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 >> 172.21.15.76:0/1929532976 conn(0x7fa4700a0030 msgr2=0x7fa47009c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 shutdown_connections 2024-09-16T06:58:00.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.340+0000 7fa474889700 1 -- 172.21.15.76:0/1929532976 wait complete. 2024-09-16T06:58:00.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.342+0000 7efeed798700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.342+0000 7efeed798700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.342+0000 7efeed798700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.347 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.342+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7efec80053b0 con 0x7efed4003740 2024-09-16T06:58:00.353 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:58:00.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:58:00.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:58:00.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.354+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v45) v1 ==== 80+0+0 (unknown 526246777 0 0) 0x7efed8012c60 con 0x7efed4003740 2024-09-16T06:58:00.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.354+0000 7efecf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.357 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.354+0000 7efecf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.464+0000 7f95c3fff700 10 monclient: get_monmap_and_config 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.464+0000 7f95c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.464+0000 7f95c3fff700 10 monclient: monmap: 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:00.471454+0000 2024-09-16T06:58:00.466 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:00.471454+0000 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:00.467 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 Processor -- start 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- start start 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: init 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:00.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f95c40964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f95c409b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f95c4137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4098560 con 0x7f95c409b1c0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4099ec0 con 0x7f95c4137e70 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c409b5f0 con 0x7f95c40964f0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient(hunting): authenticate will time out at 2027.604693s 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c2ffd700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f95c40964f0 legacy=0x7f95c40968d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c27fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f95c409b1c0 0x7f95c4136750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51372/0 (socket says 172.21.15.76:51372) 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c27fc700 1 -- 172.21.15.76:0/2416022997 learned_addr learned my addr 172.21.15.76:0/2416022997 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966199450 0 0) 0x7f95c4098560 con 0x7f95c409b1c0 2024-09-16T06:58:00.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient(hunting): my global_id is 334278 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b00057e0 con 0x7f95c409b1c0 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520832590 0 0) 0x7f95b00057e0 con 0x7f95c409b1c0 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 >> v1:172.21.15.76:6790/0 conn(0x7f95c40964f0 legacy=0x7f95c40968d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 >> v1:172.21.15.189:6789/0 conn(0x7f95c4137e70 legacy=0x7f95c413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 monclient: found mon.noname-a 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4099be0 con 0x7f95c409b1c0 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95ac001610 con 0x7f95c409b1c0 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: dump: 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 5 monclient: authenticate success, global_id 334278 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95ac001930 con 0x7f95c409b1c0 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 4 set_mon_vals no callback set 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 1 -- 172.21.15.76:0/2416022997 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95ac001d00 con 0x7f95c409b1c0 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c1ffb700 10 monclient: dump: 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.475 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: shutdown 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- 172.21.15.76:0/2416022997 >> v1:172.21.15.76:6789/0 conn(0x7f95c409b1c0 legacy=0x7f95c4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- 172.21.15.76:0/2416022997 shutdown_connections 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- 172.21.15.76:0/2416022997 >> 172.21.15.76:0/2416022997 conn(0x7f95c40a0030 msgr2=0x7f95c40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- 172.21.15.76:0/2416022997 shutdown_connections 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 1 -- 172.21.15.76:0/2416022997 wait complete. 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.468+0000 7f95c3fff700 10 monclient: monmap: 2024-09-16T06:58:00.476 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 Processor -- start 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- start start 2024-09-16T06:58:00.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: init 2024-09-16T06:58:00.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f95c40964f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f95c409b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f95c4137e70 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4091810 con 0x7f95c40964f0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4090450 con 0x7f95c409b1c0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c409bc30 con 0x7f95c4137e70 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient(hunting): authenticate will time out at 2027.608842s 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c37fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f95c4137e70 legacy=0x7f95c41d2080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c2ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f95c40964f0 0x7f95c408f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51374/0 (socket says 172.21.15.76:51374) 2024-09-16T06:58:00.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c2ffd700 1 -- 172.21.15.76:0/1647412569 learned_addr learned my addr 172.21.15.76:0/1647412569 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978808710 0 0) 0x7f95c4091810 con 0x7f95c40964f0 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): my global_id is 334281 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a40035d0 con 0x7f95c40964f0 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701837061 0 0) 0x7f95c4090450 con 0x7f95c409b1c0 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): my global_id is 284737 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c4091810 con 0x7f95c409b1c0 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297168136 0 0) 0x7f95a40035d0 con 0x7f95c40964f0 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c4090450 con 0x7f95c40964f0 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95b4004200 con 0x7f95c40964f0 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): dump: 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.486 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91718666 0 0) 0x7f95c4091810 con 0x7f95c409b1c0 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a40035d0 con 0x7f95c409b1c0 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95ac0015d0 con 0x7f95c409b1c0 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): dump: 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541609911 0 0) 0x7f95c4090450 con 0x7f95c40964f0 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6790/0 conn(0x7f95c4137e70 legacy=0x7f95c41d2080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.189:6789/0 conn(0x7f95c409b1c0 legacy=0x7f95c408fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 monclient: found mon.a 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c40a2d70 con 0x7f95c40964f0 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 5 monclient: authenticate success, global_id 334281 2024-09-16T06:58:00.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _renew_subs 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c409a540 con 0x7f95c40964f0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _renew_subs 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c4093140 con 0x7f95c40964f0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95b4004470 con 0x7f95c40964f0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95b4004780 con 0x7f95c40964f0 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c17fa700 4 set_mon_vals no callback set 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient: dump: 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.490 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95a8006590 con 0x7f95a8002830 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116122175 0 0) 0x7f95a8006590 con 0x7f95a8002830 2024-09-16T06:58:00.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 10 monclient(hunting): my global_id is 334284 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.472+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c4090450 con 0x7f95a8002830 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 994475116 0 0) 0x7f95c4090450 con 0x7f95a8002830 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a8006590 con 0x7f95a8002830 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95ac004bd0 con 0x7f95a8002830 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 10 monclient: dump: 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.492 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f95b4004960 con 0x7f95c40964f0 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f95b4043e10 con 0x7f95c40964f0 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1811448822 0 0) 0x7f95a8006590 con 0x7f95a8002830 2024-09-16T06:58:00.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a800a190 con 0x7f95a8002830 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95c37fe700 10 monclient: get_auth_request con 0x7f95a40362e0 auth_method 0 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f95ac005070 con 0x7f95a8002830 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6789/0 conn(0x7f95a8002830 legacy=0x7f95a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.476+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6789/0 conn(0x7f95a8002830 legacy=0x7f95a8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:00.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95c3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:00.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95a8009e50 con 0x7f95a8009110 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194207619 0 0) 0x7f95a8009e50 con 0x7f95a8009110 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 10 monclient(hunting): my global_id is 334287 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.484+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a8006590 con 0x7f95a8009110 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2020202606 0 0) 0x7f95a8006590 con 0x7f95a8009110 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a8009e50 con 0x7f95a8009110 2024-09-16T06:58:00.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95ac0054d0 con 0x7f95a8009110 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 10 monclient: dump: 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2392362159 0 0) 0x7f95a8009e50 con 0x7f95a8009110 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95a800c340 con 0x7f95a8009110 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 3520630803) 0x7f95ac006820 con 0x7f95a8009110 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6789/0 conn(0x7f95a8009110 legacy=0x7f95a80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95bb7fe700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6789/0 conn(0x7f95a8009110 legacy=0x7f95a80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:00.498 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 172, 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.499 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.500 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:00.502 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.503 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.506 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:00.507 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:00.508 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.509 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 7 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:00.510 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.189:6812/3098810674 conn(0x7f95a40362e0 legacy=0x7f95a40387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 10 monclient: shutdown 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 >> v1:172.21.15.76:6789/0 conn(0x7f95c40964f0 legacy=0x7f95c408f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c37fe700 1 -- 172.21.15.76:0/1647412569 reap_dead start 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 shutdown_connections 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 >> 172.21.15.76:0/1647412569 conn(0x7f95c40a0030 msgr2=0x7f95c409e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 shutdown_connections 2024-09-16T06:58:00.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.488+0000 7f95c3fff700 1 -- 172.21.15.76:0/1647412569 wait complete. 2024-09-16T06:58:00.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:58:00.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:58:00.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:58:00.529 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efeed798700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efeed798700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efeed798700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.530 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7efec8005b90 con 0x7efed4003740 2024-09-16T06:58:00.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efecf7fe700 1 -- 172.21.15.189:0/4187822987 <== mon.0 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v45) v1 ==== 80+0+0 (unknown 526246777 0 0) 0x7efed80143e0 con 0x7efed4003740 2024-09-16T06:58:00.531 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efecf7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T06:58:00.532 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.526+0000 7efecf7fe700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:58:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 >> v1:172.21.15.189:6812/3098810674 conn(0x7efed0036420 legacy=0x7efed00388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 10 monclient: shutdown 2024-09-16T06:58:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.534 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 >> v1:172.21.15.76:6789/0 conn(0x7efed4003740 legacy=0x7efee81330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 shutdown_connections 2024-09-16T06:58:00.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 >> 172.21.15.189:0/4187822987 conn(0x7efee80a4560 msgr2=0x7efee8097450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 shutdown_connections 2024-09-16T06:58:00.535 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:00.530+0000 7efeed798700 1 -- 172.21.15.189:0/4187822987 wait complete. 2024-09-16T06:58:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-16T06:58:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T06:58:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=13 2024-09-16T06:58:00.545 INFO:tasks.workunit.client.0.smithi189.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:58:00.545 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:58:00.546 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 10 monclient: get_monmap_and_config 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 10 monclient: monmap: 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:00.638445+0000 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:00.638445+0000 2024-09-16T06:58:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.634 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.634 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.634 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:00.634 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:00.634 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 1 Processor -- start 2024-09-16T06:58:00.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 1 -- start start 2024-09-16T06:58:00.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.632+0000 7f6ffbda3700 10 monclient: init 2024-09-16T06:58:00.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff40a5800 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-c con 0x7f6ff40a7300 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff40a62f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff406c950 con 0x7f6ff40a62f0 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40a6c00 con 0x7f6ff40a7300 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff4099d10 con 0x7f6ff40a5800 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient(hunting): authenticate will time out at 2027.772709s 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff9b3f700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6ff40a5800 legacy=0x7f6ff40a5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffa340700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6ff40a62f0 0x7f6ff413de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51400/0 (socket says 172.21.15.76:51400) 2024-09-16T06:58:00.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffa340700 1 -- 172.21.15.76:0/3853323040 learned_addr learned my addr 172.21.15.76:0/3853323040 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:00.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272596673 0 0) 0x7f6ff406c950 con 0x7f6ff40a62f0 2024-09-16T06:58:00.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): my global_id is 334290 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe00035d0 con 0x7f6ff40a62f0 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381264095 0 0) 0x7f6ff40a6c00 con 0x7f6ff40a7300 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient(hunting): my global_id is 284740 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff406c950 con 0x7f6ff40a7300 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877773633 0 0) 0x7f6fe00035d0 con 0x7f6ff40a62f0 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 >> v1:172.21.15.76:6790/0 conn(0x7f6ff40a5800 legacy=0x7f6ff40a5be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a7300 legacy=0x7f6ff413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 monclient: found mon.noname-a 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40a6960 con 0x7f6ff40a62f0 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 5 monclient: authenticate success, global_id 334290 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec001990 con 0x7f6ff40a62f0 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: dump: 2024-09-16T06:58:00.647 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.648 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fec001c00 con 0x7f6ff40a62f0 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 1 -- 172.21.15.76:0/3853323040 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec001f90 con 0x7f6ff40a62f0 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 4 set_mon_vals no callback set 2024-09-16T06:58:00.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ff8b3d700 10 monclient: dump: 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 10 monclient: shutdown 2024-09-16T06:58:00.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- 172.21.15.76:0/3853323040 >> v1:172.21.15.76:6789/0 conn(0x7f6ff40a62f0 legacy=0x7f6ff413de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- 172.21.15.76:0/3853323040 shutdown_connections 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- 172.21.15.76:0/3853323040 >> 172.21.15.76:0/3853323040 conn(0x7f6ff40a4500 msgr2=0x7f6ff40a4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- 172.21.15.76:0/3853323040 shutdown_connections 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.636+0000 7f6ffbda3700 1 -- 172.21.15.76:0/3853323040 wait complete. 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: build_initial_monmap 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: monmap: 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:00.651 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 Processor -- start 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- start start 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: init 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-b con 0x7f6ff40a5800 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-c con 0x7f6ff40a62f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient(hunting): picked mon.noname-a con 0x7f6ff40a7300 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff413dd70 con 0x7f6ff40a7300 2024-09-16T06:58:00.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff413df60 con 0x7f6ff40a5800 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff409ab60 con 0x7f6ff40a62f0 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ff933e700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f6ff40a62f0 legacy=0x7f6ff41cf9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient(hunting): authenticate will time out at 2027.778114s 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffa340700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6ff40a7300 0x7f6ff41d30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51404/0 (socket says 172.21.15.76:51404) 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffa340700 1 -- 172.21.15.76:0/248898361 learned_addr learned my addr 172.21.15.76:0/248898361 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767574848 0 0) 0x7f6ff413dd70 con 0x7f6ff40a7300 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): my global_id is 334296 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd80035d0 con 0x7f6ff40a7300 2024-09-16T06:58:00.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221001699 0 0) 0x7f6ff413df60 con 0x7f6ff40a5800 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): my global_id is 284746 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff413dd70 con 0x7f6ff40a5800 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165864223 0 0) 0x7f6fd80035d0 con 0x7f6ff40a7300 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff413df60 con 0x7f6ff40a7300 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec001f10 con 0x7f6ff40a7300 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): dump: 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.654 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126641165 0 0) 0x7f6ff413dd70 con 0x7f6ff40a5800 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fd80035d0 con 0x7f6ff40a5800 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe80040f0 con 0x7f6ff40a5800 2024-09-16T06:58:00.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient(hunting): dump: 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2470796772 0 0) 0x7f6ff413df60 con 0x7f6ff40a7300 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ff9b3f700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a5800 legacy=0x7f6ff412e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ff9b3f700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a5800 legacy=0x7f6ff412e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ff9b3f700 1 --1- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a5800 0x7f6ff412e3c0 :-1 s=OPENED pgs=1452 cs=1 l=1).handle_message read tag failed 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ff9b3f700 1 --1- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a5800 0x7f6ff412e3c0 :-1 s=OPENED pgs=1452 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.76:6790/0 conn(0x7f6ff40a62f0 legacy=0x7f6ff41cf9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6ff40a5800 legacy=0x7f6ff412e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: _finish_hunting 0 2024-09-16T06:58:00.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 monclient: found mon.a 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff406d5f0 con 0x7f6ff40a7300 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: _finish_auth 0 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: _check_auth_tickets 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: ms_handle_reset stray mon v1:172.21.15.189:6789/0 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 5 monclient: authenticate success, global_id 334296 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _renew_subs 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff4136f70 con 0x7f6ff40a7300 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _renew_subs 2024-09-16T06:58:00.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41346e0 con 0x7f6ff40a7300 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fec004f10 con 0x7f6ff40a7300 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fec001590 con 0x7f6ff40a7300 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe7fff700 4 set_mon_vals no callback set 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:00.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.640+0000 7f6fe67fc700 10 monclient: dump: 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.660 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6ffbda3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6fdc006590 con 0x7f6fdc002830 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f6fec0052a0 con 0x7f6ff40a7300 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022060785 0 0) 0x7f6fdc006590 con 0x7f6fdc002830 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient(hunting): my global_id is 284749 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff413df60 con 0x7f6fdc002830 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f6fec0442d0 con 0x7f6ff40a7300 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6ff933e700 10 monclient: get_auth_request con 0x7f6fd8036280 auth_method 0 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1001394432 0 0) 0x7f6ff413df60 con 0x7f6fdc002830 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fdc006590 con 0x7f6fdc002830 2024-09-16T06:58:00.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe8004780 con 0x7f6fdc002830 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient: dump: 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.662 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 55664303 0 0) 0x7f6fdc006590 con 0x7f6fdc002830 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fdc00a190 con 0x7f6fdc002830 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f6fe8006b30 con 0x7f6fdc002830 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6fdc002830 legacy=0x7f6fdc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.644+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6fdc002830 legacy=0x7f6fdc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:00.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6fdc009e30 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728445572 0 0) 0x7f6fdc009e30 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient(hunting): my global_id is 284752 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc006590 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4028887811 0 0) 0x7f6fdc006590 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fdc009e30 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f6fe8004fd0 con 0x7f6fdc009150 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:00.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient: dump: 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 748759222 0 0) 0x7f6fdc009e30 con 0x7f6fdc009150 2024-09-16T06:58:00.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6fdc00c340 con 0x7f6fdc009150 2024-09-16T06:58:00.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3871 (unknown 0 0 2656765147) 0x7f6fe8006400 con 0x7f6fdc009150 2024-09-16T06:58:00.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:00.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:00.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6fdc009150 legacy=0x7f6fdc0095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6fe67fc700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6789/0 conn(0x7f6fdc009150 legacy=0x7f6fdc0095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 172, 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:00.669 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.670 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.671 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:00.672 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:00.673 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:00.674 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:00.675 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:00.676 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.678 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:00.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:00.681 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:00.682 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:00.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.189:6812/3098810674 conn(0x7f6fd8036280 legacy=0x7f6fd8038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 10 monclient: shutdown 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 >> v1:172.21.15.76:6789/0 conn(0x7f6ff40a7300 legacy=0x7f6ff41d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffa340700 1 -- 172.21.15.76:0/248898361 reap_dead start 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffa340700 1 -- 172.21.15.76:0/248898361 reap_dead start 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 shutdown_connections 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 >> 172.21.15.76:0/248898361 conn(0x7f6ff40a4500 msgr2=0x7f6ff40a32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 shutdown_connections 2024-09-16T06:58:00.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:00.660+0000 7f6ffbda3700 1 -- 172.21.15.76:0/248898361 wait complete. 2024-09-16T06:58:00.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:58:00.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:58:20.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:58:20.688 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:58:20.688 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:58:20.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:58:20.691 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:58:20.691 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T06:58:20.691 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T06:58:20.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-16T06:58:40.692 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T06:58:40.693 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T06:58:40.693 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:58:40.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:58:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=13 2024-09-16T06:58:40.733 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T06:58:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:58:40.734 INFO:tasks.workunit.client.0.smithi189.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T06:58:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ cat 2024-09-16T06:58:40.735 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth import -i - 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: get_monmap_and_config 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: build_initial_monmap 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: monmap: 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.811 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:40.816248+0000 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:40.816248+0000 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:40.812 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 1 Processor -- start 2024-09-16T06:58:40.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 1 -- start start 2024-09-16T06:58:40.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: init 2024-09-16T06:58:40.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:40.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-b con 0x7f8ec8132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-c con 0x7f8ec806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-a con 0x7f8ec806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec806c280 con 0x7f8ec806cf50 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec806bfa0 con 0x7f8ec806d480 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec809ace0 con 0x7f8ec8132330 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient(hunting): authenticate will time out at 2067.950281s 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecce13700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8ec8132330 0x7f8ec8132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40770/0 (socket says 172.21.15.76:40770) 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecce13700 1 -- 172.21.15.76:0/3364772826 learned_addr learned my addr 172.21.15.76:0/3364772826 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:40.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940720335 0 0) 0x7f8ec809ace0 con 0x7f8ec8132330 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): my global_id is 354119 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb00035d0 con 0x7f8ec8132330 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916375512 0 0) 0x7f8ec806c280 con 0x7f8ec806cf50 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): my global_id is 334320 2024-09-16T06:58:40.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec809ace0 con 0x7f8ec806cf50 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564136818 0 0) 0x7f8ec806bfa0 con 0x7f8ec806d480 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient(hunting): my global_id is 284791 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec806c280 con 0x7f8ec806d480 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129117410 0 0) 0x7f8ec809ace0 con 0x7f8ec806cf50 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 >> v1:172.21.15.76:6790/0 conn(0x7f8ec8132330 legacy=0x7f8ec8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 >> v1:172.21.15.189:6789/0 conn(0x7f8ec806d480 legacy=0x7f8ec813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 monclient: found mon.noname-a 2024-09-16T06:58:40.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec809ae70 con 0x7f8ec806cf50 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec00018f0 con 0x7f8ec806cf50 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: dump: 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.825 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 5 monclient: authenticate success, global_id 334320 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec0001c10 con 0x7f8ec806cf50 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ecde15700 4 set_mon_vals no callback set 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 1 -- 172.21.15.76:0/3364772826 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec0001e00 con 0x7f8ec806cf50 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.814+0000 7f8ebf7fe700 10 monclient: dump: 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: shutdown 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:40.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- 172.21.15.76:0/3364772826 >> v1:172.21.15.76:6789/0 conn(0x7f8ec806cf50 legacy=0x7f8ec813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- 172.21.15.76:0/3364772826 shutdown_connections 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- 172.21.15.76:0/3364772826 >> 172.21.15.76:0/3364772826 conn(0x7f8ec80a41c0 msgr2=0x7f8ec80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- 172.21.15.76:0/3364772826 shutdown_connections 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- 172.21.15.76:0/3364772826 wait complete. 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: build_initial_monmap 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: monmap: 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:40.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 Processor -- start 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- start start 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: init 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-c con 0x7f8ec806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-b con 0x7f8ec806d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:40.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient(hunting): picked mon.noname-a con 0x7f8ec8132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec812c7e0 con 0x7f8ec8132330 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec812d740 con 0x7f8ec806d480 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec812da20 con 0x7f8ec806cf50 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecde15700 10 monclient(hunting): authenticate will time out at 2067.955120s 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecce13700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8ec806cf50 0x7f8ec8130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40784/0 (socket says 172.21.15.76:40784) 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ecce13700 1 -- 172.21.15.76:0/2653341917 learned_addr learned my addr 172.21.15.76:0/2653341917 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998494352 0 0) 0x7f8ec812da20 con 0x7f8ec806cf50 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): my global_id is 354122 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb40035d0 con 0x7f8ec806cf50 2024-09-16T06:58:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682101865 0 0) 0x7f8ec812c7e0 con 0x7f8ec8132330 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): my global_id is 334323 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec812da20 con 0x7f8ec8132330 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987246589 0 0) 0x7f8ec812d740 con 0x7f8ec806d480 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 10 monclient(hunting): my global_id is 284794 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.818+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec812c7e0 con 0x7f8ec806d480 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715450291 0 0) 0x7f8eb40035d0 con 0x7f8ec806cf50 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ec812d740 con 0x7f8ec806cf50 2024-09-16T06:58:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec40019f0 con 0x7f8ec806cf50 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): dump: 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3698805585 0 0) 0x7f8ec812c7e0 con 0x7f8ec806d480 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb40035d0 con 0x7f8ec806d480 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8eb80018b0 con 0x7f8ec806d480 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient(hunting): dump: 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712632955 0 0) 0x7f8ec812da20 con 0x7f8ec8132330 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ec812c7e0 con 0x7f8ec8132330 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163342926 0 0) 0x7f8ec812d740 con 0x7f8ec806cf50 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 >> v1:172.21.15.189:6789/0 conn(0x7f8ec806d480 legacy=0x7f8ec81cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 >> v1:172.21.15.76:6789/0 conn(0x7f8ec8132330 legacy=0x7f8ec81d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 monclient: found mon.c 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80a1540 con 0x7f8ec806cf50 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 5 monclient: authenticate success, global_id 354122 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _renew_subs 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec80a0d20 con 0x7f8ec806cf50 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _renew_subs 2024-09-16T06:58:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec8131040 con 0x7f8ec806cf50 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ec4001ca0 con 0x7f8ec806cf50 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8ec4004ce0 con 0x7f8ec806cf50 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebeffd700 4 set_mon_vals no callback set 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 10 monclient: dump: 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8005100 con 0x7f8ec806cf50 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8ec400f750 con 0x7f8ec806cf50 2024-09-16T06:58:40.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8ec4043df0 con 0x7f8ec806cf50 2024-09-16T06:58:40.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.822+0000 7f8ebffff700 10 monclient: get_auth_request con 0x7f8eb40364f0 auth_method 0 2024-09-16T06:58:40.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.826+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8ec4014560 con 0x7f8ec806cf50 2024-09-16T06:58:40.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.826+0000 7f8ebd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:40.826+0000 7f8ebd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.830+0000 7f41b2de4700 10 monclient: get_monmap_and_config 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.830+0000 7f41b2de4700 10 monclient: build_initial_monmap 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.830+0000 7f41b2de4700 10 monclient: monmap: 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:58:40.836224+0000 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:58:40.836224+0000 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:40.841 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 Processor -- start 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- start start 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient: init 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-c con 0x7f41ac09ddf0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-b con 0x7f41ac0a00e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-a con 0x7f41ac09e8e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac074a30 con 0x7f41ac09e8e0 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac0a1320 con 0x7f41ac09ddf0 2024-09-16T06:58:40.842 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac0a0980 con 0x7f41ac0a00e0 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient(hunting): authenticate will time out at 2073.348934s 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b0b80700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41ac09ddf0 0x7f41ac09e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32942/0 (socket says 172.21.15.189:32942) 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b0b80700 1 -- 172.21.15.189:0/2060686358 learned_addr learned my addr 172.21.15.189:0/2060686358 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060830727 0 0) 0x7f41ac0a0980 con 0x7f41ac0a00e0 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): my global_id is 354125 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41940035d0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802108067 0 0) 0x7f41ac0a1320 con 0x7f41ac09ddf0 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.843 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): my global_id is 284797 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac0a0980 con 0x7f41ac09ddf0 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778282114 0 0) 0x7f41ac074a30 con 0x7f41ac09e8e0 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient(hunting): my global_id is 334326 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac0a1320 con 0x7f41ac09e8e0 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491843636 0 0) 0x7f41940035d0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 >> v1:172.21.15.189:6789/0 conn(0x7f41ac09ddf0 legacy=0x7f41ac09e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 >> v1:172.21.15.76:6789/0 conn(0x7f41ac09e8e0 legacy=0x7f41ac13de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.844 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 monclient: found mon.noname-b 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac0a0c60 con 0x7f41ac0a00e0 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f419c001610 con 0x7f41ac0a00e0 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: dump: 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.845 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 5 monclient: authenticate success, global_id 354125 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f419c0018f0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 1 -- 172.21.15.189:0/2060686358 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f419c001c80 con 0x7f41ac0a00e0 2024-09-16T06:58:40.846 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41a37fe700 10 monclient: dump: 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:40.847 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 4 set_mon_vals no callback set 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 10 monclient: shutdown 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- 172.21.15.189:0/2060686358 >> v1:172.21.15.76:6790/0 conn(0x7f41ac0a00e0 legacy=0x7f41ac13a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- 172.21.15.189:0/2060686358 shutdown_connections 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- 172.21.15.189:0/2060686358 >> 172.21.15.189:0/2060686358 conn(0x7f41ac0a1920 msgr2=0x7f41ac0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- 172.21.15.189:0/2060686358 shutdown_connections 2024-09-16T06:58:40.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.834+0000 7f41b2de4700 1 -- 172.21.15.189:0/2060686358 wait complete. 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient: build_initial_monmap 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient: monmap: 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:40.852 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 1 Processor -- start 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 1 -- start start 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient: init 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-a con 0x7f41ac09ddf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-c con 0x7f41ac09e8e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient(hunting): picked mon.noname-b con 0x7f41ac0a00e0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:40.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac072750 con 0x7f41ac09ddf0 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac120ef0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac121aa0 con 0x7f41ac09e8e0 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b2de4700 10 monclient(hunting): authenticate will time out at 2073.355552s 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b1381700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f41ac0a00e0 0x7f41ac1d27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32958/0 (socket says 172.21.15.189:32958) 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41b1381700 1 -- 172.21.15.189:0/456734605 learned_addr learned my addr 172.21.15.189:0/456734605 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.838+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754465838 0 0) 0x7f41ac120ef0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): my global_id is 284800 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41980035d0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067048028 0 0) 0x7f41ac0a0980 con 0x7f41ac09ddf0 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): my global_id is 334329 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac120ef0 con 0x7f41ac09ddf0 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452843499 0 0) 0x7f41ac121aa0 con 0x7f41ac09e8e0 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): my global_id is 354128 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac0a0980 con 0x7f41ac09e8e0 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3225990985 0 0) 0x7f41980035d0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.855 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ac121aa0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41a4001c10 con 0x7f41ac0a00e0 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): dump: 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.856 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232061066 0 0) 0x7f41ac121aa0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 >> v1:172.21.15.76:6790/0 conn(0x7f41ac09e8e0 legacy=0x7f41ac1cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 >> v1:172.21.15.76:6789/0 conn(0x7f41ac09ddf0 legacy=0x7f41ac134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 monclient: found mon.b 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac121d80 con 0x7f41ac0a00e0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:40.857 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 5 monclient: authenticate success, global_id 284800 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _renew_subs 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac120c50 con 0x7f41ac0a00e0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _renew_subs 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac1c7280 con 0x7f41ac0a00e0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41a4001fc0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f41a4003e90 con 0x7f41ac0a00e0 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:40.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a2ffd700 4 set_mon_vals no callback set 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 10 monclient: dump: 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f41a4003ff0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f41a4044040 con 0x7f41ac0a00e0 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b0b80700 10 monclient: get_auth_request con 0x7f4198035f30 auth_method 0 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.842+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c005100 con 0x7f41ac0a00e0 2024-09-16T06:58:40.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.854+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f41a4014ae0 con 0x7f41ac0a00e0 2024-09-16T06:58:40.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.854+0000 7f41a17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:40.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:40.854+0000 7f41a17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.054+0000 7f8ecde15700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.054+0000 7f8ecde15700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.054+0000 7f8ecde15700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.054+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ea8005930 con 0x7f8ec806cf50 2024-09-16T06:58:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.058+0000 7f8ebd7fa700 1 -- 172.21.15.76:0/2653341917 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1851333841) 0x7f8ec4012c00 con 0x7f8ec806cf50 2024-09-16T06:58:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.058+0000 7f8ebd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.058+0000 7f8ebd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.055 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8eb40364f0 legacy=0x7f8eb40389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 10 monclient: shutdown 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 >> v1:172.21.15.76:6790/0 conn(0x7f8ec806cf50 legacy=0x7f8ec8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 shutdown_connections 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 >> 172.21.15.76:0/2653341917 conn(0x7f8ec80a41c0 msgr2=0x7f8ec8129fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 shutdown_connections 2024-09-16T06:58:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.062+0000 7f8ecde15700 1 -- 172.21.15.76:0/2653341917 wait complete. 2024-09-16T06:58:41.074 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:58:41.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:58:41.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:58:41.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.082+0000 7f41b2de4700 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T06:58:41.087 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.082+0000 7f41b2de4700 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-16T06:58:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.082+0000 7f41b2de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.088 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.082+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f418c005930 con 0x7f41ac0a00e0 2024-09-16T06:58:41.089 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-16T06:58:41.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.090+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth import"}]=0 v49) v1 ==== 59+0+0 (unknown 3513795084 0 0) 0x7f41a4012f50 con 0x7f41ac0a00e0 2024-09-16T06:58:41.094 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.090+0000 7f41a17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-16T06:58:41.095 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.090+0000 7f41a17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: monmap: 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:41.190637+0000 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:41.190637+0000 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.186 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 1 Processor -- start 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 1 -- start start 2024-09-16T06:58:41.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: init 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-b con 0x7efd9c097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-a con 0x7efd9c099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-c con 0x7efd9c139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c098990 con 0x7efd9c099090 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c097f40 con 0x7efd9c139500 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c09d610 con 0x7efd9c097b60 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.186+0000 7efd9bfff700 10 monclient(hunting): authenticate will time out at 2068.323849s 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd927fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efd9c099090 0x7efd9c09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57686/0 (socket says 172.21.15.76:57686) 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd927fc700 1 -- 172.21.15.76:0/3480347156 learned_addr learned my addr 172.21.15.76:0/3480347156 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813323786 0 0) 0x7efd9c098990 con 0x7efd9c099090 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): my global_id is 334332 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880035d0 con 0x7efd9c099090 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537497574 0 0) 0x7efd9c09d610 con 0x7efd9c097b60 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): my global_id is 354131 2024-09-16T06:58:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c098990 con 0x7efd9c097b60 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871915594 0 0) 0x7efd9c097f40 con 0x7efd9c139500 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient(hunting): my global_id is 284803 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c09d610 con 0x7efd9c139500 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464342056 0 0) 0x7efd880035d0 con 0x7efd9c099090 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 >> v1:172.21.15.76:6790/0 conn(0x7efd9c097b60 legacy=0x7efd9c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 >> v1:172.21.15.189:6789/0 conn(0x7efd9c139500 legacy=0x7efd9c13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0986b0 con 0x7efd9c099090 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 5 monclient: authenticate success, global_id 334332 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd84001610 con 0x7efd9c099090 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: dump: 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd840018f0 con 0x7efd9c099090 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 1 -- 172.21.15.76:0/3480347156 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd84001cc0 con 0x7efd9c099090 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 4 set_mon_vals no callback set 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9a7fc700 10 monclient: dump: 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 10 monclient: shutdown 2024-09-16T06:58:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 1 -- 172.21.15.76:0/3480347156 >> v1:172.21.15.76:6789/0 conn(0x7efd9c099090 legacy=0x7efd9c09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 1 -- 172.21.15.76:0/3480347156 shutdown_connections 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 1 -- 172.21.15.76:0/3480347156 >> 172.21.15.76:0/3480347156 conn(0x7efd9c0a0030 msgr2=0x7efd9c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 1 -- 172.21.15.76:0/3480347156 shutdown_connections 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.190+0000 7efd9bfff700 1 -- 172.21.15.76:0/3480347156 wait complete. 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: monmap: 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 Processor -- start 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- start start 2024-09-16T06:58:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: init 2024-09-16T06:58:41.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:41.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-a con 0x7efd9c097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-b con 0x7efd9c099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient(hunting): picked mon.noname-c con 0x7efd9c139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c09a540 con 0x7efd9c097b60 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efd9c097b60 0x7efd9c06be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57698/0 (socket says 172.21.15.76:57698) 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c06eea0 con 0x7efd9c099090 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9affd700 1 -- 172.21.15.76:0/150887391 learned_addr learned my addr 172.21.15.76:0/150887391 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c06ccb0 con 0x7efd9c139500 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient(hunting): authenticate will time out at 2068.330400s 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155396960 0 0) 0x7efd9c09a540 con 0x7efd9c097b60 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): my global_id is 334335 2024-09-16T06:58:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c0035d0 con 0x7efd9c097b60 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553900278 0 0) 0x7efd9c06ccb0 con 0x7efd9c139500 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): my global_id is 354134 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c09a540 con 0x7efd9c139500 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109292187 0 0) 0x7efd9c06eea0 con 0x7efd9c099090 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): my global_id is 284806 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c06ccb0 con 0x7efd9c099090 2024-09-16T06:58:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 104109219 0 0) 0x7efd7c0035d0 con 0x7efd9c097b60 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd9c06eea0 con 0x7efd9c097b60 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd8c001a00 con 0x7efd9c097b60 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): dump: 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3074540223 0 0) 0x7efd9c09a540 con 0x7efd9c139500 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd7c0035d0 con 0x7efd9c139500 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd94001b30 con 0x7efd9c139500 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): dump: 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2362466362 0 0) 0x7efd9c06ccb0 con 0x7efd9c099090 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd9c09a540 con 0x7efd9c099090 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd84001a60 con 0x7efd9c099090 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient(hunting): dump: 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.211 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085910 0 0) 0x7efd9c06eea0 con 0x7efd9c097b60 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6790/0 conn(0x7efd9c139500 legacy=0x7efd9c1cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.189:6789/0 conn(0x7efd9c099090 legacy=0x7efd9c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 monclient: found mon.a 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0710e0 con 0x7efd9c097b60 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd93fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 5 monclient: authenticate success, global_id 334335 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _renew_subs 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c0a2ca0 con 0x7efd9c097b60 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _renew_subs 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.194+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c06de60 con 0x7efd9c097b60 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd8c001ca0 con 0x7efd9c097b60 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd8c003af0 con 0x7efd9c097b60 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd99ffb700 4 set_mon_vals no callback set 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: dump: 2024-09-16T06:58:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd9bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7efd8c003c50 con 0x7efd9c097b60 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efd80006590 con 0x7efd80002830 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7efd8c043e70 con 0x7efd9c097b60 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646074560 0 0) 0x7efd80006590 con 0x7efd80002830 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient(hunting): my global_id is 334338 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c06eea0 con 0x7efd80002830 2024-09-16T06:58:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd9b7fe700 10 monclient: get_auth_request con 0x7efd7c0365c0 auth_method 0 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3534867793 0 0) 0x7efd9c06eea0 con 0x7efd80002830 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd80006590 con 0x7efd80002830 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd84004b90 con 0x7efd80002830 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: dump: 2024-09-16T06:58:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2247223540 0 0) 0x7efd80006590 con 0x7efd80002830 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd8000a190 con 0x7efd80002830 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7efd84004e30 con 0x7efd80002830 2024-09-16T06:58:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80002830 legacy=0x7efd80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.198+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80002830 legacy=0x7efd80004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd9bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:41.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efd80009e50 con 0x7efd80009170 2024-09-16T06:58:41.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd927fc700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 legacy=0x7efd80009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:58:41.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd927fc700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 legacy=0x7efd80009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd927fc700 1 --1- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 0x7efd80009600 :-1 s=OPENED pgs=1424 cs=1 l=1).handle_message read tag failed 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd927fc700 1 --1- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 0x7efd80009600 :-1 s=OPENED pgs=1424 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 legacy=0x7efd80009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd80009170 legacy=0x7efd80009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efd80006590 con 0x7efd7c040f00 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576722983 0 0) 0x7efd80006590 con 0x7efd7c040f00 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient(hunting): my global_id is 334344 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c06eea0 con 0x7efd7c040f00 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 357988441 0 0) 0x7efd9c06eea0 con 0x7efd7c040f00 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd80006590 con 0x7efd7c040f00 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7efd840052f0 con 0x7efd7c040f00 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 10 monclient: dump: 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3704476731 0 0) 0x7efd80006590 con 0x7efd7c040f00 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.210+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efd84004e30 con 0x7efd7c040f00 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1437604131) 0x7efd84006670 con 0x7efd7c040f00 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd93fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd93fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd7c040f00 legacy=0x7efd7c0412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd93fff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd7c040f00 legacy=0x7efd7c0412e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9b7fe700 1 -- 172.21.15.76:0/150887391 reap_dead start 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:41.223 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.224 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.225 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:41.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.227 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.229 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.230 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.231 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.232 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.233 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:41.234 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:41.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.189:6812/3098810674 conn(0x7efd7c0365c0 legacy=0x7efd7c038a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 10 monclient: shutdown 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 >> v1:172.21.15.76:6789/0 conn(0x7efd9c097b60 legacy=0x7efd9c06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 shutdown_connections 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 >> 172.21.15.76:0/150887391 conn(0x7efd9c0a0030 msgr2=0x7efd9c13aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 shutdown_connections 2024-09-16T06:58:41.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.214+0000 7efd9bfff700 1 -- 172.21.15.76:0/150887391 wait complete. 2024-09-16T06:58:41.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:41.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:58:41.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:58:41.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.258+0000 7f41b2de4700 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T06:58:41.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.258+0000 7f41b2de4700 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-16T06:58:41.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.258+0000 7f41b2de4700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.264 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.258+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7f418c002370 con 0x7f41ac0a00e0 2024-09-16T06:58:41.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.266+0000 7f41a17fa700 1 -- 172.21.15.189:0/456734605 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth import"}]=0 v50) v1 ==== 59+0+0 (unknown 2377307344 0 0) 0x7f41a40146d0 con 0x7f41ac0a00e0 2024-09-16T06:58:41.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.266+0000 7f41a17fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-16T06:58:41.271 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.266+0000 7f41a17fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:58:41.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4198035f30 legacy=0x7f41980383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 10 monclient: shutdown 2024-09-16T06:58:41.273 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 >> v1:172.21.15.189:6789/0 conn(0x7f41ac0a00e0 legacy=0x7f41ac1d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 shutdown_connections 2024-09-16T06:58:41.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 >> 172.21.15.189:0/456734605 conn(0x7f41ac0a1920 msgr2=0x7f41ac0a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 shutdown_connections 2024-09-16T06:58:41.274 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.270+0000 7f41b2de4700 1 -- 172.21.15.189:0/456734605 wait complete. 2024-09-16T06:58:41.286 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph health 2024-09-16T06:58:41.287 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep AUTH_BAD_CAP 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: monmap: 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:41.361107+0000 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:41.361107+0000 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.356 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.357 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.357 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 1 Processor -- start 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 1 -- start start 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: init 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-a con 0x7fab6809ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-c con 0x7fab68098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-b con 0x7fab68099030 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6809d560 con 0x7fab6809ed60 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6809f140 con 0x7fab68098be0 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6809d790 con 0x7fab68099030 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 10 monclient(hunting): authenticate will time out at 2068.494479s 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6e73f700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fab68099030 0x7fab6813ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40818/0 (socket says 172.21.15.76:40818) 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6e73f700 1 -- 172.21.15.76:0/190346972 learned_addr learned my addr 172.21.15.76:0/190346972 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113871698 0 0) 0x7fab6809d790 con 0x7fab68099030 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): my global_id is 354137 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab540035d0 con 0x7fab68099030 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055333279 0 0) 0x7fab6809d560 con 0x7fab6809ed60 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): my global_id is 334347 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6809d790 con 0x7fab6809ed60 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588537654 0 0) 0x7fab6809f140 con 0x7fab68098be0 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient(hunting): my global_id is 284809 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6809d560 con 0x7fab68098be0 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496237881 0 0) 0x7fab6809d790 con 0x7fab6809ed60 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 >> v1:172.21.15.76:6790/0 conn(0x7fab68099030 legacy=0x7fab6813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 >> v1:172.21.15.189:6789/0 conn(0x7fab68098be0 legacy=0x7fab681374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 monclient: found mon.noname-a 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6809f620 con 0x7fab6809ed60 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab640015f0 con 0x7fab6809ed60 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: dump: 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.365 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 5 monclient: authenticate success, global_id 334347 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6ef40700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab64001890 con 0x7fab6809ed60 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.358+0000 7fab6cf3c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6cf3c700 1 -- 172.21.15.76:0/190346972 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab64001c60 con 0x7fab6809ed60 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 4 set_mon_vals no callback set 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6cf3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6cf3c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6cf3c700 10 monclient: dump: 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: shutdown 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/190346972 >> v1:172.21.15.76:6789/0 conn(0x7fab6809ed60 legacy=0x7fab680984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/190346972 shutdown_connections 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/190346972 >> 172.21.15.76:0/190346972 conn(0x7fab680a0030 msgr2=0x7fab680a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/190346972 shutdown_connections 2024-09-16T06:58:41.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/190346972 wait complete. 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: monmap: 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 Processor -- start 2024-09-16T06:58:41.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- start start 2024-09-16T06:58:41.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: init 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-c con 0x7fab68098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-a con 0x7fab68099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient(hunting): picked mon.noname-b con 0x7fab6809ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68090e50 con 0x7fab68099030 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6df3e700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fab68098be0 0x7fab680966a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40834/0 (socket says 172.21.15.76:40834) 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68090890 con 0x7fab6809ed60 2024-09-16T06:58:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6df3e700 1 -- 172.21.15.76:0/3282288018 learned_addr learned my addr 172.21.15.76:0/3282288018 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68090b70 con 0x7fab68098be0 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.362+0000 7fab6ef40700 10 monclient(hunting): authenticate will time out at 2068.499850s 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489356194 0 0) 0x7fab68090b70 con 0x7fab68098be0 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): my global_id is 354140 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0035d0 con 0x7fab68098be0 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951795442 0 0) 0x7fab68090e50 con 0x7fab68099030 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): my global_id is 334350 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68090b70 con 0x7fab68099030 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31463012 0 0) 0x7fab68090890 con 0x7fab6809ed60 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): my global_id is 284812 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68090e50 con 0x7fab6809ed60 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2252806442 0 0) 0x7fab68090b70 con 0x7fab68099030 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab68090890 con 0x7fab68099030 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab5c0017f0 con 0x7fab68099030 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): dump: 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768127667 0 0) 0x7fab4c0035d0 con 0x7fab68098be0 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab68090b70 con 0x7fab68098be0 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab640015b0 con 0x7fab68098be0 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): dump: 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.379 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272939270 0 0) 0x7fab68090e50 con 0x7fab6809ed60 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab4c0035d0 con 0x7fab6809ed60 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab60001bd0 con 0x7fab6809ed60 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): dump: 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.380 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 540717256 0 0) 0x7fab68090890 con 0x7fab68099030 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6790/0 conn(0x7fab68098be0 legacy=0x7fab680966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.189:6789/0 conn(0x7fab6809ed60 legacy=0x7fab681ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 monclient: found mon.a 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68096db0 con 0x7fab68099030 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 5 monclient: authenticate success, global_id 334350 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 10 monclient: _renew_subs 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6808ee20 con 0x7fab68099030 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 10 monclient: _renew_subs 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6808f430 con 0x7fab68099030 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab5c0036a0 con 0x7fab68099030 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5bfff700 4 set_mon_vals no callback set 2024-09-16T06:58:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab5c001950 con 0x7fab68099030 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient: dump: 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fab50006590 con 0x7fab50002830 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fab5c00e2c0 con 0x7fab68099030 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fab5c042b40 con 0x7fab68099030 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab6e73f700 10 monclient: get_auth_request con 0x7fab4c036500 auth_method 0 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461086150 0 0) 0x7fab50006590 con 0x7fab50002830 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 10 monclient(hunting): my global_id is 354143 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68090890 con 0x7fab50002830 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2647011152 0 0) 0x7fab68090890 con 0x7fab50002830 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.366+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab50006590 con 0x7fab50002830 2024-09-16T06:58:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab64003d80 con 0x7fab50002830 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 10 monclient: dump: 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3804006406 0 0) 0x7fab50006590 con 0x7fab50002830 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab5000a190 con 0x7fab50002830 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7fab640058d0 con 0x7fab50002830 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6790/0 conn(0x7fab50002830 legacy=0x7fab50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.370+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6790/0 conn(0x7fab50002830 legacy=0x7fab50004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab6ef40700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fab50009e50 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111972328 0 0) 0x7fab50009e50 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient(hunting): my global_id is 354146 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab50006590 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 96817298 0 0) 0x7fab50006590 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab50009e50 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab64004400 con 0x7fab50009170 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 10 monclient: dump: 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1351279162 0 0) 0x7fab50009e50 con 0x7fab50009170 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.378+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fab5000c340 con 0x7fab50009170 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3331836095) 0x7fab64005a40 con 0x7fab50009170 2024-09-16T06:58:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab5a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab5a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6790/0 conn(0x7fab50009170 legacy=0x7fab50009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab5a7fc700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6790/0 conn(0x7fab50009170 legacy=0x7fab50009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:41.391 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.392 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.393 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:41.394 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.395 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.396 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.397 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.398 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.399 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.400 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:41.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:41.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.403 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:41.403 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:41.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.189:6812/3098810674 conn(0x7fab4c036500 legacy=0x7fab4c0389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 10 monclient: shutdown 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 >> v1:172.21.15.76:6789/0 conn(0x7fab68099030 legacy=0x7fab681cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6e73f700 1 -- 172.21.15.76:0/3282288018 reap_dead start 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 shutdown_connections 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 >> 172.21.15.76:0/3282288018 conn(0x7fab680a0030 msgr2=0x7fab6806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 shutdown_connections 2024-09-16T06:58:41.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.382+0000 7fab6ef40700 1 -- 172.21.15.76:0/3282288018 wait complete. 2024-09-16T06:58:41.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:41.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:58:41.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.378+0000 7f39e1f12700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.378+0000 7f39e1f12700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.378+0000 7f39e1f12700 10 monclient: monmap: 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:58:41.384324+0000 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:58:41.384324+0000 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.411 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 Processor -- start 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- start start 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient: init 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-b con 0x7f39dc09f430 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-a con 0x7f39dc09ff20 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-c con 0x7f39dc139bd0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc099b90 con 0x7f39dc09ff20 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc09d5a0 con 0x7f39dc139bd0 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc09de40 con 0x7f39dc09f430 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.412 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient(hunting): authenticate will time out at 2073.896971s 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e0f10700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f39dc09f430 0x7f39dc09f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.189:44236/0 (socket says 172.21.15.189:44236) 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e0f10700 1 -- 172.21.15.189:0/1108623975 learned_addr learned my addr 172.21.15.189:0/1108623975 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1711700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f39dc139bd0 0x7f39dc13bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32960/0 (socket says 172.21.15.189:32960) 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256421032 0 0) 0x7f39dc09d5a0 con 0x7f39dc139bd0 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): my global_id is 284815 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c40035d0 con 0x7f39dc139bd0 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783355990 0 0) 0x7f39dc09de40 con 0x7f39dc09f430 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.413 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): my global_id is 354149 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc09d5a0 con 0x7f39dc09f430 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617636736 0 0) 0x7f39dc099b90 con 0x7f39dc09ff20 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient(hunting): my global_id is 334353 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc09de40 con 0x7f39dc09ff20 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284504683 0 0) 0x7f39c40035d0 con 0x7f39dc139bd0 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 >> v1:172.21.15.76:6790/0 conn(0x7f39dc09f430 legacy=0x7f39dc09f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 >> v1:172.21.15.76:6789/0 conn(0x7f39dc09ff20 legacy=0x7f39dc1384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.414 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 monclient: found mon.noname-c 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc099f90 con 0x7f39dc139bd0 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 5 monclient: authenticate success, global_id 284815 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39d4001990 con 0x7f39dc139bd0 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.415 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: dump: 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.416 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 1 -- 172.21.15.189:0/1108623975 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d4001bf0 con 0x7f39dc139bd0 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39d37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 4 set_mon_vals no callback set 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 10 monclient: shutdown 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- 172.21.15.189:0/1108623975 >> v1:172.21.15.189:6789/0 conn(0x7f39dc139bd0 legacy=0x7f39dc13bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- 172.21.15.189:0/1108623975 shutdown_connections 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- 172.21.15.189:0/1108623975 >> 172.21.15.189:0/1108623975 conn(0x7f39dc0a0b50 msgr2=0x7f39dc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.417 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- 172.21.15.189:0/1108623975 shutdown_connections 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.382+0000 7f39e1f12700 1 -- 172.21.15.189:0/1108623975 wait complete. 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient: monmap: 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:58:41.418 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 1 Processor -- start 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 1 -- start start 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient: init 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-b con 0x7f39dc09f430 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-a con 0x7f39dc09ff20 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient(hunting): picked mon.noname-c con 0x7f39dc139bd0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1363e0 con 0x7f39dc09ff20 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc137950 con 0x7f39dc09f430 2024-09-16T06:58:41.419 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1352b0 con 0x7f39dc139bd0 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e0f10700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f39dc09f430 0x7f39dc134490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32966/0 (socket says 172.21.15.189:32966) 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e0f10700 1 -- 172.21.15.189:0/2213854708 learned_addr learned my addr 172.21.15.189:0/2213854708 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39e1f12700 10 monclient(hunting): authenticate will time out at 2073.903074s 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858539206 0 0) 0x7f39dc137950 con 0x7f39dc09f430 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): my global_id is 284818 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c80035d0 con 0x7f39dc09f430 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353292230 0 0) 0x7f39c80035d0 con 0x7f39dc09f430 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39dc137950 con 0x7f39dc09f430 2024-09-16T06:58:41.420 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048574317 0 0) 0x7f39dc1363e0 con 0x7f39dc09ff20 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): my global_id is 334356 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c80035d0 con 0x7f39dc09ff20 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39d80012b0 con 0x7f39dc09f430 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): dump: 2024-09-16T06:58:41.421 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327118126 0 0) 0x7f39dc1352b0 con 0x7f39dc139bd0 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.422 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 10 monclient(hunting): my global_id is 354152 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.386+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc1363e0 con 0x7f39dc139bd0 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2168784285 0 0) 0x7f39dc137950 con 0x7f39dc09f430 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 >> v1:172.21.15.76:6790/0 conn(0x7f39dc139bd0 legacy=0x7f39dc1cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 >> v1:172.21.15.76:6789/0 conn(0x7f39dc09ff20 legacy=0x7f39dc134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 monclient: found mon.b 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc135f40 con 0x7f39dc09f430 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.423 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 5 monclient: authenticate success, global_id 284818 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _renew_subs 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc070720 con 0x7f39dc09f430 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _renew_subs 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc137670 con 0x7f39dc09f430 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d8001640 con 0x7f39dc09f430 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f39d8004ce0 con 0x7f39dc09f430 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d2ffd700 4 set_mon_vals no callback set 2024-09-16T06:58:41.424 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 10 monclient: dump: 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f39d800f7d0 con 0x7f39dc09f430 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f39d8043fd0 con 0x7f39dc09f430 2024-09-16T06:58:41.425 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39d3fff700 10 monclient: get_auth_request con 0x7f39c80362f0 auth_method 0 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.390+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc005100 con 0x7f39dc09f430 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.394+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f39d8014910 con 0x7f39dc09f430 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.394+0000 7f39d17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.426 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.394+0000 7f39d17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 10 monclient: monmap: 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:41.531137+0000 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:41.531137+0000 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.527 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.527 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 1 Processor -- start 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 1 -- start start 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.526+0000 7f35051a3700 10 monclient: init 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-a con 0x7f3500097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-b con 0x7f3500099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-c con 0x7f3500139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3500098990 con 0x7f3500097b60 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3500097f40 con 0x7f3500139500 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350009d610 con 0x7f3500099090 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient(hunting): authenticate will time out at 2068.664432s 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34fffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3500097b60 0x7f350009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57756/0 (socket says 172.21.15.76:57756) 2024-09-16T06:58:41.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34fffff700 1 -- 172.21.15.76:0/2029630476 learned_addr learned my addr 172.21.15.76:0/2029630476 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281110603 0 0) 0x7f3500098990 con 0x7f3500097b60 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): my global_id is 334359 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec0035d0 con 0x7f3500097b60 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949975823 0 0) 0x7f350009d610 con 0x7f3500099090 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): my global_id is 354155 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500098990 con 0x7f3500099090 2024-09-16T06:58:41.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861569133 0 0) 0x7f3500097f40 con 0x7f3500139500 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient(hunting): my global_id is 284821 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350009d610 con 0x7f3500139500 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1145309996 0 0) 0x7f3500098990 con 0x7f3500099090 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 >> v1:172.21.15.189:6789/0 conn(0x7f3500139500 legacy=0x7f350013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 >> v1:172.21.15.76:6789/0 conn(0x7f3500097b60 legacy=0x7f350009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 monclient: found mon.noname-b 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35000986b0 con 0x7f3500099090 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34e80015f0 con 0x7f3500099090 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: dump: 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.534 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 5 monclient: authenticate success, global_id 354155 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e8001910 con 0x7f3500099090 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 1 -- 172.21.15.76:0/2029630476 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34e8001ca0 con 0x7f3500099090 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 4 set_mon_vals no callback set 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f34ff7fe700 10 monclient: dump: 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.535 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: shutdown 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- 172.21.15.76:0/2029630476 >> v1:172.21.15.76:6790/0 conn(0x7f3500099090 legacy=0x7f350009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- 172.21.15.76:0/2029630476 shutdown_connections 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- 172.21.15.76:0/2029630476 >> 172.21.15.76:0/2029630476 conn(0x7f35000a0030 msgr2=0x7f35000a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- 172.21.15.76:0/2029630476 shutdown_connections 2024-09-16T06:58:41.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 1 -- 172.21.15.76:0/2029630476 wait complete. 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.530+0000 7f35051a3700 10 monclient: monmap: 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.537 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 Processor -- start 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- start start 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: init 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-b con 0x7f3500097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-a con 0x7f3500099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient(hunting): picked mon.noname-c con 0x7f3500139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3500070470 con 0x7f3500099090 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350006ef40 con 0x7f3500097b60 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350006ece0 con 0x7f3500139500 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35049a2700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f3500139500 0x7f35001cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40876/0 (socket says 172.21.15.76:40876) 2024-09-16T06:58:41.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35049a2700 1 -- 172.21.15.76:0/3099205695 learned_addr learned my addr 172.21.15.76:0/3099205695 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient(hunting): authenticate will time out at 2068.670309s 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071158632 0 0) 0x7f350006ece0 con 0x7f3500139500 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): my global_id is 354158 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e00035d0 con 0x7f3500139500 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532496843 0 0) 0x7f3500070470 con 0x7f3500099090 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): my global_id is 334362 2024-09-16T06:58:41.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350006ece0 con 0x7f3500099090 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018419710 0 0) 0x7f350006ef40 con 0x7f3500097b60 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): my global_id is 284824 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500070470 con 0x7f3500097b60 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3507645223 0 0) 0x7f34e00035d0 con 0x7f3500139500 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350006ef40 con 0x7f3500139500 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34f8001c90 con 0x7f3500139500 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:58:41.545 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854047484 0 0) 0x7f350006ece0 con 0x7f3500099090 2024-09-16T06:58:41.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34e00035d0 con 0x7f3500099090 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34e80015f0 con 0x7f3500099090 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.547 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856409312 0 0) 0x7f3500070470 con 0x7f3500097b60 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350006ece0 con 0x7f3500097b60 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34f0001850 con 0x7f3500097b60 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient(hunting): dump: 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.548 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1486324248 0 0) 0x7f350006ef40 con 0x7f3500139500 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6789/0 conn(0x7f3500097b60 legacy=0x7f350006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.76:6789/0 conn(0x7f3500099090 legacy=0x7f350006d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 monclient: found mon.c 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350006fc40 con 0x7f3500139500 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 5 monclient: authenticate success, global_id 354158 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _renew_subs 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350006dd00 con 0x7f3500139500 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34f8002040 con 0x7f3500139500 2024-09-16T06:58:41.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34f8003e90 con 0x7f3500139500 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34fd7fa700 10 monclient: dump: 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.551 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f34feffd700 4 set_mon_vals no callback set 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _renew_subs 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.534+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3500070190 con 0x7f3500139500 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f35051a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f350013bb10 con 0x7f3500138d00 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f34f8004070 con 0x7f3500139500 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f34f8044180 con 0x7f3500139500 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245788499 0 0) 0x7f350013bb10 con 0x7f3500138d00 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient(hunting): my global_id is 284827 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350006ef40 con 0x7f3500138d00 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34f77fe700 10 monclient: get_auth_request con 0x7f34e00365a0 auth_method 0 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4100870872 0 0) 0x7f350006ef40 con 0x7f3500138d00 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350013bb10 con 0x7f3500138d00 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34f0004890 con 0x7f3500138d00 2024-09-16T06:58:41.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient: dump: 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.556 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3308827581 0 0) 0x7f350013bb10 con 0x7f3500138d00 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3500070890 con 0x7f3500138d00 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f34f0006af0 con 0x7f3500138d00 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6789/0 conn(0x7f3500138d00 legacy=0x7f3500135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.538+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6789/0 conn(0x7f3500138d00 legacy=0x7f3500135800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f350006e9d0 con 0x7f350002c6b0 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785078589 0 0) 0x7f350006e9d0 con 0x7f350002c6b0 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient(hunting): my global_id is 284830 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f350013bb10 con 0x7f350002c6b0 2024-09-16T06:58:41.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 74213749 0 0) 0x7f350013bb10 con 0x7f350002c6b0 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350006e9d0 con 0x7f350002c6b0 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f34f0005020 con 0x7f350002c6b0 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient: dump: 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.560 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3948630871 0 0) 0x7f350006e9d0 con 0x7f350002c6b0 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3500099500 con 0x7f350002c6b0 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 2700428390) 0x7f34f0006410 con 0x7f350002c6b0 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6789/0 conn(0x7f350002c6b0 legacy=0x7f3500135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f34fd7fa700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6789/0 conn(0x7f350002c6b0 legacy=0x7f3500135f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:41.561 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.562 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.563 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.564 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:41.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:41.566 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:41.570 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:41.574 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.189:6812/3098810674 conn(0x7f34e00365a0 legacy=0x7f34e0038a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 10 monclient: shutdown 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 >> v1:172.21.15.76:6790/0 conn(0x7f3500139500 legacy=0x7f35001cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35049a2700 1 -- 172.21.15.76:0/3099205695 reap_dead start 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35049a2700 1 -- 172.21.15.76:0/3099205695 reap_dead start 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 shutdown_connections 2024-09-16T06:58:41.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 >> 172.21.15.76:0/3099205695 conn(0x7f35000a0030 msgr2=0x7f35000a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 shutdown_connections 2024-09-16T06:58:41.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.550+0000 7f35051a3700 1 -- 172.21.15.76:0/3099205695 wait complete. 2024-09-16T06:58:41.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:41.576 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:58:41.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39e1f12700 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39e1f12700 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39e1f12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f39bc005bc0 con 0x7f39dc09f430 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7f39d8012d80 con 0x7f39dc09f430 2024-09-16T06:58:41.604 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39d17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-16T06:58:41.605 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.598+0000 7f39d17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: monmap: 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:41.692837+0000 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:41.692837+0000 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 1 Processor -- start 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 1 -- start start 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: init 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-c con 0x7f8e500a27d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-b con 0x7f8e50071520 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-a con 0x7f8e500a32c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50070760 con 0x7f8e500a32c0 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5009d2c0 con 0x7f8e500a27d0 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5009a3a0 con 0x7f8e50071520 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient(hunting): authenticate will time out at 2068.826259s 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5611e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8e500a32c0 0x7f8e5013dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57774/0 (socket says 172.21.15.76:57774) 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5611e700 1 -- 172.21.15.76:0/2560421629 learned_addr learned my addr 172.21.15.76:0/2560421629 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919715603 0 0) 0x7f8e50070760 con 0x7f8e500a32c0 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): my global_id is 334365 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c0035d0 con 0x7f8e500a32c0 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104989864 0 0) 0x7f8e5009a3a0 con 0x7f8e50071520 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): my global_id is 354161 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e50070760 con 0x7f8e50071520 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755388012 0 0) 0x7f8e5009d2c0 con 0x7f8e500a27d0 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient(hunting): my global_id is 284833 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5009a3a0 con 0x7f8e500a27d0 2024-09-16T06:58:41.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526043144 0 0) 0x7f8e50070760 con 0x7f8e50071520 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 >> v1:172.21.15.189:6789/0 conn(0x7f8e500a27d0 legacy=0x7f8e500a2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 >> v1:172.21.15.76:6789/0 conn(0x7f8e500a32c0 legacy=0x7f8e5013dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 monclient: found mon.noname-b 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50070ba0 con 0x7f8e50071520 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 5 monclient: authenticate success, global_id 354161 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e4c001630 con 0x7f8e50071520 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: dump: 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e4c0018c0 con 0x7f8e50071520 2024-09-16T06:58:41.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 1 -- 172.21.15.76:0/2560421629 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e4c001c90 con 0x7f8e50071520 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5691f700 4 set_mon_vals no callback set 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.690+0000 7f8e5491b700 10 monclient: dump: 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.697 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: shutdown 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- 172.21.15.76:0/2560421629 >> v1:172.21.15.76:6790/0 conn(0x7f8e50071520 legacy=0x7f8e5013a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- 172.21.15.76:0/2560421629 shutdown_connections 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- 172.21.15.76:0/2560421629 >> 172.21.15.76:0/2560421629 conn(0x7f8e500a03c0 msgr2=0x7f8e500a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- 172.21.15.76:0/2560421629 shutdown_connections 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- 172.21.15.76:0/2560421629 wait complete. 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: monmap: 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.698 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.699 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 Processor -- start 2024-09-16T06:58:41.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- start start 2024-09-16T06:58:41.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: init 2024-09-16T06:58:41.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-b con 0x7f8e50071520 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-c con 0x7f8e500a27d0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient(hunting): picked mon.noname-a con 0x7f8e500a32c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50137370 con 0x7f8e500a32c0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50072520 con 0x7f8e50071520 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5009aa10 con 0x7f8e500a27d0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5691f700 10 monclient(hunting): authenticate will time out at 2068.830848s 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5511c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8e500a27d0 0x7f8e50135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40894/0 (socket says 172.21.15.76:40894) 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e5511c700 1 -- 172.21.15.76:0/2962719146 learned_addr learned my addr 172.21.15.76:0/2962719146 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156511001 0 0) 0x7f8e5009aa10 con 0x7f8e500a27d0 2024-09-16T06:58:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): my global_id is 354164 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e340035d0 con 0x7f8e500a27d0 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587550524 0 0) 0x7f8e50137370 con 0x7f8e500a32c0 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): my global_id is 334368 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5009aa10 con 0x7f8e500a32c0 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710154273 0 0) 0x7f8e50072520 con 0x7f8e50071520 2024-09-16T06:58:41.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): my global_id is 284836 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e50137370 con 0x7f8e50071520 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013372829 0 0) 0x7f8e340035d0 con 0x7f8e500a27d0 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e50072520 con 0x7f8e500a27d0 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e4c0016b0 con 0x7f8e500a27d0 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): dump: 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.705 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.694+0000 7f8e427fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 83983449 0 0) 0x7f8e50072520 con 0x7f8e500a27d0 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 >> v1:172.21.15.189:6789/0 conn(0x7f8e50071520 legacy=0x7f8e50134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 >> v1:172.21.15.76:6789/0 conn(0x7f8e500a32c0 legacy=0x7f8e501d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 monclient: found mon.c 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5013d550 con 0x7f8e500a27d0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 5 monclient: authenticate success, global_id 354164 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _renew_subs 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e50138940 con 0x7f8e500a27d0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _renew_subs 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e50136ed0 con 0x7f8e500a27d0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e4c003ac0 con 0x7f8e500a27d0 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8e4c003e50 con 0x7f8e500a27d0 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e437fe700 4 set_mon_vals no callback set 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 10 monclient: dump: 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.708 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8e4c004030 con 0x7f8e500a27d0 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e38005100 con 0x7f8e500a27d0 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8e4c044020 con 0x7f8e500a27d0 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.698+0000 7f8e5591d700 10 monclient: get_auth_request con 0x7f8e340363b0 auth_method 0 2024-09-16T06:58:41.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.706+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8e4c014b60 con 0x7f8e500a27d0 2024-09-16T06:58:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.706+0000 7f8e427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.706+0000 7f8e427fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T06:58:41.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-16T06:58:41.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f39bc005680 con 0x7f39dc09f430 2024-09-16T06:58:41.738 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39d17fa700 1 -- 172.21.15.189:0/2213854708 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7f39d8014500 con 0x7f39dc09f430 2024-09-16T06:58:41.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39d17fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-16T06:58:41.739 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39d17fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:58:41.740 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 >> v1:172.21.15.189:6812/3098810674 conn(0x7f39c80362f0 legacy=0x7f39c80387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 10 monclient: shutdown 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 >> v1:172.21.15.189:6789/0 conn(0x7f39dc09f430 legacy=0x7f39dc134490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 shutdown_connections 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.734+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 >> 172.21.15.189:0/2213854708 conn(0x7f39dc0a0b50 msgr2=0x7f39dc0a2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.738+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 shutdown_connections 2024-09-16T06:58:41.741 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.738+0000 7f39e1f12700 1 -- 172.21.15.189:0/2213854708 wait complete. 2024-09-16T06:58:41.751 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph health detail 2024-09-16T06:58:41.751 INFO:tasks.workunit.client.0.smithi189.stderr:+ grep client.bad 2024-09-16T06:58:41.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: get_monmap_and_config 2024-09-16T06:58:41.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.845 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: monmap: 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:58:41.845040+0000 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:58:41.845040+0000 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:41.846 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.848 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 Processor -- start 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- start start 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: init 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-c con 0x7f9b8c06da20 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-a con 0x7f9b8c0a2e60 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-b con 0x7f9b8c0a3270 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c06de00 con 0x7f9b8c0a2e60 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c09f460 con 0x7f9b8c06da20 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c0a38a0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.849 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient(hunting): authenticate will time out at 2074.357683s 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b90f9f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b8c06da20 0x7f9b8c0a2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32978/0 (socket says 172.21.15.189:32978) 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b90f9f700 1 -- 172.21.15.189:0/3398726523 learned_addr learned my addr 172.21.15.189:0/3398726523 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128585125 0 0) 0x7f9b8c09f460 con 0x7f9b8c06da20 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient(hunting): my global_id is 284839 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780035d0 con 0x7f9b8c06da20 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868917887 0 0) 0x7f9b780035d0 con 0x7f9b8c06da20 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 >> v1:172.21.15.76:6790/0 conn(0x7f9b8c0a3270 legacy=0x7f9b8c13e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 >> v1:172.21.15.76:6789/0 conn(0x7f9b8c0a2e60 legacy=0x7f9b8c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.850 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 monclient: found mon.noname-c 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0a21e0 con 0x7f9b8c06da20 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9b84001990 con 0x7f9b8c06da20 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: dump: 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.851 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 5 monclient: authenticate success, global_id 284839 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b84001cb0 con 0x7f9b8c06da20 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.852 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 1 -- 172.21.15.189:0/3398726523 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9b84002040 con 0x7f9b8c06da20 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b8b7fe700 10 monclient: dump: 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.853 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 4 set_mon_vals no callback set 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 10 monclient: shutdown 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- 172.21.15.189:0/3398726523 >> v1:172.21.15.189:6789/0 conn(0x7f9b8c06da20 legacy=0x7f9b8c0a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- 172.21.15.189:0/3398726523 shutdown_connections 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.842+0000 7f9b93203700 1 -- 172.21.15.189:0/3398726523 >> 172.21.15.189:0/3398726523 conn(0x7f9b8c074d70 msgr2=0x7f9b8c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- 172.21.15.189:0/3398726523 shutdown_connections 2024-09-16T06:58:41.854 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- 172.21.15.189:0/3398726523 wait complete. 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient: build_initial_monmap 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient: monmap: 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:58:41.858 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 Processor -- start 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- start start 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient: init 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-a con 0x7f9b8c06da20 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-c con 0x7f9b8c0a2e60 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient(hunting): picked mon.noname-b con 0x7f9b8c0a3270 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c1cace0 con 0x7f9b8c06da20 2024-09-16T06:58:41.859 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c09f0b0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c135710 con 0x7f9b8c0a2e60 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b93203700 10 monclient(hunting): authenticate will time out at 2074.362868s 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b917a0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b8c0a3270 0x7f9b8c1d2830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:32986/0 (socket says 172.21.15.189:32986) 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b917a0700 1 -- 172.21.15.189:0/900606188 learned_addr learned my addr 172.21.15.189:0/900606188 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551456132 0 0) 0x7f9b8c09f0b0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.860 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): my global_id is 284842 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b6c0035d0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176875976 0 0) 0x7f9b6c0035d0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b8c09f0b0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9b7c0018f0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.861 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915392334 0 0) 0x7f9b8c135710 con 0x7f9b8c0a2e60 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): my global_id is 354170 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b6c0035d0 con 0x7f9b8c0a2e60 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146913036 0 0) 0x7f9b8c1cace0 con 0x7f9b8c06da20 2024-09-16T06:58:41.862 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 10 monclient(hunting): my global_id is 334374 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.846+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b8c135710 con 0x7f9b8c06da20 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590059228 0 0) 0x7f9b8c09f0b0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 >> v1:172.21.15.76:6790/0 conn(0x7f9b8c0a2e60 legacy=0x7f9b8c1cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 >> v1:172.21.15.76:6789/0 conn(0x7f9b8c06da20 legacy=0x7f9b8c1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 monclient: found mon.b 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.863 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0a7360 con 0x7f9b8c0a3270 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 5 monclient: authenticate success, global_id 284842 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _renew_subs 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c122080 con 0x7f9b8c0a3270 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _renew_subs 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.864 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c120950 con 0x7f9b8c0a3270 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b7c001cc0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b8affd700 4 set_mon_vals no callback set 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9b7c003b90 con 0x7f9b8c0a3270 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 10 monclient: dump: 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.865 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9b7c003d70 con 0x7f9b8c0a3270 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b70005100 con 0x7f9b8c0a3270 2024-09-16T06:58:41.866 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f9b7c043ed0 con 0x7f9b8c0a3270 2024-09-16T06:58:41.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.850+0000 7f9b90f9f700 10 monclient: get_auth_request con 0x7f9b6c036400 auth_method 0 2024-09-16T06:58:41.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.858+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f9b7c014690 con 0x7f9b8c0a3270 2024-09-16T06:58:41.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.858+0000 7f9b897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:41.867 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:41.858+0000 7f9b897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:41.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e5691f700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:58:41.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e5691f700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:58:41.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e5691f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:41.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8e38005bc0 con 0x7f8e500a27d0 2024-09-16T06:58:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e427fc700 1 -- 172.21.15.76:0/2962719146 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f8e4c014750 con 0x7f8e500a27d0 2024-09-16T06:58:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:58:41.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.938+0000 7f8e427fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=13,logm=47} crc {auth=4191894895,config=1097430493,health=433167869,kv=73414735,logm=242826913}) 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=3449163291,config=1097430493,health=2606195453,kv=3901684014,logm=3962100202}) 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=13,logm=47} crc {auth=4191894895,config=1097430493,health=433167869,kv=73414735,logm=242826913}) 2024-09-16T06:58:41.939 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.934+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=687800372,config=1097430493,health=2606195453,kv=1001081567,logm=1479492227}) 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8e340363b0 legacy=0x7f8e34038870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 10 monclient: shutdown 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 >> v1:172.21.15.76:6790/0 conn(0x7f8e500a27d0 legacy=0x7f8e50135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 shutdown_connections 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 >> 172.21.15.76:0/2962719146 conn(0x7f8e500a03c0 msgr2=0x7f8e5009bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 shutdown_connections 2024-09-16T06:58:41.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:41.942+0000 7f8e5691f700 1 -- 172.21.15.76:0/2962719146 wait complete. 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682646163}) 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791780772}) 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682646163}) 2024-09-16T06:58:41.942 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=262357360}) 2024-09-16T06:58:41.944 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.944 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657690990}) 2024-09-16T06:58:41.944 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522636973}) 2024-09-16T06:58:41.944 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.945 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657690990}) 2024-09-16T06:58:41.945 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.938+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977108908}) 2024-09-16T06:58:41.947 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.947 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819264016}) 2024-09-16T06:58:41.947 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346156306}) 2024-09-16T06:58:41.947 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.948 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819264016}) 2024-09-16T06:58:41.948 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.942+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317809697}) 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503141482}) 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930842222}) 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503141482}) 2024-09-16T06:58:41.950 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909081036}) 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064127911}) 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861500474}) 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064127911}) 2024-09-16T06:58:41.951 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209130315}) 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259457963}) 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737992769}) 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259457963}) 2024-09-16T06:58:41.952 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.946+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452508436}) 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967009401}) 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783550303}) 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967009401}) 2024-09-16T06:58:41.953 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434313527}) 2024-09-16T06:58:41.954 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.954 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647926453}) 2024-09-16T06:58:41.955 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751384390}) 2024-09-16T06:58:41.955 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.955 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647926453}) 2024-09-16T06:58:41.955 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708767337}) 2024-09-16T06:58:41.955 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360986417}) 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320527677}) 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360986417}) 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389583722}) 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.956 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059553574}) 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397206580}) 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059553574}) 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.950+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633926620}) 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.957 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446913742}) 2024-09-16T06:58:41.958 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581685014}) 2024-09-16T06:58:41.958 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.958 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446913742}) 2024-09-16T06:58:41.958 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423240482}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668355907}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197660181}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668355907}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173907482}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186693110}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632575747}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186693110}) 2024-09-16T06:58:41.959 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906700885}) 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357031995}) 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406527644}) 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357031995}) 2024-09-16T06:58:41.960 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.954+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465879688}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397850362}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794381786}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397850362}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585788886}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164914887}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108566306}) 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.962 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164914887}) 2024-09-16T06:58:41.963 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3906805259}) 2024-09-16T06:58:41.963 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.963 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921512174}) 2024-09-16T06:58:41.963 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577643006}) 2024-09-16T06:58:41.963 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.964 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921512174}) 2024-09-16T06:58:41.964 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936333779}) 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213544371}) 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929713836}) 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213544371}) 2024-09-16T06:58:41.965 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.958+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950481112}) 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188062321}) 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272046483}) 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188062321}) 2024-09-16T06:58:41.966 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=26730639}) 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099145768}) 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161589424}) 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099145768}) 2024-09-16T06:58:41.968 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.962+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393208228}) 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251882567}) 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183535980}) 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251882567}) 2024-09-16T06:58:41.969 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096066339}) 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086084897}) 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244339004}) 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086084897}) 2024-09-16T06:58:41.971 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1596754651}) 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759720883}) 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643545334}) 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759720883}) 2024-09-16T06:58:41.973 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.966+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529072900}) 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867563921}) 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65307451}) 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867563921}) 2024-09-16T06:58:41.974 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93222010}) 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004110858}) 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252590481}) 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004110858}) 2024-09-16T06:58:41.975 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379499965}) 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713322575}) 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294319595}) 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713322575}) 2024-09-16T06:58:41.976 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.970+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231634672}) 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728395914}) 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299951738}) 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728395914}) 2024-09-16T06:58:41.977 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517163337}) 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3246135504}) 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315879803}) 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3246135504}) 2024-09-16T06:58:41.978 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108203124}) 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970384863}) 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331828256}) 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970384863}) 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1345198788}) 2024-09-16T06:58:41.979 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.980 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082295290}) 2024-09-16T06:58:41.980 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147202673}) 2024-09-16T06:58:41.980 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.980 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082295290}) 2024-09-16T06:58:41.980 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.974+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803938264}) 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916731863}) 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835171024}) 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916731863}) 2024-09-16T06:58:41.981 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426632906}) 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279612460}) 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644142036}) 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279612460}) 2024-09-16T06:58:41.982 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2295321228}) 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265820360}) 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983117804}) 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265820360}) 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167111340}) 2024-09-16T06:58:41.983 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.984 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062777115}) 2024-09-16T06:58:41.984 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127135713}) 2024-09-16T06:58:41.984 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.984 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062777115}) 2024-09-16T06:58:41.984 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.978+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266460921}) 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3108422845}) 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910646496}) 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3108422845}) 2024-09-16T06:58:41.985 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=674159446}) 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939178330}) 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181776993}) 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939178330}) 2024-09-16T06:58:41.986 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610023084}) 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314881831}) 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3052924063}) 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314881831}) 2024-09-16T06:58:41.987 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223396791}) 2024-09-16T06:58:41.988 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.988 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759792133}) 2024-09-16T06:58:41.988 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329146025}) 2024-09-16T06:58:41.988 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.989 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759792133}) 2024-09-16T06:58:41.989 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.982+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=333996032}) 2024-09-16T06:58:41.989 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.989 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939927516}) 2024-09-16T06:58:41.989 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373589761}) 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939927516}) 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=203496008}) 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745325529}) 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2684801359}) 2024-09-16T06:58:41.990 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745325529}) 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532667349}) 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80287865}) 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855817378}) 2024-09-16T06:58:41.991 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.992 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80287865}) 2024-09-16T06:58:41.992 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2289294015}) 2024-09-16T06:58:41.992 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.992 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261335396}) 2024-09-16T06:58:41.992 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588458973}) 2024-09-16T06:58:41.993 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.993 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261335396}) 2024-09-16T06:58:41.993 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.986+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3560696071}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255072395}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804180140}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255072395}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733111744}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925509923}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826515714}) 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.994 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925509923}) 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2350411583}) 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2265272155}) 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739003910}) 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.995 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2265272155}) 2024-09-16T06:58:41.996 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2034688841}) 2024-09-16T06:58:41.996 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.996 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850722844}) 2024-09-16T06:58:41.997 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632483829}) 2024-09-16T06:58:41.997 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.997 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850722844}) 2024-09-16T06:58:41.997 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.990+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021860665}) 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913725106}) 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367806380}) 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913725106}) 2024-09-16T06:58:41.998 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166988252}) 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=415366216}) 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759945521}) 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=415366216}) 2024-09-16T06:58:41.999 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250599860}) 2024-09-16T06:58:42.000 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.000 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212860791}) 2024-09-16T06:58:42.001 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052767189}) 2024-09-16T06:58:42.001 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.001 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212860791}) 2024-09-16T06:58:42.001 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.994+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041724108}) 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804807230}) 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419073454}) 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804807230}) 2024-09-16T06:58:42.002 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801804760}) 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4110608954}) 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399370852}) 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4110608954}) 2024-09-16T06:58:42.003 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849390162}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634113411}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3313727857}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634113411}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160612115}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552413403}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183607207}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552413403}) 2024-09-16T06:58:42.005 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:41.998+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871700117}) 2024-09-16T06:58:42.006 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119715084}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964431811}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119715084}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=938948516}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1516879536}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115052596}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1516879536}) 2024-09-16T06:58:42.007 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1877828456}) 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384889964}) 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614260199}) 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384889964}) 2024-09-16T06:58:42.008 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.002+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2983866032}) 2024-09-16T06:58:42.009 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.009 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465425239}) 2024-09-16T06:58:42.009 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007937095}) 2024-09-16T06:58:42.009 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.009 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465425239}) 2024-09-16T06:58:42.010 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537223581}) 2024-09-16T06:58:42.010 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.010 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638719621}) 2024-09-16T06:58:42.010 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556205358}) 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638719621}) 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2370934089}) 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984020501}) 2024-09-16T06:58:42.011 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420666922}) 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984020501}) 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149118215}) 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082904931}) 2024-09-16T06:58:42.012 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205409265}) 2024-09-16T06:58:42.013 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.013 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082904931}) 2024-09-16T06:58:42.013 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.006+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745544247}) 2024-09-16T06:58:42.013 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.013 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231979540}) 2024-09-16T06:58:42.014 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111585484}) 2024-09-16T06:58:42.014 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.014 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231979540}) 2024-09-16T06:58:42.014 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554298965}) 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685004557}) 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410195476}) 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685004557}) 2024-09-16T06:58:42.015 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655775237}) 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3578105712}) 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617160636}) 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3578105712}) 2024-09-16T06:58:42.016 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803612257}) 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080213571}) 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2609055549}) 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080213571}) 2024-09-16T06:58:42.017 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.010+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1974214}) 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957167471}) 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773999429}) 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957167471}) 2024-09-16T06:58:42.018 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024907564}) 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804505571}) 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773636312}) 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804505571}) 2024-09-16T06:58:42.019 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720506246}) 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083426063}) 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160737213}) 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083426063}) 2024-09-16T06:58:42.020 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.014+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101336175}) 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=74140529}) 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104286744}) 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=74140529}) 2024-09-16T06:58:42.021 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622478363}) 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685169065}) 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167501380}) 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685169065}) 2024-09-16T06:58:42.022 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821530869}) 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342136301}) 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588112593}) 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342136301}) 2024-09-16T06:58:42.023 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294107121}) 2024-09-16T06:58:42.024 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.024 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874959799}) 2024-09-16T06:58:42.024 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576340538}) 2024-09-16T06:58:42.024 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.025 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874959799}) 2024-09-16T06:58:42.025 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.018+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404247199}) 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526178329}) 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684850504}) 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526178329}) 2024-09-16T06:58:42.026 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951288765}) 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299728470}) 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176525443}) 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299728470}) 2024-09-16T06:58:42.027 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2751334737}) 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143189743}) 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810236583}) 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143189743}) 2024-09-16T06:58:42.028 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.022+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=528482770}) 2024-09-16T06:58:42.029 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.029 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3065690229}) 2024-09-16T06:58:42.029 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168626776}) 2024-09-16T06:58:42.029 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.029 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3065690229}) 2024-09-16T06:58:42.030 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643917996}) 2024-09-16T06:58:42.030 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.030 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2665471374}) 2024-09-16T06:58:42.030 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058878743}) 2024-09-16T06:58:42.031 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.031 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2665471374}) 2024-09-16T06:58:42.031 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=558378716}) 2024-09-16T06:58:42.031 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2681094559}) 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884415435}) 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2681094559}) 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.026+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622370390}) 2024-09-16T06:58:42.032 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.033 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839611052}) 2024-09-16T06:58:42.033 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168484968}) 2024-09-16T06:58:42.033 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.033 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839611052}) 2024-09-16T06:58:42.033 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630833693}) 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243426801}) 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814791131}) 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243426801}) 2024-09-16T06:58:42.034 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1632504675}) 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798823112}) 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697171538}) 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798823112}) 2024-09-16T06:58:42.035 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831323499}) 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153941988}) 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299105328}) 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153941988}) 2024-09-16T06:58:42.036 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.030+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093860625}) 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420301066}) 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3142295955}) 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420301066}) 2024-09-16T06:58:42.037 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410003912}) 2024-09-16T06:58:42.038 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825242400}) 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409703732}) 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825242400}) 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945142930}) 2024-09-16T06:58:42.039 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.040 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745731452}) 2024-09-16T06:58:42.040 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884913223}) 2024-09-16T06:58:42.040 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.040 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745731452}) 2024-09-16T06:58:42.040 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.034+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767737457}) 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2210963107}) 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193625771}) 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2210963107}) 2024-09-16T06:58:42.041 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1647617781}) 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821159574}) 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956451672}) 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821159574}) 2024-09-16T06:58:42.042 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153289571}) 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793345880}) 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354707179}) 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793345880}) 2024-09-16T06:58:42.043 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715723865}) 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254078401}) 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639097095}) 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254078401}) 2024-09-16T06:58:42.044 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.038+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936764865}) 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465309821}) 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895025244}) 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465309821}) 2024-09-16T06:58:42.045 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003726925}) 2024-09-16T06:58:42.046 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.046 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677681020}) 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923818902}) 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677681020}) 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656295092}) 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.047 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543254877}) 2024-09-16T06:58:42.048 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188948216}) 2024-09-16T06:58:42.048 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.048 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543254877}) 2024-09-16T06:58:42.048 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4133889847}) 2024-09-16T06:58:42.048 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.049 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641433096}) 2024-09-16T06:58:42.049 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225414770}) 2024-09-16T06:58:42.049 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.049 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641433096}) 2024-09-16T06:58:42.049 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.042+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829192429}) 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114060770}) 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684724530}) 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114060770}) 2024-09-16T06:58:42.050 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616733261}) 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965197075}) 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964396669}) 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965197075}) 2024-09-16T06:58:42.051 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293886397}) 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972123595}) 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232191478}) 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972123595}) 2024-09-16T06:58:42.052 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106303750}) 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.046+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695639491}) 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173922649}) 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695639491}) 2024-09-16T06:58:42.053 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018452551}) 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629052578}) 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138237038}) 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629052578}) 2024-09-16T06:58:42.054 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333097029}) 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449937887}) 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307445506}) 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449937887}) 2024-09-16T06:58:42.055 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439424679}) 2024-09-16T06:58:42.056 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.056 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2144241857}) 2024-09-16T06:58:42.056 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487206773}) 2024-09-16T06:58:42.056 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.056 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2144241857}) 2024-09-16T06:58:42.057 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.050+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201637181}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282177023}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462184054}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282177023}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3603962087}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30526047}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967639857}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30526047}) 2024-09-16T06:58:42.058 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868094019}) 2024-09-16T06:58:42.059 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.059 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523156368}) 2024-09-16T06:58:42.059 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100301174}) 2024-09-16T06:58:42.059 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.059 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523156368}) 2024-09-16T06:58:42.060 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=710715393}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527580501}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931308195}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1527580501}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.054+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487817348}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560468876}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4083046411}) 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.061 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3560468876}) 2024-09-16T06:58:42.062 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042307408}) 2024-09-16T06:58:42.062 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704684698}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996747082}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704684698}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056963174}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614681205}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70720809}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614681205}) 2024-09-16T06:58:42.063 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458380702}) 2024-09-16T06:58:42.064 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886852698}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422999279}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886852698}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371175625}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.058+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539741720}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2622467625}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539741720}) 2024-09-16T06:58:42.065 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808456070}) 2024-09-16T06:58:42.066 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717506262}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942818042}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717506262}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884510106}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2274532024}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426822627}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2274532024}) 2024-09-16T06:58:42.067 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880762747}) 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782297016}) 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741277202}) 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782297016}) 2024-09-16T06:58:42.068 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.062+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230792378}) 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796324648}) 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121171292}) 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796324648}) 2024-09-16T06:58:42.069 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4235088424}) 2024-09-16T06:58:42.070 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.070 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575900175}) 2024-09-16T06:58:42.070 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486919283}) 2024-09-16T06:58:42.070 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.071 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575900175}) 2024-09-16T06:58:42.071 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633825842}) 2024-09-16T06:58:42.071 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.071 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918806828}) 2024-09-16T06:58:42.072 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626272994}) 2024-09-16T06:58:42.072 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.072 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918806828}) 2024-09-16T06:58:42.072 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=13010152}) 2024-09-16T06:58:42.072 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.073 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454672045}) 2024-09-16T06:58:42.073 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386368731}) 2024-09-16T06:58:42.073 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.073 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454672045}) 2024-09-16T06:58:42.073 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.066+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504260702}) 2024-09-16T06:58:42.074 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.074 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1315873325}) 2024-09-16T06:58:42.074 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=672624581}) 2024-09-16T06:58:42.074 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.074 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1315873325}) 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000501399}) 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113067586}) 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23608158}) 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113067586}) 2024-09-16T06:58:42.075 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704051466}) 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555865583}) 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165755674}) 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555865583}) 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100555708}) 2024-09-16T06:58:42.076 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516218850}) 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976576136}) 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516218850}) 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.070+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403787724}) 2024-09-16T06:58:42.077 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.078 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268304644}) 2024-09-16T06:58:42.078 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643921005}) 2024-09-16T06:58:42.078 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.078 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268304644}) 2024-09-16T06:58:42.078 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1357548931}) 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725254994}) 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786774861}) 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725254994}) 2024-09-16T06:58:42.079 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883073449}) 2024-09-16T06:58:42.080 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.080 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704880002}) 2024-09-16T06:58:42.080 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482236838}) 2024-09-16T06:58:42.080 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.080 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704880002}) 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963677539}) 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2645706837}) 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824234239}) 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2645706837}) 2024-09-16T06:58:42.081 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.074+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167210325}) 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850464411}) 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039444361}) 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850464411}) 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184506178}) 2024-09-16T06:58:42.082 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.083 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699421894}) 2024-09-16T06:58:42.083 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440629826}) 2024-09-16T06:58:42.083 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.083 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699421894}) 2024-09-16T06:58:42.083 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608643529}) 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=3139611441,mdsmap=1531151340,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2282629782,mon_config_key=3810543884,monmap=1820574642,osd_metadata=304085106,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4200930638}) 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1977530298,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2707839295,mon_config_key=4229390698,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227}) 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=20,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=1} crc {logm=3139611441,mdsmap=1531151340,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2282629782,mon_config_key=3810543884,monmap=1820574642,osd_metadata=304085106,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=4200930638}) 2024-09-16T06:58:42.084 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.078+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1649660498,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2707839295,mon_config_key=1247158487,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227}) 2024-09-16T06:58:42.091 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.092 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=2260189}) 2024-09-16T06:58:42.092 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=1732975789}) 2024-09-16T06:58:42.092 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:58:42.092 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=2260189}) 2024-09-16T06:58:42.092 INFO:tasks.ceph.mon.b.smithi189.stderr:2024-09-16T06:58:42.086+0000 7f0f3b0f6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=2240742001}) 2024-09-16T06:58:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.090+0000 7f9b93203700 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.090+0000 7f9b93203700 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.090+0000 7f9b93203700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.096 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.090+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9b700053b0 con 0x7f9b8c0a3270 2024-09-16T06:58:42.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.094+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7f9b7c012d30 con 0x7f9b8c0a3270 2024-09-16T06:58:42.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.094+0000 7f9b897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.097 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.094+0000 7f9b897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9b70005680 con 0x7f9b8c0a3270 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b897fa700 1 -- 172.21.15.189:0/900606188 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7f9b7c0144b0 con 0x7f9b8c0a3270 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b897fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T06:58:42.245 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b897fa700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:58:42.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9b6c036400 legacy=0x7f9b6c0388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.247 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 10 monclient: shutdown 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 >> v1:172.21.15.189:6789/0 conn(0x7f9b8c0a3270 legacy=0x7f9b8c1d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 shutdown_connections 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 >> 172.21.15.189:0/900606188 conn(0x7f9b8c074d70 msgr2=0x7f9b8c1bd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 shutdown_connections 2024-09-16T06:58:42.248 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.242+0000 7f9b93203700 1 -- 172.21.15.189:0/900606188 wait complete. 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T06:58:42.261 INFO:tasks.workunit.client.0.smithi189.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T06:58:42.262 INFO:tasks.workunit.client.0.smithi189.stderr:+ ceph auth rm client.bad 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 10 monclient: get_monmap_and_config 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 10 monclient: build_initial_monmap 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 10 monclient: monmap: 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:58:42.354461+0000 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:58:42.354461+0000 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:42.355 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:42.356 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 1 Processor -- start 2024-09-16T06:58:42.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 1 -- start start 2024-09-16T06:58:42.356 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.350+0000 7fc23ada5700 10 monclient: init 2024-09-16T06:58:42.358 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-a con 0x7fc234074af0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-c con 0x7fc2340a1740 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-b con 0x7fc23413b1a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234131e30 con 0x7fc234074af0 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234074ed0 con 0x7fc2340a1740 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234099ed0 con 0x7fc23413b1a0 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient(hunting): authenticate will time out at 2074.868455s 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc2395a2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc2340a1740 0x7fc2340a1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33002/0 (socket says 172.21.15.189:33002) 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc2395a2700 1 -- 172.21.15.189:0/3349302429 learned_addr learned my addr 172.21.15.189:0/3349302429 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820403206 0 0) 0x7fc234074ed0 con 0x7fc2340a1740 2024-09-16T06:58:42.359 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): my global_id is 284845 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2200035d0 con 0x7fc2340a1740 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925393314 0 0) 0x7fc234131e30 con 0x7fc234074af0 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): my global_id is 334377 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc234074ed0 con 0x7fc234074af0 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981708495 0 0) 0x7fc234099ed0 con 0x7fc23413b1a0 2024-09-16T06:58:42.360 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient(hunting): my global_id is 354173 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc234131e30 con 0x7fc23413b1a0 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661080393 0 0) 0x7fc2200035d0 con 0x7fc2340a1740 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 >> v1:172.21.15.76:6790/0 conn(0x7fc23413b1a0 legacy=0x7fc23413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 >> v1:172.21.15.76:6789/0 conn(0x7fc234074af0 legacy=0x7fc234132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: _finish_hunting 0 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 monclient: found mon.noname-c 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234066dd0 con 0x7fc2340a1740 2024-09-16T06:58:42.361 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: _finish_auth 0 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: _check_auth_tickets 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 5 monclient: authenticate success, global_id 284845 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc230001610 con 0x7fc2340a1740 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: dump: 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:42.362 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc230001880 con 0x7fc2340a1740 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 1 -- 172.21.15.189:0/3349302429 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc230001c10 con 0x7fc2340a1740 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 4 set_mon_vals no callback set 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:42.363 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc238da1700 10 monclient: dump: 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:42.364 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: shutdown 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- 172.21.15.189:0/3349302429 >> v1:172.21.15.189:6789/0 conn(0x7fc2340a1740 legacy=0x7fc2340a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- 172.21.15.189:0/3349302429 shutdown_connections 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- 172.21.15.189:0/3349302429 >> 172.21.15.189:0/3349302429 conn(0x7fc234073f20 msgr2=0x7fc234074330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- 172.21.15.189:0/3349302429 shutdown_connections 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 1 -- 172.21.15.189:0/3349302429 wait complete. 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: build_initial_monmap 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.354+0000 7fc23ada5700 10 monclient: monmap: 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 0 2024-09-16T06:58:42.369 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 0.000000 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:created 0.000000 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 1 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 Processor -- start 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- start start 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: init 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-b con 0x7fc234074af0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:42.370 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-a con 0x7fc2340a1740 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient(hunting): picked mon.noname-c con 0x7fc23413b1a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234132ea0 con 0x7fc2340a1740 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23412ffb0 con 0x7fc234074af0 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23413af70 con 0x7fc23413b1a0 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient(hunting): authenticate will time out at 2074.872933s 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc239da3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc234074af0 0x7fc23409dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:33014/0 (socket says 172.21.15.189:33014) 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc239da3700 1 -- 172.21.15.189:0/3610980569 learned_addr learned my addr 172.21.15.189:0/3610980569 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063908453 0 0) 0x7fc23412ffb0 con 0x7fc234074af0 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): my global_id is 284848 2024-09-16T06:58:42.371 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2180035d0 con 0x7fc234074af0 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290607659 0 0) 0x7fc23413af70 con 0x7fc23413b1a0 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): my global_id is 354176 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23412ffb0 con 0x7fc23413b1a0 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267644927 0 0) 0x7fc234132ea0 con 0x7fc2340a1740 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): my global_id is 334380 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23413af70 con 0x7fc2340a1740 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2813880270 0 0) 0x7fc2180035d0 con 0x7fc234074af0 2024-09-16T06:58:42.372 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc234132ea0 con 0x7fc234074af0 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc2240018f0 con 0x7fc234074af0 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): dump: 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:42.373 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1352215140 0 0) 0x7fc234132ea0 con 0x7fc234074af0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 >> v1:172.21.15.76:6790/0 conn(0x7fc23413b1a0 legacy=0x7fc23409ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 >> v1:172.21.15.76:6789/0 conn(0x7fc2340a1740 legacy=0x7fc23409e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 monclient: found mon.b 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23406e320 con 0x7fc234074af0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:42.374 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 5 monclient: authenticate success, global_id 284848 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _renew_subs 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc234132bc0 con 0x7fc234074af0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _renew_subs 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2340a0250 con 0x7fc234074af0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc224001c80 con 0x7fc234074af0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fc224003b10 con 0x7fc234074af0 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc23ada5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:42.375 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22a7fc700 10 monclient: dump: 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:epoch 1 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:election_strategy: 3 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr: 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fc224003cf0 con 0x7fc234074af0 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.358+0000 7fc22bfff700 4 set_mon_vals no callback set 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fc224043ed0 con 0x7fc234074af0 2024-09-16T06:58:42.376 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc23ada5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc23ada5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2340008d0 con 0x7fc234074af0 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.362+0000 7fc2395a2700 10 monclient: get_auth_request con 0x7fc218036390 auth_method 0 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.370+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fc224014890 con 0x7fc234074af0 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.370+0000 7fc22a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:42.377 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.370+0000 7fc22a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:42.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.586+0000 7fc23ada5700 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.586+0000 7fc23ada5700 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.591 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.586+0000 7fc23ada5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.592 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.586+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fc23413ecb0 con 0x7fc234074af0 2024-09-16T06:58:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.590+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v51) v1 ==== 79+0+0 (unknown 1132354964 0 0) 0x7fc224012d00 con 0x7fc234074af0 2024-09-16T06:58:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.590+0000 7fc22a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.596 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.590+0000 7fc22a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:42.745 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7fc2341d2f30 con 0x7fc234074af0 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc22a7fc700 1 -- 172.21.15.189:0/3610980569 <== mon.1 v1:172.21.15.189:6789/0 11 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v51) v1 ==== 79+0+0 (unknown 1132354964 0 0) 0x7fc224014480 con 0x7fc234074af0 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc22a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T06:58:42.746 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc22a7fc700 10 monclient: _finish_command 3 = system:0 2024-09-16T06:58:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 >> v1:172.21.15.189:6812/3098810674 conn(0x7fc218036390 legacy=0x7fc218038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 10 monclient: shutdown 2024-09-16T06:58:42.748 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 >> v1:172.21.15.189:6789/0 conn(0x7fc234074af0 legacy=0x7fc23409dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 shutdown_connections 2024-09-16T06:58:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 >> 172.21.15.189:0/3610980569 conn(0x7fc234073f20 msgr2=0x7fc23413d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 shutdown_connections 2024-09-16T06:58:42.749 INFO:tasks.workunit.client.0.smithi189.stderr:2024-09-16T06:58:42.742+0000 7fc23ada5700 1 -- 172.21.15.189:0/3610980569 wait complete. 2024-09-16T06:58:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-16T06:58:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-16T06:58:42.762 INFO:tasks.workunit.client.0.smithi189.stderr:+ expected_ret=1 2024-09-16T06:58:42.762 INFO:tasks.workunit.client.0.smithi189.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-16T06:58:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T06:58:42.763 INFO:tasks.workunit.client.0.smithi189.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T06:58:42.956 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:58:42.956 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:58:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ ret=1 2024-09-16T06:58:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ [[ 1 -ne 1 ]] 2024-09-16T06:58:43.069 INFO:tasks.workunit.client.0.smithi189.stderr:+ return 0 2024-09-16T06:58:43.069 INFO:tasks.workunit.client.0.smithi189.stdout:OK 2024-09-16T06:58:43.070 INFO:tasks.workunit.client.0.smithi189.stderr:+ echo OK 2024-09-16T06:58:43.070 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:58:43.070 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.074+0000 7f5d91a7a700 10 monclient: get_monmap_and_config 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.074+0000 7f5d91a7a700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.074+0000 7f5d91a7a700 10 monclient: monmap: 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:43.079731+0000 2024-09-16T06:58:43.074 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:43.079731+0000 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 1 Processor -- start 2024-09-16T06:58:43.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 1 -- start start 2024-09-16T06:58:43.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient: init 2024-09-16T06:58:43.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:43.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c0a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c1334a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c133910 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0a1620 con 0x7f5d8c1334a0 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0a4c20 con 0x7f5d8c0a4840 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c071f10 con 0x7f5d8c133910 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 10 monclient(hunting): authenticate will time out at 2070.214711s 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91279700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5d8c133910 0x7f5d8c13b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40898/0 (socket says 172.21.15.76:40898) 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5d8c1334a0 0x7f5d8c1380b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57778/0 (socket says 172.21.15.76:57778) 2024-09-16T06:58:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91279700 1 -- 172.21.15.76:0/2269754781 learned_addr learned my addr 172.21.15.76:0/2269754781 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527644769 0 0) 0x7f5d8c071f10 con 0x7f5d8c133910 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): my global_id is 354185 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0035d0 con 0x7f5d8c133910 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169974641 0 0) 0x7f5d8c0a1620 con 0x7f5d8c1334a0 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): my global_id is 334389 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c071f10 con 0x7f5d8c1334a0 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514231764 0 0) 0x7f5d8c0a4c20 con 0x7f5d8c0a4840 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient(hunting): my global_id is 284857 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c0a1620 con 0x7f5d8c0a4840 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219490247 0 0) 0x7f5d7c0035d0 con 0x7f5d8c133910 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 >> v1:172.21.15.189:6789/0 conn(0x7f5d8c0a4840 legacy=0x7f5d8c132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 >> v1:172.21.15.76:6789/0 conn(0x7f5d8c1334a0 legacy=0x7f5d8c1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 monclient: found mon.noname-b 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c09e1e0 con 0x7f5d8c133910 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 5 monclient: authenticate success, global_id 354185 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d91a7a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d84001990 con 0x7f5d8c133910 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.078+0000 7f5d8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: dump: 2024-09-16T06:58:43.085 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d84001bb0 con 0x7f5d8c133910 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 4 set_mon_vals no callback set 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 1 -- 172.21.15.76:0/2269754781 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d84001f40 con 0x7f5d8c133910 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d8b7fe700 10 monclient: dump: 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.087 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: shutdown 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2269754781 >> v1:172.21.15.76:6790/0 conn(0x7f5d8c133910 legacy=0x7f5d8c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2269754781 shutdown_connections 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2269754781 >> 172.21.15.76:0/2269754781 conn(0x7f5d8c0a4430 msgr2=0x7f5d8c0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2269754781 shutdown_connections 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2269754781 wait complete. 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: monmap: 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:43.088 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 Processor -- start 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- start start 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: init 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c0a4840 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c1334a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c133910 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c13c6b0 con 0x7f5d8c0a4840 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c133200 con 0x7f5d8c133910 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c132cb0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d91a7a700 10 monclient(hunting): authenticate will time out at 2070.218469s 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d90a78700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5d8c0a4840 0x7f5d8c1389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57782/0 (socket says 172.21.15.76:57782) 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d90a78700 1 -- 172.21.15.76:0/2196333698 learned_addr learned my addr 172.21.15.76:0/2196333698 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168452785 0 0) 0x7f5d8c13c6b0 con 0x7f5d8c0a4840 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): my global_id is 334392 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d700035d0 con 0x7f5d8c0a4840 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074656939 0 0) 0x7f5d8c132cb0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): my global_id is 354188 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c13c6b0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322174223 0 0) 0x7f5d700035d0 con 0x7f5d8c0a4840 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d8c132cb0 con 0x7f5d8c0a4840 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d80001990 con 0x7f5d8c0a4840 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2000807020 0 0) 0x7f5d8c13c6b0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d700035d0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656720885 0 0) 0x7f5d8c133200 con 0x7f5d8c133910 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): my global_id is 284860 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c13c6b0 con 0x7f5d8c133910 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d780018b0 con 0x7f5d8c1334a0 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.082+0000 7f5d897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347809485 0 0) 0x7f5d8c13c6b0 con 0x7f5d8c133910 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d8c133200 con 0x7f5d8c133910 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d84001d20 con 0x7f5d8c133910 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.094 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.095 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.097 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-16T06:58:43.097 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T06:58:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 534338606 0 0) 0x7f5d8c133200 con 0x7f5d8c133910 2024-09-16T06:58:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 >> v1:172.21.15.76:6790/0 conn(0x7f5d8c1334a0 legacy=0x7f5d8c1390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 >> v1:172.21.15.76:6789/0 conn(0x7f5d8c0a4840 legacy=0x7f5d8c1389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 monclient: found mon.b 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c09b0f0 con 0x7f5d8c133910 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 5 monclient: authenticate success, global_id 284860 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 10 monclient: _renew_subs 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c13c3d0 con 0x7f5d8c133910 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 10 monclient: _renew_subs 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c13a350 con 0x7f5d8c133910 2024-09-16T06:58:43.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d84001f90 con 0x7f5d8c133910 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5d84005060 con 0x7f5d8c133910 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d8affd700 4 set_mon_vals no callback set 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.086+0000 7f5d897fa700 10 monclient: dump: 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.100 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5d8400fad0 con 0x7f5d8c133910 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d91a7a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d91a7a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d91a7a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5d84044180 con 0x7f5d8c133910 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c005100 con 0x7f5d8c133910 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.090+0000 7f5d90a78700 10 monclient: get_auth_request con 0x7f5d700365b0 auth_method 0 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.098+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5d84014be0 con 0x7f5d8c133910 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.098+0000 7f5d897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.098+0000 7f5d897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:43.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d91a7a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:43.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d91a7a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:43.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d91a7a700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:43.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5d6c005930 con 0x7f5d8c133910 2024-09-16T06:58:43.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d897fa700 1 -- 172.21.15.76:0/2196333698 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 441963614) 0x7f5d840147d0 con 0x7f5d8c133910 2024-09-16T06:58:43.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:43.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.338+0000 7f5d897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:43.339 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5d700365b0 legacy=0x7f5d70038a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 10 monclient: shutdown 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 >> v1:172.21.15.189:6789/0 conn(0x7f5d8c133910 legacy=0x7f5d8c1d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 shutdown_connections 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 >> 172.21.15.76:0/2196333698 conn(0x7f5d8c0a4430 msgr2=0x7f5d8c09bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 shutdown_connections 2024-09-16T06:58:43.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.346+0000 7f5d91a7a700 1 -- 172.21.15.76:0/2196333698 wait complete. 2024-09-16T06:58:43.355 DEBUG:teuthology.parallel:result is None 2024-09-16T06:58:43.355 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:58:43.357 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:58:43.357 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T06:58:43.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:58:43.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:58:43.368 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:58:43.368 DEBUG:teuthology.orchestra.run.smithi189:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T06:58:43.378 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:58:43.378 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T06:58:43.387 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: get_monmap_and_config 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: monmap: 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:43.476973+0000 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:43.476973+0000 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.473 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:43.473 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:43.473 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 Processor -- start 2024-09-16T06:58:43.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 -- start start 2024-09-16T06:58:43.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: init 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7dac097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7dac099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7dac139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac098990 con 0x7f7dac097b60 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac097f40 con 0x7f7dac099090 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac09d610 con 0x7f7dac139500 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient(hunting): authenticate will time out at 2070.610223s 2024-09-16T06:58:43.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7daad9d700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7dac139500 0x7f7dac13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40918/0 (socket says 172.21.15.76:40918) 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7daad9d700 1 -- 172.21.15.76:0/544527065 learned_addr learned my addr 172.21.15.76:0/544527065 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305657636 0 0) 0x7f7dac09d610 con 0x7f7dac139500 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): my global_id is 354191 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d980035d0 con 0x7f7dac139500 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180389425 0 0) 0x7f7dac098990 con 0x7f7dac097b60 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): my global_id is 334395 2024-09-16T06:58:43.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac09d610 con 0x7f7dac097b60 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557094050 0 0) 0x7f7dac097f40 con 0x7f7dac099090 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient(hunting): my global_id is 284863 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac098990 con 0x7f7dac099090 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1803980317 0 0) 0x7f7d980035d0 con 0x7f7dac139500 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 >> v1:172.21.15.189:6789/0 conn(0x7f7dac099090 legacy=0x7f7dac09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 >> v1:172.21.15.76:6789/0 conn(0x7f7dac097b60 legacy=0x7f7dac09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 monclient: found mon.noname-b 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0986b0 con 0x7f7dac139500 2024-09-16T06:58:43.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 5 monclient: authenticate success, global_id 354191 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7da4001990 con 0x7f7dac139500 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: dump: 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7da4001c70 con 0x7f7dac139500 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 1 -- 172.21.15.76:0/544527065 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7da4002000 con 0x7f7dac139500 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 4 set_mon_vals no callback set 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7da9d9b700 10 monclient: dump: 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.486 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 10 monclient: shutdown 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.474+0000 7f7dab59e700 1 -- 172.21.15.76:0/544527065 >> v1:172.21.15.76:6790/0 conn(0x7f7dac139500 legacy=0x7f7dac13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/544527065 shutdown_connections 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/544527065 >> 172.21.15.76:0/544527065 conn(0x7f7dac0a0030 msgr2=0x7f7dac0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/544527065 shutdown_connections 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/544527065 wait complete. 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: monmap: 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:43.488 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 Processor -- start 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- start start 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: init 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-b con 0x7f7dac097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-c con 0x7f7dac099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient(hunting): picked mon.noname-a con 0x7f7dac139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac13c3d0 con 0x7f7dac139500 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac09a540 con 0x7f7dac097b60 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac0a2bd0 con 0x7f7dac099090 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient(hunting): authenticate will time out at 2070.613860s 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f7dac099090 0x7f7dac1cced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40934/0 (socket says 172.21.15.76:40934) 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da3fff700 1 -- 172.21.15.76:0/4041628988 learned_addr learned my addr 172.21.15.76:0/4041628988 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851852525 0 0) 0x7f7dac0a2bd0 con 0x7f7dac099090 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): my global_id is 354194 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c0035d0 con 0x7f7dac099090 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115119056 0 0) 0x7f7dac0986b0 con 0x7f7dac139500 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): my global_id is 334398 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0a2bd0 con 0x7f7dac139500 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146599071 0 0) 0x7f7dac09a540 con 0x7f7dac097b60 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): my global_id is 284866 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0986b0 con 0x7f7dac097b60 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537158272 0 0) 0x7f7d8c0035d0 con 0x7f7dac099090 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac09a540 con 0x7f7dac099090 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d9c001990 con 0x7f7dac099090 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:43.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): dump: 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213751005 0 0) 0x7f7dac0a2bd0 con 0x7f7dac139500 2024-09-16T06:58:43.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d8c0035d0 con 0x7f7dac139500 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7da40017b0 con 0x7f7dac139500 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): dump: 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1282418331 0 0) 0x7f7dac0986b0 con 0x7f7dac097b60 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac0a2bd0 con 0x7f7dac097b60 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d940018b0 con 0x7f7dac097b60 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient(hunting): dump: 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.494 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1250541228 0 0) 0x7f7dac09a540 con 0x7f7dac099090 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.189:6789/0 conn(0x7f7dac097b60 legacy=0x7f7dac138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6789/0 conn(0x7f7dac139500 legacy=0x7f7dac1d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 monclient: found mon.c 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac08f750 con 0x7f7dac099090 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 5 monclient: authenticate success, global_id 354194 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _renew_subs 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac08eeb0 con 0x7f7dac099090 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _renew_subs 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac096f30 con 0x7f7dac099090 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d9c001d40 con 0x7f7dac099090 2024-09-16T06:58:43.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da959a700 4 set_mon_vals no callback set 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d9c003b10 con 0x7f7dac099090 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.478+0000 7f7da2ffd700 10 monclient: dump: 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7dab59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7d9c003c70 con 0x7f7dac099090 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d90006590 con 0x7f7d90002830 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f7d9c043e30 con 0x7f7dac099090 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455615830 0 0) 0x7f7d90006590 con 0x7f7d90002830 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient(hunting): my global_id is 334401 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac09a540 con 0x7f7d90002830 2024-09-16T06:58:43.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7daad9d700 10 monclient: get_auth_request con 0x7f7d8c036580 auth_method 0 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 566365680 0 0) 0x7f7dac09a540 con 0x7f7d90002830 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d90006590 con 0x7f7d90002830 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d940036a0 con 0x7f7d90002830 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient: dump: 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.499 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3451410900 0 0) 0x7f7d90006590 con 0x7f7d90002830 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d9000a190 con 0x7f7d90002830 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7d94005840 con 0x7f7d90002830 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6789/0 conn(0x7f7d90002830 legacy=0x7f7d90004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.482+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6789/0 conn(0x7f7d90002830 legacy=0x7f7d90004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7d90009e30 con 0x7f7d90009150 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698642037 0 0) 0x7f7d90009e30 con 0x7f7d90009150 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient(hunting): my global_id is 334404 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d90006590 con 0x7f7d90009150 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 218127598 0 0) 0x7f7d90006590 con 0x7f7d90009150 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d90009e30 con 0x7f7d90009150 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7d94003d10 con 0x7f7d90009150 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient: dump: 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.503 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2771158063 0 0) 0x7f7d90009e30 con 0x7f7d90009150 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d9000c340 con 0x7f7d90009150 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 549308107) 0x7f7d940059b0 con 0x7f7d90009150 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6789/0 conn(0x7f7d90009150 legacy=0x7f7d900095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7da2ffd700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6789/0 conn(0x7f7d90009150 legacy=0x7f7d900095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:43.505 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.506 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.507 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.509 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.510 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:43.511 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:43.512 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.513 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.516 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:43.517 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7d8c036580 legacy=0x7f7d8c038a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 10 monclient: shutdown 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 >> v1:172.21.15.76:6790/0 conn(0x7f7dac099090 legacy=0x7f7dac1cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7daad9d700 1 -- 172.21.15.76:0/4041628988 reap_dead start 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7daad9d700 1 -- 172.21.15.76:0/4041628988 reap_dead start 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 shutdown_connections 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 >> 172.21.15.76:0/4041628988 conn(0x7f7dac0a0030 msgr2=0x7f7dac0a1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 shutdown_connections 2024-09-16T06:58:43.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.494+0000 7f7dab59e700 1 -- 172.21.15.76:0/4041628988 wait complete. 2024-09-16T06:58:43.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:43.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:58:43.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:58:43.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: get_monmap_and_config 2024-09-16T06:58:43.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: monmap: 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:43.645326+0000 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:43.645326+0000 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 1 Processor -- start 2024-09-16T06:58:43.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 1 -- start start 2024-09-16T06:58:43.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: init 2024-09-16T06:58:43.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-a con 0x7f62f809b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-b con 0x7f62f80964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-c con 0x7f62f8138870 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8098560 con 0x7f62f809b1c0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8099ec0 con 0x7f62f8138870 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8097030 con 0x7f62f80964f0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient(hunting): authenticate will time out at 2070.778670s 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fcf13700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f62f80964f0 0x7f62f8096920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40938/0 (socket says 172.21.15.76:40938) 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fcf13700 1 -- 172.21.15.76:0/953586068 learned_addr learned my addr 172.21.15.76:0/953586068 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900857723 0 0) 0x7f62f8097030 con 0x7f62f80964f0 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): my global_id is 354197 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e00035d0 con 0x7f62f80964f0 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012631624 0 0) 0x7f62f8098560 con 0x7f62f809b1c0 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): my global_id is 334407 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8097030 con 0x7f62f809b1c0 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669756756 0 0) 0x7f62f8099ec0 con 0x7f62f8138870 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient(hunting): my global_id is 284869 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8098560 con 0x7f62f8138870 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 366818462 0 0) 0x7f62f8097030 con 0x7f62f809b1c0 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 >> v1:172.21.15.76:6790/0 conn(0x7f62f80964f0 legacy=0x7f62f8096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 >> v1:172.21.15.189:6789/0 conn(0x7f62f8138870 legacy=0x7f62f813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 monclient: found mon.noname-a 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f809b5a0 con 0x7f62f809b1c0 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62e80015f0 con 0x7f62f809b1c0 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 5 monclient: authenticate success, global_id 334407 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: dump: 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.652 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62e8001810 con 0x7f62f809b1c0 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 1 -- 172.21.15.76:0/953586068 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62e8001c20 con 0x7f62f809b1c0 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62fe716700 4 set_mon_vals no callback set 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.642+0000 7f62effff700 10 monclient: dump: 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.653 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: shutdown 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- 172.21.15.76:0/953586068 >> v1:172.21.15.76:6789/0 conn(0x7f62f809b1c0 legacy=0x7f62f809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- 172.21.15.76:0/953586068 shutdown_connections 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- 172.21.15.76:0/953586068 >> 172.21.15.76:0/953586068 conn(0x7f62f80a0030 msgr2=0x7f62f80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- 172.21.15.76:0/953586068 shutdown_connections 2024-09-16T06:58:43.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- 172.21.15.76:0/953586068 wait complete. 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: monmap: 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 Processor -- start 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- start start 2024-09-16T06:58:43.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: init 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-c con 0x7f62f80964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-b con 0x7f62f809b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient(hunting): picked mon.noname-a con 0x7f62f8138870 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f808e110 con 0x7f62f8138870 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f809c1e0 con 0x7f62f809b1c0 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f809be90 con 0x7f62f80964f0 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fe716700 10 monclient(hunting): authenticate will time out at 2070.782517s 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fd714700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f62f80964f0 0x7f62f808bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40944/0 (socket says 172.21.15.76:40944) 2024-09-16T06:58:43.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62fd714700 1 -- 172.21.15.76:0/3426914988 learned_addr learned my addr 172.21.15.76:0/3426914988 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464453328 0 0) 0x7f62f808e110 con 0x7f62f8138870 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): my global_id is 334410 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e40035d0 con 0x7f62f8138870 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041859973 0 0) 0x7f62f809be90 con 0x7f62f80964f0 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): my global_id is 354200 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f808e110 con 0x7f62f80964f0 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951401037 0 0) 0x7f62f809c1e0 con 0x7f62f809b1c0 2024-09-16T06:58:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): my global_id is 284872 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f809be90 con 0x7f62f809b1c0 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369383832 0 0) 0x7f62e40035d0 con 0x7f62f8138870 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f809c1e0 con 0x7f62f8138870 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62f00018b0 con 0x7f62f8138870 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): dump: 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.658 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378991440 0 0) 0x7f62f808e110 con 0x7f62f80964f0 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e40035d0 con 0x7f62f80964f0 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62e8001b20 con 0x7f62f80964f0 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): dump: 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3811049909 0 0) 0x7f62f809be90 con 0x7f62f809b1c0 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f808e110 con 0x7f62f809b1c0 2024-09-16T06:58:43.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62f4001b10 con 0x7f62f809b1c0 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.646+0000 7f62edffb700 10 monclient(hunting): dump: 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.661 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1593255303 0 0) 0x7f62f809c1e0 con 0x7f62f8138870 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6790/0 conn(0x7f62f80964f0 legacy=0x7f62f808bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.189:6789/0 conn(0x7f62f809b1c0 legacy=0x7f62f808c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 monclient: found mon.a 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f809a540 con 0x7f62f8138870 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 5 monclient: authenticate success, global_id 334410 2024-09-16T06:58:43.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 10 monclient: _renew_subs 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f808f790 con 0x7f62f8138870 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 10 monclient: _renew_subs 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f00036c0 con 0x7f62f8138870 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f808d710 con 0x7f62f8138870 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62f00039d0 con 0x7f62f8138870 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62ef7fe700 4 set_mon_vals no callback set 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: dump: 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f62f80008d0 con 0x7f62f808cba0 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f62f000e440 con 0x7f62f8138870 2024-09-16T06:58:43.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f62f0042bd0 con 0x7f62f8138870 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62fcf13700 10 monclient: get_auth_request con 0x7f62e40364f0 auth_method 0 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487279881 0 0) 0x7f62f80008d0 con 0x7f62f808cba0 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient(hunting): my global_id is 354203 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f809c1e0 con 0x7f62f808cba0 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2345015188 0 0) 0x7f62f809c1e0 con 0x7f62f808cba0 2024-09-16T06:58:43.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f80008d0 con 0x7f62f808cba0 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62e8003e20 con 0x7f62f808cba0 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: dump: 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.669 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1529037358 0 0) 0x7f62f80008d0 con 0x7f62f808cba0 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62f809d2d0 con 0x7f62f808cba0 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f62e8004120 con 0x7f62f808cba0 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6790/0 conn(0x7f62f808cba0 legacy=0x7f62f8061580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.650+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6790/0 conn(0x7f62f808cba0 legacy=0x7f62f8061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.662+0000 7f62fe716700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.662+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f62f802cc30 con 0x7f62f802c820 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814490636 0 0) 0x7f62f802cc30 con 0x7f62f802c820 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient(hunting): my global_id is 354206 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f80008d0 con 0x7f62f802c820 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1168454217 0 0) 0x7f62f80008d0 con 0x7f62f802c820 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f802cc30 con 0x7f62f802c820 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f62e80058f0 con 0x7f62f802c820 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient: dump: 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1712722924 0 0) 0x7f62f802cc30 con 0x7f62f802c820 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f62f808f1d0 con 0x7f62f802c820 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 1403837840) 0x7f62e8007af0 con 0x7f62f802c820 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6790/0 conn(0x7f62f802c820 legacy=0x7f62f806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62edffb700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6790/0 conn(0x7f62f802c820 legacy=0x7f62f806eeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:43.673 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 2, 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.674 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.675 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:43.676 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.677 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:43.678 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.683 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:43.684 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.189:6812/3098810674 conn(0x7f62e40364f0 legacy=0x7f62e40389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 10 monclient: shutdown 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 >> v1:172.21.15.76:6789/0 conn(0x7f62f8138870 legacy=0x7f62f81cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fdf15700 1 -- 172.21.15.76:0/3426914988 reap_dead start 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fdf15700 1 -- 172.21.15.76:0/3426914988 reap_dead start 2024-09-16T06:58:43.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 shutdown_connections 2024-09-16T06:58:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 >> 172.21.15.76:0/3426914988 conn(0x7f62f80a0030 msgr2=0x7f62f80a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 shutdown_connections 2024-09-16T06:58:43.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.666+0000 7f62fe716700 1 -- 172.21.15.76:0/3426914988 wait complete. 2024-09-16T06:58:43.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:43.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:58:43.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: get_monmap_and_config 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: monmap: 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:43.813313+0000 2024-09-16T06:58:43.808 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:43.813313+0000 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 1 Processor -- start 2024-09-16T06:58:43.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 1 -- start start 2024-09-16T06:58:43.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: init 2024-09-16T06:58:43.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:43.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-c con 0x7f57f012fbf0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.810+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-a con 0x7f57f012f7e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-b con 0x7f57f0139720 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f009db90 con 0x7f57f012f7e0 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f009d0f0 con 0x7f57f012fbf0 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f01306e0 con 0x7f57f0139720 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient(hunting): authenticate will time out at 2070.948128s 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f4ff7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f57f012f7e0 0x7f57f0138000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57850/0 (socket says 172.21.15.76:57850) 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f4ff7700 1 -- 172.21.15.76:0/984219275 learned_addr learned my addr 172.21.15.76:0/984219275 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257349804 0 0) 0x7f57f009db90 con 0x7f57f012f7e0 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): my global_id is 334413 2024-09-16T06:58:43.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57dc0035d0 con 0x7f57f012f7e0 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335513883 0 0) 0x7f57f01306e0 con 0x7f57f0139720 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): my global_id is 354209 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f009db90 con 0x7f57f0139720 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005698834 0 0) 0x7f57f009d0f0 con 0x7f57f012fbf0 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient(hunting): my global_id is 284875 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f01306e0 con 0x7f57f012fbf0 2024-09-16T06:58:43.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1648886120 0 0) 0x7f57f009db90 con 0x7f57f0139720 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 >> v1:172.21.15.189:6789/0 conn(0x7f57f012fbf0 legacy=0x7f57f012ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 >> v1:172.21.15.76:6789/0 conn(0x7f57f012f7e0 legacy=0x7f57f0138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 monclient: found mon.noname-b 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f009cc70 con 0x7f57f0139720 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57e80018a0 con 0x7f57f0139720 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: dump: 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 5 monclient: authenticate success, global_id 354209 2024-09-16T06:58:43.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57e8001b40 con 0x7f57f0139720 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 1 -- 172.21.15.76:0/984219275 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57e8001f10 con 0x7f57f0139720 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57e7fff700 10 monclient: dump: 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.817 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.818 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.818 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:43.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 4 set_mon_vals no callback set 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient: shutdown 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- 172.21.15.76:0/984219275 >> v1:172.21.15.76:6790/0 conn(0x7f57f0139720 legacy=0x7f57f013bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- 172.21.15.76:0/984219275 shutdown_connections 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- 172.21.15.76:0/984219275 >> 172.21.15.76:0/984219275 conn(0x7f57f00a5870 msgr2=0x7f57f00a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- 172.21.15.76:0/984219275 shutdown_connections 2024-09-16T06:58:43.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 1 -- 172.21.15.76:0/984219275 wait complete. 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient: build_initial_monmap 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.814+0000 7f57f67fa700 10 monclient: monmap: 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:43.824 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 Processor -- start 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- start start 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: init 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-c con 0x7f57f012f7e0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-a con 0x7f57f012fbf0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient(hunting): picked mon.noname-b con 0x7f57f0139720 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f01c88d0 con 0x7f57f012fbf0 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f006ea50 con 0x7f57f0139720 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f006fd40 con 0x7f57f012f7e0 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient(hunting): authenticate will time out at 2070.953235s 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f4ff7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f57f012fbf0 0x7f57f0071b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57860/0 (socket says 172.21.15.76:57860) 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f4ff7700 1 -- 172.21.15.76:0/1805784189 learned_addr learned my addr 172.21.15.76:0/1805784189 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413174034 0 0) 0x7f57f01c88d0 con 0x7f57f012fbf0 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): my global_id is 334416 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d40035d0 con 0x7f57f012fbf0 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904380387 0 0) 0x7f57f006fd40 con 0x7f57f012f7e0 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): my global_id is 354212 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f01c88d0 con 0x7f57f012f7e0 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841123688 0 0) 0x7f57f006ea50 con 0x7f57f0139720 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): my global_id is 284878 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f006fd40 con 0x7f57f0139720 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 493861094 0 0) 0x7f57d40035d0 con 0x7f57f012fbf0 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57f006ea50 con 0x7f57f012fbf0 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57e0001770 con 0x7f57f012fbf0 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:43.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): dump: 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269467775 0 0) 0x7f57f01c88d0 con 0x7f57f012f7e0 2024-09-16T06:58:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d40035d0 con 0x7f57f012f7e0 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57ec0018b0 con 0x7f57f012f7e0 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient(hunting): dump: 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.829 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 430052174 0 0) 0x7f57f006ea50 con 0x7f57f012fbf0 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.76:6790/0 conn(0x7f57f012f7e0 legacy=0x7f57f01c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6789/0 conn(0x7f57f0139720 legacy=0x7f57f01cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 monclient: found mon.a 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f0070300 con 0x7f57f012fbf0 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:43.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 5 monclient: authenticate success, global_id 334416 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _renew_subs 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f0070020 con 0x7f57f012fbf0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _renew_subs 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f0133480 con 0x7f57f012fbf0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57e0001d40 con 0x7f57f012fbf0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57e0004a30 con 0x7f57f012fbf0 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e5ffb700 10 monclient: dump: 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57e77fe700 4 set_mon_vals no callback set 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.818+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57d0006590 con 0x7f57d0002830 2024-09-16T06:58:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f57e000f520 con 0x7f57f012fbf0 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f57e0043d10 con 0x7f57f012fbf0 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57f5ff9700 10 monclient: get_auth_request con 0x7f57d4036430 auth_method 0 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121879378 0 0) 0x7f57d0006590 con 0x7f57d0002830 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient(hunting): my global_id is 284881 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f006ea50 con 0x7f57d0002830 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2548891321 0 0) 0x7f57f006ea50 con 0x7f57d0002830 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d0006590 con 0x7f57d0002830 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57ec003860 con 0x7f57d0002830 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient: dump: 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.834 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2225245463 0 0) 0x7f57d0006590 con 0x7f57d0002830 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57d000a190 con 0x7f57d0002830 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f57ec0046f0 con 0x7f57d0002830 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6789/0 conn(0x7f57d0002830 legacy=0x7f57d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.822+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6789/0 conn(0x7f57d0002830 legacy=0x7f57d0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.830+0000 7f57f67fa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.830+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f57d0009e50 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633293648 0 0) 0x7f57d0009e50 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient(hunting): my global_id is 284884 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d0006590 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3929878129 0 0) 0x7f57d0006590 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d0009e50 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f57ec003fb0 con 0x7f57d0009170 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient: dump: 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 836973901 0 0) 0x7f57d0009e50 con 0x7f57d0009170 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57d000c340 con 0x7f57d0009170 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4048 (unknown 0 0 2703476867) 0x7f57ec004290 con 0x7f57d0009170 2024-09-16T06:58:43.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6789/0 conn(0x7f57d0009170 legacy=0x7f57d0009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57e5ffb700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6789/0 conn(0x7f57d0009170 legacy=0x7f57d0009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 176, 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:43.839 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 3, 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.840 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.841 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:43.842 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:43.843 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:43.844 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:43.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:43.848 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 3 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:43.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:43.850 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:43.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.189:6812/3098810674 conn(0x7f57d4036430 legacy=0x7f57d40388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 10 monclient: shutdown 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 >> v1:172.21.15.76:6789/0 conn(0x7f57f012fbf0 legacy=0x7f57f0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f5ff9700 1 -- 172.21.15.76:0/1805784189 reap_dead start 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f5ff9700 1 -- 172.21.15.76:0/1805784189 reap_dead start 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 shutdown_connections 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 >> 172.21.15.76:0/1805784189 conn(0x7f57f00a5870 msgr2=0x7f57f009b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 shutdown_connections 2024-09-16T06:58:43.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:43.834+0000 7f57f67fa700 1 -- 172.21.15.76:0/1805784189 wait complete. 2024-09-16T06:58:43.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:58:43.853 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T06:58:43.853 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T06:58:43.853 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T06:58:43.853 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:58:43.853 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T06:58:43.853 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T06:58:43.890 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T06:58:43.890 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T06:58:43.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:58:44.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:58:44.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: build_initial_monmap 2024-09-16T06:58:44.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: monmap: 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:44.012525+0000 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:44.012525+0000 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:44.008 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 1 Processor -- start 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 1 -- start start 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: init 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fab6401f4c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fab64025410 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fab64025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6401f8a0 con 0x7fab64025410 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab640233f0 con 0x7fab6401f4c0 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab64020050 con 0x7fab64025860 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:44.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient(hunting): authenticate will time out at 2071.145849s 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6b7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fab64025860 legacy=0x7fab640c2320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fab64025410 0x7fab640bebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38976/0 (socket says 172.21.15.76:38976) 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6a7fc700 1 -- 172.21.15.76:0/3854468543 learned_addr learned my addr 172.21.15.76:0/3854468543 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962970599 0 0) 0x7fab6401f8a0 con 0x7fab64025410 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): my global_id is 334419 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab580035d0 con 0x7fab64025410 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151766167 0 0) 0x7fab640233f0 con 0x7fab6401f4c0 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:44.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient(hunting): my global_id is 284887 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6401f8a0 con 0x7fab6401f4c0 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94207354 0 0) 0x7fab580035d0 con 0x7fab64025410 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 >> v1:172.21.15.76:6790/0 conn(0x7fab64025860 legacy=0x7fab640c2320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 >> v1:172.21.15.189:6789/0 conn(0x7fab6401f4c0 legacy=0x7fab64024d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 monclient: found mon.noname-a 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64021a90 con 0x7fab64025410 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:44.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab5c001610 con 0x7fab64025410 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 5 monclient: authenticate success, global_id 334419 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: dump: 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:44.019 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab5c001880 con 0x7fab64025410 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 1 -- 172.21.15.76:0/3854468543 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab5c001c10 con 0x7fab64025410 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab6bfff700 4 set_mon_vals no callback set 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.010+0000 7fab69ffb700 10 monclient: dump: 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:44.020 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: shutdown 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/3854468543 >> v1:172.21.15.76:6789/0 conn(0x7fab64025410 legacy=0x7fab640bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/3854468543 shutdown_connections 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/3854468543 >> 172.21.15.76:0/3854468543 conn(0x7fab64005b40 msgr2=0x7fab64005f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:44.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/3854468543 shutdown_connections 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/3854468543 wait complete. 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: build_initial_monmap 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: monmap: 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 Processor -- start 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- start start 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: init 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fab6401f4c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fab64025410 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fab64025860 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab64028d80 con 0x7fab6401f4c0 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6412f2f0 con 0x7fab64025410 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab640204a0 con 0x7fab64025860 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6b7fe700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7fab64025860 legacy=0x7fab64158730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient(hunting): authenticate will time out at 2071.150419s 2024-09-16T06:58:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fab6401f4c0 0x7fab6414bd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38990/0 (socket says 172.21.15.76:38990) 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6affd700 1 -- 172.21.15.76:0/2222426598 learned_addr learned my addr 172.21.15.76:0/2222426598 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130294385 0 0) 0x7fab64028d80 con 0x7fab6401f4c0 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): my global_id is 334422 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0035d0 con 0x7fab6401f4c0 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3273696186 0 0) 0x7fab4c0035d0 con 0x7fab6401f4c0 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab64028d80 con 0x7fab6401f4c0 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab54001950 con 0x7fab6401f4c0 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): dump: 2024-09-16T06:58:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703577228 0 0) 0x7fab6412f2f0 con 0x7fab64025410 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient(hunting): my global_id is 284890 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0035d0 con 0x7fab64025410 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3627924786 0 0) 0x7fab64028d80 con 0x7fab6401f4c0 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 >> v1:172.21.15.76:6790/0 conn(0x7fab64025860 legacy=0x7fab64158730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 >> v1:172.21.15.189:6789/0 conn(0x7fab64025410 legacy=0x7fab64155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 monclient: found mon.a 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64130de0 con 0x7fab6401f4c0 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:44.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 5 monclient: authenticate success, global_id 334422 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _renew_subs 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab641310c0 con 0x7fab6401f4c0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _renew_subs 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab641300e0 con 0x7fab6401f4c0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab54001d60 con 0x7fab6401f4c0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fab54003b90 con 0x7fab6401f4c0 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab697fa700 4 set_mon_vals no callback set 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.014+0000 7fab637fe700 10 monclient: dump: 2024-09-16T06:58:44.027 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab6bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab6bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab50005100 con 0x7fab6401f4c0 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fab54003db0 con 0x7fab6401f4c0 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fab54043f10 con 0x7fab6401f4c0 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.018+0000 7fab6a7fc700 10 monclient: get_auth_request con 0x7fab4c036180 auth_method 0 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.026+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fab54014720 con 0x7fab6401f4c0 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.026+0000 7fab637fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.026+0000 7fab637fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:44.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:44.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:44.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:44.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab50005930 con 0x7fab6401f4c0 2024-09-16T06:58:44.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab637fe700 1 -- 172.21.15.76:0/2222426598 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 2728823649) 0x7fab54012dc0 con 0x7fab6401f4c0 2024-09-16T06:58:44.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab637fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:44.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab637fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:44.256 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:44.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 >> v1:172.21.15.189:6812/3098810674 conn(0x7fab4c036180 legacy=0x7fab4c038640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:44.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 10 monclient: shutdown 2024-09-16T06:58:44.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:44.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 >> v1:172.21.15.76:6789/0 conn(0x7fab6401f4c0 legacy=0x7fab6414bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:44.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 shutdown_connections 2024-09-16T06:58:44.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 >> 172.21.15.76:0/2222426598 conn(0x7fab64005b40 msgr2=0x7fab6401a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:44.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 shutdown_connections 2024-09-16T06:58:44.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:44.258+0000 7fab6bfff700 1 -- 172.21.15.76:0/2222426598 wait complete. 2024-09-16T06:58:47.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: get_monmap_and_config 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: build_initial_monmap 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: monmap: 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:47.392873+0000 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:47.392873+0000 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:47.388 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:47.389 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:47.389 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:47.389 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 1 Processor -- start 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 1 -- start start 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: init 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:58:47.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient(hunting): picked mon.noname-b con 0x7f783809b1c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient(hunting): picked mon.noname-c con 0x7f78380964f0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient(hunting): picked mon.noname-a con 0x7f7838138870 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838098560 con 0x7f7838138870 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838099ec0 con 0x7f78380964f0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838097030 con 0x7f783809b1c0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient(hunting): authenticate will time out at 2074.526120s 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783cf74700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f783809b1c0 legacy=0x7f783809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783d775700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7838138870 0x7f783813ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38992/0 (socket says 172.21.15.76:38992) 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783d775700 1 -- 172.21.15.76:0/4049843128 learned_addr learned my addr 172.21.15.76:0/4049843128 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803039684 0 0) 0x7f7838099ec0 con 0x7f78380964f0 2024-09-16T06:58:47.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient(hunting): my global_id is 284893 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78200057e0 con 0x7f78380964f0 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69220484 0 0) 0x7f78200057e0 con 0x7f78380964f0 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 >> v1:172.21.15.76:6790/0 conn(0x7f783809b1c0 legacy=0x7f783809ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 >> v1:172.21.15.76:6789/0 conn(0x7f7838138870 legacy=0x7f783813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 monclient: found mon.noname-c 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783809b5a0 con 0x7f78380964f0 2024-09-16T06:58:47.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f782c001610 con 0x7f78380964f0 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 5 monclient: authenticate success, global_id 284893 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: dump: 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f782c001930 con 0x7f78380964f0 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 1 -- 172.21.15.76:0/4049843128 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f782c001cc0 con 0x7f78380964f0 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:47.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f78377fe700 10 monclient: dump: 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.390+0000 7f783df76700 4 set_mon_vals no callback set 2024-09-16T06:58:47.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: shutdown 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- 172.21.15.76:0/4049843128 >> v1:172.21.15.189:6789/0 conn(0x7f78380964f0 legacy=0x7f7838096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- 172.21.15.76:0/4049843128 shutdown_connections 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- 172.21.15.76:0/4049843128 >> 172.21.15.76:0/4049843128 conn(0x7f78380a0030 msgr2=0x7f78380a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- 172.21.15.76:0/4049843128 shutdown_connections 2024-09-16T06:58:47.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- 172.21.15.76:0/4049843128 wait complete. 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: build_initial_monmap 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: monmap: 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:47.402 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 Processor -- start 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 1 -- start start 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: init 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.394+0000 7f783df76700 10 monclient(hunting): picked mon.noname-c con 0x7f78380964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient(hunting): picked mon.noname-a con 0x7f783809b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:47.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient(hunting): picked mon.noname-b con 0x7f7838138870 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78381386c0 con 0x7f783809b1c0 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838093aa0 con 0x7f7838138870 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838093d80 con 0x7f78380964f0 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient(hunting): authenticate will time out at 2074.532080s 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783cf74700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f78380964f0 legacy=0x7f7838137890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f7837fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f783809b1c0 0x7f78381cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:38994/0 (socket says 172.21.15.76:38994) 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f7837fff700 1 -- 172.21.15.76:0/33775566 learned_addr learned my addr 172.21.15.76:0/33775566 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645618893 0 0) 0x7f7838093aa0 con 0x7f7838138870 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): my global_id is 284896 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78240035d0 con 0x7f7838138870 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020494912 0 0) 0x7f78240035d0 con 0x7f7838138870 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7838093aa0 con 0x7f7838138870 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7830001aa0 con 0x7f7838138870 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): dump: 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1532354175 0 0) 0x7f7838093aa0 con 0x7f7838138870 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 >> v1:172.21.15.76:6790/0 conn(0x7f78380964f0 legacy=0x7f7838137890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 >> v1:172.21.15.76:6789/0 conn(0x7f783809b1c0 legacy=0x7f78381cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 monclient: found mon.b 2024-09-16T06:58:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78380925f0 con 0x7f7838138870 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 5 monclient: authenticate success, global_id 284896 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient: _renew_subs 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783809c030 con 0x7f7838138870 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient: _renew_subs 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78380937c0 con 0x7f7838138870 2024-09-16T06:58:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78300020a0 con 0x7f7838138870 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7830004db0 con 0x7f7838138870 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78357fa700 10 monclient: dump: 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:47.408 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.398+0000 7f78367fc700 4 set_mon_vals no callback set 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f783df76700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f783df76700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f783df76700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7818005100 con 0x7f7838138870 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7830004f90 con 0x7f7838138870 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f78300441a0 con 0x7f7838138870 2024-09-16T06:58:47.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.402+0000 7f783cf74700 10 monclient: get_auth_request con 0x7f78240360a0 auth_method 0 2024-09-16T06:58:47.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.414+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f7830014850 con 0x7f7838138870 2024-09-16T06:58:47.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.414+0000 7f78357fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:47.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.414+0000 7f78357fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:47.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:47.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:47.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-09-16T06:58:47.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7818005930 con 0x7f7838138870 2024-09-16T06:58:47.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f78357fa700 1 -- 172.21.15.76:0/33775566 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 4064953664) 0x7f783001d0d0 con 0x7f7838138870 2024-09-16T06:58:47.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f78357fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:47.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f78357fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:47.644 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 >> v1:172.21.15.189:6812/3098810674 conn(0x7f78240360a0 legacy=0x7f7824038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 10 monclient: shutdown 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 >> v1:172.21.15.189:6789/0 conn(0x7f7838138870 legacy=0x7f78381d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 shutdown_connections 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 >> 172.21.15.76:0/33775566 conn(0x7f78380a0030 msgr2=0x7f78380a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 shutdown_connections 2024-09-16T06:58:47.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:47.646+0000 7f783df76700 1 -- 172.21.15.76:0/33775566 wait complete. 2024-09-16T06:58:50.657 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:58:50.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.778+0000 7f5c9e1c0700 10 monclient: get_monmap_and_config 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.778+0000 7f5c9e1c0700 10 monclient: build_initial_monmap 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.778+0000 7f5c9e1c0700 10 monclient: monmap: 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:50.783731+0000 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:50.783731+0000 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:50.779 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:50.780 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:50.780 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 1 Processor -- start 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 1 -- start start 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient: init 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-b con 0x7f5c98097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-a con 0x7f5c98099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-c con 0x7f5c98139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c98098990 con 0x7f5c98099090 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c98097f40 con 0x7f5c98139500 2024-09-16T06:58:50.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9809d610 con 0x7f5c98097b60 2024-09-16T06:58:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9e1c0700 10 monclient(hunting): authenticate will time out at 2077.916964s 2024-09-16T06:58:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9d1be700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c9809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9c9bd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5c98099090 0x7f5c9809f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:39004/0 (socket says 172.21.15.76:39004) 2024-09-16T06:58:50.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.782+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/2505820963 learned_addr learned my addr 172.21.15.76:0/2505820963 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:50.986+0000 7f5c9d1be700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c9809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:51.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:51.390+0000 7f5c9d1be700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c9809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:52.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:52.194+0000 7f5c9d1be700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c9809ed60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:53.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): tick 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 monclient(hunting): continuing hunt 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c9809ed60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.189:6789/0 conn(0x7f5c98139500 legacy=0x7f5c9813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:53.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6789/0 conn(0x7f5c98099090 legacy=0x7f5c9809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-c con 0x7f5c7c003740 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-b con 0x7f5c98097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-a con 0x7f5c98099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980986b0 con 0x7f5c98099090 2024-09-16T06:58:53.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9809d610 con 0x7f5c7c003740 2024-09-16T06:58:53.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:53.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c0103f0 con 0x7f5c98097b60 2024-09-16T06:58:53.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.782+0000 7f5c8effd700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:53.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:53.986+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:54.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:54.390+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:55.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:55.194+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:56.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): tick 2024-09-16T06:58:56.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T06:58:56.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 monclient(hunting): continuing hunt 2024-09-16T06:58:56.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T06:58:56.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.189:6789/0 conn(0x7f5c7c003740 legacy=0x7f5c7c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6789/0 conn(0x7f5c98099090 legacy=0x7f5c7c00cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-a con 0x7f5c7c010630 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-c con 0x7f5c7c0128c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): picked mon.noname-b con 0x7f5c98097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:56.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c011600 con 0x7f5c7c010630 2024-09-16T06:58:56.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c0103f0 con 0x7f5c7c0128c0 2024-09-16T06:58:56.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c007410 con 0x7f5c98097b60 2024-09-16T06:58:56.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c9d9bf700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c00cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:56.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.782+0000 7f5c8effd700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:56.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:56.986+0000 7f5c9d9bf700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c00cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:57.390+0000 7f5c9d9bf700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c00cc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387161132 0 0) 0x7f5c7c0103f0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient(hunting): my global_id is 284899 2024-09-16T06:58:58.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c840035d0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813664434 0 0) 0x7f5c840035d0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6790/0 conn(0x7f5c98097b60 legacy=0x7f5c7c00cc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.76:6789/0 conn(0x7f5c7c010630 legacy=0x7f5c7c010a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 monclient: found mon.noname-c 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-09-16T06:58:58.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0085b0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c880015c0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c8ffff700 10 monclient: dump: 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.130 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.131 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.131 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.131 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c9e1c0700 5 monclient: authenticate success, global_id 284899 2024-09-16T06:58:58.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.130+0000 7f5c9e1c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c88001860 con 0x7f5c7c0128c0 2024-09-16T06:58:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c9e1c0700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 1 -- 172.21.15.76:0/2505820963 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c88001bf0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c9e1c0700 4 set_mon_vals no callback set 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.134+0000 7f5c8ffff700 10 monclient: dump: 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.135 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 10 monclient: shutdown 2024-09-16T06:58:58.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/2505820963 >> v1:172.21.15.189:6789/0 conn(0x7f5c7c0128c0 legacy=0x7f5c7c014cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/2505820963 shutdown_connections 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/2505820963 >> 172.21.15.76:0/2505820963 conn(0x7f5c980a0030 msgr2=0x7f5c980a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/2505820963 shutdown_connections 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/2505820963 wait complete. 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 10 monclient: build_initial_monmap 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.138+0000 7f5c9e1c0700 10 monclient: monmap: 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:58.137 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:58.138 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 Processor -- start 2024-09-16T06:58:58.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- start start 2024-09-16T06:58:58.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: init 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-c con 0x7f5c7c010630 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-a con 0x7f5c7c0128c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient(hunting): picked mon.noname-b con 0x7f5c98097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c980923a0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c98093e50 con 0x7f5c98097b60 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9809a540 con 0x7f5c7c010630 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9d1be700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f5c7c010630 legacy=0x7f5c980908f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient(hunting): authenticate will time out at 2085.277281s 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9c9bd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5c7c0128c0 0x7f5c98091000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57760/0 (socket says 172.21.15.76:57760) 2024-09-16T06:58:58.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9c9bd700 1 -- 172.21.15.76:0/381480817 learned_addr learned my addr 172.21.15.76:0/381480817 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:58.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413950796 0 0) 0x7f5c980923a0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): my global_id is 334425 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c7c010ce0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057005823 0 0) 0x7f5c98093e50 con 0x7f5c98097b60 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): my global_id is 284902 2024-09-16T06:58:58.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c980923a0 con 0x7f5c98097b60 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2510397645 0 0) 0x7f5c7c010ce0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c98093e50 con 0x7f5c7c0128c0 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c88001bb0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): dump: 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.146 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194999458 0 0) 0x7f5c980923a0 con 0x7f5c98097b60 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c7c010ce0 con 0x7f5c98097b60 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c90001a10 con 0x7f5c98097b60 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient(hunting): dump: 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3752288371 0 0) 0x7f5c98093e50 con 0x7f5c7c0128c0 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 >> v1:172.21.15.76:6790/0 conn(0x7f5c7c010630 legacy=0x7f5c980908f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 >> v1:172.21.15.189:6789/0 conn(0x7f5c98097b60 legacy=0x7f5c98091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 monclient: found mon.a 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c980925d0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 5 monclient: authenticate success, global_id 334425 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _renew_subs 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c980a1320 con 0x7f5c7c0128c0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _renew_subs 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98092ef0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c880011a0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8f7fe700 4 set_mon_vals no callback set 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5c88004ce0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.142+0000 7f5c8dffb700 10 monclient: dump: 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.150 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5c88004ec0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c9e1c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c9e1c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c9e1c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c78005100 con 0x7f5c7c0128c0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5c88043ff0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.146+0000 7f5c9d1be700 10 monclient: get_auth_request con 0x7f5c7c0376c0 auth_method 0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.150+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f5c880064b0 con 0x7f5c7c0128c0 2024-09-16T06:58:58.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.154+0000 7f5c8dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.154+0000 7f5c8dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:58.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.385+0000 7f5c9e1c0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:58:58.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.385+0000 7f5c9e1c0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:58.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.385+0000 7f5c9e1c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.385+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c78005930 con 0x7f5c7c0128c0 2024-09-16T06:58:58.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c8dffb700 1 -- 172.21.15.76:0/381480817 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1384 (unknown 254746657 0 3481090799) 0x7f5c88012e60 con 0x7f5c7c0128c0 2024-09-16T06:58:58.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c8dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:58:58.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c8dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:58.388 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":180,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:58:58.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5c7c0376c0 legacy=0x7f5c7c039b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 10 monclient: shutdown 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 >> v1:172.21.15.76:6789/0 conn(0x7f5c7c0128c0 legacy=0x7f5c98091000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 shutdown_connections 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 >> 172.21.15.76:0/381480817 conn(0x7f5c980a0030 msgr2=0x7f5c98070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 shutdown_connections 2024-09-16T06:58:58.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.389+0000 7f5c9e1c0700 1 -- 172.21.15.76:0/381480817 wait complete. 2024-09-16T06:58:58.401 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T06:58:58.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T06:58:58.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:58:58.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.509+0000 7f775480f700 10 monclient: get_monmap_and_config 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.509+0000 7f775480f700 10 monclient: build_initial_monmap 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.509+0000 7f775480f700 10 monclient: monmap: 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:58.514807+0000 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:58.514807+0000 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:58.510 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 Processor -- start 2024-09-16T06:58:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- start start 2024-09-16T06:58:58.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient: init 2024-09-16T06:58:58.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient(hunting): picked mon.noname-b con 0x7f77500a4840 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient(hunting): picked mon.noname-a con 0x7f77501334a0 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient(hunting): picked mon.noname-c con 0x7f7750133910 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500a1620 con 0x7f77501334a0 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500a4c20 con 0x7f7750133910 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774ed9d700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f77500a4840 legacy=0x7f7750132d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7750071f10 con 0x7f77500a4840 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient(hunting): authenticate will time out at 2085.649720s 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f77501334a0 0x7f77501380b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57766/0 (socket says 172.21.15.76:57766) 2024-09-16T06:58:58.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774659c700 1 -- 172.21.15.76:0/3066184146 learned_addr learned my addr 172.21.15.76:0/3066184146 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677115188 0 0) 0x7f77500a1620 con 0x7f77501334a0 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient(hunting): my global_id is 334428 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c0035d0 con 0x7f77501334a0 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 741689829 0 0) 0x7f773c0035d0 con 0x7f77501334a0 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 >> v1:172.21.15.76:6790/0 conn(0x7f77500a4840 legacy=0x7f7750132d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 >> v1:172.21.15.189:6789/0 conn(0x7f7750133910 legacy=0x7f775013b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 monclient: found mon.noname-a 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775009e1e0 con 0x7f77501334a0 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 5 monclient: authenticate success, global_id 334428 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7738001610 con 0x7f77501334a0 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: dump: 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.517 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7738001930 con 0x7f77501334a0 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 1 -- 172.21.15.76:0/3066184146 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7738001d00 con 0x7f77501334a0 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 4 set_mon_vals no callback set 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f774e59c700 10 monclient: dump: 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 10 monclient: shutdown 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- 172.21.15.76:0/3066184146 >> v1:172.21.15.76:6789/0 conn(0x7f77501334a0 legacy=0x7f77501380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- 172.21.15.76:0/3066184146 shutdown_connections 2024-09-16T06:58:58.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- 172.21.15.76:0/3066184146 >> 172.21.15.76:0/3066184146 conn(0x7f77500a4430 msgr2=0x7f77500a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- 172.21.15.76:0/3066184146 shutdown_connections 2024-09-16T06:58:58.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.513+0000 7f775480f700 1 -- 172.21.15.76:0/3066184146 wait complete. 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: build_initial_monmap 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: monmap: 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:58.525 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 Processor -- start 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- start start 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: init 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient(hunting): picked mon.noname-b con 0x7f77500a4840 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient(hunting): picked mon.noname-c con 0x7f77501334a0 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient(hunting): picked mon.noname-a con 0x7f7750133910 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:58.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775013ad10 con 0x7f7750133910 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775013cb00 con 0x7f77500a4840 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7750133000 con 0x7f77501334a0 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient(hunting): authenticate will time out at 2085.654185s 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f774659c700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f77501334a0 legacy=0x7f7750138ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f774f59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7750133910 0x7f77501395d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57770/0 (socket says 172.21.15.76:57770) 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f774f59e700 1 -- 172.21.15.76:0/2702383870 learned_addr learned my addr 172.21.15.76:0/2702383870 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358484888 0 0) 0x7f775013ad10 con 0x7f7750133910 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): my global_id is 334431 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77300035d0 con 0x7f7750133910 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969883835 0 0) 0x7f775013cb00 con 0x7f77500a4840 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): my global_id is 284908 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775013ad10 con 0x7f77500a4840 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1420885308 0 0) 0x7f77300035d0 con 0x7f7750133910 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f775013cb00 con 0x7f7750133910 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7748001cc0 con 0x7f7750133910 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:58.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): dump: 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741678773 0 0) 0x7f775013ad10 con 0x7f77500a4840 2024-09-16T06:58:58.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77300035d0 con 0x7f77500a4840 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025658559 0 0) 0x7f775013cb00 con 0x7f7750133910 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6790/0 conn(0x7f77501334a0 legacy=0x7f7750138ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.189:6789/0 conn(0x7f77500a4840 legacy=0x7f77501387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 monclient: found mon.a 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775009b0f0 con 0x7f7750133910 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 5 monclient: authenticate success, global_id 334431 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _renew_subs 2024-09-16T06:58:58.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7750132b70 con 0x7f7750133910 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _renew_subs 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f775013be40 con 0x7f7750133910 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7748002030 con 0x7f7750133910 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7748003f10 con 0x7f7750133910 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.517+0000 7f7747fff700 10 monclient: dump: 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f774dd9b700 4 set_mon_vals no callback set 2024-09-16T06:58:58.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f7748004070 con 0x7f7750133910 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f775480f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7734006590 con 0x7f7734002830 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f77480442d0 con 0x7f7750133910 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775921149 0 0) 0x7f7734006590 con 0x7f7734002830 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient(hunting): my global_id is 334434 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775013cb00 con 0x7f7734002830 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f774659c700 10 monclient: get_auth_request con 0x7f7730036260 auth_method 0 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2722099474 0 0) 0x7f775013cb00 con 0x7f7734002830 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7734006590 con 0x7f7734002830 2024-09-16T06:58:58.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7740004800 con 0x7f7734002830 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient: dump: 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2946369655 0 0) 0x7f7734006590 con 0x7f7734002830 2024-09-16T06:58:58.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f773400a190 con 0x7f7734002830 2024-09-16T06:58:58.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f7740006bb0 con 0x7f7734002830 2024-09-16T06:58:58.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:58.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6789/0 conn(0x7f7734002830 legacy=0x7f7734004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.521+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6789/0 conn(0x7f7734002830 legacy=0x7f7734004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:58.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:58.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7734009e50 con 0x7f7734009170 2024-09-16T06:58:58.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052189499 0 0) 0x7f7734009e50 con 0x7f7734009170 2024-09-16T06:58:58.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient(hunting): my global_id is 334437 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7734006590 con 0x7f7734009170 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1276323106 0 0) 0x7f7734006590 con 0x7f7734009170 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7734009e50 con 0x7f7734009170 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f7740005060 con 0x7f7734009170 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient: dump: 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.537 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2020757941 0 0) 0x7f7734009e50 con 0x7f7734009170 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f773400c340 con 0x7f7734009170 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4201 (unknown 0 0 3900738623) 0x7f7740006490 con 0x7f7734009170 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6789/0 conn(0x7f7734009170 legacy=0x7f7734009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f7747fff700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6789/0 conn(0x7f7734009170 legacy=0x7f7734009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 180, 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 6 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:58.551 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.189:6812/3098810674 conn(0x7f7730036260 legacy=0x7f7730038720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 10 monclient: shutdown 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f774f59e700 1 -- 172.21.15.76:0/2702383870 reap_dead start 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 >> v1:172.21.15.76:6789/0 conn(0x7f7750133910 legacy=0x7f77501395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 shutdown_connections 2024-09-16T06:58:58.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 >> 172.21.15.76:0/2702383870 conn(0x7f77500a4430 msgr2=0x7f775009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 shutdown_connections 2024-09-16T06:58:58.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.533+0000 7f775480f700 1 -- 172.21.15.76:0/2702383870 wait complete. 2024-09-16T06:58:58.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:58:58.553 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T06:58:58.553 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:58:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.669+0000 7f3291cc1700 10 monclient: get_monmap_and_config 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.669+0000 7f3291cc1700 10 monclient: build_initial_monmap 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.669+0000 7f3291cc1700 10 monclient: monmap: 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:58:58.674706+0000 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:58:58.674706+0000 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:58:58.670 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 Processor -- start 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- start start 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: init 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-c con 0x7f328c09ee40 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-b con 0x7f328c138570 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-a con 0x7f328c13c080 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c09f930 con 0x7f328c13c080 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c06d9b0 con 0x7f328c09ee40 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c06cb30 con 0x7f328c138570 2024-09-16T06:58:58.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient(hunting): authenticate will time out at 2085.808831s 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f328c138570 legacy=0x7f328c13a960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f32914c0700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f328c13c080 0x7f328c13e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57794/0 (socket says 172.21.15.76:57794) 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f32914c0700 1 -- 172.21.15.76:0/4098865856 learned_addr learned my addr 172.21.15.76:0/4098865856 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270863730 0 0) 0x7f3284001490 con 0x7f328c13c080 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient(hunting): my global_id is 334440 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c0035d0 con 0x7f328c13c080 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801444581 0 0) 0x7f327c0035d0 con 0x7f328c13c080 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 >> v1:172.21.15.76:6790/0 conn(0x7f328c138570 legacy=0x7f328c13a960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T06:58:58.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 >> v1:172.21.15.189:6789/0 conn(0x7f328c09ee40 legacy=0x7f328c09f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c09d7d0 con 0x7f328c13c080 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3284001980 con 0x7f328c13c080 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: dump: 2024-09-16T06:58:58.675 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 5 monclient: authenticate success, global_id 334440 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3284001c70 con 0x7f328c13c080 2024-09-16T06:58:58.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 1 -- 172.21.15.76:0/4098865856 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3284002040 con 0x7f328c13c080 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f328b7fe700 10 monclient: dump: 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.677 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: get_monmap_and_config success 2024-09-16T06:58:58.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 4 set_mon_vals no callback set 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: shutdown 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- 172.21.15.76:0/4098865856 >> v1:172.21.15.76:6789/0 conn(0x7f328c13c080 legacy=0x7f328c13e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- 172.21.15.76:0/4098865856 shutdown_connections 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- 172.21.15.76:0/4098865856 >> 172.21.15.76:0/4098865856 conn(0x7f328c0a2300 msgr2=0x7f328c0a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- 172.21.15.76:0/4098865856 shutdown_connections 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 1 -- 172.21.15.76:0/4098865856 wait complete. 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: build_initial_monmap 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.673+0000 7f3291cc1700 10 monclient: monmap: 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:58:58.684 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 Processor -- start 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- start start 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: init 2024-09-16T06:58:58.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _reopen_session rank -1 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-b con 0x7f328c09ee40 addr v1:172.21.15.189:6789/0 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-c con 0x7f328c138570 addr v1:172.21.15.76:6790/0 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient(hunting): picked mon.noname-a con 0x7f328c13c080 addr v1:172.21.15.76:6789/0 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c0725f0 con 0x7f328c13c080 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c138390 con 0x7f328c09ee40 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c090690 con 0x7f328c138570 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient(hunting): _renew_subs 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient(hunting): authenticate will time out at 2085.812634s 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f328bfff700 1 -- >> v1:172.21.15.76:6790/0 conn(0x7f328c138570 legacy=0x7f328c137c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.76:6790/0 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32914c0700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f328c13c080 0x7f328c1d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57796/0 (socket says 172.21.15.76:57796) 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32914c0700 1 -- 172.21.15.76:0/3725195614 learned_addr learned my addr 172.21.15.76:0/3725195614 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:58:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970943867 0 0) 0x7f328c0725f0 con 0x7f328c13c080 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): my global_id is 334443 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32700035d0 con 0x7f328c13c080 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178564388 0 0) 0x7f328c138390 con 0x7f328c09ee40 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): my global_id is 284911 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c0725f0 con 0x7f328c09ee40 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 579046698 0 0) 0x7f32700035d0 con 0x7f328c13c080 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f328c138390 con 0x7f328c13c080 2024-09-16T06:58:58.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3284001360 con 0x7f328c13c080 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211332749 0 0) 0x7f328c0725f0 con 0x7f328c09ee40 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32700035d0 con 0x7f328c09ee40 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3280001810 con 0x7f328c09ee40 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient(hunting): dump: 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.689 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1412943103 0 0) 0x7f328c138390 con 0x7f328c13c080 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.76:6790/0 conn(0x7f328c138570 legacy=0x7f328c137c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6789/0 conn(0x7f328c09ee40 legacy=0x7f328c137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: _finish_hunting 0 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 monclient: found mon.a 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c071ea0 con 0x7f328c13c080 2024-09-16T06:58:58.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: _finish_auth 0 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: _check_auth_tickets 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 5 monclient: authenticate success, global_id 334443 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _renew_subs 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c0a0780 con 0x7f328c13c080 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _renew_subs 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c0903b0 con 0x7f328c13c080 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3284005070 con 0x7f328c13c080 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3284005380 con 0x7f328c13c080 2024-09-16T06:58:58.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f328affd700 4 set_mon_vals no callback set 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 10 monclient: dump: 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.692 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f326c006590 con 0x7f326c002830 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.677+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f32840055a0 con 0x7f328c13c080 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f32840444c0 con 0x7f328c13c080 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999956464 0 0) 0x7f326c006590 con 0x7f326c002830 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient(hunting): my global_id is 284914 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c138390 con 0x7f326c002830 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f328bfff700 10 monclient: get_auth_request con 0x7f32700362a0 auth_method 0 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1044258091 0 0) 0x7f328c138390 con 0x7f326c002830 2024-09-16T06:58:58.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f326c006590 con 0x7f326c002830 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f32800036a0 con 0x7f326c002830 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient: dump: 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.694 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 384430769 0 0) 0x7f326c006590 con 0x7f326c002830 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f326c00a190 con 0x7f326c002830 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f32800046d0 con 0x7f326c002830 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6789/0 conn(0x7f326c002830 legacy=0x7f326c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.681+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6789/0 conn(0x7f326c002830 legacy=0x7f326c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:58.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f3291cc1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f326c009e50 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132434456 0 0) 0x7f326c009e50 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient(hunting): _negotiate 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient(hunting): my global_id is 284917 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c006590 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3144879981 0 0) 0x7f326c006590 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f326c009e50 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3280003df0 con 0x7f326c009170 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:58:58.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.689+0000 7f32897fa700 10 monclient: dump: 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 393644486 0 0) 0x7f326c009e50 con 0x7f326c009170 2024-09-16T06:58:58.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f326c00c340 con 0x7f326c009170 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4037 (unknown 0 0 2444423774) 0x7f3280004120 con 0x7f326c009170 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6789/0 conn(0x7f326c009170 legacy=0x7f326c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32897fa700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6789/0 conn(0x7f326c009170 legacy=0x7f326c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:58:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 180, 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:58:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:58:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:58:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:58:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:58:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:58:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:58:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:58:58.711 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.189:6812/3098810674 conn(0x7f32700362a0 legacy=0x7f3270038760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 10 monclient: shutdown 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 >> v1:172.21.15.76:6789/0 conn(0x7f328c13c080 legacy=0x7f328c1d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32914c0700 1 -- 172.21.15.76:0/3725195614 reap_dead start 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f32914c0700 1 -- 172.21.15.76:0/3725195614 reap_dead start 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 shutdown_connections 2024-09-16T06:58:58.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 >> 172.21.15.76:0/3725195614 conn(0x7f328c0a2300 msgr2=0x7f328c09e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:58:58.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 shutdown_connections 2024-09-16T06:58:58.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:58:58.693+0000 7f3291cc1700 1 -- 172.21.15.76:0/3725195614 wait complete. 2024-09-16T06:58:58.713 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T06:58:58.713 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T06:59:18.714 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T06:59:18.714 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T06:59:18.714 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T06:59:18.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T06:59:18.717 INFO:tasks.ceph.mon.c:Started 2024-09-16T06:59:18.717 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:59:18.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:59:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: get_monmap_and_config 2024-09-16T06:59:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: build_initial_monmap 2024-09-16T06:59:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: monmap: 2024-09-16T06:59:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:18.820 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:18.824592+0000 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:18.824592+0000 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 1 Processor -- start 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 1 -- start start 2024-09-16T06:59:18.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: init 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-b con 0x7f337c135470 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-c con 0x7f337c071640 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-a con 0x7f337c13b9f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c071400 con 0x7f337c13b9f0 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c099d60 con 0x7f337c071640 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c09a040 con 0x7f337c135470 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:18.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient(hunting): authenticate will time out at 2105.958651s 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3382158700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f337c135470 0x7f337c135850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53938/0 (socket says 172.21.15.76:53938) 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3382158700 1 -- 172.21.15.76:0/3597167985 learned_addr learned my addr 172.21.15.76:0/3597167985 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423505948 0 0) 0x7f337c071400 con 0x7f337c13b9f0 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _negotiate 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): my global_id is 334446 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33680035d0 con 0x7f337c13b9f0 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582002653 0 0) 0x7f337c099d60 con 0x7f337c071640 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _negotiate 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:18.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient(hunting): my global_id is 284920 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f337c071400 con 0x7f337c071640 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144657499 0 0) 0x7f33680035d0 con 0x7f337c13b9f0 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 >> v1:172.21.15.76:6790/0 conn(0x7f337c135470 legacy=0x7f337c135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 >> v1:172.21.15.189:6789/0 conn(0x7f337c071640 legacy=0x7f337c13a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: _finish_hunting 0 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 monclient: found mon.noname-a 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c070fc0 con 0x7f337c13b9f0 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: _finish_auth 0 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: _check_auth_tickets 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 5 monclient: authenticate success, global_id 334446 2024-09-16T06:59:18.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3374001990 con 0x7f337c13b9f0 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: dump: 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:18.828 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3374001bb0 con 0x7f337c13b9f0 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f33843bc700 4 set_mon_vals no callback set 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 1 -- 172.21.15.76:0/3597167985 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3374001f40 con 0x7f337c13b9f0 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.821+0000 7f3381156700 10 monclient: dump: 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.829 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 10 monclient: shutdown 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- 172.21.15.76:0/3597167985 >> v1:172.21.15.76:6789/0 conn(0x7f337c13b9f0 legacy=0x7f337c13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- 172.21.15.76:0/3597167985 shutdown_connections 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- 172.21.15.76:0/3597167985 >> 172.21.15.76:0/3597167985 conn(0x7f337c0a3e50 msgr2=0x7f337c0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- 172.21.15.76:0/3597167985 shutdown_connections 2024-09-16T06:59:18.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- 172.21.15.76:0/3597167985 wait complete. 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 10 monclient: build_initial_monmap 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 10 monclient: monmap: 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:18.834 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 Processor -- start 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 1 -- start start 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.825+0000 7f33843bc700 10 monclient: init 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-b con 0x7f337c071640 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-a con 0x7f337c135470 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient(hunting): picked mon.noname-c con 0x7f337c13b9f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c139f10 con 0x7f337c135470 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3381957700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f337c135470 0x7f337c1369a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44374/0 (socket says 172.21.15.76:44374) 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3381957700 1 -- 172.21.15.76:0/61777712 learned_addr learned my addr 172.21.15.76:0/61777712 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c072450 con 0x7f337c071640 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c1370b0 con 0x7f337c13b9f0 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient(hunting): authenticate will time out at 2105.965451s 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647286991 0 0) 0x7f337c139f10 con 0x7f337c135470 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:18.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): my global_id is 334449 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33600035d0 con 0x7f337c135470 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220998090 0 0) 0x7f33600035d0 con 0x7f337c135470 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f337c139f10 con 0x7f337c135470 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f336c001930 con 0x7f337c135470 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): dump: 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373826623 0 0) 0x7f337c072450 con 0x7f337c071640 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient(hunting): my global_id is 284923 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33600035d0 con 0x7f337c071640 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666399863 0 0) 0x7f337c139f10 con 0x7f337c135470 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 >> v1:172.21.15.76:6790/0 conn(0x7f337c13b9f0 legacy=0x7f337c1d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 >> v1:172.21.15.189:6789/0 conn(0x7f337c071640 legacy=0x7f337c136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 monclient: found mon.a 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c09c520 con 0x7f337c135470 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: _finish_auth 0 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 5 monclient: authenticate success, global_id 334449 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _renew_subs 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c13f260 con 0x7f337c135470 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _renew_subs 2024-09-16T06:59:18.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c1387e0 con 0x7f337c135470 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f336c001c50 con 0x7f337c135470 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f336c003b90 con 0x7f337c135470 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3380955700 4 set_mon_vals no callback set 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 10 monclient: dump: 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:18.840 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f336c003df0 con 0x7f337c135470 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.829+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f336c043f00 con 0x7f337c135470 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.833+0000 7f33843bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.833+0000 7f33843bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.833+0000 7f33843bc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.833+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3364005100 con 0x7f337c135470 2024-09-16T06:59:18.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.833+0000 7f3382158700 10 monclient: get_auth_request con 0x7f3360036310 auth_method 0 2024-09-16T06:59:18.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.837+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f336c0148a0 con 0x7f337c135470 2024-09-16T06:59:18.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.837+0000 7f3372ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:18.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:18.837+0000 7f3372ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:19.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:59:19.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:59:19.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3364005930 con 0x7f337c135470 2024-09-16T06:59:19.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f3372ffd700 1 -- 172.21.15.76:0/61777712 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 271970198) 0x7f336c012d50 con 0x7f337c135470 2024-09-16T06:59:19.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f3372ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:59:19.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f3372ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:19.069 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:59:19.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3360036310 legacy=0x7f33600387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 10 monclient: shutdown 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 >> v1:172.21.15.76:6789/0 conn(0x7f337c135470 legacy=0x7f337c1369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 shutdown_connections 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 >> 172.21.15.76:0/61777712 conn(0x7f337c0a3e50 msgr2=0x7f337c09aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 shutdown_connections 2024-09-16T06:59:19.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.069+0000 7f33843bc700 1 -- 172.21.15.76:0/61777712 wait complete. 2024-09-16T06:59:19.083 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:59:19.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T06:59:19.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T06:59:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: get_monmap_and_config 2024-09-16T06:59:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: monmap: 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:19.197548+0000 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:19.197548+0000 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:19.193 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 1 Processor -- start 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 1 -- start start 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: init 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient(hunting): picked mon.noname-b con 0x7f873409ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient(hunting): picked mon.noname-a con 0x7f8734098be0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient(hunting): picked mon.noname-c con 0x7f8734099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873409d560 con 0x7f8734098be0 2024-09-16T06:59:19.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873409f140 con 0x7f8734099030 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873409d790 con 0x7f873409ed60 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient(hunting): authenticate will time out at 2106.330791s 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873a506700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f873409ed60 0x7f87340984d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53962/0 (socket says 172.21.15.76:53962) 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873a506700 1 -- 172.21.15.76:0/2279224315 learned_addr learned my addr 172.21.15.76:0/2279224315 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455940025 0 0) 0x7f873409d790 con 0x7f873409ed60 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): my global_id is 374099 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87200035d0 con 0x7f873409ed60 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671550183 0 0) 0x7f873409d560 con 0x7f8734098be0 2024-09-16T06:59:19.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient(hunting): my global_id is 334452 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f873409d790 con 0x7f8734098be0 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4012739648 0 0) 0x7f87200035d0 con 0x7f873409ed60 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 >> v1:172.21.15.189:6789/0 conn(0x7f8734099030 legacy=0x7f873413ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 >> v1:172.21.15.76:6789/0 conn(0x7f8734098be0 legacy=0x7f87341374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 monclient: found mon.noname-b 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873409f620 con 0x7f873409ed60 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f872c001990 con 0x7f873409ed60 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: dump: 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.207 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 5 monclient: authenticate success, global_id 374099 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f872c001c70 con 0x7f873409ed60 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 1 -- 172.21.15.76:0/2279224315 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f872c002000 con 0x7f873409ed60 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f873b508700 4 set_mon_vals no callback set 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.193+0000 7f8739504700 10 monclient: dump: 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.208 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.209 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: shutdown 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2279224315 >> v1:172.21.15.76:6790/0 conn(0x7f873409ed60 legacy=0x7f87340984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2279224315 shutdown_connections 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2279224315 >> 172.21.15.76:0/2279224315 conn(0x7f87340a0030 msgr2=0x7f87340a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2279224315 shutdown_connections 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2279224315 wait complete. 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: monmap: 2024-09-16T06:59:19.210 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 Processor -- start 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- start start 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: init 2024-09-16T06:59:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient(hunting): picked mon.noname-c con 0x7f8734098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient(hunting): picked mon.noname-b con 0x7f8734099030 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient(hunting): picked mon.noname-a con 0x7f873409ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340a1680 con 0x7f873409ed60 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8734093e90 con 0x7f8734099030 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8734094170 con 0x7f8734098be0 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient(hunting): authenticate will time out at 2106.334427s 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873a506700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8734098be0 0x7f873406be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53970/0 (socket says 172.21.15.76:53970) 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873a506700 1 -- 172.21.15.76:0/2865335218 learned_addr learned my addr 172.21.15.76:0/2865335218 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873ad07700 1 --1- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f873409ed60 0x7f87341d2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44392/0 (socket says 172.21.15.76:44392) 2024-09-16T06:59:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287672387 0 0) 0x7f87340a1680 con 0x7f873409ed60 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): my global_id is 334455 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87180035d0 con 0x7f873409ed60 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244530691 0 0) 0x7f8734094170 con 0x7f8734098be0 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): my global_id is 374102 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87340a1680 con 0x7f8734098be0 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696324437 0 0) 0x7f8734093e90 con 0x7f8734099030 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): my global_id is 284929 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734094170 con 0x7f8734099030 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1510539408 0 0) 0x7f87340a1680 con 0x7f8734098be0 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8734093e90 con 0x7f8734098be0 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f872c001950 con 0x7f8734098be0 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): dump: 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764693708 0 0) 0x7f87180035d0 con 0x7f873409ed60 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87340a1680 con 0x7f873409ed60 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f87300017e0 con 0x7f873409ed60 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient(hunting): dump: 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.216 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2624996144 0 0) 0x7f8734093e90 con 0x7f8734098be0 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.189:6789/0 conn(0x7f8734099030 legacy=0x7f873406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f873409ed60 legacy=0x7f87341d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 monclient: found mon.c 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873406ccb0 con 0x7f8734098be0 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f872affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 5 monclient: authenticate success, global_id 374102 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: _renew_subs 2024-09-16T06:59:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.197+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87340a1250 con 0x7f8734098be0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873b508700 10 monclient: _renew_subs 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873b508700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8734092c50 con 0x7f8734098be0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f872c005020 con 0x7f8734098be0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f872c0053b0 con 0x7f8734098be0 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f8738d03700 4 set_mon_vals no callback set 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: dump: 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873b508700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f871c006590 con 0x7f871c002830 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f872c005510 con 0x7f8734098be0 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f872c044250 con 0x7f8734098be0 2024-09-16T06:59:19.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f873ad07700 10 monclient: get_auth_request con 0x7f87180363f0 auth_method 0 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111911288 0 0) 0x7f871c006590 con 0x7f871c002830 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient(hunting): my global_id is 334458 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734093e90 con 0x7f871c002830 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2763717566 0 0) 0x7f8734093e90 con 0x7f871c002830 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c006590 con 0x7f871c002830 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8724001660 con 0x7f871c002830 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: dump: 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.220 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3583748757 0 0) 0x7f871c006590 con 0x7f871c002830 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f871c00a190 con 0x7f871c002830 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f8724001a90 con 0x7f871c002830 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:19.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f871c002830 legacy=0x7f871c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.201+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f871c002830 legacy=0x7f871c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.213+0000 7f873b508700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.213+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f871c009e50 con 0x7f871c009170 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471960056 0 0) 0x7f871c009e50 con 0x7f871c009170 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient(hunting): my global_id is 334461 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c006590 con 0x7f871c009170 2024-09-16T06:59:19.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1714567716 0 0) 0x7f871c006590 con 0x7f871c009170 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c009e50 con 0x7f871c009170 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8724003840 con 0x7f871c009170 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient: dump: 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 246765517 0 0) 0x7f871c009e50 con 0x7f871c009170 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 --> v1:172.21.15.76:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f871c00c340 con 0x7f871c009170 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 <== mon.0 v1:172.21.15.76:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4212 (unknown 0 0 815510436) 0x7f8724005b00 con 0x7f871c009170 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f871c009170 legacy=0x7f871c009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f872affd700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6789/0 conn(0x7f871c009170 legacy=0x7f871c009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.225 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 184, 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:59:19.226 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.227 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.228 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:59:19.229 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.230 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.231 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.232 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:59:19.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.234 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.236 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:59:19.237 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 5 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr": [ 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:59:19.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.240 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:59:19.240 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.189:6812/3098810674 conn(0x7f87180363f0 legacy=0x7f87180388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 10 monclient: shutdown 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 >> v1:172.21.15.76:6790/0 conn(0x7f8734098be0 legacy=0x7f873406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873ad07700 1 -- 172.21.15.76:0/2865335218 reap_dead start 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873ad07700 1 -- 172.21.15.76:0/2865335218 reap_dead start 2024-09-16T06:59:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 shutdown_connections 2024-09-16T06:59:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 >> 172.21.15.76:0/2865335218 conn(0x7f87340a0030 msgr2=0x7f87341bb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 shutdown_connections 2024-09-16T06:59:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.217+0000 7f873b508700 1 -- 172.21.15.76:0/2865335218 wait complete. 2024-09-16T06:59:19.243 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:59:19.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T06:59:19.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: get_monmap_and_config 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: monmap: 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:19.366015+0000 2024-09-16T06:59:19.361 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:19.366015+0000 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:19.362 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 1 Processor -- start 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 1 -- start start 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: init 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-a con 0x7f46e40973c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-b con 0x7f46e4097eb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-c con 0x7f46e41400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e4099350 con 0x7f46e40973c0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e409f690 con 0x7f46e41400d0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e4099bf0 con 0x7f46e4097eb0 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e9f1e700 10 monclient(hunting): authenticate will time out at 2106.499308s 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e3fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f46e4097eb0 0x7f46e413e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53972/0 (socket says 172.21.15.76:53972) 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e3fff700 1 -- 172.21.15.76:0/2639448031 learned_addr learned my addr 172.21.15.76:0/2639448031 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451376481 0 0) 0x7f46e4099bf0 con 0x7f46e4097eb0 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.361+0000 7f46e37fe700 10 monclient(hunting): my global_id is 374105 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46cc0035d0 con 0x7f46e4097eb0 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835395360 0 0) 0x7f46e4099350 con 0x7f46e40973c0 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): my global_id is 334464 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4099bf0 con 0x7f46e40973c0 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873833158 0 0) 0x7f46e409f690 con 0x7f46e41400d0 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient(hunting): my global_id is 284932 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4099350 con 0x7f46e41400d0 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1522800522 0 0) 0x7f46cc0035d0 con 0x7f46e4097eb0 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 >> v1:172.21.15.189:6789/0 conn(0x7f46e41400d0 legacy=0x7f46e41424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 >> v1:172.21.15.76:6789/0 conn(0x7f46e40973c0 legacy=0x7f46e40977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 monclient: found mon.noname-b 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4098360 con 0x7f46e4097eb0 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 5 monclient: authenticate success, global_id 374105 2024-09-16T06:59:19.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d4001610 con 0x7f46e4097eb0 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: dump: 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.378 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.379 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.379 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.379 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.379 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.379 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d40018f0 con 0x7f46e4097eb0 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 4 set_mon_vals no callback set 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 1 -- 172.21.15.76:0/2639448031 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d4001c80 con 0x7f46e4097eb0 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e37fe700 10 monclient: dump: 2024-09-16T06:59:19.381 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.383 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.383 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.383 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: shutdown 2024-09-16T06:59:19.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- 172.21.15.76:0/2639448031 >> v1:172.21.15.76:6790/0 conn(0x7f46e4097eb0 legacy=0x7f46e413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- 172.21.15.76:0/2639448031 shutdown_connections 2024-09-16T06:59:19.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- 172.21.15.76:0/2639448031 >> 172.21.15.76:0/2639448031 conn(0x7f46e40a0030 msgr2=0x7f46e40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- 172.21.15.76:0/2639448031 shutdown_connections 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- 172.21.15.76:0/2639448031 wait complete. 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: monmap: 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.385 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:19.386 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 Processor -- start 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- start start 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: init 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-c con 0x7f46e40973c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-a con 0x7f46e4097eb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient(hunting): picked mon.noname-b con 0x7f46e41400d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e4072660 con 0x7f46e4097eb0 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e413f450 con 0x7f46e41400d0 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e406c5a0 con 0x7f46e40973c0 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e9f1e700 10 monclient(hunting): authenticate will time out at 2106.503474s 2024-09-16T06:59:19.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e8f1c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f46e40973c0 0x7f46e413c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53978/0 (socket says 172.21.15.76:53978) 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e8f1c700 1 -- 172.21.15.76:0/1174074881 learned_addr learned my addr 172.21.15.76:0/1174074881 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f46e4097eb0 0x7f46e413cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44434/0 (socket says 172.21.15.76:44434) 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343463158 0 0) 0x7f46e406c5a0 con 0x7f46e40973c0 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.365+0000 7f46e17fa700 10 monclient(hunting): my global_id is 374108 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d00035d0 con 0x7f46e40973c0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631684661 0 0) 0x7f46e4072660 con 0x7f46e4097eb0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): my global_id is 334467 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e406c5a0 con 0x7f46e4097eb0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 622515336 0 0) 0x7f46d00035d0 con 0x7f46e40973c0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e4072660 con 0x7f46e40973c0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d80019d0 con 0x7f46e40973c0 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): dump: 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:19.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146296758 0 0) 0x7f46e413f450 con 0x7f46e41400d0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): my global_id is 284935 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d00035d0 con 0x7f46e41400d0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1600453921 0 0) 0x7f46e4072660 con 0x7f46e40973c0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.189:6789/0 conn(0x7f46e41400d0 legacy=0x7f46e41d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6789/0 conn(0x7f46e4097eb0 legacy=0x7f46e413cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 monclient: found mon.c 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e413fda0 con 0x7f46e40973c0 2024-09-16T06:59:19.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 5 monclient: authenticate success, global_id 374108 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 10 monclient: _renew_subs 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e409a540 con 0x7f46e40973c0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 10 monclient: _renew_subs 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e413f930 con 0x7f46e40973c0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d8001c40 con 0x7f46e40973c0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d8003b70 con 0x7f46e40973c0 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e2ffd700 4 set_mon_vals no callback set 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: dump: 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46c4006590 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f46d8003d50 con 0x7f46e40973c0 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f46d8043de0 con 0x7f46e40973c0 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440276101 0 0) 0x7f46c4006590 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient(hunting): my global_id is 374111 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e4072660 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e971d700 10 monclient: get_auth_request con 0x7f46d00360d0 auth_method 0 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2743131264 0 0) 0x7f46e4072660 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c4006590 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d4004a30 con 0x7f46c4002830 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 10 monclient: dump: 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1625095798 0 0) 0x7f46c4006590 con 0x7f46c4002830 2024-09-16T06:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.369+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46c400a190 con 0x7f46c4002830 2024-09-16T06:59:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.373+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f46d4004e20 con 0x7f46c4002830 2024-09-16T06:59:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.373+0000 7f46e17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.373+0000 7f46e17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.373+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6790/0 conn(0x7f46c4002830 legacy=0x7f46c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.373+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6790/0 conn(0x7f46c4002830 legacy=0x7f46c4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f46c4009e50 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566679567 0 0) 0x7f46c4009e50 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient(hunting): my global_id is 374114 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c4006590 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3293431094 0 0) 0x7f46c4006590 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c4009e50 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f46d4005480 con 0x7f46c4009170 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient: dump: 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1767602089 0 0) 0x7f46c4009e50 con 0x7f46c4009170 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 --> v1:172.21.15.76:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46c400c340 con 0x7f46c4009170 2024-09-16T06:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 <== mon.2 v1:172.21.15.76:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3882 (unknown 0 0 3796476391) 0x7f46d4006810 con 0x7f46c4009170 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6790/0 conn(0x7f46c4009170 legacy=0x7f46c4009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e17fa700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6790/0 conn(0x7f46c4009170 legacy=0x7f46c4009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "leader", 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 184, 2024-09-16T06:59:19.401 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.402 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.403 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:59:19.404 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.405 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:59:19.406 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.407 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.408 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:59:19.409 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:59:19.410 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.410 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.410 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.410 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.411 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.412 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.413 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.414 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.415 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.416 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:59:19.417 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.418 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.418 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.418 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.418 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.418 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:59:19.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.420 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 2 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.421 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.422 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:59:19.422 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:19.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.189:6812/3098810674 conn(0x7f46d00360d0 legacy=0x7f46d0038590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 10 monclient: shutdown 2024-09-16T06:59:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 >> v1:172.21.15.76:6790/0 conn(0x7f46e40973c0 legacy=0x7f46e413c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e971d700 1 -- 172.21.15.76:0/1174074881 reap_dead start 2024-09-16T06:59:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 shutdown_connections 2024-09-16T06:59:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.381+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 >> 172.21.15.76:0/1174074881 conn(0x7f46e40a0030 msgr2=0x7f46e4096260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.385+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 shutdown_connections 2024-09-16T06:59:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.385+0000 7f46e9f1e700 1 -- 172.21.15.76:0/1174074881 wait complete. 2024-09-16T06:59:19.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:59:19.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T06:59:19.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.549+0000 7f5f6bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.549+0000 7f5f6bfff700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.549+0000 7f5f6bfff700 10 monclient: monmap: 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:19.555863+0000 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:19.555863+0000 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.551 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.552 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.552 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:19.552 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:19.552 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 Processor -- start 2024-09-16T06:59:19.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- start start 2024-09-16T06:59:19.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient: init 2024-09-16T06:59:19.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c09ed60 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c098be0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c09d560 con 0x7f5f6c099030 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c09f140 con 0x7f5f6c09ed60 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c09d790 con 0x7f5f6c098be0 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient(hunting): authenticate will time out at 2106.689205s 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6b7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5f6c099030 0x7f5f6c13ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44446/0 (socket says 172.21.15.76:44446) 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6a7fc700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f5f6c098be0 0x7f5f6c1374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:53996/0 (socket says 172.21.15.76:53996) 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6b7fe700 1 -- 172.21.15.76:0/1105957223 learned_addr learned my addr 172.21.15.76:0/1105957223 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696435821 0 0) 0x7f5f6c09d560 con 0x7f5f6c099030 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): my global_id is 334470 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f580035d0 con 0x7f5f6c099030 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691098305 0 0) 0x7f5f6c09d790 con 0x7f5f6c098be0 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): my global_id is 374117 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c09d560 con 0x7f5f6c098be0 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928663959 0 0) 0x7f5f6c09f140 con 0x7f5f6c09ed60 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient(hunting): my global_id is 284938 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c09d790 con 0x7f5f6c09ed60 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394159460 0 0) 0x7f5f580035d0 con 0x7f5f6c099030 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 >> v1:172.21.15.76:6790/0 conn(0x7f5f6c098be0 legacy=0x7f5f6c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 >> v1:172.21.15.189:6789/0 conn(0x7f5f6c09ed60 legacy=0x7f5f6c0984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 monclient: found mon.noname-a 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c09f620 con 0x7f5f6c099030 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f54001610 con 0x7f5f6c099030 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 5 monclient: authenticate success, global_id 334470 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: dump: 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.560 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f540018f0 con 0x7f5f6c099030 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 1 -- 172.21.15.76:0/1105957223 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f54001cc0 con 0x7f5f6c099030 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 4 set_mon_vals no callback set 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f69ffb700 10 monclient: dump: 2024-09-16T06:59:19.561 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 10 monclient: shutdown 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- 172.21.15.76:0/1105957223 >> v1:172.21.15.76:6789/0 conn(0x7f5f6c099030 legacy=0x7f5f6c13ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- 172.21.15.76:0/1105957223 shutdown_connections 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.553+0000 7f5f6bfff700 1 -- 172.21.15.76:0/1105957223 >> 172.21.15.76:0/1105957223 conn(0x7f5f6c0a0030 msgr2=0x7f5f6c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- 172.21.15.76:0/1105957223 shutdown_connections 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- 172.21.15.76:0/1105957223 wait complete. 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient: monmap: 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.563 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:19.564 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:19.564 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 Processor -- start 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- start start 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient: init 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c098be0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c099030 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c09ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c09a540 con 0x7f5f6c099030 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c091b30 con 0x7f5f6c098be0 2024-09-16T06:59:19.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c091780 con 0x7f5f6c09ed60 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6bfff700 10 monclient(hunting): authenticate will time out at 2106.695642s 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6a7fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5f6c099030 0x7f5f6c091070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44458/0 (socket says 172.21.15.76:44458) 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.557+0000 7f5f6a7fc700 1 -- 172.21.15.76:0/3188496494 learned_addr learned my addr 172.21.15.76:0/3188496494 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621717132 0 0) 0x7f5f6c09a540 con 0x7f5f6c099030 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): my global_id is 334473 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c0035d0 con 0x7f5f6c099030 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107530134 0 0) 0x7f5f6c091b30 con 0x7f5f6c098be0 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): my global_id is 284941 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c09a540 con 0x7f5f6c098be0 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765668711 0 0) 0x7f5f6c091780 con 0x7f5f6c09ed60 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): my global_id is 374120 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c091b30 con 0x7f5f6c09ed60 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393926426 0 0) 0x7f5f4c0035d0 con 0x7f5f6c099030 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f6c091780 con 0x7f5f6c099030 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f5c001910 con 0x7f5f6c099030 2024-09-16T06:59:19.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): dump: 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159676214 0 0) 0x7f5f6c09a540 con 0x7f5f6c098be0 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f4c0035d0 con 0x7f5f6c098be0 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f64001ae0 con 0x7f5f6c098be0 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): dump: 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.573 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909474787 0 0) 0x7f5f6c091b30 con 0x7f5f6c09ed60 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f6c09a540 con 0x7f5f6c09ed60 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f54001a60 con 0x7f5f6c09ed60 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): dump: 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.574 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710075518 0 0) 0x7f5f6c091780 con 0x7f5f6c099030 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.76:6790/0 conn(0x7f5f6c09ed60 legacy=0x7f5f6c1d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6789/0 conn(0x7f5f6c098be0 legacy=0x7f5f6c090960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 monclient: found mon.a 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c091ea0 con 0x7f5f6c099030 2024-09-16T06:59:19.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 5 monclient: authenticate success, global_id 334473 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 10 monclient: _renew_subs 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c092cf0 con 0x7f5f6c099030 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 10 monclient: _renew_subs 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c092fd0 con 0x7f5f6c099030 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f5c004890 con 0x7f5f6c099030 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f5c004ba0 con 0x7f5f6c099030 2024-09-16T06:59:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f697fa700 4 set_mon_vals no callback set 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient: dump: 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5f50006590 con 0x7f5f50002830 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f5f5c00f610 con 0x7f5f6c099030 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f5f5c043e50 con 0x7f5f6c099030 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226225754 0 0) 0x7f5f50006590 con 0x7f5f50002830 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 10 monclient(hunting): my global_id is 284944 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c091780 con 0x7f5f50002830 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.561+0000 7f5f6b7fe700 10 monclient: get_auth_request con 0x7f5f4c036700 auth_method 0 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2578817672 0 0) 0x7f5f6c091780 con 0x7f5f50002830 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f50006590 con 0x7f5f50002830 2024-09-16T06:59:19.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f64003a40 con 0x7f5f50002830 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 10 monclient: dump: 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 298388339 0 0) 0x7f5f50006590 con 0x7f5f50002830 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f5000a190 con 0x7f5f50002830 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (unknown 0 0 2944672742) 0x7f5f64003d90 con 0x7f5f50002830 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6789/0 conn(0x7f5f50002830 legacy=0x7f5f50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.565+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6789/0 conn(0x7f5f50002830 legacy=0x7f5f50004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f6bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5f50009e50 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528039914 0 0) 0x7f5f50009e50 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient(hunting): my global_id is 284947 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f50006590 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4117318487 0 0) 0x7f5f50006590 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f50009e50 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f5f640043a0 con 0x7f5f50009170 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 10 monclient: dump: 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3915817560 0 0) 0x7f5f50009e50 con 0x7f5f50009170 2024-09-16T06:59:19.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.577+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 --> v1:172.21.15.189:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f5000c340 con 0x7f5f50009170 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 <== mon.1 v1:172.21.15.189:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4046 (unknown 0 0 3590712530) 0x7f5f640057c0 con 0x7f5f50009170 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f637fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f637fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6789/0 conn(0x7f5f50009170 legacy=0x7f5f50009600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f637fe700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6789/0 conn(0x7f5f50009170 legacy=0x7f5f50009600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:59:19.584 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "peon", 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "election_epoch": 184, 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum": [ 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_age": 0, 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "required_con": "2449958758054445060", 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "required_mon": [ 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.585 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_con": "4540138322906710015", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "quorum_mon": [ 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.586 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "outside_quorum": [], 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "extra_probe_peers": [], 2024-09-16T06:59:19.587 INFO:teuthology.orchestra.run.smithi076.stdout: "sync_provider": [], 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": { 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1, 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "53fe3d11-e8d6-4d19-8f38-046f335b6765", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2024-09-16T06:37:45.879044Z", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release": 18, 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min_mon_release_name": "reef", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "election_strategy": 3, 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "disallowed_leaders: ": "", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false, 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "tiebreaker_mon": "", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_ranks: ": "", 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "features": { 2024-09-16T06:59:19.588 INFO:teuthology.orchestra.run.smithi076.stdout: "persistent": [ 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "kraken", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "luminous", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "mimic", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap-prune", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "nautilus", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "elector-pinging", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy", 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "reef" 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: "optional": [] 2024-09-16T06:59:19.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "mons": [ 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 0, 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "a", 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789", 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6789/0", 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 1, 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "b", 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.591 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789", 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.189:6789/0", 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "rank": 2, 2024-09-16T06:59:19.592 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "c", 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790", 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 0 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6790/0", 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "priority": 0, 2024-09-16T06:59:19.593 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "{}" 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "feature_map": { 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "mon": [ 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": [ 2024-09-16T06:59:19.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 4 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "client": [ 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "release": "reef", 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 1 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-09-16T06:59:19.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-09-16T06:59:19.596 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": false 2024-09-16T06:59:19.596 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.189:6812/3098810674 conn(0x7f5f4c036700 legacy=0x7f5f4c038bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 10 monclient: shutdown 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 >> v1:172.21.15.76:6789/0 conn(0x7f5f6c099030 legacy=0x7f5f6c091070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6b7fe700 1 -- 172.21.15.76:0/3188496494 reap_dead start 2024-09-16T06:59:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6b7fe700 1 -- 172.21.15.76:0/3188496494 reap_dead start 2024-09-16T06:59:19.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 shutdown_connections 2024-09-16T06:59:19.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 >> 172.21.15.76:0/3188496494 conn(0x7f5f6c0a0030 msgr2=0x7f5f6c06e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 shutdown_connections 2024-09-16T06:59:19.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.581+0000 7f5f6bfff700 1 -- 172.21.15.76:0/3188496494 wait complete. 2024-09-16T06:59:19.598 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T06:59:19.598 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T06:59:19.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T06:59:19.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.713+0000 7ff679847700 10 monclient: get_monmap_and_config 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.713+0000 7ff679847700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.713+0000 7ff679847700 10 monclient: monmap: 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:19.719669+0000 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:19.719669+0000 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.716 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.716 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:19.716 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:19.716 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 Processor -- start 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- start start 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient: init 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient(hunting): picked mon.noname-a con 0x7ff674132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient(hunting): picked mon.noname-c con 0x7ff67406d480 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient(hunting): picked mon.noname-b con 0x7ff67406cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67406c280 con 0x7ff674132330 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67406bfa0 con 0x7ff67406d480 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67409ace0 con 0x7ff67406cf50 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient(hunting): authenticate will time out at 2106.853138s 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff678845700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff674132330 0x7ff674132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44468/0 (socket says 172.21.15.76:44468) 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff678845700 1 -- 172.21.15.76:0/2882413729 learned_addr learned my addr 172.21.15.76:0/2882413729 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379553007 0 0) 0x7ff67406c280 con 0x7ff674132330 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): my global_id is 334476 2024-09-16T06:59:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c0035d0 con 0x7ff674132330 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442136642 0 0) 0x7ff67409ace0 con 0x7ff67406cf50 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): my global_id is 374123 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67406c280 con 0x7ff67406cf50 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201234160 0 0) 0x7ff67406bfa0 con 0x7ff67406d480 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient(hunting): my global_id is 284950 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67409ace0 con 0x7ff67406d480 2024-09-16T06:59:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522223627 0 0) 0x7ff65c0035d0 con 0x7ff674132330 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 >> v1:172.21.15.76:6790/0 conn(0x7ff67406cf50 legacy=0x7ff67413da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 >> v1:172.21.15.189:6789/0 conn(0x7ff67406d480 legacy=0x7ff67413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 monclient: found mon.noname-a 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67409ae70 con 0x7ff674132330 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 5 monclient: authenticate success, global_id 334476 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff670001610 con 0x7ff674132330 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: dump: 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.723 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6700018f0 con 0x7ff674132330 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 1 -- 172.21.15.76:0/2882413729 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff670001ae0 con 0x7ff674132330 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 4 set_mon_vals no callback set 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:19.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff66b7fe700 10 monclient: dump: 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 10 monclient: shutdown 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- 172.21.15.76:0/2882413729 >> v1:172.21.15.76:6789/0 conn(0x7ff674132330 legacy=0x7ff674132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- 172.21.15.76:0/2882413729 shutdown_connections 2024-09-16T06:59:19.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- 172.21.15.76:0/2882413729 >> 172.21.15.76:0/2882413729 conn(0x7ff6740a41c0 msgr2=0x7ff6740a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- 172.21.15.76:0/2882413729 shutdown_connections 2024-09-16T06:59:19.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.717+0000 7ff679847700 1 -- 172.21.15.76:0/2882413729 wait complete. 2024-09-16T06:59:19.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient: build_initial_monmap 2024-09-16T06:59:19.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient: monmap: 2024-09-16T06:59:19.731 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:19.731 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:19.731 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 1 Processor -- start 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 1 -- start start 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient: init 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient(hunting): picked mon.noname-b con 0x7ff67406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient(hunting): picked mon.noname-c con 0x7ff67406d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient(hunting): picked mon.noname-a con 0x7ff674132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff674133270 con 0x7ff674132330 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67409e8b0 con 0x7ff67406cf50 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff674131800 con 0x7ff67406d480 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff679847700 10 monclient(hunting): authenticate will time out at 2106.859282s 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff66bfff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 0x7ff67412db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54026/0 (socket says 172.21.15.76:54026) 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff66bfff700 1 -- 172.21.15.76:0/3960773687 learned_addr learned my addr 172.21.15.76:0/3960773687 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148744438 0 0) 0x7ff674131800 con 0x7ff67406d480 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): my global_id is 374126 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6600035d0 con 0x7ff67406d480 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949224105 0 0) 0x7ff674133270 con 0x7ff674132330 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): my global_id is 334479 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674131800 con 0x7ff674132330 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892345498 0 0) 0x7ff67409e8b0 con 0x7ff67406cf50 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 10 monclient(hunting): my global_id is 284953 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.721+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674133270 con 0x7ff67406cf50 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1773135579 0 0) 0x7ff6600035d0 con 0x7ff67406d480 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67409e8b0 con 0x7ff67406d480 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff664001990 con 0x7ff67406d480 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): dump: 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.735 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273315282 0 0) 0x7ff67409e8b0 con 0x7ff67406d480 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.189:6789/0 conn(0x7ff67406cf50 legacy=0x7ff67412d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6789/0 conn(0x7ff674132330 legacy=0x7ff67412e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 monclient: found mon.c 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff674130db0 con 0x7ff67406d480 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 5 monclient: authenticate success, global_id 374126 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 monclient: _renew_subs 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6741312d0 con 0x7ff67406d480 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 monclient: _renew_subs 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67412efa0 con 0x7ff67406d480 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff66bfff700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 legacy=0x7ff67412db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T06:59:19.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff66bfff700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 legacy=0x7ff67412db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff66bfff700 1 --1- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 0x7ff67412db90 :-1 s=OPENED pgs=1830 cs=1 l=1).handle_message read tag failed 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff66bfff700 1 --1- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 0x7ff67412db90 :-1 s=OPENED pgs=1830 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6790/0 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff67406d480 legacy=0x7ff67412db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): picked mon.c con 0x7ff660004aa0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): picked mon.a con 0x7ff674132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): picked mon.b con 0x7ff67406cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:19.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff67409e8b0 con 0x7ff674132330 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff674131800 con 0x7ff67406cf50 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff674133270 con 0x7ff660004aa0 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff679847700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504541024 0 0) 0x7ff674133270 con 0x7ff660004aa0 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7ff66000fe60 con 0x7ff660004aa0 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663852667 0 0) 0x7ff67409e8b0 con 0x7ff674132330 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:59:19.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7ff674133270 con 0x7ff674132330 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360663145 0 0) 0x7ff674131800 con 0x7ff67406cf50 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.189:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7ff67409e8b0 con 0x7ff67406cf50 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1215043874 0 0) 0x7ff66000fe60 con 0x7ff660004aa0 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.189:6789/0 conn(0x7ff67406cf50 legacy=0x7ff66000e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6789/0 conn(0x7ff674132330 legacy=0x7ff66000aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 monclient: found mon.c 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7ff6600035d0 con 0x7ff660004aa0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67412fc40 con 0x7ff660004aa0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff674133270 con 0x7ff660004aa0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff670004a30 con 0x7ff660004aa0 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: dump: 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:19.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff670004c00 con 0x7ff660004aa0 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff6697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.725+0000 7ff66a7fc700 4 set_mon_vals no callback set 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.729+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff67000f700 con 0x7ff660004aa0 2024-09-16T06:59:19.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.729+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 6 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff670043c20 con 0x7ff660004aa0 2024-09-16T06:59:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.729+0000 7ff66bfff700 10 monclient: get_auth_request con 0x7ff67406d480 auth_method 0 2024-09-16T06:59:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.733+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff670014530 con 0x7ff660004aa0 2024-09-16T06:59:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.733+0000 7ff6697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:19.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.733+0000 7ff6697fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:19.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.737+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff670012bd0 con 0x7ff660004aa0 2024-09-16T06:59:19.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.737+0000 7ff6697fa700 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T06:59:19.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.969+0000 7ff679847700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T06:59:19.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.969+0000 7ff679847700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T06:59:19.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.969+0000 7ff679847700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:19.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.969+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7ff6740672a0 con 0x7ff660004aa0 2024-09-16T06:59:19.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.973+0000 7ff6697fa700 1 -- 172.21.15.76:0/3960773687 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7ff670014350 con 0x7ff660004aa0 2024-09-16T06:59:19.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.973+0000 7ff6697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T06:59:19.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.973+0000 7ff6697fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:19.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=1905443432,config=1097430493,health=3153103335,kv=1008284096,logm=3032129792}) 2024-09-16T06:59:19.974 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=12,logm=45} crc {auth=1861548831,config=1097430493,health=141815541,kv=1588170259,logm=3744033835}) 2024-09-16T06:59:19.975 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.975 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=1905443432,config=1097430493,health=3153103335,kv=1008284096,logm=3032129792}) 2024-09-16T06:59:19.975 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.973+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=12,logm=46} crc {auth=3854372304,config=1097430493,health=1429719135,kv=86277986,logm=1533479007}) 2024-09-16T06:59:19.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007491755}) 2024-09-16T06:59:19.977 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920549453}) 2024-09-16T06:59:19.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007491755}) 2024-09-16T06:59:19.978 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.977+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401913684}) 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff67406d480 legacy=0x7ff660043920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 10 monclient: shutdown 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 >> v1:172.21.15.76:6790/0 conn(0x7ff660004aa0 legacy=0x7ff660004e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 shutdown_connections 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 >> 172.21.15.76:0/3960773687 conn(0x7ff6740a41c0 msgr2=0x7ff67409d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 shutdown_connections 2024-09-16T06:59:19.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:19.977+0000 7ff679847700 1 -- 172.21.15.76:0/3960773687 wait complete. 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937726558}) 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039702914}) 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937726558}) 2024-09-16T06:59:19.980 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.981+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921406048}) 2024-09-16T06:59:19.983 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.983 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227193522}) 2024-09-16T06:59:19.983 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851841802}) 2024-09-16T06:59:19.984 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.984 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227193522}) 2024-09-16T06:59:19.984 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339200397}) 2024-09-16T06:59:19.987 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.987 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.985+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187762795}) 2024-09-16T06:59:19.987 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158646870}) 2024-09-16T06:59:19.987 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.988 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187762795}) 2024-09-16T06:59:19.988 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036670401}) 2024-09-16T06:59:19.988 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.989 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931723766}) 2024-09-16T06:59:19.989 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453959871}) 2024-09-16T06:59:19.989 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.989 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931723766}) 2024-09-16T06:59:19.989 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4112112811}) 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=26223699}) 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957345400}) 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=26223699}) 2024-09-16T06:59:19.990 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168195001}) 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065352737}) 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626799722}) 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065352737}) 2024-09-16T06:59:19.991 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.989+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483030618}) 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150713096}) 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219036460}) 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150713096}) 2024-09-16T06:59:19.992 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674990514}) 2024-09-16T06:59:19.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T06:59:19.994 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.994 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517848005}) 2024-09-16T06:59:19.994 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235025006}) 2024-09-16T06:59:19.994 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.994 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517848005}) 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3534596671}) 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727743983}) 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643066981}) 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727743983}) 2024-09-16T06:59:19.995 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.993+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580934018}) 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030578790}) 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380471174}) 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030578790}) 2024-09-16T06:59:19.996 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339855708}) 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308637096}) 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156154033}) 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308637096}) 2024-09-16T06:59:19.997 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273540782}) 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366984450}) 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155187654}) 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366984450}) 2024-09-16T06:59:19.998 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:19.997+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208374287}) 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152915677}) 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562397370}) 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152915677}) 2024-09-16T06:59:19.999 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3078113065}) 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732484953}) 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798813797}) 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732484953}) 2024-09-16T06:59:20.000 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663486350}) 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2886909356}) 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825415707}) 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2886909356}) 2024-09-16T06:59:20.001 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584238435}) 2024-09-16T06:59:20.002 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.002 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820132692}) 2024-09-16T06:59:20.002 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717917911}) 2024-09-16T06:59:20.002 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820132692}) 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.001+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1758692757}) 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660887315}) 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623758797}) 2024-09-16T06:59:20.003 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.004 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660887315}) 2024-09-16T06:59:20.004 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013942231}) 2024-09-16T06:59:20.004 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.004 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229727539}) 2024-09-16T06:59:20.004 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110453325}) 2024-09-16T06:59:20.005 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.005 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229727539}) 2024-09-16T06:59:20.005 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769082028}) 2024-09-16T06:59:20.005 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.005 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257896119}) 2024-09-16T06:59:20.006 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1116112309}) 2024-09-16T06:59:20.006 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.006 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257896119}) 2024-09-16T06:59:20.006 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=862557762}) 2024-09-16T06:59:20.006 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62795075}) 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626363370}) 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62795075}) 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.005+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3577915199}) 2024-09-16T06:59:20.007 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916564752}) 2024-09-16T06:59:20.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108850710}) 2024-09-16T06:59:20.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916564752}) 2024-09-16T06:59:20.008 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608461436}) 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743611345}) 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569592842}) 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743611345}) 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2162089640}) 2024-09-16T06:59:20.009 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059286483}) 2024-09-16T06:59:20.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979026166}) 2024-09-16T06:59:20.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059286483}) 2024-09-16T06:59:20.010 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664838060}) 2024-09-16T06:59:20.011 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.011 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679462169}) 2024-09-16T06:59:20.011 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826313598}) 2024-09-16T06:59:20.011 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.011 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679462169}) 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.009+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1564747380}) 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732880087}) 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016159663}) 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732880087}) 2024-09-16T06:59:20.012 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018641463}) 2024-09-16T06:59:20.013 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=682367175}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931874692}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=682367175}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1981989843}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334723755}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523538728}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334723755}) 2024-09-16T06:59:20.014 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470344661}) 2024-09-16T06:59:20.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.015 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563097367}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398481488}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563097367}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.013+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175831326}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280877904}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185290467}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280877904}) 2024-09-16T06:59:20.016 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671588250}) 2024-09-16T06:59:20.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544542340}) 2024-09-16T06:59:20.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322297891}) 2024-09-16T06:59:20.017 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544542340}) 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007426386}) 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252170900}) 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052663701}) 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4252170900}) 2024-09-16T06:59:20.018 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503696200}) 2024-09-16T06:59:20.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.019 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143934083}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626511008}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143934083}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.017+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440014840}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059181560}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995630980}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059181560}) 2024-09-16T06:59:20.020 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1458142070}) 2024-09-16T06:59:20.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938814551}) 2024-09-16T06:59:20.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538744197}) 2024-09-16T06:59:20.021 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938814551}) 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199313348}) 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=423613549}) 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732725896}) 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=423613549}) 2024-09-16T06:59:20.022 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3104286796}) 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=576309206}) 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3730814531}) 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=576309206}) 2024-09-16T06:59:20.023 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.021+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244527046}) 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492856020}) 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211697173}) 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492856020}) 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116706780}) 2024-09-16T06:59:20.024 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678427417}) 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179401369}) 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678427417}) 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115544330}) 2024-09-16T06:59:20.025 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140172070}) 2024-09-16T06:59:20.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976333636}) 2024-09-16T06:59:20.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140172070}) 2024-09-16T06:59:20.026 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.025+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322515623}) 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697678395}) 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789912967}) 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697678395}) 2024-09-16T06:59:20.027 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471695240}) 2024-09-16T06:59:20.028 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.028 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236939485}) 2024-09-16T06:59:20.028 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045411731}) 2024-09-16T06:59:20.028 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236939485}) 2024-09-16T06:59:20.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869275287}) 2024-09-16T06:59:20.029 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531559517}) 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760821756}) 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531559517}) 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903696745}) 2024-09-16T06:59:20.030 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987053859}) 2024-09-16T06:59:20.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171152873}) 2024-09-16T06:59:20.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.029+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987053859}) 2024-09-16T06:59:20.031 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29970425}) 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898356134}) 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847668376}) 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898356134}) 2024-09-16T06:59:20.032 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=501407627}) 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059256412}) 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761121633}) 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059256412}) 2024-09-16T06:59:20.033 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=701626666}) 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240338058}) 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717347386}) 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240338058}) 2024-09-16T06:59:20.034 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154139661}) 2024-09-16T06:59:20.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.033+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814657704}) 2024-09-16T06:59:20.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=564419993}) 2024-09-16T06:59:20.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.035 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814657704}) 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4256575971}) 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2049612416}) 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184061491}) 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.036 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2049612416}) 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449571991}) 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745692871}) 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936419799}) 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.037 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745692871}) 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769972456}) 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519933969}) 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829792142}) 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519933969}) 2024-09-16T06:59:20.038 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.037+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454055898}) 2024-09-16T06:59:20.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1768942592}) 2024-09-16T06:59:20.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808869260}) 2024-09-16T06:59:20.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.039 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1768942592}) 2024-09-16T06:59:20.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3460852135}) 2024-09-16T06:59:20.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987921872}) 2024-09-16T06:59:20.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731414587}) 2024-09-16T06:59:20.040 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.041 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987921872}) 2024-09-16T06:59:20.041 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4063229977}) 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438745049}) 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1774249170}) 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438745049}) 2024-09-16T06:59:20.042 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.041+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875635975}) 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165777064}) 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066304592}) 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165777064}) 2024-09-16T06:59:20.043 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168559003}) 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786933234}) 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035472741}) 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786933234}) 2024-09-16T06:59:20.044 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369079937}) 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223915713}) 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722024732}) 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223915713}) 2024-09-16T06:59:20.045 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166397654}) 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450029130}) 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016017879}) 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450029130}) 2024-09-16T06:59:20.046 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.045+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853230214}) 2024-09-16T06:59:20.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157025172}) 2024-09-16T06:59:20.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634278885}) 2024-09-16T06:59:20.047 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157025172}) 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3008592878}) 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549952191}) 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2978026181}) 2024-09-16T06:59:20.048 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549952191}) 2024-09-16T06:59:20.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341084017}) 2024-09-16T06:59:20.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205009725}) 2024-09-16T06:59:20.049 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871798304}) 2024-09-16T06:59:20.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=205009725}) 2024-09-16T06:59:20.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293346184}) 2024-09-16T06:59:20.050 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1227472881}) 2024-09-16T06:59:20.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791424468}) 2024-09-16T06:59:20.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1227472881}) 2024-09-16T06:59:20.051 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.049+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510972316}) 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085727142}) 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460997703}) 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085727142}) 2024-09-16T06:59:20.052 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079593615}) 2024-09-16T06:59:20.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397716941}) 2024-09-16T06:59:20.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=786205965}) 2024-09-16T06:59:20.053 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.054 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397716941}) 2024-09-16T06:59:20.054 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369059461}) 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483022020}) 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423214466}) 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483022020}) 2024-09-16T06:59:20.055 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.053+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2782572214}) 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610335480}) 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414744877}) 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610335480}) 2024-09-16T06:59:20.056 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1404526858}) 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249813453}) 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189529259}) 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249813453}) 2024-09-16T06:59:20.057 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483870807}) 2024-09-16T06:59:20.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161407334}) 2024-09-16T06:59:20.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172341578}) 2024-09-16T06:59:20.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.058 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161407334}) 2024-09-16T06:59:20.059 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.057+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008178249}) 2024-09-16T06:59:20.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807194242}) 2024-09-16T06:59:20.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278959800}) 2024-09-16T06:59:20.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.061 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807194242}) 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=949901783}) 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172869553}) 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874614778}) 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.062 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172869553}) 2024-09-16T06:59:20.063 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=289352328}) 2024-09-16T06:59:20.067 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549356133}) 2024-09-16T06:59:20.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328774958}) 2024-09-16T06:59:20.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549356133}) 2024-09-16T06:59:20.068 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.061+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221846854}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2730336072}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540215351}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2730336072}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978182657}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3867240728}) 2024-09-16T06:59:20.069 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102283002}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3867240728}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767981214}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084361595}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206036641}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084361595}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610791612}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472698030}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827259885}) 2024-09-16T06:59:20.070 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472698030}) 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.065+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3120404375}) 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844469173}) 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855306831}) 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844469173}) 2024-09-16T06:59:20.071 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777200313}) 2024-09-16T06:59:20.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.074 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087349186}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779869420}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087349186}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243170581}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660389364}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758889936}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660389364}) 2024-09-16T06:59:20.075 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2375236761}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583641272}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3894225697}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583641272}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.069+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2601075073}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819725910}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572830452}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819725910}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859050769}) 2024-09-16T06:59:20.076 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147952764}) 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026264109}) 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147952764}) 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565248879}) 2024-09-16T06:59:20.077 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164384117}) 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574529725}) 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164384117}) 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3473057626}) 2024-09-16T06:59:20.078 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628290490}) 2024-09-16T06:59:20.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917156326}) 2024-09-16T06:59:20.079 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628290490}) 2024-09-16T06:59:20.080 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.073+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436982519}) 2024-09-16T06:59:20.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754018500}) 2024-09-16T06:59:20.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983652779}) 2024-09-16T06:59:20.095 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754018500}) 2024-09-16T06:59:20.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023706986}) 2024-09-16T06:59:20.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371637922}) 2024-09-16T06:59:20.096 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524929411}) 2024-09-16T06:59:20.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371637922}) 2024-09-16T06:59:20.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.077+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1095943871}) 2024-09-16T06:59:20.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.097 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206610606}) 2024-09-16T06:59:20.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501009508}) 2024-09-16T06:59:20.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206610606}) 2024-09-16T06:59:20.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135452528}) 2024-09-16T06:59:20.098 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446244587}) 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016818860}) 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446244587}) 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130136831}) 2024-09-16T06:59:20.099 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890627492}) 2024-09-16T06:59:20.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2689260597}) 2024-09-16T06:59:20.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890627492}) 2024-09-16T06:59:20.100 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.081+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755858477}) 2024-09-16T06:59:20.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352836484}) 2024-09-16T06:59:20.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987177776}) 2024-09-16T06:59:20.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.101 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352836484}) 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247846690}) 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111748576}) 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987640668}) 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.102 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111748576}) 2024-09-16T06:59:20.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3676363371}) 2024-09-16T06:59:20.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724035202}) 2024-09-16T06:59:20.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965155400}) 2024-09-16T06:59:20.103 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724035202}) 2024-09-16T06:59:20.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860062392}) 2024-09-16T06:59:20.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600877367}) 2024-09-16T06:59:20.104 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724171363}) 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600877367}) 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.085+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245450505}) 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388577733}) 2024-09-16T06:59:20.105 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118333435}) 2024-09-16T06:59:20.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388577733}) 2024-09-16T06:59:20.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018295283}) 2024-09-16T06:59:20.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.106 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2369710822}) 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148306640}) 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2369710822}) 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1946242687}) 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.107 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1330302988}) 2024-09-16T06:59:20.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181607570}) 2024-09-16T06:59:20.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1330302988}) 2024-09-16T06:59:20.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3368027306}) 2024-09-16T06:59:20.108 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251596029}) 2024-09-16T06:59:20.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031804175}) 2024-09-16T06:59:20.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251596029}) 2024-09-16T06:59:20.109 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.089+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2913626313}) 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908063611}) 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556336006}) 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908063611}) 2024-09-16T06:59:20.110 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781589065}) 2024-09-16T06:59:20.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748018149}) 2024-09-16T06:59:20.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945051439}) 2024-09-16T06:59:20.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.111 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748018149}) 2024-09-16T06:59:20.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2973134365}) 2024-09-16T06:59:20.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710800737}) 2024-09-16T06:59:20.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753729383}) 2024-09-16T06:59:20.112 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710800737}) 2024-09-16T06:59:20.113 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.093+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2301998647}) 2024-09-16T06:59:20.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063322934}) 2024-09-16T06:59:20.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330600923}) 2024-09-16T06:59:20.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.134 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063322934}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004867426}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010286846}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339107065}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010286846}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4075549107}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880197452}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485241139}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880197452}) 2024-09-16T06:59:20.135 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.097+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587264538}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231394150}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287721554}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231394150}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663885586}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152010489}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145231283}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152010489}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952258260}) 2024-09-16T06:59:20.136 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150322981}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332755780}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150322981}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993339483}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562086531}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989918389}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562086531}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.101+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450855375}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598863901}) 2024-09-16T06:59:20.137 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319994759}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598863901}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896900796}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730655468}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023381687}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730655468}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3173437799}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416818873}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937827488}) 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.138 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416818873}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2169512180}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070662493}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282431424}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070662493}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.105+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071345737}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1286256881}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646611223}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1286256881}) 2024-09-16T06:59:20.139 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1303912648}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251291336}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067182831}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251291336}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101068938}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430525172}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263964904}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430525172}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757682830}) 2024-09-16T06:59:20.140 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792829536}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392767926}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792829536}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.109+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3730613083}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472667856}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333180077}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472667856}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1556141490}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198815865}) 2024-09-16T06:59:20.141 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713623412}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198815865}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1684980300}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965847613}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283404018}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965847613}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.113+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159349904}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3987314616}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576223932}) 2024-09-16T06:59:20.142 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3987314616}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4150692459}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957338105}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789276737}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957338105}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470200521}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1314286262}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158539673}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1314286262}) 2024-09-16T06:59:20.143 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866482021}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003367593}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921252508}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003367593}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.117+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291568369}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288368161}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403621953}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288368161}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1240945967}) 2024-09-16T06:59:20.144 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038728499}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745524319}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038728499}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874124150}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201692047}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565708055}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201692047}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.121+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202997811}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=676134485}) 2024-09-16T06:59:20.145 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080116860}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=676134485}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315088682}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462637294}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214404255}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462637294}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805572656}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700064154}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548440153}) 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.146 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700064154}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.125+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414605000}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=20} crc {logm=2950227499,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3369849016,mon_config_key=3504926273,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=4249038086,osdmap=2352400960}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=19} crc {logm=2972287758,mdsmap=1531151339,mgr=3454917385,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2460250801,mon_config_key=1137914535,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=2545952040}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=20} crc {logm=2950227499,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=3369849016,mon_config_key=3504926273,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=4249038086,osdmap=2352400960}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.129+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=17} crc {logm=1254273832,mdsmap=1531151339,mgr=1752639018,mgr_command_descs=794974419,mgr_metadata=1347778423,mgrstat=2460250801,mon_config_key=1174667612,monmap=1887442722,osd_metadata=2101314684,osd_pg_creating=3462100867,osd_snap=2033388227,osdmap=3031933149}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=3968615094}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=172646897}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=3968615094}) 2024-09-16T06:59:20.147 INFO:tasks.ceph.mon.c.smithi076.stderr:2024-09-16T06:59:20.133+0000 7f3c7c8ee700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=2783599768}) 2024-09-16T06:59:20.994 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T06:59:20.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: get_monmap_and_config 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: monmap: 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:21.108134+0000 2024-09-16T06:59:21.103 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:21.108134+0000 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:21.104 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 1 Processor -- start 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 1 -- start start 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: init 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-b con 0x7f19c0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-a con 0x7f19c006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-c con 0x7f19c006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c006c280 con 0x7f19c006d480 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c006bfa0 con 0x7f19c006cf50 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c009ace0 con 0x7f19c0132330 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient(hunting): authenticate will time out at 2108.241980s 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19beffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f19c006d480 0x7f19c013a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44486/0 (socket says 172.21.15.76:44486) 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19beffd700 1 -- 172.21.15.76:0/3507926714 learned_addr learned my addr 172.21.15.76:0/3507926714 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420821061 0 0) 0x7f19c006c280 con 0x7f19c006d480 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): my global_id is 334482 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a80035d0 con 0x7f19c006d480 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070368504 0 0) 0x7f19c009ace0 con 0x7f19c0132330 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): my global_id is 374129 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c006c280 con 0x7f19c0132330 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693694312 0 0) 0x7f19c006bfa0 con 0x7f19c006cf50 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient(hunting): my global_id is 284956 2024-09-16T06:59:21.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c009ace0 con 0x7f19c006cf50 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4104812057 0 0) 0x7f19a80035d0 con 0x7f19c006d480 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 >> v1:172.21.15.76:6790/0 conn(0x7f19c0132330 legacy=0x7f19c0132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 >> v1:172.21.15.189:6789/0 conn(0x7f19c006cf50 legacy=0x7f19c013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 monclient: found mon.noname-a 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c009ae70 con 0x7f19c006d480 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 5 monclient: authenticate success, global_id 334482 2024-09-16T06:59:21.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19b0001610 con 0x7f19c006d480 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: dump: 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.112 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b00018f0 con 0x7f19c006d480 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 1 -- 172.21.15.76:0/3507926714 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19b0001b20 con 0x7f19c006d480 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19c4b6d700 4 set_mon_vals no callback set 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.105+0000 7f19be7fc700 10 monclient: dump: 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.113 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: shutdown 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- 172.21.15.76:0/3507926714 >> v1:172.21.15.76:6789/0 conn(0x7f19c006d480 legacy=0x7f19c013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- 172.21.15.76:0/3507926714 shutdown_connections 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- 172.21.15.76:0/3507926714 >> 172.21.15.76:0/3507926714 conn(0x7f19c00a41c0 msgr2=0x7f19c00a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- 172.21.15.76:0/3507926714 shutdown_connections 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- 172.21.15.76:0/3507926714 wait complete. 2024-09-16T06:59:21.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: monmap: 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:21.115 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 Processor -- start 2024-09-16T06:59:21.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- start start 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: init 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-b con 0x7f19c006cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-a con 0x7f19c006d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient(hunting): picked mon.noname-c con 0x7f19c0132330 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c012f9e0 con 0x7f19c006d480 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0131570 con 0x7f19c006cf50 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c012f2f0 con 0x7f19c0132330 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19c4b6d700 10 monclient(hunting): authenticate will time out at 2108.246699s 2024-09-16T06:59:21.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19beffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f19c006d480 0x7f19c01cea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44496/0 (socket says 172.21.15.76:44496) 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19beffd700 1 -- 172.21.15.76:0/1855054207 learned_addr learned my addr 172.21.15.76:0/1855054207 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089192642 0 0) 0x7f19c012f2f0 con 0x7f19c0132330 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): my global_id is 374132 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac0035d0 con 0x7f19c0132330 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027491916 0 0) 0x7f19c012f9e0 con 0x7f19c006d480 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): my global_id is 334485 2024-09-16T06:59:21.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c012f2f0 con 0x7f19c006d480 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202906606 0 0) 0x7f19c0131570 con 0x7f19c006cf50 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): my global_id is 284959 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c012f9e0 con 0x7f19c006cf50 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1029829048 0 0) 0x7f19c012f2f0 con 0x7f19c006d480 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c0131570 con 0x7f19c006d480 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007292179 0 0) 0x7f19ac0035d0 con 0x7f19c0132330 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c012f2f0 con 0x7f19c0132330 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19b0004890 con 0x7f19c006d480 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): dump: 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:21.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19b8001bb0 con 0x7f19c0132330 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.109+0000 7f19a7fff700 10 monclient(hunting): dump: 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.125 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1321907981 0 0) 0x7f19c012f2f0 con 0x7f19c0132330 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 >> v1:172.21.15.189:6789/0 conn(0x7f19c006cf50 legacy=0x7f19c013e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 >> v1:172.21.15.76:6789/0 conn(0x7f19c006d480 legacy=0x7f19c01cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 monclient: found mon.c 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c012ee10 con 0x7f19c0132330 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 5 monclient: authenticate success, global_id 374132 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _renew_subs 2024-09-16T06:59:21.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c013eb70 con 0x7f19c0132330 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _renew_subs 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c0131070 con 0x7f19c0132330 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b8003a40 con 0x7f19c0132330 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f19b8003d50 con 0x7f19c0132330 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19bdffb700 4 set_mon_vals no callback set 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 10 monclient: dump: 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.127 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f19b800e7c0 con 0x7f19c0132330 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f19b8042d90 con 0x7f19c0132330 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0005100 con 0x7f19c0132330 2024-09-16T06:59:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.113+0000 7f19bf7fe700 10 monclient: get_auth_request con 0x7f19ac0361e0 auth_method 0 2024-09-16T06:59:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.121+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f19b8013730 con 0x7f19c0132330 2024-09-16T06:59:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.121+0000 7f19a7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.121+0000 7f19a7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:21.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.349+0000 7f19c4b6d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T06:59:21.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.349+0000 7f19c4b6d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T06:59:21.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.349+0000 7f19c4b6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.349+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f19a0005930 con 0x7f19c0132330 2024-09-16T06:59:21.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.349+0000 7f19a7fff700 1 -- 172.21.15.76:0/1855054207 <== mon.2 v1:172.21.15.76:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1390 (unknown 254746657 0 1681598233) 0x7f19b8011ba0 con 0x7f19c0132330 2024-09-16T06:59:21.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19a7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T06:59:21.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19a7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:21.351 INFO:teuthology.orchestra.run.smithi076.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","modified":"2024-09-16T06:37:45.879044Z","created":"2024-09-16T06:37:45.879044Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T06:59:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 >> v1:172.21.15.189:6812/3098810674 conn(0x7f19ac0361e0 legacy=0x7f19ac0386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 10 monclient: shutdown 2024-09-16T06:59:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 >> v1:172.21.15.76:6790/0 conn(0x7f19c0132330 legacy=0x7f19c01d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 shutdown_connections 2024-09-16T06:59:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.353+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 >> 172.21.15.76:0/1855054207 conn(0x7f19c00a41c0 msgr2=0x7f19c009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.357+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 shutdown_connections 2024-09-16T06:59:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.357+0000 7f19c4b6d700 1 -- 172.21.15.76:0/1855054207 wait complete. 2024-09-16T06:59:21.370 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T06:59:21.370 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T06:59:21.387 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T06:59:21.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: get_monmap_and_config 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: monmap: 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:21.508048+0000 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:21.508048+0000 2024-09-16T06:59:21.503 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.504 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.504 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.504 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:21.504 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:21.504 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 Processor -- start 2024-09-16T06:59:21.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- start start 2024-09-16T06:59:21.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: init 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3b4c1342c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3b4c06cfb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3b4c06d410 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0a31f0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c06c4f0 con 0x7f3b4c06d410 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c06bbe0 con 0x7f3b4c1342c0 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient(hunting): authenticate will time out at 2108.641336s 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b427fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3b4c06cfb0 0x7f3b4c13a280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44500/0 (socket says 172.21.15.76:44500) 2024-09-16T06:59:21.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b427fc700 1 -- 172.21.15.76:0/3226022539 learned_addr learned my addr 172.21.15.76:0/3226022539 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262591540 0 0) 0x7f3b4c0a31f0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): my global_id is 334488 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b380035d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960326491 0 0) 0x7f3b4c06bbe0 con 0x7f3b4c1342c0 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): my global_id is 374135 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b4c0a31f0 con 0x7f3b4c1342c0 2024-09-16T06:59:21.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141109172 0 0) 0x7f3b4c06c4f0 con 0x7f3b4c06d410 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient(hunting): my global_id is 284962 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b4c06bbe0 con 0x7f3b4c06d410 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1596915217 0 0) 0x7f3b380035d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 >> v1:172.21.15.76:6790/0 conn(0x7f3b4c1342c0 legacy=0x7f3b4c1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 >> v1:172.21.15.189:6789/0 conn(0x7f3b4c06d410 legacy=0x7f3b4c13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c09ace0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 5 monclient: authenticate success, global_id 334488 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b34001610 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:21.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: dump: 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b340018f0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 4 set_mon_vals no callback set 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 1 -- 172.21.15.76:0/3226022539 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b34001c80 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4a7fc700 10 monclient: dump: 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.513 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 10 monclient: shutdown 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3226022539 >> v1:172.21.15.76:6789/0 conn(0x7f3b4c06cfb0 legacy=0x7f3b4c13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3226022539 shutdown_connections 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3226022539 >> 172.21.15.76:0/3226022539 conn(0x7f3b4c0a41c0 msgr2=0x7f3b4c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3226022539 shutdown_connections 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.505+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3226022539 wait complete. 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient: monmap: 2024-09-16T06:59:21.514 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:21.515 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 1 Processor -- start 2024-09-16T06:59:21.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 1 -- start start 2024-09-16T06:59:21.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient: init 2024-09-16T06:59:21.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3b4c06cfb0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3b4c06d410 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3b4c1342c0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0a5ec0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4affd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3b4c06cfb0 0x7f3b4c137520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44508/0 (socket says 172.21.15.76:44508) 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c138340 con 0x7f3b4c06d410 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4affd700 1 -- 172.21.15.76:0/3078376850 learned_addr learned my addr 172.21.15.76:0/3078376850 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0a5970 con 0x7f3b4c1342c0 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b4bfff700 10 monclient(hunting): authenticate will time out at 2108.647188s 2024-09-16T06:59:21.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960050680 0 0) 0x7f3b4c0a5ec0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): my global_id is 334491 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0035d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198520565 0 0) 0x7f3b4c0a5970 con 0x7f3b4c1342c0 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): my global_id is 374138 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b4c0a5ec0 con 0x7f3b4c1342c0 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3207880898 0 0) 0x7f3b2c0035d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b4c0a5970 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b3c0019d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): dump: 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.523 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759637312 0 0) 0x7f3b4c138340 con 0x7f3b4c06d410 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.509+0000 7f3b48ff9700 10 monclient(hunting): my global_id is 284965 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0035d0 con 0x7f3b4c06d410 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4094406474 0 0) 0x7f3b4c0a5ec0 con 0x7f3b4c1342c0 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b4c138340 con 0x7f3b4c1342c0 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b44001c30 con 0x7f3b4c1342c0 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:21.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient(hunting): dump: 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1286170647 0 0) 0x7f3b4c0a5970 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 >> v1:172.21.15.76:6790/0 conn(0x7f3b4c1342c0 legacy=0x7f3b4c1d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 >> v1:172.21.15.189:6789/0 conn(0x7f3b4c06d410 legacy=0x7f3b4c137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 monclient: found mon.a 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c09e8b0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 5 monclient: authenticate success, global_id 334491 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _renew_subs 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c13e0d0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _renew_subs 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b3c0036a0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c139ac0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f3b3c0039b0 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 10 monclient: dump: 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.527 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b49ffb700 4 set_mon_vals no callback set 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f3b3c003b10 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b30005100 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f3b3c043e30 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.513+0000 7f3b427fc700 10 monclient: get_auth_request con 0x7f3b2c036400 auth_method 0 2024-09-16T06:59:21.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.517+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f3b3c014790 con 0x7f3b4c06cfb0 2024-09-16T06:59:21.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.517+0000 7f3b48ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.517+0000 7f3b48ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.689+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b30000b90 con 0x7f3b2c036400 2024-09-16T06:59:21.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b48ff9700 1 -- 172.21.15.76:0/3078376850 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98767 (unknown 2965378022 0 1926146659) 0x7f3b30000b90 con 0x7f3b2c036400 2024-09-16T06:59:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:21.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 >> v1:172.21.15.189:6812/3098810674 conn(0x7f3b2c036400 legacy=0x7f3b2c0388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 10 monclient: shutdown 2024-09-16T06:59:21.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:21.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 >> v1:172.21.15.76:6789/0 conn(0x7f3b4c06cfb0 legacy=0x7f3b4c137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 shutdown_connections 2024-09-16T06:59:21.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 >> 172.21.15.76:0/3078376850 conn(0x7f3b4c0a41c0 msgr2=0x7f3b4c0a60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:21.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 shutdown_connections 2024-09-16T06:59:21.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.701+0000 7f3b4bfff700 1 -- 172.21.15.76:0/3078376850 wait complete. 2024-09-16T06:59:21.701 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:59:21.714 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-09-16T06:59:20.039427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191536,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039504,"statfs":{"total":579820584960,"available":579624452096,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003918"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:21.071754+0000","last_change":"2024-09-16T06:44:52.856113+0000","last_active":"2024-09-16T06:55:21.071754+0000","last_peered":"2024-09-16T06:55:21.071754+0000","last_clean":"2024-09-16T06:55:21.071754+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:55:21.071754+0000","last_undegraded":"2024-09-16T06:55:21.071754+0000","last_fullsized":"2024-09-16T06:55:21.071754+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:26.598082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191190+0000","last_change":"2024-09-16T06:44:52.853100+0000","last_active":"2024-09-16T06:55:45.191190+0000","last_peered":"2024-09-16T06:55:45.191190+0000","last_clean":"2024-09-16T06:55:45.191190+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:55:45.191190+0000","last_undegraded":"2024-09-16T06:55:45.191190+0000","last_fullsized":"2024-09-16T06:55:45.191190+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:16.588106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.072562+0000","last_change":"2024-09-16T06:44:52.855874+0000","last_active":"2024-09-16T06:55:26.072562+0000","last_peered":"2024-09-16T06:55:26.072562+0000","last_clean":"2024-09-16T06:55:26.072562+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:55:26.072562+0000","last_undegraded":"2024-09-16T06:55:26.072562+0000","last_fullsized":"2024-09-16T06:55:26.072562+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:55.805033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191145+0000","last_change":"2024-09-16T06:44:52.854782+0000","last_active":"2024-09-16T06:55:45.191145+0000","last_peered":"2024-09-16T06:55:45.191145+0000","last_clean":"2024-09-16T06:55:45.191145+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:55:45.191145+0000","last_undegraded":"2024-09-16T06:55:45.191145+0000","last_fullsized":"2024-09-16T06:55:45.191145+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:49.724068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.193218+0000","last_change":"2024-09-16T06:44:52.859034+0000","last_active":"2024-09-16T06:55:45.193218+0000","last_peered":"2024-09-16T06:55:45.193218+0000","last_clean":"2024-09-16T06:55:45.193218+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:55:45.193218+0000","last_undegraded":"2024-09-16T06:55:45.193218+0000","last_fullsized":"2024-09-16T06:55:45.193218+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:48:37.653672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056468+0000","last_change":"2024-09-16T06:44:52.859319+0000","last_active":"2024-09-16T06:55:25.056468+0000","last_peered":"2024-09-16T06:55:25.056468+0000","last_clean":"2024-09-16T06:55:25.056468+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:55:25.056468+0000","last_undegraded":"2024-09-16T06:55:25.056468+0000","last_fullsized":"2024-09-16T06:55:25.056468+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:36.527527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":163,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:01.211371+0000","last_change":"2024-09-16T06:44:52.857054+0000","last_active":"2024-09-16T06:56:01.211371+0000","last_peered":"2024-09-16T06:56:01.211371+0000","last_clean":"2024-09-16T06:56:01.211371+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:56:01.211371+0000","last_undegraded":"2024-09-16T06:56:01.211371+0000","last_fullsized":"2024-09-16T06:56:01.211371+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:13.406841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796081+0000","last_change":"2024-09-16T06:44:52.854292+0000","last_active":"2024-09-16T06:55:18.796081+0000","last_peered":"2024-09-16T06:55:18.796081+0000","last_clean":"2024-09-16T06:55:18.796081+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:55:18.796081+0000","last_undegraded":"2024-09-16T06:55:18.796081+0000","last_fullsized":"2024-09-16T06:55:18.796081+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:55.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201207+0000","last_change":"2024-09-16T06:44:52.857665+0000","last_active":"2024-09-16T06:56:55.201207+0000","last_peered":"2024-09-16T06:56:55.201207+0000","last_clean":"2024-09-16T06:56:55.201207+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:56:55.201207+0000","last_undegraded":"2024-09-16T06:56:55.201207+0000","last_fullsized":"2024-09-16T06:56:55.201207+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:58:55.693264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":165,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207121+0000","last_change":"2024-09-16T06:44:52.857164+0000","last_active":"2024-09-16T06:55:31.207121+0000","last_peered":"2024-09-16T06:55:31.207121+0000","last_clean":"2024-09-16T06:55:31.207121+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:55:31.207121+0000","last_undegraded":"2024-09-16T06:55:31.207121+0000","last_fullsized":"2024-09-16T06:55:31.207121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:24.291104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":170,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207067+0000","last_change":"2024-09-16T06:44:52.854453+0000","last_active":"2024-09-16T06:55:31.207067+0000","last_peered":"2024-09-16T06:55:31.207067+0000","last_clean":"2024-09-16T06:55:31.207067+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:55:31.207067+0000","last_undegraded":"2024-09-16T06:55:31.207067+0000","last_fullsized":"2024-09-16T06:55:31.207067+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:49.308777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796044+0000","last_change":"2024-09-16T06:44:52.856374+0000","last_active":"2024-09-16T06:55:18.796044+0000","last_peered":"2024-09-16T06:55:18.796044+0000","last_clean":"2024-09-16T06:55:18.796044+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:55:18.796044+0000","last_undegraded":"2024-09-16T06:55:18.796044+0000","last_fullsized":"2024-09-16T06:55:18.796044+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:52.978402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.055715+0000","last_change":"2024-09-16T06:44:53.857627+0000","last_active":"2024-09-16T06:55:20.055715+0000","last_peered":"2024-09-16T06:55:20.055715+0000","last_clean":"2024-09-16T06:55:20.055715+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:55:20.055715+0000","last_undegraded":"2024-09-16T06:55:20.055715+0000","last_fullsized":"2024-09-16T06:55:20.055715+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:18:22.103962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.795979+0000","last_change":"2024-09-16T06:44:53.857508+0000","last_active":"2024-09-16T06:55:18.795979+0000","last_peered":"2024-09-16T06:55:18.795979+0000","last_clean":"2024-09-16T06:55:18.795979+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:55:18.795979+0000","last_undegraded":"2024-09-16T06:55:18.795979+0000","last_fullsized":"2024-09-16T06:55:18.795979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:09.945570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":116,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:16.079853+0000","last_change":"2024-09-16T06:44:53.857597+0000","last_active":"2024-09-16T06:56:16.079853+0000","last_peered":"2024-09-16T06:56:16.079853+0000","last_clean":"2024-09-16T06:56:16.079853+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:56:16.079853+0000","last_undegraded":"2024-09-16T06:56:16.079853+0000","last_fullsized":"2024-09-16T06:56:16.079853+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:53.321544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:00.070430+0000","last_change":"2024-09-16T06:44:52.857229+0000","last_active":"2024-09-16T06:57:00.070430+0000","last_peered":"2024-09-16T06:57:00.070430+0000","last_clean":"2024-09-16T06:57:00.070430+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:57:00.070430+0000","last_undegraded":"2024-09-16T06:57:00.070430+0000","last_fullsized":"2024-09-16T06:57:00.070430+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:03.913790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.203557+0000","last_change":"2024-09-16T06:44:55.870279+0000","last_active":"2024-09-16T06:56:55.203557+0000","last_peered":"2024-09-16T06:56:55.203557+0000","last_clean":"2024-09-16T06:56:55.203557+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:56:55.203557+0000","last_undegraded":"2024-09-16T06:56:55.203557+0000","last_fullsized":"2024-09-16T06:56:55.203557+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:03.350848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":245,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.189959+0000","last_change":"2024-09-16T06:44:52.855544+0000","last_active":"2024-09-16T06:55:25.189959+0000","last_peered":"2024-09-16T06:55:25.189959+0000","last_clean":"2024-09-16T06:55:25.189959+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:55:25.189959+0000","last_undegraded":"2024-09-16T06:55:25.189959+0000","last_fullsized":"2024-09-16T06:55:25.189959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:35.351132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201179+0000","last_change":"2024-09-16T06:44:52.858722+0000","last_active":"2024-09-16T06:56:55.201179+0000","last_peered":"2024-09-16T06:56:55.201179+0000","last_clean":"2024-09-16T06:56:55.201179+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:56:55.201179+0000","last_undegraded":"2024-09-16T06:56:55.201179+0000","last_fullsized":"2024-09-16T06:56:55.201179+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:45.458295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796021+0000","last_change":"2024-09-16T06:44:52.856469+0000","last_active":"2024-09-16T06:55:18.796021+0000","last_peered":"2024-09-16T06:55:18.796021+0000","last_clean":"2024-09-16T06:55:18.796021+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:55:18.796021+0000","last_undegraded":"2024-09-16T06:55:18.796021+0000","last_fullsized":"2024-09-16T06:55:18.796021+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:31.216149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":280,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.206353+0000","last_change":"2024-09-16T06:44:52.866089+0000","last_active":"2024-09-16T06:55:26.206353+0000","last_peered":"2024-09-16T06:55:26.206353+0000","last_clean":"2024-09-16T06:55:26.206353+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:55:26.206353+0000","last_undegraded":"2024-09-16T06:55:26.206353+0000","last_fullsized":"2024-09-16T06:55:26.206353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:55.504470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056440+0000","last_change":"2024-09-16T06:44:52.856612+0000","last_active":"2024-09-16T06:55:25.056440+0000","last_peered":"2024-09-16T06:55:25.056440+0000","last_clean":"2024-09-16T06:55:25.056440+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:55:25.056440+0000","last_undegraded":"2024-09-16T06:55:25.056440+0000","last_fullsized":"2024-09-16T06:55:25.056440+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:02.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.189258+0000","last_change":"2024-09-16T06:44:52.844681+0000","last_active":"2024-09-16T06:55:20.189258+0000","last_peered":"2024-09-16T06:55:20.189258+0000","last_clean":"2024-09-16T06:55:20.189258+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:55:20.189258+0000","last_undegraded":"2024-09-16T06:55:20.189258+0000","last_fullsized":"2024-09-16T06:55:20.189258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:47:26.834608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:46.075513+0000","last_change":"2024-09-16T06:44:52.848857+0000","last_active":"2024-09-16T06:55:46.075513+0000","last_peered":"2024-09-16T06:55:46.075513+0000","last_clean":"2024-09-16T06:55:46.075513+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:55:46.075513+0000","last_undegraded":"2024-09-16T06:55:46.075513+0000","last_fullsized":"2024-09-16T06:55:46.075513+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:46.816057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:55.194847+0000","last_change":"2024-09-16T06:44:53.857350+0000","last_active":"2024-09-16T06:55:55.194847+0000","last_peered":"2024-09-16T06:55:55.194847+0000","last_clean":"2024-09-16T06:55:55.194847+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:55:55.194847+0000","last_undegraded":"2024-09-16T06:55:55.194847+0000","last_fullsized":"2024-09-16T06:55:55.194847+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:30.632072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.751}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":0.873},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":0.90600000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575102,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.622}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":1.1910000000000001}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":1.3109999999999999}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":1.232},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":1.2849999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":0.94299999999999995}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.98099999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607805,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":1.044},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.997},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.60699999999999998},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.83799999999999997,"15min":1.0549999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.60999999999999999,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.86399999999999999,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.71399999999999997,"5min":0.61899999999999999,"15min":0.57499999999999996},"max":{"1min":0.96499999999999997,"5min":1.073,"15min":1.3500000000000001},"last":0.871},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97099999999999997,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79600000000000004}]},{"osd":5,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79900000000000004,"15min":0.83699999999999997},"min":{"1min":0.65700000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.5780000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84099999999999997,"15min":0.85399999999999998},"min":{"1min":0.71699999999999997,"5min":0.60699999999999998,"15min":0.59599999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.83999999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.498},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.71499999999999997}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.81100000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46800000000000003,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.60299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0109999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.92800000000000005}]},{"osd":1,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.58699999999999997,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":1.0149999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.872,"15min":0.88500000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":1.0529999999999999,"5min":1.153,"15min":1.4910000000000001},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.74099999999999999,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":1.008,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.018},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.121},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.88,"15min":0.86699999999999999},"min":{"1min":0.65900000000000003,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1140000000000001,"5min":1.125,"15min":1.1899999999999999},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:59:21.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:59:21.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: get_monmap_and_config 2024-09-16T06:59:21.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: monmap: 2024-09-16T06:59:21.827 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:21.832340+0000 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:21.832340+0000 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:21.828 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 1 Processor -- start 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 1 -- start start 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: init 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-a con 0x7f1ea809f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-c con 0x7f1ea809fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-b con 0x7f1ea8139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea809cd80 con 0x7f1ea809f230 2024-09-16T06:59:21.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea8099fd0 con 0x7f1ea809fd20 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea809a410 con 0x7f1ea8139b70 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient(hunting): authenticate will time out at 2108.965596s 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eac9d9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1ea809f230 0x7f1ea809f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44512/0 (socket says 172.21.15.76:44512) 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ead1da700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1ea8139b70 0x7f1ea813bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54076/0 (socket says 172.21.15.76:54076) 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eac9d9700 1 -- 172.21.15.76:0/1349153200 learned_addr learned my addr 172.21.15.76:0/1349153200 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995168848 0 0) 0x7f1ea809a410 con 0x7f1ea8139b70 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): my global_id is 374141 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e980035d0 con 0x7f1ea8139b70 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435170144 0 0) 0x7f1ea8099fd0 con 0x7f1ea809fd20 2024-09-16T06:59:21.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): my global_id is 284968 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea809a410 con 0x7f1ea809fd20 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124909645 0 0) 0x7f1ea809cd80 con 0x7f1ea809f230 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient(hunting): my global_id is 334494 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8099fd0 con 0x7f1ea809f230 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199627840 0 0) 0x7f1e980035d0 con 0x7f1ea8139b70 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 >> v1:172.21.15.189:6789/0 conn(0x7f1ea809fd20 legacy=0x7f1ea8138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 >> v1:172.21.15.76:6789/0 conn(0x7f1ea809f230 legacy=0x7f1ea809f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 monclient: found mon.noname-b 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea809de90 con 0x7f1ea8139b70 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1ea0001990 con 0x7f1ea8139b70 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: dump: 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.835 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 5 monclient: authenticate success, global_id 374141 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ea0001bb0 con 0x7f1ea8139b70 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 1 -- 172.21.15.76:0/1349153200 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1ea0001ec0 con 0x7f1ea8139b70 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1eaec3d700 4 set_mon_vals no callback set 2024-09-16T06:59:21.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.829+0000 7f1ea77fe700 10 monclient: dump: 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 10 monclient: shutdown 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1349153200 >> v1:172.21.15.76:6790/0 conn(0x7f1ea8139b70 legacy=0x7f1ea813bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1349153200 shutdown_connections 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1349153200 >> 172.21.15.76:0/1349153200 conn(0x7f1ea80a3e50 msgr2=0x7f1ea80a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1349153200 shutdown_connections 2024-09-16T06:59:21.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1349153200 wait complete. 2024-09-16T06:59:21.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 10 monclient: build_initial_monmap 2024-09-16T06:59:21.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 10 monclient: monmap: 2024-09-16T06:59:21.842 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:21.842 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 Processor -- start 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 1 -- start start 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 10 monclient: init 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.833+0000 7f1eaec3d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:21.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-a con 0x7f1ea809f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-c con 0x7f1ea809fd20 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient(hunting): picked mon.noname-b con 0x7f1ea8139b70 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea811c810 con 0x7f1ea809f230 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea811c250 con 0x7f1ea8139b70 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea811c530 con 0x7f1ea809fd20 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient(hunting): authenticate will time out at 2108.972138s 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea7fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f1ea809fd20 0x7f1ea811a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54082/0 (socket says 172.21.15.76:54082) 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea7fff700 1 -- 172.21.15.76:0/1236976792 learned_addr learned my addr 172.21.15.76:0/1236976792 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055571712 0 0) 0x7f1ea811c530 con 0x7f1ea809fd20 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): my global_id is 374144 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8c0035d0 con 0x7f1ea809fd20 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358598596 0 0) 0x7f1ea811c810 con 0x7f1ea809f230 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): my global_id is 334497 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea811c530 con 0x7f1ea809f230 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289326404 0 0) 0x7f1e8c0035d0 con 0x7f1ea809fd20 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea811c810 con 0x7f1ea809fd20 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e94001970 con 0x7f1ea809fd20 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:21.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): dump: 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303984856 0 0) 0x7f1ea811c530 con 0x7f1ea809f230 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e8c0035d0 con 0x7f1ea809f230 2024-09-16T06:59:21.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e9c001830 con 0x7f1ea809f230 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient(hunting): dump: 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208277808 0 0) 0x7f1ea811c810 con 0x7f1ea809fd20 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 >> v1:172.21.15.189:6789/0 conn(0x7f1ea8139b70 legacy=0x7f1ea81cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 >> v1:172.21.15.76:6789/0 conn(0x7f1ea809f230 legacy=0x7f1ea8119e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 monclient: found mon.c 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea811ac90 con 0x7f1ea809fd20 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e94001c50 con 0x7f1ea809fd20 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f1e94003b10 con 0x7f1ea809fd20 2024-09-16T06:59:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 10 monclient: dump: 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 5 monclient: authenticate success, global_id 374144 2024-09-16T06:59:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea6ffd700 4 set_mon_vals no callback set 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _renew_subs 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea8138be0 con 0x7f1ea809fd20 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _renew_subs 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea80a1780 con 0x7f1ea809fd20 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f1e94003dc0 con 0x7f1ea809fd20 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea8067890 con 0x7f1ea809fd20 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f1e94043ed0 con 0x7f1ea809fd20 2024-09-16T06:59:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.837+0000 7f1eac9d9700 10 monclient: get_auth_request con 0x7f1e8c036340 auth_method 0 2024-09-16T06:59:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.845+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f1e940145f0 con 0x7f1ea809fd20 2024-09-16T06:59:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.845+0000 7f1ea57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:21.845+0000 7f1ea57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:22.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.013+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ea809ab70 con 0x7f1e8c036340 2024-09-16T06:59:22.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.017+0000 7f1ea57fa700 1 -- 172.21.15.76:0/1236976792 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98767 (unknown 2965378022 0 1926146659) 0x7f1e9c001d20 con 0x7f1e8c036340 2024-09-16T06:59:22.021 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1e8c036340 legacy=0x7f1e8c038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 10 monclient: shutdown 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 >> v1:172.21.15.76:6790/0 conn(0x7f1ea809fd20 legacy=0x7f1ea811a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 shutdown_connections 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 >> 172.21.15.76:0/1236976792 conn(0x7f1ea80a3e50 msgr2=0x7f1ea809af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 shutdown_connections 2024-09-16T06:59:22.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.021+0000 7f1eaec3d700 1 -- 172.21.15.76:0/1236976792 wait complete. 2024-09-16T06:59:22.024 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:59:22.034 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-09-16T06:59:20.039427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191536,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039504,"statfs":{"total":579820584960,"available":579624452096,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003918"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:21.071754+0000","last_change":"2024-09-16T06:44:52.856113+0000","last_active":"2024-09-16T06:55:21.071754+0000","last_peered":"2024-09-16T06:55:21.071754+0000","last_clean":"2024-09-16T06:55:21.071754+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:55:21.071754+0000","last_undegraded":"2024-09-16T06:55:21.071754+0000","last_fullsized":"2024-09-16T06:55:21.071754+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:26.598082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191190+0000","last_change":"2024-09-16T06:44:52.853100+0000","last_active":"2024-09-16T06:55:45.191190+0000","last_peered":"2024-09-16T06:55:45.191190+0000","last_clean":"2024-09-16T06:55:45.191190+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:55:45.191190+0000","last_undegraded":"2024-09-16T06:55:45.191190+0000","last_fullsized":"2024-09-16T06:55:45.191190+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:16.588106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.072562+0000","last_change":"2024-09-16T06:44:52.855874+0000","last_active":"2024-09-16T06:55:26.072562+0000","last_peered":"2024-09-16T06:55:26.072562+0000","last_clean":"2024-09-16T06:55:26.072562+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:55:26.072562+0000","last_undegraded":"2024-09-16T06:55:26.072562+0000","last_fullsized":"2024-09-16T06:55:26.072562+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:55.805033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191145+0000","last_change":"2024-09-16T06:44:52.854782+0000","last_active":"2024-09-16T06:55:45.191145+0000","last_peered":"2024-09-16T06:55:45.191145+0000","last_clean":"2024-09-16T06:55:45.191145+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:55:45.191145+0000","last_undegraded":"2024-09-16T06:55:45.191145+0000","last_fullsized":"2024-09-16T06:55:45.191145+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:49.724068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.193218+0000","last_change":"2024-09-16T06:44:52.859034+0000","last_active":"2024-09-16T06:55:45.193218+0000","last_peered":"2024-09-16T06:55:45.193218+0000","last_clean":"2024-09-16T06:55:45.193218+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:55:45.193218+0000","last_undegraded":"2024-09-16T06:55:45.193218+0000","last_fullsized":"2024-09-16T06:55:45.193218+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:48:37.653672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056468+0000","last_change":"2024-09-16T06:44:52.859319+0000","last_active":"2024-09-16T06:55:25.056468+0000","last_peered":"2024-09-16T06:55:25.056468+0000","last_clean":"2024-09-16T06:55:25.056468+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:55:25.056468+0000","last_undegraded":"2024-09-16T06:55:25.056468+0000","last_fullsized":"2024-09-16T06:55:25.056468+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:36.527527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":163,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:01.211371+0000","last_change":"2024-09-16T06:44:52.857054+0000","last_active":"2024-09-16T06:56:01.211371+0000","last_peered":"2024-09-16T06:56:01.211371+0000","last_clean":"2024-09-16T06:56:01.211371+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:56:01.211371+0000","last_undegraded":"2024-09-16T06:56:01.211371+0000","last_fullsized":"2024-09-16T06:56:01.211371+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:13.406841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796081+0000","last_change":"2024-09-16T06:44:52.854292+0000","last_active":"2024-09-16T06:55:18.796081+0000","last_peered":"2024-09-16T06:55:18.796081+0000","last_clean":"2024-09-16T06:55:18.796081+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:55:18.796081+0000","last_undegraded":"2024-09-16T06:55:18.796081+0000","last_fullsized":"2024-09-16T06:55:18.796081+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:55.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201207+0000","last_change":"2024-09-16T06:44:52.857665+0000","last_active":"2024-09-16T06:56:55.201207+0000","last_peered":"2024-09-16T06:56:55.201207+0000","last_clean":"2024-09-16T06:56:55.201207+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:56:55.201207+0000","last_undegraded":"2024-09-16T06:56:55.201207+0000","last_fullsized":"2024-09-16T06:56:55.201207+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:58:55.693264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":165,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207121+0000","last_change":"2024-09-16T06:44:52.857164+0000","last_active":"2024-09-16T06:55:31.207121+0000","last_peered":"2024-09-16T06:55:31.207121+0000","last_clean":"2024-09-16T06:55:31.207121+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:55:31.207121+0000","last_undegraded":"2024-09-16T06:55:31.207121+0000","last_fullsized":"2024-09-16T06:55:31.207121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:24.291104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":170,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207067+0000","last_change":"2024-09-16T06:44:52.854453+0000","last_active":"2024-09-16T06:55:31.207067+0000","last_peered":"2024-09-16T06:55:31.207067+0000","last_clean":"2024-09-16T06:55:31.207067+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:55:31.207067+0000","last_undegraded":"2024-09-16T06:55:31.207067+0000","last_fullsized":"2024-09-16T06:55:31.207067+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:49.308777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796044+0000","last_change":"2024-09-16T06:44:52.856374+0000","last_active":"2024-09-16T06:55:18.796044+0000","last_peered":"2024-09-16T06:55:18.796044+0000","last_clean":"2024-09-16T06:55:18.796044+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:55:18.796044+0000","last_undegraded":"2024-09-16T06:55:18.796044+0000","last_fullsized":"2024-09-16T06:55:18.796044+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:52.978402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.055715+0000","last_change":"2024-09-16T06:44:53.857627+0000","last_active":"2024-09-16T06:55:20.055715+0000","last_peered":"2024-09-16T06:55:20.055715+0000","last_clean":"2024-09-16T06:55:20.055715+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:55:20.055715+0000","last_undegraded":"2024-09-16T06:55:20.055715+0000","last_fullsized":"2024-09-16T06:55:20.055715+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:18:22.103962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.795979+0000","last_change":"2024-09-16T06:44:53.857508+0000","last_active":"2024-09-16T06:55:18.795979+0000","last_peered":"2024-09-16T06:55:18.795979+0000","last_clean":"2024-09-16T06:55:18.795979+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:55:18.795979+0000","last_undegraded":"2024-09-16T06:55:18.795979+0000","last_fullsized":"2024-09-16T06:55:18.795979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:09.945570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":116,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:16.079853+0000","last_change":"2024-09-16T06:44:53.857597+0000","last_active":"2024-09-16T06:56:16.079853+0000","last_peered":"2024-09-16T06:56:16.079853+0000","last_clean":"2024-09-16T06:56:16.079853+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:56:16.079853+0000","last_undegraded":"2024-09-16T06:56:16.079853+0000","last_fullsized":"2024-09-16T06:56:16.079853+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:53.321544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:00.070430+0000","last_change":"2024-09-16T06:44:52.857229+0000","last_active":"2024-09-16T06:57:00.070430+0000","last_peered":"2024-09-16T06:57:00.070430+0000","last_clean":"2024-09-16T06:57:00.070430+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:57:00.070430+0000","last_undegraded":"2024-09-16T06:57:00.070430+0000","last_fullsized":"2024-09-16T06:57:00.070430+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:03.913790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.203557+0000","last_change":"2024-09-16T06:44:55.870279+0000","last_active":"2024-09-16T06:56:55.203557+0000","last_peered":"2024-09-16T06:56:55.203557+0000","last_clean":"2024-09-16T06:56:55.203557+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:56:55.203557+0000","last_undegraded":"2024-09-16T06:56:55.203557+0000","last_fullsized":"2024-09-16T06:56:55.203557+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:03.350848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":245,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.189959+0000","last_change":"2024-09-16T06:44:52.855544+0000","last_active":"2024-09-16T06:55:25.189959+0000","last_peered":"2024-09-16T06:55:25.189959+0000","last_clean":"2024-09-16T06:55:25.189959+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:55:25.189959+0000","last_undegraded":"2024-09-16T06:55:25.189959+0000","last_fullsized":"2024-09-16T06:55:25.189959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:35.351132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201179+0000","last_change":"2024-09-16T06:44:52.858722+0000","last_active":"2024-09-16T06:56:55.201179+0000","last_peered":"2024-09-16T06:56:55.201179+0000","last_clean":"2024-09-16T06:56:55.201179+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:56:55.201179+0000","last_undegraded":"2024-09-16T06:56:55.201179+0000","last_fullsized":"2024-09-16T06:56:55.201179+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:45.458295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796021+0000","last_change":"2024-09-16T06:44:52.856469+0000","last_active":"2024-09-16T06:55:18.796021+0000","last_peered":"2024-09-16T06:55:18.796021+0000","last_clean":"2024-09-16T06:55:18.796021+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:55:18.796021+0000","last_undegraded":"2024-09-16T06:55:18.796021+0000","last_fullsized":"2024-09-16T06:55:18.796021+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:31.216149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":280,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.206353+0000","last_change":"2024-09-16T06:44:52.866089+0000","last_active":"2024-09-16T06:55:26.206353+0000","last_peered":"2024-09-16T06:55:26.206353+0000","last_clean":"2024-09-16T06:55:26.206353+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:55:26.206353+0000","last_undegraded":"2024-09-16T06:55:26.206353+0000","last_fullsized":"2024-09-16T06:55:26.206353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:55.504470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056440+0000","last_change":"2024-09-16T06:44:52.856612+0000","last_active":"2024-09-16T06:55:25.056440+0000","last_peered":"2024-09-16T06:55:25.056440+0000","last_clean":"2024-09-16T06:55:25.056440+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:55:25.056440+0000","last_undegraded":"2024-09-16T06:55:25.056440+0000","last_fullsized":"2024-09-16T06:55:25.056440+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:02.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.189258+0000","last_change":"2024-09-16T06:44:52.844681+0000","last_active":"2024-09-16T06:55:20.189258+0000","last_peered":"2024-09-16T06:55:20.189258+0000","last_clean":"2024-09-16T06:55:20.189258+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:55:20.189258+0000","last_undegraded":"2024-09-16T06:55:20.189258+0000","last_fullsized":"2024-09-16T06:55:20.189258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:47:26.834608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:46.075513+0000","last_change":"2024-09-16T06:44:52.848857+0000","last_active":"2024-09-16T06:55:46.075513+0000","last_peered":"2024-09-16T06:55:46.075513+0000","last_clean":"2024-09-16T06:55:46.075513+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:55:46.075513+0000","last_undegraded":"2024-09-16T06:55:46.075513+0000","last_fullsized":"2024-09-16T06:55:46.075513+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:46.816057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:55.194847+0000","last_change":"2024-09-16T06:44:53.857350+0000","last_active":"2024-09-16T06:55:55.194847+0000","last_peered":"2024-09-16T06:55:55.194847+0000","last_clean":"2024-09-16T06:55:55.194847+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:55:55.194847+0000","last_undegraded":"2024-09-16T06:55:55.194847+0000","last_fullsized":"2024-09-16T06:55:55.194847+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:30.632072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.751}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":0.873},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.79500000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":0.90600000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575102,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.622}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":1.1910000000000001}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":1.3109999999999999}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":1.232},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":1.2849999999999999}]}]},{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":0.94299999999999995}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.98099999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.89400000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607805,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":1.044},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.997},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.60699999999999998},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.83799999999999997,"15min":1.0549999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.60999999999999999,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.86399999999999999,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.71399999999999997,"5min":0.61899999999999999,"15min":0.57499999999999996},"max":{"1min":0.96499999999999997,"5min":1.073,"15min":1.3500000000000001},"last":0.871},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97099999999999997,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.79600000000000004}]},{"osd":5,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79900000000000004,"15min":0.83699999999999997},"min":{"1min":0.65700000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.5780000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84099999999999997,"15min":0.85399999999999998},"min":{"1min":0.71699999999999997,"5min":0.60699999999999998,"15min":0.59599999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.83999999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.498},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.71499999999999997}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.90300000000000002}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.81100000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46800000000000003,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.60299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0109999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.92800000000000005}]},{"osd":1,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.58699999999999997,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":1.0149999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.872,"15min":0.88500000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":1.0529999999999999,"5min":1.153,"15min":1.4910000000000001},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.74099999999999999,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":1.008,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.018},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.121},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.88,"15min":0.86699999999999999},"min":{"1min":0.65900000000000003,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1140000000000001,"5min":1.125,"15min":1.1899999999999999},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:59:22.037 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T06:59:22.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: get_monmap_and_config 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: build_initial_monmap 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: monmap: 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:22.149737+0000 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:22.149737+0000 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.145 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:22.146 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:22.146 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:22.146 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 1 Processor -- start 2024-09-16T06:59:22.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 1 -- start start 2024-09-16T06:59:22.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: init 2024-09-16T06:59:22.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-c con 0x7f0e78097b60 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-b con 0x7f0e78099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-a con 0x7f0e78139500 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78098990 con 0x7f0e78139500 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e78097f40 con 0x7f0e78097b60 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7809d610 con 0x7f0e78099090 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7e76e700 10 monclient(hunting): authenticate will time out at 2109.283009s 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7df6d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0e78139500 0x7f0e7813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44528/0 (socket says 172.21.15.76:44528) 2024-09-16T06:59:22.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e7df6d700 1 -- 172.21.15.76:0/482233217 learned_addr learned my addr 172.21.15.76:0/482233217 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414455688 0 0) 0x7f0e78098990 con 0x7f0e78139500 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): my global_id is 334500 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e640035d0 con 0x7f0e78139500 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879672181 0 0) 0x7f0e7809d610 con 0x7f0e78099090 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): my global_id is 374147 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e78098990 con 0x7f0e78099090 2024-09-16T06:59:22.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690172130 0 0) 0x7f0e78097f40 con 0x7f0e78097b60 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 10 monclient(hunting): my global_id is 284974 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.145+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7809d610 con 0x7f0e78097b60 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471140842 0 0) 0x7f0e78098990 con 0x7f0e78099090 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 >> v1:172.21.15.189:6789/0 conn(0x7f0e78097b60 legacy=0x7f0e7809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 >> v1:172.21.15.76:6789/0 conn(0x7f0e78139500 legacy=0x7f0e7813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 monclient: found mon.noname-b 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:22.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e780986b0 con 0x7f0e78099090 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: _finish_auth 0 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e680015f0 con 0x7f0e78099090 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 5 monclient: authenticate success, global_id 374147 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: dump: 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e68001910 con 0x7f0e78099090 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:22.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 1 -- 172.21.15.76:0/482233217 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e68001ca0 con 0x7f0e78099090 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 4 set_mon_vals no callback set 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e6ffff700 10 monclient: dump: 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.157 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 10 monclient: shutdown 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 1 -- 172.21.15.76:0/482233217 >> v1:172.21.15.76:6790/0 conn(0x7f0e78099090 legacy=0x7f0e7809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 1 -- 172.21.15.76:0/482233217 shutdown_connections 2024-09-16T06:59:22.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 1 -- 172.21.15.76:0/482233217 >> 172.21.15.76:0/482233217 conn(0x7f0e780a0030 msgr2=0x7f0e780a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 1 -- 172.21.15.76:0/482233217 shutdown_connections 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 1 -- 172.21.15.76:0/482233217 wait complete. 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 10 monclient: build_initial_monmap 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.149+0000 7f0e7e76e700 10 monclient: monmap: 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:22.159 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:22.160 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:22.160 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 Processor -- start 2024-09-16T06:59:22.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- start start 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: init 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-c con 0x7f0e78097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-a con 0x7f0e78099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient(hunting): picked mon.noname-b con 0x7f0e78139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7809a540 con 0x7f0e78099090 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7806cc90 con 0x7f0e78139500 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7d76c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0e78097b60 0x7f0e7806be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54100/0 (socket says 172.21.15.76:54100) 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7d76c700 1 -- 172.21.15.76:0/3326080778 learned_addr learned my addr 172.21.15.76:0/3326080778 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7806d090 con 0x7f0e78097b60 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:22.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient(hunting): authenticate will time out at 2109.289676s 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570368405 0 0) 0x7f0e7806d090 con 0x7f0e78097b60 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 374150 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c0035d0 con 0x7f0e78097b60 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404704633 0 0) 0x7f0e7809a540 con 0x7f0e78099090 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 334503 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7806d090 con 0x7f0e78099090 2024-09-16T06:59:22.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901318627 0 0) 0x7f0e7806cc90 con 0x7f0e78139500 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): my global_id is 284977 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7809a540 con 0x7f0e78139500 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3046667593 0 0) 0x7f0e7806d090 con 0x7f0e78099090 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7806cc90 con 0x7f0e78099090 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e680015f0 con 0x7f0e78099090 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): dump: 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.167 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888360899 0 0) 0x7f0e5c0035d0 con 0x7f0e78097b60 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7806d090 con 0x7f0e78097b60 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e740018b0 con 0x7f0e78097b60 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:22.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient(hunting): dump: 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316788443 0 0) 0x7f0e7806cc90 con 0x7f0e78099090 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 >> v1:172.21.15.76:6790/0 conn(0x7f0e78097b60 legacy=0x7f0e7806be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 >> v1:172.21.15.189:6789/0 conn(0x7f0e78139500 legacy=0x7f0e781cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: _finish_hunting 0 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 monclient: found mon.a 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7806e9f0 con 0x7f0e78099090 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: _finish_auth 0 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: _check_auth_tickets 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 5 monclient: authenticate success, global_id 334503 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _renew_subs 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7806ecd0 con 0x7f0e78099090 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _renew_subs 2024-09-16T06:59:22.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7806ddb0 con 0x7f0e78099090 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e68003a10 con 0x7f0e78099090 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0e68003d20 con 0x7f0e78099090 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6f7fe700 4 set_mon_vals no callback set 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.153+0000 7f0e6dffb700 10 monclient: dump: 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.171 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0e6800e790 con 0x7f0e78099090 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0e68042dc0 con 0x7f0e78099090 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e7d76c700 10 monclient: get_auth_request con 0x7f0e5c036250 auth_method 0 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e7e76e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e7e76e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e7e76e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.157+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e58005100 con 0x7f0e78099090 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.165+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0e68013760 con 0x7f0e78099090 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.165+0000 7f0e6dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.165+0000 7f0e6dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:22.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.333+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e58000b90 con 0x7f0e5c036250 2024-09-16T06:59:22.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.341+0000 7f0e6dffb700 1 -- 172.21.15.76:0/3326080778 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98712 (unknown 2965378022 0 1098537555) 0x7f0e58000b90 con 0x7f0e5c036250 2024-09-16T06:59:22.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0e5c036250 legacy=0x7f0e5c038710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 10 monclient: shutdown 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 >> v1:172.21.15.76:6789/0 conn(0x7f0e78099090 legacy=0x7f0e7806c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 shutdown_connections 2024-09-16T06:59:22.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 >> 172.21.15.76:0/3326080778 conn(0x7f0e780a0030 msgr2=0x7f0e780a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:22.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 shutdown_connections 2024-09-16T06:59:22.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.345+0000 7f0e7e76e700 1 -- 172.21.15.76:0/3326080778 wait complete. 2024-09-16T06:59:22.345 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:59:22.357 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-09-16T06:59:22.040011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191536,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039504,"statfs":{"total":579820584960,"available":579624452096,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:21.071754+0000","last_change":"2024-09-16T06:44:52.856113+0000","last_active":"2024-09-16T06:55:21.071754+0000","last_peered":"2024-09-16T06:55:21.071754+0000","last_clean":"2024-09-16T06:55:21.071754+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:55:21.071754+0000","last_undegraded":"2024-09-16T06:55:21.071754+0000","last_fullsized":"2024-09-16T06:55:21.071754+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:26.598082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191190+0000","last_change":"2024-09-16T06:44:52.853100+0000","last_active":"2024-09-16T06:55:45.191190+0000","last_peered":"2024-09-16T06:55:45.191190+0000","last_clean":"2024-09-16T06:55:45.191190+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:55:45.191190+0000","last_undegraded":"2024-09-16T06:55:45.191190+0000","last_fullsized":"2024-09-16T06:55:45.191190+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:16.588106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.072562+0000","last_change":"2024-09-16T06:44:52.855874+0000","last_active":"2024-09-16T06:55:26.072562+0000","last_peered":"2024-09-16T06:55:26.072562+0000","last_clean":"2024-09-16T06:55:26.072562+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:55:26.072562+0000","last_undegraded":"2024-09-16T06:55:26.072562+0000","last_fullsized":"2024-09-16T06:55:26.072562+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:55.805033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191145+0000","last_change":"2024-09-16T06:44:52.854782+0000","last_active":"2024-09-16T06:55:45.191145+0000","last_peered":"2024-09-16T06:55:45.191145+0000","last_clean":"2024-09-16T06:55:45.191145+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:55:45.191145+0000","last_undegraded":"2024-09-16T06:55:45.191145+0000","last_fullsized":"2024-09-16T06:55:45.191145+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:49.724068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.193218+0000","last_change":"2024-09-16T06:44:52.859034+0000","last_active":"2024-09-16T06:55:45.193218+0000","last_peered":"2024-09-16T06:55:45.193218+0000","last_clean":"2024-09-16T06:55:45.193218+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:55:45.193218+0000","last_undegraded":"2024-09-16T06:55:45.193218+0000","last_fullsized":"2024-09-16T06:55:45.193218+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:48:37.653672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056468+0000","last_change":"2024-09-16T06:44:52.859319+0000","last_active":"2024-09-16T06:55:25.056468+0000","last_peered":"2024-09-16T06:55:25.056468+0000","last_clean":"2024-09-16T06:55:25.056468+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:55:25.056468+0000","last_undegraded":"2024-09-16T06:55:25.056468+0000","last_fullsized":"2024-09-16T06:55:25.056468+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:36.527527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":163,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:01.211371+0000","last_change":"2024-09-16T06:44:52.857054+0000","last_active":"2024-09-16T06:56:01.211371+0000","last_peered":"2024-09-16T06:56:01.211371+0000","last_clean":"2024-09-16T06:56:01.211371+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:56:01.211371+0000","last_undegraded":"2024-09-16T06:56:01.211371+0000","last_fullsized":"2024-09-16T06:56:01.211371+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:13.406841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796081+0000","last_change":"2024-09-16T06:44:52.854292+0000","last_active":"2024-09-16T06:55:18.796081+0000","last_peered":"2024-09-16T06:55:18.796081+0000","last_clean":"2024-09-16T06:55:18.796081+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:55:18.796081+0000","last_undegraded":"2024-09-16T06:55:18.796081+0000","last_fullsized":"2024-09-16T06:55:18.796081+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:55.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201207+0000","last_change":"2024-09-16T06:44:52.857665+0000","last_active":"2024-09-16T06:56:55.201207+0000","last_peered":"2024-09-16T06:56:55.201207+0000","last_clean":"2024-09-16T06:56:55.201207+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:56:55.201207+0000","last_undegraded":"2024-09-16T06:56:55.201207+0000","last_fullsized":"2024-09-16T06:56:55.201207+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:58:55.693264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":165,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207121+0000","last_change":"2024-09-16T06:44:52.857164+0000","last_active":"2024-09-16T06:55:31.207121+0000","last_peered":"2024-09-16T06:55:31.207121+0000","last_clean":"2024-09-16T06:55:31.207121+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:55:31.207121+0000","last_undegraded":"2024-09-16T06:55:31.207121+0000","last_fullsized":"2024-09-16T06:55:31.207121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:24.291104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":170,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207067+0000","last_change":"2024-09-16T06:44:52.854453+0000","last_active":"2024-09-16T06:55:31.207067+0000","last_peered":"2024-09-16T06:55:31.207067+0000","last_clean":"2024-09-16T06:55:31.207067+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:55:31.207067+0000","last_undegraded":"2024-09-16T06:55:31.207067+0000","last_fullsized":"2024-09-16T06:55:31.207067+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:49.308777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796044+0000","last_change":"2024-09-16T06:44:52.856374+0000","last_active":"2024-09-16T06:55:18.796044+0000","last_peered":"2024-09-16T06:55:18.796044+0000","last_clean":"2024-09-16T06:55:18.796044+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:55:18.796044+0000","last_undegraded":"2024-09-16T06:55:18.796044+0000","last_fullsized":"2024-09-16T06:55:18.796044+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:52.978402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.055715+0000","last_change":"2024-09-16T06:44:53.857627+0000","last_active":"2024-09-16T06:55:20.055715+0000","last_peered":"2024-09-16T06:55:20.055715+0000","last_clean":"2024-09-16T06:55:20.055715+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:55:20.055715+0000","last_undegraded":"2024-09-16T06:55:20.055715+0000","last_fullsized":"2024-09-16T06:55:20.055715+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:18:22.103962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.795979+0000","last_change":"2024-09-16T06:44:53.857508+0000","last_active":"2024-09-16T06:55:18.795979+0000","last_peered":"2024-09-16T06:55:18.795979+0000","last_clean":"2024-09-16T06:55:18.795979+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:55:18.795979+0000","last_undegraded":"2024-09-16T06:55:18.795979+0000","last_fullsized":"2024-09-16T06:55:18.795979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:09.945570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":116,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:16.079853+0000","last_change":"2024-09-16T06:44:53.857597+0000","last_active":"2024-09-16T06:56:16.079853+0000","last_peered":"2024-09-16T06:56:16.079853+0000","last_clean":"2024-09-16T06:56:16.079853+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:56:16.079853+0000","last_undegraded":"2024-09-16T06:56:16.079853+0000","last_fullsized":"2024-09-16T06:56:16.079853+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:53.321544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:00.070430+0000","last_change":"2024-09-16T06:44:52.857229+0000","last_active":"2024-09-16T06:57:00.070430+0000","last_peered":"2024-09-16T06:57:00.070430+0000","last_clean":"2024-09-16T06:57:00.070430+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:57:00.070430+0000","last_undegraded":"2024-09-16T06:57:00.070430+0000","last_fullsized":"2024-09-16T06:57:00.070430+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:03.913790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.203557+0000","last_change":"2024-09-16T06:44:55.870279+0000","last_active":"2024-09-16T06:56:55.203557+0000","last_peered":"2024-09-16T06:56:55.203557+0000","last_clean":"2024-09-16T06:56:55.203557+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:56:55.203557+0000","last_undegraded":"2024-09-16T06:56:55.203557+0000","last_fullsized":"2024-09-16T06:56:55.203557+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:03.350848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":245,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.189959+0000","last_change":"2024-09-16T06:44:52.855544+0000","last_active":"2024-09-16T06:55:25.189959+0000","last_peered":"2024-09-16T06:55:25.189959+0000","last_clean":"2024-09-16T06:55:25.189959+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:55:25.189959+0000","last_undegraded":"2024-09-16T06:55:25.189959+0000","last_fullsized":"2024-09-16T06:55:25.189959+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:35.351132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201179+0000","last_change":"2024-09-16T06:44:52.858722+0000","last_active":"2024-09-16T06:56:55.201179+0000","last_peered":"2024-09-16T06:56:55.201179+0000","last_clean":"2024-09-16T06:56:55.201179+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:56:55.201179+0000","last_undegraded":"2024-09-16T06:56:55.201179+0000","last_fullsized":"2024-09-16T06:56:55.201179+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:45.458295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796021+0000","last_change":"2024-09-16T06:44:52.856469+0000","last_active":"2024-09-16T06:55:18.796021+0000","last_peered":"2024-09-16T06:55:18.796021+0000","last_clean":"2024-09-16T06:55:18.796021+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:55:18.796021+0000","last_undegraded":"2024-09-16T06:55:18.796021+0000","last_fullsized":"2024-09-16T06:55:18.796021+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:31.216149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":280,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.206353+0000","last_change":"2024-09-16T06:44:52.866089+0000","last_active":"2024-09-16T06:55:26.206353+0000","last_peered":"2024-09-16T06:55:26.206353+0000","last_clean":"2024-09-16T06:55:26.206353+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:55:26.206353+0000","last_undegraded":"2024-09-16T06:55:26.206353+0000","last_fullsized":"2024-09-16T06:55:26.206353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:55.504470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056440+0000","last_change":"2024-09-16T06:44:52.856612+0000","last_active":"2024-09-16T06:55:25.056440+0000","last_peered":"2024-09-16T06:55:25.056440+0000","last_clean":"2024-09-16T06:55:25.056440+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:55:25.056440+0000","last_undegraded":"2024-09-16T06:55:25.056440+0000","last_fullsized":"2024-09-16T06:55:25.056440+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:02.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.189258+0000","last_change":"2024-09-16T06:44:52.844681+0000","last_active":"2024-09-16T06:55:20.189258+0000","last_peered":"2024-09-16T06:55:20.189258+0000","last_clean":"2024-09-16T06:55:20.189258+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:55:20.189258+0000","last_undegraded":"2024-09-16T06:55:20.189258+0000","last_fullsized":"2024-09-16T06:55:20.189258+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:47:26.834608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:46.075513+0000","last_change":"2024-09-16T06:44:52.848857+0000","last_active":"2024-09-16T06:55:46.075513+0000","last_peered":"2024-09-16T06:55:46.075513+0000","last_clean":"2024-09-16T06:55:46.075513+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:55:46.075513+0000","last_undegraded":"2024-09-16T06:55:46.075513+0000","last_fullsized":"2024-09-16T06:55:46.075513+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:46.816057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:55.194847+0000","last_change":"2024-09-16T06:44:53.857350+0000","last_active":"2024-09-16T06:55:55.194847+0000","last_peered":"2024-09-16T06:55:55.194847+0000","last_clean":"2024-09-16T06:55:55.194847+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:55:55.194847+0000","last_undegraded":"2024-09-16T06:55:55.194847+0000","last_fullsized":"2024-09-16T06:55:55.194847+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:48:30.632072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575101,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.71699999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":0.872},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575103,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":0.71299999999999997}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":0.747}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":1.036},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":1.099}]}]},{"osd":5,"up_from":12,"seq":51539607807,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":0.92100000000000004}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.74099999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607806,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":1.044},"last":0.495},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.997},"last":0.58699999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.60699999999999998},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.83799999999999997,"15min":1.0549999999999999},"last":0.625},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.60999999999999999,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.86399999999999999,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.879}]},{"osd":3,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.71399999999999997,"5min":0.61899999999999999,"15min":0.57499999999999996},"max":{"1min":0.96499999999999997,"5min":1.073,"15min":1.3500000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97099999999999997,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.93100000000000005}]},{"osd":5,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79900000000000004,"15min":0.83699999999999997},"min":{"1min":0.65700000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.5780000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84099999999999997,"15min":0.85399999999999998},"min":{"1min":0.71699999999999997,"5min":0.60699999999999998,"15min":0.59599999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.77400000000000002}]}]},{"osd":1,"up_from":12,"seq":51539607808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.75},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.81799999999999995}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.012},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.96299999999999997}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.1759999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46800000000000003,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.60299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0109999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.92800000000000005}]},{"osd":1,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.58699999999999997,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":1.0149999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.872,"15min":0.88500000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":1.0529999999999999,"5min":1.153,"15min":1.4910000000000001},"last":0.86099999999999999}]},{"osd":2,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.74099999999999999,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":1.008,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.018},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.121},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.88,"15min":0.86699999999999999},"min":{"1min":0.65900000000000003,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1140000000000001,"5min":1.125,"15min":1.1899999999999999},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:59:22.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: get_monmap_and_config 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: build_initial_monmap 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: monmap: 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.473 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:22.478071+0000 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:22.478071+0000 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:22.474 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 1 Processor -- start 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 1 -- start start 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: init 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-c con 0x7f404c132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-b con 0x7f404c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-a con 0x7f404c06cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c06c280 con 0x7f404c06cf50 2024-09-16T06:59:22.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c06bfa0 con 0x7f404c132330 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c09ace0 con 0x7f404c06d480 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f4052a2b700 10 monclient(hunting): authenticate will time out at 2109.611657s 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f405222a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f404c06cf50 0x7f404c13da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44550/0 (socket says 172.21.15.76:44550) 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.473+0000 7f405222a700 1 -- 172.21.15.76:0/2691771018 learned_addr learned my addr 172.21.15.76:0/2691771018 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:22.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731108250 0 0) 0x7f404c09ace0 con 0x7f404c06d480 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): my global_id is 374153 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40380035d0 con 0x7f404c06d480 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706063252 0 0) 0x7f404c06c280 con 0x7f404c06cf50 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): my global_id is 334506 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c09ace0 con 0x7f404c06cf50 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172689380 0 0) 0x7f404c06bfa0 con 0x7f404c132330 2024-09-16T06:59:22.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient(hunting): my global_id is 284980 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c06c280 con 0x7f404c132330 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948451090 0 0) 0x7f404c09ace0 con 0x7f404c06cf50 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 >> v1:172.21.15.76:6790/0 conn(0x7f404c06d480 legacy=0x7f404c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 >> v1:172.21.15.189:6789/0 conn(0x7f404c132330 legacy=0x7f404c132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: _finish_hunting 0 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 monclient: found mon.noname-a 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c09ae70 con 0x7f404c06cf50 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: _finish_auth 0 2024-09-16T06:59:22.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: _check_auth_tickets 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4044001570 con 0x7f404c06cf50 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 5 monclient: authenticate success, global_id 334506 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: dump: 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.482 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4044001890 con 0x7f404c06cf50 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 1 -- 172.21.15.76:0/2691771018 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4044001a80 con 0x7f404c06cf50 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 4 set_mon_vals no callback set 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4050a27700 10 monclient: dump: 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.483 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 10 monclient: shutdown 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- 172.21.15.76:0/2691771018 >> v1:172.21.15.76:6789/0 conn(0x7f404c06cf50 legacy=0x7f404c13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- 172.21.15.76:0/2691771018 shutdown_connections 2024-09-16T06:59:22.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- 172.21.15.76:0/2691771018 >> 172.21.15.76:0/2691771018 conn(0x7f404c0a41c0 msgr2=0x7f404c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- 172.21.15.76:0/2691771018 shutdown_connections 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- 172.21.15.76:0/2691771018 wait complete. 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 10 monclient: build_initial_monmap 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 10 monclient: monmap: 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:22.485 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 Processor -- start 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 1 -- start start 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.477+0000 7f4052a2b700 10 monclient: init 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-b con 0x7f404c06cf50 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-c con 0x7f404c06d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient(hunting): picked mon.noname-a con 0x7f404c132330 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c1ca670 con 0x7f404c132330 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c131620 con 0x7f404c06cf50 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c09e8b0 con 0x7f404c06d480 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:22.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient(hunting): authenticate will time out at 2109.616769s 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f405222a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f404c132330 0x7f404c1d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44566/0 (socket says 172.21.15.76:44566) 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f405222a700 1 -- 172.21.15.76:0/4288659604 learned_addr learned my addr 172.21.15.76:0/4288659604 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001750102 0 0) 0x7f404c1ca670 con 0x7f404c132330 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): my global_id is 334509 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f402c0035d0 con 0x7f404c132330 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759646397 0 0) 0x7f404c09e8b0 con 0x7f404c06d480 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): my global_id is 374156 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c1ca670 con 0x7f404c06d480 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209021989 0 0) 0x7f404c131620 con 0x7f404c06cf50 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): my global_id is 284983 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c09e8b0 con 0x7f404c06cf50 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3339132095 0 0) 0x7f402c0035d0 con 0x7f404c132330 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f404c131620 con 0x7f404c132330 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4044001d50 con 0x7f404c132330 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): dump: 2024-09-16T06:59:22.492 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.493 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3547979663 0 0) 0x7f404c1ca670 con 0x7f404c06d480 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f402c0035d0 con 0x7f404c06d480 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4034001c50 con 0x7f404c06d480 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient(hunting): dump: 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.494 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982329395 0 0) 0x7f404c131620 con 0x7f404c132330 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 >> v1:172.21.15.76:6790/0 conn(0x7f404c06d480 legacy=0x7f404c1c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 >> v1:172.21.15.189:6789/0 conn(0x7f404c06cf50 legacy=0x7f404c1c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 monclient: found mon.a 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c12dd70 con 0x7f404c132330 2024-09-16T06:59:22.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 5 monclient: authenticate success, global_id 334509 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _renew_subs 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c12e4c0 con 0x7f404c132330 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _renew_subs 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c12eb60 con 0x7f404c132330 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4044001130 con 0x7f404c132330 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4044004db0 con 0x7f404c132330 2024-09-16T06:59:22.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 10 monclient: dump: 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404bfff700 4 set_mon_vals no callback set 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f4044004f90 con 0x7f404c132330 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4030005100 con 0x7f404c132330 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.481+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f4044043ee0 con 0x7f404c132330 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.485+0000 7f4051a29700 10 monclient: get_auth_request con 0x7f402c036390 auth_method 0 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.493+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f40440147f0 con 0x7f404c132330 2024-09-16T06:59:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.493+0000 7f404a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:22.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.493+0000 7f404a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.665+0000 7f4052a2b700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:59:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.665+0000 7f4052a2b700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:59:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.665+0000 7f4052a2b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.665+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40300053b0 con 0x7f404c132330 2024-09-16T06:59:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f404a7fc700 1 -- 172.21.15.76:0/4288659604 <== mon.0 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+20242 (unknown 593233210 0 2756273128) 0x7f4044012e90 con 0x7f404c132330 2024-09-16T06:59:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f404a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T06:59:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f404a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T06:59:22.668 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:22.668 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":255,"fsid":"53fe3d11-e8d6-4d19-8f38-046f335b6765","created":"2024-09-16T06:38:16.134351+0000","modified":"2024-09-16T06:47:30.981927+0000","last_up_change":"2024-09-16T06:38:25.048660+0000","last_in_change":"2024-09-16T06:42:18.083613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":183,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T06:38:25.702868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T06:38:25.996213+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-16T06:42:57.732693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"barset","create_time":"2024-09-16T06:42:58.681305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"85115802-a6ae-4a37-9bfa-b343e08d4648","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6808","nonce":3838562525}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6809","nonce":3838562525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6811","nonce":3838562525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6810","nonce":3838562525}]},"public_addr":"172.21.15.76:6808/3838562525","cluster_addr":"172.21.15.76:6809/3838562525","heartbeat_back_addr":"172.21.15.76:6811/3838562525","heartbeat_front_addr":"172.21.15.76:6810/3838562525","state":["exists","up"]},{"osd":1,"uuid":"99a99267-bac0-4365-af54-14423447fbc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6800","nonce":3502405955}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6801","nonce":3502405955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6803","nonce":3502405955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6802","nonce":3502405955}]},"public_addr":"172.21.15.76:6800/3502405955","cluster_addr":"172.21.15.76:6801/3502405955","heartbeat_back_addr":"172.21.15.76:6803/3502405955","heartbeat_front_addr":"172.21.15.76:6802/3502405955","state":["exists","up"]},{"osd":2,"uuid":"6d2d3abd-a22c-443d-a7c7-8c53e04088ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6804","nonce":779009499}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6805","nonce":779009499}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6807","nonce":779009499}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.76:6806","nonce":779009499}]},"public_addr":"172.21.15.76:6804/779009499","cluster_addr":"172.21.15.76:6805/779009499","heartbeat_back_addr":"172.21.15.76:6807/779009499","heartbeat_front_addr":"172.21.15.76:6806/779009499","state":["exists","up"]},{"osd":3,"uuid":"b39276aa-291e-4283-9006-7b82b23dd012","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6808","nonce":3809009238}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6809","nonce":3809009238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6811","nonce":3809009238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6810","nonce":3809009238}]},"public_addr":"172.21.15.189:6808/3809009238","cluster_addr":"172.21.15.189:6809/3809009238","heartbeat_back_addr":"172.21.15.189:6811/3809009238","heartbeat_front_addr":"172.21.15.189:6810/3809009238","state":["exists","up"]},{"osd":4,"uuid":"74f6ec14-1f4a-4f78-bac5-f20ccd7a3c7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6804","nonce":407912501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6805","nonce":407912501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6807","nonce":407912501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6806","nonce":407912501}]},"public_addr":"172.21.15.189:6804/407912501","cluster_addr":"172.21.15.189:6805/407912501","heartbeat_back_addr":"172.21.15.189:6807/407912501","heartbeat_front_addr":"172.21.15.189:6806/407912501","state":["exists","up"]},{"osd":5,"uuid":"9b794539-b7ca-4f2b-8653-a98904253c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6800","nonce":2128366135}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6801","nonce":2128366135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6803","nonce":2128366135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.189:6802","nonce":2128366135}]},"public_addr":"172.21.15.189:6800/2128366135","cluster_addr":"172.21.15.189:6801/2128366135","heartbeat_back_addr":"172.21.15.189:6803/2128366135","heartbeat_front_addr":"172.21.15.189:6802/2128366135","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"12c53d86-6cb3-4f7e-a16f-7f1b9e8aabc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.209148+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.408945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.850780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:21.901722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:23.166044+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T06:38:22.780816+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 >> v1:172.21.15.189:6812/3098810674 conn(0x7f402c036390 legacy=0x7f402c038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 10 monclient: shutdown 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 >> v1:172.21.15.76:6789/0 conn(0x7f404c132330 legacy=0x7f404c1d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 shutdown_connections 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 >> 172.21.15.76:0/4288659604 conn(0x7f404c0a41c0 msgr2=0x7f404c09d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 shutdown_connections 2024-09-16T06:59:22.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:22.669+0000 7f4052a2b700 1 -- 172.21.15.76:0/4288659604 wait complete. 2024-09-16T06:59:23.683 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T06:59:23.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: get_monmap_and_config 2024-09-16T06:59:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: build_initial_monmap 2024-09-16T06:59:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: monmap: 2024-09-16T06:59:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:23.817224+0000 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:23.817224+0000 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:23.813 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 1 Processor -- start 2024-09-16T06:59:23.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 1 -- start start 2024-09-16T06:59:23.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: init 2024-09-16T06:59:23.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:23.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient(hunting): picked mon.noname-a con 0x7f48200a26c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient(hunting): picked mon.noname-c con 0x7f482006cfa0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient(hunting): picked mon.noname-b con 0x7f48200a31b0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482006cd00 con 0x7f48200a26c0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482009fad0 con 0x7f482006cfa0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200a3710 con 0x7f48200a31b0 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.813+0000 7f4825845700 10 monclient(hunting): authenticate will time out at 2110.951797s 2024-09-16T06:59:23.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481f7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f48200a31b0 0x7f482013de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:54134/0 (socket says 172.21.15.76:54134) 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481f7fe700 1 -- 172.21.15.76:0/773353344 learned_addr learned my addr 172.21.15.76:0/773353344 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481effd700 1 -- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6789/0 conn(0x7f48200a26c0 legacy=0x7f48200a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481effd700 1 -- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6789/0 conn(0x7f48200a26c0 legacy=0x7f48200a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481effd700 1 --1- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6789/0 conn(0x7f48200a26c0 0x7f48200a2aa0 :-1 s=OPENED pgs=1502 cs=1 l=1).handle_message read tag failed 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481effd700 1 --1- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6789/0 conn(0x7f48200a26c0 0x7f48200a2aa0 :-1 s=OPENED pgs=1502 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:59:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026299151 0 0) 0x7f48200a3710 con 0x7f48200a31b0 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): my global_id is 374159 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c0035d0 con 0x7f48200a31b0 2024-09-16T06:59:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435830430 0 0) 0x7f482009fad0 con 0x7f482006cfa0 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient(hunting): my global_id is 284986 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48200a3710 con 0x7f482006cfa0 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588068427 0 0) 0x7f480c0035d0 con 0x7f48200a31b0 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 >> v1:172.21.15.189:6789/0 conn(0x7f482006cfa0 legacy=0x7f482013a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6789/0 conn(0x7f48200a26c0 legacy=0x7f48200a2aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 monclient: found mon.noname-b 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200a5140 con 0x7f48200a31b0 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4818001990 con 0x7f48200a31b0 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 5 monclient: authenticate success, global_id 374159 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: dump: 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4818001c70 con 0x7f48200a31b0 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 1 -- 172.21.15.76:0/773353344 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4818002000 con 0x7f48200a31b0 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 4 set_mon_vals no callback set 2024-09-16T06:59:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f481e7fc700 10 monclient: dump: 2024-09-16T06:59:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: shutdown 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- 172.21.15.76:0/773353344 >> v1:172.21.15.76:6790/0 conn(0x7f48200a31b0 legacy=0x7f482013de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- 172.21.15.76:0/773353344 shutdown_connections 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- 172.21.15.76:0/773353344 >> 172.21.15.76:0/773353344 conn(0x7f48200a82f0 msgr2=0x7f48200a47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- 172.21.15.76:0/773353344 shutdown_connections 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- 172.21.15.76:0/773353344 wait complete. 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: build_initial_monmap 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: monmap: 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:23.827 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 Processor -- start 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 1 -- start start 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: init 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.817+0000 7f4825845700 10 monclient(hunting): picked mon.noname-c con 0x7f482006cfa0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient(hunting): picked mon.noname-b con 0x7f48200a26c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient(hunting): picked mon.noname-a con 0x7f48200a31b0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4820136c40 con 0x7f48200a31b0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200a6980 con 0x7f48200a26c0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482006d950 con 0x7f482006cfa0 2024-09-16T06:59:23.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient(hunting): authenticate will time out at 2110.956072s 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f48200a31b0 0x7f48201d27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:44578/0 (socket says 172.21.15.76:44578) 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481f7fe700 1 -- 172.21.15.76:0/1805727049 learned_addr learned my addr 172.21.15.76:0/1805727049 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674050304 0 0) 0x7f4820136c40 con 0x7f48200a31b0 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): my global_id is 334512 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48000035d0 con 0x7f48200a31b0 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201175917 0 0) 0x7f482006d950 con 0x7f482006cfa0 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): my global_id is 374162 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820136c40 con 0x7f482006cfa0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192161856 0 0) 0x7f48200a6980 con 0x7f48200a26c0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): my global_id is 284989 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482006d950 con 0x7f48200a26c0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1579006949 0 0) 0x7f48000035d0 con 0x7f48200a31b0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48200a6980 con 0x7f48200a31b0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4818001f80 con 0x7f48200a31b0 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): dump: 2024-09-16T06:59:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880618958 0 0) 0x7f4820136c40 con 0x7f482006cfa0 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48000035d0 con 0x7f482006cfa0 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f4810001770 con 0x7f482006cfa0 2024-09-16T06:59:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient(hunting): dump: 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1731804772 0 0) 0x7f48200a6980 con 0x7f48200a31b0 2024-09-16T06:59:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 >> v1:172.21.15.76:6790/0 conn(0x7f482006cfa0 legacy=0x7f4820132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 >> v1:172.21.15.189:6789/0 conn(0x7f48200a26c0 legacy=0x7f48201334c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: _finish_hunting 0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 monclient: found mon.a 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200a6ed0 con 0x7f48200a31b0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: _finish_auth 0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: _check_auth_tickets 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 5 monclient: authenticate success, global_id 334512 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient: _renew_subs 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820134600 con 0x7f48200a31b0 2024-09-16T06:59:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient: _renew_subs 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48201353f0 con 0x7f48200a31b0 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4818001480 con 0x7f48200a31b0 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481dffb700 4 set_mon_vals no callback set 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f48180051c0 con 0x7f48200a31b0 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: dump: 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f481800fc30 con 0x7f48200a31b0 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4804000f10 con 0x7f48200a31b0 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f4818044290 con 0x7f48200a31b0 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6808/3838562525 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4800043a70 con 0x7f480003fe40 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=255) v2 ==== 24+0+0 (unknown 2784100248 0 0) 0x7f4818014cf0 con 0x7f48200a31b0 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481cff9700 10 monclient: handle_get_version_reply finishing 1 version 255 2024-09-16T06:59:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.821+0000 7f481effd700 10 monclient: get_auth_request con 0x7f4800036190 auth_method 0 2024-09-16T06:59:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.825+0000 7f48167fc700 10 monclient: get_auth_request con 0x7f480003fe40 auth_method 0 2024-09-16T06:59:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.825+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== osd.0 v1:172.21.15.76:6808/3838562525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f4800043a70 con 0x7f480003fe40 2024-09-16T06:59:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.841+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 --> v1:172.21.15.76:6808/3838562525 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4804002c00 con 0x7f480003fe40 2024-09-16T06:59:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f481cff9700 1 -- 172.21.15.76:0/1805727049 <== osd.0 v1:172.21.15.76:6808/3838562525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f4804002c00 con 0x7f480003fe40 2024-09-16T06:59:23.843 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:23.843 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:23.843 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 >> v1:172.21.15.76:6808/3838562525 conn(0x7f480003fe40 legacy=0x7f48000422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 >> v1:172.21.15.189:6812/3098810674 conn(0x7f4800036190 legacy=0x7f4800038650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 10 monclient: shutdown 2024-09-16T06:59:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 >> v1:172.21.15.76:6789/0 conn(0x7f48200a31b0 legacy=0x7f48201d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f481f7fe700 1 -- 172.21.15.76:0/1805727049 reap_dead start 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 shutdown_connections 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 >> 172.21.15.76:0/1805727049 conn(0x7f48200a82f0 msgr2=0x7f48200a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 shutdown_connections 2024-09-16T06:59:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.845+0000 7f4825845700 1 -- 172.21.15.76:0/1805727049 wait complete. 2024-09-16T06:59:23.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: get_monmap_and_config 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: build_initial_monmap 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: monmap: 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.974 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:23.978688+0000 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:23.978688+0000 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 1 Processor -- start 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 1 -- start start 2024-09-16T06:59:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: init 2024-09-16T06:59:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-b con 0x7f8f6c0964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-c con 0x7f8f6c09b1c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-a con 0x7f8f6c137e70 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c098560 con 0x7f8f6c137e70 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c099ec0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c09b5f0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.973+0000 7f8f7292d700 10 monclient(hunting): authenticate will time out at 2111.111885s 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7212c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8f6c137e70 0x7f8f6c13a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47940/0 (socket says 172.21.15.76:47940) 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7212c700 1 -- 172.21.15.76:0/3679849983 learned_addr learned my addr 172.21.15.76:0/3679849983 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986073519 0 0) 0x7f8f6c098560 con 0x7f8f6c137e70 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): my global_id is 334515 2024-09-16T06:59:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f580035d0 con 0x7f8f6c137e70 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310165886 0 0) 0x7f8f6c09b5f0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): my global_id is 374165 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c098560 con 0x7f8f6c0964f0 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89900148 0 0) 0x7f8f6c099ec0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient(hunting): my global_id is 284992 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c09b5f0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124474654 0 0) 0x7f8f6c098560 con 0x7f8f6c0964f0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 >> v1:172.21.15.189:6789/0 conn(0x7f8f6c09b1c0 legacy=0x7f8f6c136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 >> v1:172.21.15.76:6789/0 conn(0x7f8f6c137e70 legacy=0x7f8f6c13a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: _finish_hunting 0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 monclient: found mon.noname-b 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c099be0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: _finish_auth 0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: _check_auth_tickets 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f680015f0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:23.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: dump: 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 5 monclient: authenticate success, global_id 374165 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f68001910 con 0x7f8f6c0964f0 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:23.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 4 set_mon_vals no callback set 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 1 -- 172.21.15.76:0/3679849983 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f68001ca0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f70929700 10 monclient: dump: 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 10 monclient: shutdown 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 1 -- 172.21.15.76:0/3679849983 >> v1:172.21.15.76:6790/0 conn(0x7f8f6c0964f0 legacy=0x7f8f6c0968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 1 -- 172.21.15.76:0/3679849983 shutdown_connections 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 1 -- 172.21.15.76:0/3679849983 >> 172.21.15.76:0/3679849983 conn(0x7f8f6c0a0030 msgr2=0x7f8f6c0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 1 -- 172.21.15.76:0/3679849983 shutdown_connections 2024-09-16T06:59:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 1 -- 172.21.15.76:0/3679849983 wait complete. 2024-09-16T06:59:23.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.977+0000 7f8f7292d700 10 monclient: build_initial_monmap 2024-09-16T06:59:23.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: monmap: 2024-09-16T06:59:23.989 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:23.989 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:23.989 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 Processor -- start 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- start start 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: init 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:23.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-c con 0x7f8f6c0964f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-a con 0x7f8f6c09b1c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient(hunting): picked mon.noname-b con 0x7f8f6c137e70 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c071df0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c0a2b80 con 0x7f8f6c137e70 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c09a540 con 0x7f8f6c0964f0 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient(hunting): authenticate will time out at 2111.117850s 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7192b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f8f6c0964f0 0x7f8f6c0716e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40292/0 (socket says 172.21.15.76:40292) 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7192b700 1 -- 172.21.15.76:0/4223753903 learned_addr learned my addr 172.21.15.76:0/4223753903 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:23.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267440176 0 0) 0x7f8f6c09a540 con 0x7f8f6c0964f0 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): my global_id is 374168 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f500035d0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803421819 0 0) 0x7f8f6c071df0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): my global_id is 334518 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c09a540 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135806452 0 0) 0x7f8f500035d0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f6c071df0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f680015f0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): dump: 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.993 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586673500 0 0) 0x7f8f580035d0 con 0x7f8f6c137e70 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): my global_id is 284995 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f500035d0 con 0x7f8f6c137e70 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013976250 0 0) 0x7f8f6c09a540 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f580035d0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f5c0018b0 con 0x7f8f6c09b1c0 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient(hunting): dump: 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2354578668 0 0) 0x7f8f6c071df0 con 0x7f8f6c0964f0 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 >> v1:172.21.15.189:6789/0 conn(0x7f8f6c137e70 legacy=0x7f8f6c1cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 >> v1:172.21.15.76:6789/0 conn(0x7f8f6c09b1c0 legacy=0x7f8f6c1cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 monclient: found mon.c 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c13e760 con 0x7f8f6c0964f0 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 5 monclient: authenticate success, global_id 374168 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _renew_subs 2024-09-16T06:59:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c13d820 con 0x7f8f6c0964f0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _renew_subs 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c13cf80 con 0x7f8f6c0964f0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f68003a10 con 0x7f8f6c0964f0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f8f68003d20 con 0x7f8f6c0964f0 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f63fff700 4 set_mon_vals no callback set 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.981+0000 7f8f627fc700 10 monclient: dump: 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f8f68003e80 con 0x7f8f6c0964f0 2024-09-16T06:59:23.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f7292d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f7292d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f7292d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f54005100 con 0x7f8f6c0964f0 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f8f68044050 con 0x7f8f6c0964f0 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.985+0000 7f8f7112a700 10 monclient: get_auth_request con 0x7f8f50036450 auth_method 0 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.989+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f8f68014a50 con 0x7f8f6c0964f0 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.989+0000 7f8f627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:23.989+0000 7f8f627fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:24.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.157+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f54000b90 con 0x7f8f50036450 2024-09-16T06:59:24.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.157+0000 7f8f627fc700 1 -- 172.21.15.76:0/4223753903 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7f8f54000b90 con 0x7f8f50036450 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 >> v1:172.21.15.189:6812/3098810674 conn(0x7f8f50036450 legacy=0x7f8f50038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 10 monclient: shutdown 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 >> v1:172.21.15.76:6790/0 conn(0x7f8f6c0964f0 legacy=0x7f8f6c0716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 shutdown_connections 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 >> 172.21.15.76:0/4223753903 conn(0x7f8f6c0a0030 msgr2=0x7f8f6c09bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 shutdown_connections 2024-09-16T06:59:24.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.161+0000 7f8f7292d700 1 -- 172.21.15.76:0/4223753903 wait complete. 2024-09-16T06:59:24.162 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T06:59:24.177 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T06:59:24.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:24.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.297+0000 7fb6b93d0700 10 monclient: get_monmap_and_config 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.297+0000 7fb6b93d0700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.297+0000 7fb6b93d0700 10 monclient: monmap: 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:24.302620+0000 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:24.302620+0000 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.298 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:24.299 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 Processor -- start 2024-09-16T06:59:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 -- start start 2024-09-16T06:59:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient: init 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b40a55f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b40a6da0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b413b410 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40a71b0 con 0x7fb6b40a55f0 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb6b40a55f0 0x7fb6b40a59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47956/0 (socket says 172.21.15.76:47956) 2024-09-16T06:59:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40a1470 con 0x7fb6b40a6da0 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b3fff700 1 -- 172.21.15.76:0/69802152 learned_addr learned my addr 172.21.15.76:0/69802152 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b4099c70 con 0x7fb6b413b410 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient(hunting): authenticate will time out at 2111.437905s 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373385738 0 0) 0x7fb6b40a71b0 con 0x7fb6b40a55f0 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): my global_id is 334521 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0035d0 con 0x7fb6b40a55f0 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114800364 0 0) 0x7fb6b4099c70 con 0x7fb6b413b410 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): my global_id is 374171 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b40a71b0 con 0x7fb6b413b410 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593225708 0 0) 0x7fb6b40a1470 con 0x7fb6b40a6da0 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient(hunting): my global_id is 284998 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b4099c70 con 0x7fb6b40a6da0 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152092488 0 0) 0x7fb69c0035d0 con 0x7fb6b40a55f0 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 >> v1:172.21.15.76:6790/0 conn(0x7fb6b413b410 legacy=0x7fb6b413d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 >> v1:172.21.15.189:6789/0 conn(0x7fb6b40a6da0 legacy=0x7fb6b40a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 monclient: found mon.noname-a 2024-09-16T06:59:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b409a930 con 0x7fb6b40a55f0 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6a4001610 con 0x7fb6b40a55f0 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: dump: 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 5 monclient: authenticate success, global_id 334521 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a4001870 con 0x7fb6b40a55f0 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 1 -- 172.21.15.76:0/69802152 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6a4001c40 con 0x7fb6b40a55f0 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 4 set_mon_vals no callback set 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b2ffd700 10 monclient: dump: 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 10 monclient: shutdown 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.301+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 >> v1:172.21.15.76:6789/0 conn(0x7fb6b40a55f0 legacy=0x7fb6b40a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 shutdown_connections 2024-09-16T06:59:24.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 >> 172.21.15.76:0/69802152 conn(0x7fb6b40a42f0 msgr2=0x7fb6b40a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 shutdown_connections 2024-09-16T06:59:24.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- 172.21.15.76:0/69802152 wait complete. 2024-09-16T06:59:24.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient: monmap: 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 Processor -- start 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- start start 2024-09-16T06:59:24.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient: init 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b40a55f0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b40a6da0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b413b410 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b409d640 con 0x7fb6b40a55f0 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40a0150 con 0x7fb6b413b410 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b409cf70 con 0x7fb6b40a6da0 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b93d0700 10 monclient(hunting): authenticate will time out at 2111.442904s 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb6b40a55f0 0x7fb6b409c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47964/0 (socket says 172.21.15.76:47964) 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b37fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fb6b40a6da0 0x7fb6b409c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40316/0 (socket says 172.21.15.76:40316) 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b3fff700 1 -- 172.21.15.76:0/2138493126 learned_addr learned my addr 172.21.15.76:0/2138493126 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458687881 0 0) 0x7fb6b409cf70 con 0x7fb6b40a6da0 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): my global_id is 374174 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a00035d0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891035924 0 0) 0x7fb6b409d640 con 0x7fb6b40a55f0 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): my global_id is 334524 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b409cf70 con 0x7fb6b40a55f0 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900249867 0 0) 0x7fb6b40a0150 con 0x7fb6b413b410 2024-09-16T06:59:24.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): my global_id is 285001 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b409d640 con 0x7fb6b413b410 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471904435 0 0) 0x7fb6a00035d0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b40a0150 con 0x7fb6b40a6da0 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6a80018f0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): dump: 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.317 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3774678374 0 0) 0x7fb6b409cf70 con 0x7fb6b40a55f0 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a00035d0 con 0x7fb6b40a55f0 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.305+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6a4001b80 con 0x7fb6b40a55f0 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient(hunting): dump: 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.318 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 423874360 0 0) 0x7fb6b40a0150 con 0x7fb6b40a6da0 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 >> v1:172.21.15.189:6789/0 conn(0x7fb6b413b410 legacy=0x7fb6b41d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 >> v1:172.21.15.76:6789/0 conn(0x7fb6b40a55f0 legacy=0x7fb6b409c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 monclient: found mon.c 2024-09-16T06:59:24.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b413e0a0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 5 monclient: authenticate success, global_id 374174 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 10 monclient: _renew_subs 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b40a3da0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 10 monclient: _renew_subs 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b409fe70 con 0x7fb6b40a6da0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a80036c0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fb6a80039d0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b27fc700 4 set_mon_vals no callback set 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: dump: 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fb6a8003b30 con 0x7fb6b40a6da0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb694000f10 con 0x7fb6b40a6da0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fb6a8043e90 con 0x7fb6b40a6da0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6800/3502405955 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6a0043d50 con 0x7fb6a0040120 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_get_version_reply(handle=1 version=255) v2 ==== 24+0+0 (unknown 2784100248 0 0) 0x7fb6a80441e0 con 0x7fb6b40a6da0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b0ff9700 10 monclient: handle_get_version_reply finishing 1 version 255 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b3fff700 10 monclient: get_auth_request con 0x7fb6a0036470 auth_method 0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.309+0000 7fb6b8bcf700 10 monclient: get_auth_request con 0x7fb6a0040120 auth_method 0 2024-09-16T06:59:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.313+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== osd.1 v1:172.21.15.76:6800/3502405955 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fb6a0043d50 con 0x7fb6a0040120 2024-09-16T06:59:24.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 --> v1:172.21.15.76:6800/3502405955 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb694002c00 con 0x7fb6a0040120 2024-09-16T06:59:24.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b0ff9700 1 -- 172.21.15.76:0/2138493126 <== osd.1 v1:172.21.15.76:6800/3502405955 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fb694002c00 con 0x7fb6a0040120 2024-09-16T06:59:24.324 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:24.325 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:24.325 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 >> v1:172.21.15.76:6800/3502405955 conn(0x7fb6a0040120 legacy=0x7fb6a0042580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 >> v1:172.21.15.189:6812/3098810674 conn(0x7fb6a0036470 legacy=0x7fb6a0038930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 10 monclient: shutdown 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 >> v1:172.21.15.76:6790/0 conn(0x7fb6b40a6da0 legacy=0x7fb6b409c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b8bcf700 1 -- 172.21.15.76:0/2138493126 reap_dead start 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 shutdown_connections 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 >> 172.21.15.76:0/2138493126 conn(0x7fb6b40a42f0 msgr2=0x7fb6b413a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 shutdown_connections 2024-09-16T06:59:24.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.325+0000 7fb6b93d0700 1 -- 172.21.15.76:0/2138493126 wait complete. 2024-09-16T06:59:24.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T06:59:24.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.449+0000 7f4806e65700 10 monclient: get_monmap_and_config 2024-09-16T06:59:24.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.449+0000 7f4806e65700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.449+0000 7f4806e65700 10 monclient: monmap: 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:24.455269+0000 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:24.455269+0000 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:24.451 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 Processor -- start 2024-09-16T06:59:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- start start 2024-09-16T06:59:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient: init 2024-09-16T06:59:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-b con 0x7f4800097b60 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-a con 0x7f4800099090 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-c con 0x7f4800139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800098990 con 0x7f4800099090 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800097f40 con 0x7f4800139500 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480009d610 con 0x7f4800097b60 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient(hunting): authenticate will time out at 2111.589208s 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4805e63700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4800097b60 0x7f480009ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40332/0 (socket says 172.21.15.76:40332) 2024-09-16T06:59:24.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4805e63700 1 -- 172.21.15.76:0/4066558018 learned_addr learned my addr 172.21.15.76:0/4066558018 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171791003 0 0) 0x7f480009d610 con 0x7f4800097b60 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): my global_id is 374177 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ec0035d0 con 0x7f4800097b60 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095203415 0 0) 0x7f4800098990 con 0x7f4800099090 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): my global_id is 334527 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480009d610 con 0x7f4800099090 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58649924 0 0) 0x7f4800097f40 con 0x7f4800139500 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient(hunting): my global_id is 285004 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800098990 con 0x7f4800139500 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2646263067 0 0) 0x7f47ec0035d0 con 0x7f4800097b60 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 >> v1:172.21.15.189:6789/0 conn(0x7f4800139500 legacy=0x7f480013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 >> v1:172.21.15.76:6789/0 conn(0x7f4800099090 legacy=0x7f480009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 monclient: found mon.noname-b 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000986b0 con 0x7f4800097b60 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc001990 con 0x7f4800097b60 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 5 monclient: authenticate success, global_id 374177 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: dump: 2024-09-16T06:59:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47fc001bf0 con 0x7f4800097b60 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 1 -- 172.21.15.76:0/4066558018 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc001fc0 con 0x7f4800097b60 2024-09-16T06:59:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 4 set_mon_vals no callback set 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4804e61700 10 monclient: dump: 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.460 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 10 monclient: shutdown 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- 172.21.15.76:0/4066558018 >> v1:172.21.15.76:6790/0 conn(0x7f4800097b60 legacy=0x7f480009ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- 172.21.15.76:0/4066558018 shutdown_connections 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- 172.21.15.76:0/4066558018 >> 172.21.15.76:0/4066558018 conn(0x7f48000a0030 msgr2=0x7f48000a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- 172.21.15.76:0/4066558018 shutdown_connections 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.453+0000 7f4806e65700 1 -- 172.21.15.76:0/4066558018 wait complete. 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: monmap: 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:24.461 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 Processor -- start 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- start start 2024-09-16T06:59:24.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: init 2024-09-16T06:59:24.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:24.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-a con 0x7f4800097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-b con 0x7f4800099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient(hunting): picked mon.noname-c con 0x7f4800139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480009a540 con 0x7f4800097b60 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800133400 con 0x7f4800099090 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4800133120 con 0x7f4800139500 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient(hunting): authenticate will time out at 2111.594099s 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806664700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f4800139500 0x7f48001cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40338/0 (socket says 172.21.15.76:40338) 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806664700 1 -- 172.21.15.76:0/4089829109 learned_addr learned my addr 172.21.15.76:0/4089829109 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292778181 0 0) 0x7f480009a540 con 0x7f4800097b60 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): my global_id is 334530 2024-09-16T06:59:24.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e40035d0 con 0x7f4800097b60 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474957197 0 0) 0x7f4800133120 con 0x7f4800139500 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): my global_id is 374180 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480009a540 con 0x7f4800139500 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421123298 0 0) 0x7f4800133400 con 0x7f4800099090 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): my global_id is 285007 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800133120 con 0x7f4800099090 2024-09-16T06:59:24.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320632524 0 0) 0x7f47e40035d0 con 0x7f4800097b60 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4800133400 con 0x7f4800097b60 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc001f00 con 0x7f4800097b60 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): dump: 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.470 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857347934 0 0) 0x7f480009a540 con 0x7f4800139500 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47e40035d0 con 0x7f4800139500 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47f80018f0 con 0x7f4800139500 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient(hunting): dump: 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.471 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2891182486 0 0) 0x7f4800133120 con 0x7f4800099090 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f480009a540 con 0x7f4800099090 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 229095893 0 0) 0x7f4800133400 con 0x7f4800097b60 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 >> v1:172.21.15.76:6790/0 conn(0x7f4800139500 legacy=0x7f48001cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 >> v1:172.21.15.189:6789/0 conn(0x7f4800099090 legacy=0x7f480006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 monclient: found mon.a 2024-09-16T06:59:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480006fe40 con 0x7f4800097b60 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f47f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 5 monclient: authenticate success, global_id 334530 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _renew_subs 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4800132e40 con 0x7f4800097b60 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _renew_subs 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.457+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4800132890 con 0x7f4800097b60 2024-09-16T06:59:24.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47fc004f40 con 0x7f4800097b60 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f3fff700 4 set_mon_vals no callback set 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f47fc001590 con 0x7f4800097b60 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 10 monclient: dump: 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.474 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f47fc0052d0 con 0x7f4800097b60 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f4806e65700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f4806e65700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f4806e65700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e8005100 con 0x7f4800097b60 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f47fc044440 con 0x7f4800097b60 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.461+0000 7f4805662700 10 monclient: get_auth_request con 0x7f47e40364e0 auth_method 0 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.465+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f47fc014bb0 con 0x7f4800097b60 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.465+0000 7f47f27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.465+0000 7f47f27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:24.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.637+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f47e8000b90 con 0x7f47e40364e0 2024-09-16T06:59:24.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.637+0000 7f47f27fc700 1 -- 172.21.15.76:0/4089829109 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7f47e8000b90 con 0x7f47e40364e0 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 >> v1:172.21.15.189:6812/3098810674 conn(0x7f47e40364e0 legacy=0x7f47e40389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 10 monclient: shutdown 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 >> v1:172.21.15.76:6789/0 conn(0x7f4800097b60 legacy=0x7f480006f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 shutdown_connections 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 >> 172.21.15.76:0/4089829109 conn(0x7f48000a0030 msgr2=0x7f48000a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 shutdown_connections 2024-09-16T06:59:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.641+0000 7f4806e65700 1 -- 172.21.15.76:0/4089829109 wait complete. 2024-09-16T06:59:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T06:59:24.651 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T06:59:24.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: get_monmap_and_config 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: monmap: 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:24.777717+0000 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:24.777717+0000 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.773 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.774 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.774 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:24.774 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:24.774 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 1 Processor -- start 2024-09-16T06:59:24.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 1 -- start start 2024-09-16T06:59:24.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: init 2024-09-16T06:59:24.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-b con 0x7f9c4c135430 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-c con 0x7f9c4c06c570 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-a con 0x7f9c4c0a4db0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c06bbe0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c06c330 con 0x7f9c4c06c570 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c06bfb0 con 0x7f9c4c135430 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c50acf700 10 monclient(hunting): authenticate will time out at 2111.911650s 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c4bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9c4c0a4db0 0x7f9c4c13db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:47994/0 (socket says 172.21.15.76:47994) 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.772+0000 7f9c4bfff700 1 -- 172.21.15.76:0/3545970546 learned_addr learned my addr 172.21.15.76:0/3545970546 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345603319 0 0) 0x7f9c4c06bbe0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): my global_id is 334533 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c340035d0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899409932 0 0) 0x7f9c4c06bfb0 con 0x7f9c4c135430 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): my global_id is 374183 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c4c06bbe0 con 0x7f9c4c135430 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833893048 0 0) 0x7f9c4c06c330 con 0x7f9c4c06c570 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient(hunting): my global_id is 285010 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c4c06bfb0 con 0x7f9c4c06c570 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129474536 0 0) 0x7f9c340035d0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 >> v1:172.21.15.76:6790/0 conn(0x7f9c4c135430 legacy=0x7f9c4c135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 >> v1:172.21.15.189:6789/0 conn(0x7f9c4c06c570 legacy=0x7f9c4c13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 monclient: found mon.noname-a 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c06d2d0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 5 monclient: authenticate success, global_id 334533 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c44001990 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: dump: 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.785 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c44001bf0 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:24.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 1 -- 172.21.15.76:0/3545970546 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c44001f80 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 4 set_mon_vals no callback set 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4a7fc700 10 monclient: dump: 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: shutdown 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- 172.21.15.76:0/3545970546 >> v1:172.21.15.76:6789/0 conn(0x7f9c4c0a4db0 legacy=0x7f9c4c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- 172.21.15.76:0/3545970546 shutdown_connections 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- 172.21.15.76:0/3545970546 >> 172.21.15.76:0/3545970546 conn(0x7f9c4c0a5e00 msgr2=0x7f9c4c0a6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- 172.21.15.76:0/3545970546 shutdown_connections 2024-09-16T06:59:24.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- 172.21.15.76:0/3545970546 wait complete. 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: monmap: 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 Processor -- start 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- start start 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: init 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-c con 0x7f9c4c06c570 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-b con 0x7f9c4c0a4db0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient(hunting): picked mon.noname-a con 0x7f9c4c135430 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c131b80 con 0x7f9c4c135430 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c131540 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c072650 con 0x7f9c4c06c570 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c50acf700 10 monclient(hunting): authenticate will time out at 2111.915800s 2024-09-16T06:59:24.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4b7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f9c4c06c570 0x7f9c4c130720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40350/0 (socket says 172.21.15.76:40350) 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.776+0000 7f9c4b7fe700 1 -- 172.21.15.76:0/3516744835 learned_addr learned my addr 172.21.15.76:0/3516744835 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120400902 0 0) 0x7f9c4c072650 con 0x7f9c4c06c570 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): my global_id is 374186 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c380035d0 con 0x7f9c4c06c570 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 127799661 0 0) 0x7f9c380035d0 con 0x7f9c4c06c570 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c4c072650 con 0x7f9c4c06c570 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c400012d0 con 0x7f9c4c06c570 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:24.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): dump: 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528098922 0 0) 0x7f9c4c131b80 con 0x7f9c4c135430 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): my global_id is 334536 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c380035d0 con 0x7f9c4c135430 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123691284 0 0) 0x7f9c4c131540 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient(hunting): my global_id is 285013 2024-09-16T06:59:24.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c4c131b80 con 0x7f9c4c0a4db0 2024-09-16T06:59:24.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241927627 0 0) 0x7f9c4c072650 con 0x7f9c4c06c570 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 >> v1:172.21.15.189:6789/0 conn(0x7f9c4c0a4db0 legacy=0x7f9c4c130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 >> v1:172.21.15.76:6789/0 conn(0x7f9c4c135430 legacy=0x7f9c4c1d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 monclient: found mon.c 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c133670 con 0x7f9c4c06c570 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 5 monclient: authenticate success, global_id 374186 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 10 monclient: _renew_subs 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c1322a0 con 0x7f9c4c06c570 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 10 monclient: _renew_subs 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c134a20 con 0x7f9c4c06c570 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c40001580 con 0x7f9c4c06c570 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9c40003af0 con 0x7f9c4c06c570 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 10 monclient: dump: 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c49ffb700 4 set_mon_vals no callback set 2024-09-16T06:59:24.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9c40003c50 con 0x7f9c4c06c570 2024-09-16T06:59:24.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f9c40043e30 con 0x7f9c4c06c570 2024-09-16T06:59:24.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6804/779009499 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c2c005330 con 0x7f9c2c0015c0 2024-09-16T06:59:24.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c4affd700 10 monclient: get_auth_request con 0x7f9c38036320 auth_method 0 2024-09-16T06:59:24.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.780+0000 7f9c4bfff700 10 monclient: get_auth_request con 0x7f9c2c0015c0 auth_method 0 2024-09-16T06:59:24.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.784+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== osd.2 v1:172.21.15.76:6804/779009499 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9c2c005330 con 0x7f9c2c0015c0 2024-09-16T06:59:24.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 --> v1:172.21.15.76:6804/779009499 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9c2c006ff0 con 0x7f9c2c0015c0 2024-09-16T06:59:24.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c33fff700 1 -- 172.21.15.76:0/3516744835 <== osd.2 v1:172.21.15.76:6804/779009499 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f9c2c006ff0 con 0x7f9c2c0015c0 2024-09-16T06:59:24.800 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:24.800 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:24.800 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:24.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 >> v1:172.21.15.76:6804/779009499 conn(0x7f9c2c0015c0 legacy=0x7f9c2c003a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9c38036320 legacy=0x7f9c380387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 10 monclient: shutdown 2024-09-16T06:59:24.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 >> v1:172.21.15.76:6790/0 conn(0x7f9c4c06c570 legacy=0x7f9c4c130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c4bfff700 1 -- 172.21.15.76:0/3516744835 reap_dead start 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 shutdown_connections 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 >> 172.21.15.76:0/3516744835 conn(0x7f9c4c0a5e00 msgr2=0x7f9c4c13e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 shutdown_connections 2024-09-16T06:59:24.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.800+0000 7f9c50acf700 1 -- 172.21.15.76:0/3516744835 wait complete. 2024-09-16T06:59:24.815 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T06:59:24.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 10 monclient: get_monmap_and_config 2024-09-16T06:59:24.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 10 monclient: monmap: 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:24.935207+0000 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:24.935207+0000 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:24.931 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 1 Processor -- start 2024-09-16T06:59:24.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 1 -- start start 2024-09-16T06:59:24.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.928+0000 7fd62359e700 10 monclient: init 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd624097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd624099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd624139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd624098990 con 0x7fd624097b60 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd624097f40 con 0x7fd624139500 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62409d610 con 0x7fd624099090 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient(hunting): authenticate will time out at 2112.068467s 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd621d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd624099090 0x7fd62409f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40352/0 (socket says 172.21.15.76:40352) 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd621d9b700 1 -- 172.21.15.76:0/3049998958 learned_addr learned my addr 172.21.15.76:0/3049998958 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62259c700 1 --1- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 0x7fd62409ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48024/0 (socket says 172.21.15.76:48024) 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509856831 0 0) 0x7fd62409d610 con 0x7fd624099090 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): my global_id is 374189 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62259c700 1 -- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 legacy=0x7fd62409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62259c700 1 -- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 legacy=0x7fd62409ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62259c700 1 --1- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 0x7fd62409ed60 :-1 s=OPENED pgs=1512 cs=1 l=1).handle_message read tag failed 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62259c700 1 --1- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 0x7fd62409ed60 :-1 s=OPENED pgs=1512 cs=1 l=1).fault on lossy channel, failing 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6100035d0 con 0x7fd624099090 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.76:6789/0 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916531071 0 0) 0x7fd624097f40 con 0x7fd624139500 2024-09-16T06:59:24.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient(hunting): my global_id is 285016 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd62409d610 con 0x7fd624139500 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243464582 0 0) 0x7fd6100035d0 con 0x7fd624099090 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 >> v1:172.21.15.189:6789/0 conn(0x7fd624139500 legacy=0x7fd62413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 legacy=0x7fd62409ed60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 monclient: found mon.noname-b 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240986b0 con 0x7fd624099090 2024-09-16T06:59:24.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd614001990 con 0x7fd624099090 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 5 monclient: authenticate success, global_id 374189 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: dump: 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.943 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd614001c70 con 0x7fd624099090 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 1 -- 172.21.15.76:0/3049998958 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd614002000 con 0x7fd624099090 2024-09-16T06:59:24.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 4 set_mon_vals no callback set 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62159a700 10 monclient: dump: 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.945 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: shutdown 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- 172.21.15.76:0/3049998958 >> v1:172.21.15.76:6790/0 conn(0x7fd624099090 legacy=0x7fd62409f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- 172.21.15.76:0/3049998958 shutdown_connections 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- 172.21.15.76:0/3049998958 >> 172.21.15.76:0/3049998958 conn(0x7fd6240a0030 msgr2=0x7fd6240a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- 172.21.15.76:0/3049998958 shutdown_connections 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- 172.21.15.76:0/3049998958 wait complete. 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: build_initial_monmap 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: monmap: 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:24.946 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 Processor -- start 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 1 -- start start 2024-09-16T06:59:24.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.932+0000 7fd62359e700 10 monclient: init 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd624097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd624099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd624139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6241368d0 con 0x7fd624097b60 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd624138210 con 0x7fd624139500 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6241387d0 con 0x7fd624099090 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient(hunting): authenticate will time out at 2112.072391s 2024-09-16T06:59:24.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd621d9b700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fd624099090 0x7fd6241361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40364/0 (socket says 172.21.15.76:40364) 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd621d9b700 1 -- 172.21.15.76:0/221652809 learned_addr learned my addr 172.21.15.76:0/221652809 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664705655 0 0) 0x7fd6241368d0 con 0x7fd624097b60 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): my global_id is 334542 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6040035d0 con 0x7fd624097b60 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435077196 0 0) 0x7fd6241387d0 con 0x7fd624099090 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): my global_id is 374192 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6241368d0 con 0x7fd624099090 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324675590 0 0) 0x7fd624138210 con 0x7fd624139500 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): my global_id is 285019 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6241387d0 con 0x7fd624139500 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507401258 0 0) 0x7fd6241368d0 con 0x7fd624099090 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd624138210 con 0x7fd624099090 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809360697 0 0) 0x7fd6040035d0 con 0x7fd624097b60 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6241368d0 con 0x7fd624097b60 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd60c0017e0 con 0x7fd624097b60 2024-09-16T06:59:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): dump: 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd614001990 con 0x7fd624099090 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient(hunting): dump: 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3832214171 0 0) 0x7fd624138210 con 0x7fd624099090 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 >> v1:172.21.15.189:6789/0 conn(0x7fd624139500 legacy=0x7fd6241d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 >> v1:172.21.15.76:6789/0 conn(0x7fd624097b60 legacy=0x7fd624135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 monclient: found mon.c 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624138ab0 con 0x7fd624099090 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: _finish_auth 0 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 5 monclient: authenticate success, global_id 374192 2024-09-16T06:59:24.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _renew_subs 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6241384f0 con 0x7fd624099090 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _renew_subs 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd624137320 con 0x7fd624099090 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd614004f80 con 0x7fd624099090 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fd614005290 con 0x7fd624099090 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61bfff700 4 set_mon_vals no callback set 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 10 monclient: dump: 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:24.954 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fd614005470 con 0x7fd624099090 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fd614044430 con 0x7fd624099090 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62259c700 10 monclient: get_auth_request con 0x7fd604036430 auth_method 0 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:24.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.936+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608005100 con 0x7fd624099090 2024-09-16T06:59:24.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.944+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fd614014e90 con 0x7fd624099090 2024-09-16T06:59:24.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.944+0000 7fd61affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:24.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:24.944+0000 7fd61affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:25.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.108+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd608000b90 con 0x7fd604036430 2024-09-16T06:59:25.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.108+0000 7fd61affd700 1 -- 172.21.15.76:0/221652809 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7fd608000b90 con 0x7fd604036430 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 >> v1:172.21.15.189:6812/3098810674 conn(0x7fd604036430 legacy=0x7fd6040388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 10 monclient: shutdown 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 >> v1:172.21.15.76:6790/0 conn(0x7fd624099090 legacy=0x7fd6241361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 shutdown_connections 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 >> 172.21.15.76:0/221652809 conn(0x7fd6240a0030 msgr2=0x7fd6240702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 shutdown_connections 2024-09-16T06:59:25.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.112+0000 7fd62359e700 1 -- 172.21.15.76:0/221652809 wait complete. 2024-09-16T06:59:25.114 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T06:59:25.125 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T06:59:25.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:25.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: get_monmap_and_config 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: monmap: 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:25.245525+0000 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:25.245525+0000 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 1 Processor -- start 2024-09-16T06:59:25.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 1 -- start start 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: init 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-b con 0x7f36ec09da30 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-a con 0x7f36ec09b3e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-c con 0x7f36ec138140 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec09a880 con 0x7f36ec09b3e0 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec09ff00 con 0x7f36ec138140 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec09b130 con 0x7f36ec09da30 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f2f74700 10 monclient(hunting): authenticate will time out at 2112.379323s 2024-09-16T06:59:25.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f0d10700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f36ec09da30 0x7f36ec099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40368/0 (socket says 172.21.15.76:40368) 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36f0d10700 1 -- 172.21.15.76:0/2646372367 learned_addr learned my addr 172.21.15.76:0/2646372367 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36ebfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f36ec09b3e0 0x7f36ec09a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48040/0 (socket says 172.21.15.76:48040) 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.240+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110577353 0 0) 0x7f36ec09b130 con 0x7f36ec09da30 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): my global_id is 374195 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d80035d0 con 0x7f36ec09da30 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835246155 0 0) 0x7f36ec09a880 con 0x7f36ec09b3e0 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): my global_id is 334545 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec09b130 con 0x7f36ec09b3e0 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853730753 0 0) 0x7f36ec09ff00 con 0x7f36ec138140 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient(hunting): my global_id is 285022 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec09a880 con 0x7f36ec138140 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248536290 0 0) 0x7f36d80035d0 con 0x7f36ec09da30 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 >> v1:172.21.15.189:6789/0 conn(0x7f36ec138140 legacy=0x7f36ec13a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 >> v1:172.21.15.76:6789/0 conn(0x7f36ec09b3e0 legacy=0x7f36ec09a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 monclient: found mon.noname-b 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec09bca0 con 0x7f36ec09da30 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 5 monclient: authenticate success, global_id 374195 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36e4001990 con 0x7f36ec09da30 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: dump: 2024-09-16T06:59:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e4001c70 con 0x7f36ec09da30 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 1 -- 172.21.15.76:0/2646372367 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36e4002000 con 0x7f36ec09da30 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 4 set_mon_vals no callback set 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36eb7fe700 10 monclient: dump: 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.252 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 10 monclient: shutdown 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- 172.21.15.76:0/2646372367 >> v1:172.21.15.76:6790/0 conn(0x7f36ec09da30 legacy=0x7f36ec099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- 172.21.15.76:0/2646372367 shutdown_connections 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- 172.21.15.76:0/2646372367 >> 172.21.15.76:0/2646372367 conn(0x7f36ec0a3f60 msgr2=0x7f36ec0a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- 172.21.15.76:0/2646372367 shutdown_connections 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- 172.21.15.76:0/2646372367 wait complete. 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 10 monclient: monmap: 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:25.253 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 Processor -- start 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 1 -- start start 2024-09-16T06:59:25.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.244+0000 7f36f2f74700 10 monclient: init 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-a con 0x7f36ec09b3e0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-c con 0x7f36ec09da30 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient(hunting): picked mon.noname-b con 0x7f36ec138140 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec09df90 con 0x7f36ec09b3e0 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec11d5e0 con 0x7f36ec138140 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ec0a2390 con 0x7f36ec09da30 2024-09-16T06:59:25.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient(hunting): authenticate will time out at 2112.384770s 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f0d10700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f36ec09b3e0 0x7f36ec1334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48046/0 (socket says 172.21.15.76:48046) 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f0d10700 1 -- 172.21.15.76:0/1558709058 learned_addr learned my addr 172.21.15.76:0/1558709058 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729610269 0 0) 0x7f36ec0a2390 con 0x7f36ec09da30 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): my global_id is 374198 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc0035d0 con 0x7f36ec09da30 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602309459 0 0) 0x7f36ec09df90 con 0x7f36ec09b3e0 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): my global_id is 334548 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec0a2390 con 0x7f36ec09b3e0 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194321653 0 0) 0x7f36ec11d5e0 con 0x7f36ec138140 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): my global_id is 285025 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec09df90 con 0x7f36ec138140 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 938417799 0 0) 0x7f36ec0a2390 con 0x7f36ec09b3e0 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ec11d5e0 con 0x7f36ec09b3e0 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36e4001f80 con 0x7f36ec09b3e0 2024-09-16T06:59:25.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): dump: 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433558314 0 0) 0x7f36cc0035d0 con 0x7f36ec09da30 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ec0a2390 con 0x7f36ec09da30 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36d40019d0 con 0x7f36ec09da30 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient(hunting): dump: 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.263 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4113092168 0 0) 0x7f36ec11d5e0 con 0x7f36ec09b3e0 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 >> v1:172.21.15.76:6790/0 conn(0x7f36ec09da30 legacy=0x7f36ec1ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 >> v1:172.21.15.189:6789/0 conn(0x7f36ec138140 legacy=0x7f36ec1cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 monclient: found mon.a 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec0a2950 con 0x7f36ec09b3e0 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 5 monclient: authenticate success, global_id 334548 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _renew_subs 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec0725c0 con 0x7f36ec09b3e0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _renew_subs 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec133c00 con 0x7f36ec09b3e0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e4005020 con 0x7f36ec09b3e0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f36e4005330 con 0x7f36ec09b3e0 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36eaffd700 4 set_mon_vals no callback set 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 10 monclient: dump: 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36d0000f10 con 0x7f36ec09b3e0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.248+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f36e400fda0 con 0x7f36ec09b3e0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f36e4044270 con 0x7f36ec09b3e0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.189:6808/3809009238 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36cc043db0 con 0x7f36cc0400e0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=255) v2 ==== 24+0+0 (unknown 2784100248 0 0) 0x7f36e4044700 con 0x7f36ec09b3e0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36e97fa700 10 monclient: handle_get_version_reply finishing 1 version 255 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36ebfff700 10 monclient: get_auth_request con 0x7f36cc036430 auth_method 0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36f1511700 10 monclient: get_auth_request con 0x7f36cc0400e0 auth_method 0 2024-09-16T06:59:25.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.252+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== osd.3 v1:172.21.15.189:6808/3809009238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f36cc043db0 con 0x7f36cc0400e0 2024-09-16T06:59:25.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 --> v1:172.21.15.189:6808/3809009238 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f36d0002c00 con 0x7f36cc0400e0 2024-09-16T06:59:25.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36e97fa700 1 -- 172.21.15.76:0/1558709058 <== osd.3 v1:172.21.15.189:6808/3809009238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f36d0002c00 con 0x7f36cc0400e0 2024-09-16T06:59:25.272 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:25.272 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:25.272 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:25.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 >> v1:172.21.15.189:6808/3809009238 conn(0x7f36cc0400e0 legacy=0x7f36cc042540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 >> v1:172.21.15.189:6812/3098810674 conn(0x7f36cc036430 legacy=0x7f36cc0388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 10 monclient: shutdown 2024-09-16T06:59:25.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 >> v1:172.21.15.76:6789/0 conn(0x7f36ec09b3e0 legacy=0x7f36ec1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f1511700 1 -- 172.21.15.76:0/1558709058 reap_dead start 2024-09-16T06:59:25.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 shutdown_connections 2024-09-16T06:59:25.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 >> 172.21.15.76:0/1558709058 conn(0x7f36ec0a3f60 msgr2=0x7f36ec0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 shutdown_connections 2024-09-16T06:59:25.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.272+0000 7f36f2f74700 1 -- 172.21.15.76:0/1558709058 wait complete. 2024-09-16T06:59:25.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.392+0000 7f80b7fff700 10 monclient: get_monmap_and_config 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: monmap: 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:25.399968+0000 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:25.399968+0000 2024-09-16T06:59:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:25.396 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 Processor -- start 2024-09-16T06:59:25.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 -- start start 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: init 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f80b8097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f80b8099090 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f80b8139500 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b8098990 con 0x7f80b8097b60 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b8097f40 con 0x7f80b8139500 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b809d610 con 0x7f80b8099090 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient(hunting): authenticate will time out at 2112.533309s 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b6ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f80b8097b60 0x7f80b809ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48058/0 (socket says 172.21.15.76:48058) 2024-09-16T06:59:25.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b6ffd700 1 -- 172.21.15.76:0/440065310 learned_addr learned my addr 172.21.15.76:0/440065310 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111552516 0 0) 0x7f80b8098990 con 0x7f80b8097b60 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): my global_id is 334551 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a40035d0 con 0x7f80b8097b60 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672713547 0 0) 0x7f80b809d610 con 0x7f80b8099090 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): my global_id is 374201 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8098990 con 0x7f80b8099090 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355448072 0 0) 0x7f80b8097f40 con 0x7f80b8139500 2024-09-16T06:59:25.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient(hunting): my global_id is 285028 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b809d610 con 0x7f80b8139500 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3760422328 0 0) 0x7f80b8098990 con 0x7f80b8099090 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 >> v1:172.21.15.189:6789/0 conn(0x7f80b8139500 legacy=0x7f80b813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 >> v1:172.21.15.76:6789/0 conn(0x7f80b8097b60 legacy=0x7f80b809ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 monclient: found mon.noname-b 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b80986b0 con 0x7f80b8099090 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80b0001970 con 0x7f80b8099090 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: dump: 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.403 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 5 monclient: authenticate success, global_id 374201 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80b0001c90 con 0x7f80b8099090 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 1 -- 172.21.15.76:0/440065310 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80b0002060 con 0x7f80b8099090 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 4 set_mon_vals no callback set 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b5ffb700 10 monclient: dump: 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.404 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 10 monclient: shutdown 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.396+0000 7f80b7fff700 1 -- 172.21.15.76:0/440065310 >> v1:172.21.15.76:6790/0 conn(0x7f80b8099090 legacy=0x7f80b809f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- 172.21.15.76:0/440065310 shutdown_connections 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- 172.21.15.76:0/440065310 >> 172.21.15.76:0/440065310 conn(0x7f80b80a0030 msgr2=0x7f80b80a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- 172.21.15.76:0/440065310 shutdown_connections 2024-09-16T06:59:25.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- 172.21.15.76:0/440065310 wait complete. 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient: monmap: 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:25.410 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 Processor -- start 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- start start 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient: init 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f80b8097b60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f80b8099090 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f80b8139500 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b809a540 con 0x7f80b8097b60 2024-09-16T06:59:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b806ff40 con 0x7f80b8099090 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b81384e0 con 0x7f80b8139500 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b7fff700 10 monclient(hunting): authenticate will time out at 2112.539371s 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b6ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f80b8097b60 0x7f80b806f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48062/0 (socket says 172.21.15.76:48062) 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80b6ffd700 1 -- 172.21.15.76:0/3374514082 learned_addr learned my addr 172.21.15.76:0/3374514082 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447900412 0 0) 0x7f80b809a540 con 0x7f80b8097b60 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 10 monclient(hunting): my global_id is 334554 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.400+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80980035d0 con 0x7f80b8097b60 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238470014 0 0) 0x7f80b81384e0 con 0x7f80b8139500 2024-09-16T06:59:25.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): my global_id is 374204 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b809a540 con 0x7f80b8139500 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085618271 0 0) 0x7f80b806ff40 con 0x7f80b8099090 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): my global_id is 285031 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b81384e0 con 0x7f80b8099090 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1117106337 0 0) 0x7f80b809a540 con 0x7f80b8139500 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80b806ff40 con 0x7f80b8139500 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80a80018b0 con 0x7f80b8139500 2024-09-16T06:59:25.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): dump: 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:25.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 248511933 0 0) 0x7f80980035d0 con 0x7f80b8097b60 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80b809a540 con 0x7f80b8097b60 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80a00019f0 con 0x7f80b8097b60 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): dump: 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3575106785 0 0) 0x7f80b81384e0 con 0x7f80b8099090 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80980035d0 con 0x7f80b8099090 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80b0001930 con 0x7f80b8099090 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient(hunting): dump: 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.416 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612897335 0 0) 0x7f80b806ff40 con 0x7f80b8139500 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 >> v1:172.21.15.189:6789/0 conn(0x7f80b8099090 legacy=0x7f80b806f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 >> v1:172.21.15.76:6789/0 conn(0x7f80b8097b60 legacy=0x7f80b806f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 monclient: found mon.c 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b8132a00 con 0x7f80b8139500 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 5 monclient: authenticate success, global_id 374204 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _renew_subs 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b8132fb0 con 0x7f80b8139500 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _renew_subs 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b8071630 con 0x7f80b8139500 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80a80036c0 con 0x7f80b8139500 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f80a80039d0 con 0x7f80b8139500 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b57fa700 4 set_mon_vals no callback set 2024-09-16T06:59:25.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 10 monclient: dump: 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f80a800e440 con 0x7f80b8139500 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c005100 con 0x7f80b8139500 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f80a8042bb0 con 0x7f80b8139500 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.404+0000 7f80b6ffd700 10 monclient: get_auth_request con 0x7f80980364e0 auth_method 0 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.412+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f80a80132c0 con 0x7f80b8139500 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.412+0000 7f80af7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.412+0000 7f80af7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:25.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.580+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f809c000b90 con 0x7f80980364e0 2024-09-16T06:59:25.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.580+0000 7f80af7fe700 1 -- 172.21.15.76:0/3374514082 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7f809c000b90 con 0x7f80980364e0 2024-09-16T06:59:25.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 >> v1:172.21.15.189:6812/3098810674 conn(0x7f80980364e0 legacy=0x7f80980389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 10 monclient: shutdown 2024-09-16T06:59:25.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 >> v1:172.21.15.76:6790/0 conn(0x7f80b8139500 legacy=0x7f80b81cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 shutdown_connections 2024-09-16T06:59:25.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 >> 172.21.15.76:0/3374514082 conn(0x7f80b80a0030 msgr2=0x7f80b80a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 shutdown_connections 2024-09-16T06:59:25.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.584+0000 7f80b7fff700 1 -- 172.21.15.76:0/3374514082 wait complete. 2024-09-16T06:59:25.586 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T06:59:25.597 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T06:59:25.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:25.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.716+0000 7f954f59e700 10 monclient: get_monmap_and_config 2024-09-16T06:59:25.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.716+0000 7f954f59e700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.716+0000 7f954f59e700 10 monclient: monmap: 2024-09-16T06:59:25.718 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:25.723189+0000 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:25.723189+0000 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:25.719 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 Processor -- start 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- start start 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient: init 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f95500a55f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f95500a6da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f955013b410 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95500a71b0 con 0x7f95500a6da0 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95500a1470 con 0x7f955013b410 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9550099c70 con 0x7f95500a55f0 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient(hunting): authenticate will time out at 2112.857629s 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f95500a55f0 0x7f95500a59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40396/0 (socket says 172.21.15.76:40396) 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954e59c700 1 -- 172.21.15.76:0/2387961681 learned_addr learned my addr 172.21.15.76:0/2387961681 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639800031 0 0) 0x7f9550099c70 con 0x7f95500a55f0 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): my global_id is 374207 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95380035d0 con 0x7f95500a55f0 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80317884 0 0) 0x7f95500a71b0 con 0x7f95500a6da0 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient(hunting): my global_id is 334557 2024-09-16T06:59:25.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9550099c70 con 0x7f95500a6da0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277009650 0 0) 0x7f95380035d0 con 0x7f95500a55f0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 >> v1:172.21.15.189:6789/0 conn(0x7f955013b410 legacy=0x7f955013d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 >> v1:172.21.15.76:6789/0 conn(0x7f95500a6da0 legacy=0x7f95500a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 monclient: found mon.noname-b 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955009a930 con 0x7f95500a55f0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9540001610 con 0x7f95500a55f0 2024-09-16T06:59:25.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: dump: 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 5 monclient: authenticate success, global_id 374207 2024-09-16T06:59:25.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95400018f0 con 0x7f95500a55f0 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 1 -- 172.21.15.76:0/2387961681 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9540001c80 con 0x7f95500a55f0 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954d59a700 10 monclient: dump: 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.728 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 4 set_mon_vals no callback set 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 10 monclient: shutdown 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- 172.21.15.76:0/2387961681 >> v1:172.21.15.76:6790/0 conn(0x7f95500a55f0 legacy=0x7f95500a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- 172.21.15.76:0/2387961681 shutdown_connections 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- 172.21.15.76:0/2387961681 >> 172.21.15.76:0/2387961681 conn(0x7f95500a42f0 msgr2=0x7f95500a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- 172.21.15.76:0/2387961681 shutdown_connections 2024-09-16T06:59:25.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.720+0000 7f954f59e700 1 -- 172.21.15.76:0/2387961681 wait complete. 2024-09-16T06:59:25.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient: monmap: 2024-09-16T06:59:25.735 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 1 Processor -- start 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 1 -- start start 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient: init 2024-09-16T06:59:25.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f95500a55f0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f95500a6da0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f955013b410 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95500a05f0 con 0x7f95500a6da0 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955013e2e0 con 0x7f955013b410 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95500a3d20 con 0x7f95500a55f0 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954f59e700 10 monclient(hunting): authenticate will time out at 2112.862775s 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954e59c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f95500a55f0 0x7f955009c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40412/0 (socket says 172.21.15.76:40412) 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954dd9b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f95500a6da0 0x7f955009cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48084/0 (socket says 172.21.15.76:48084) 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f954e59c700 1 -- 172.21.15.76:0/502914879 learned_addr learned my addr 172.21.15.76:0/502914879 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789845871 0 0) 0x7f95500a05f0 con 0x7f95500a6da0 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): my global_id is 334560 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95300035d0 con 0x7f95500a6da0 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367459002 0 0) 0x7f95500a3d20 con 0x7f95500a55f0 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): my global_id is 374210 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95500a05f0 con 0x7f95500a55f0 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061829375 0 0) 0x7f955013e2e0 con 0x7f955013b410 2024-09-16T06:59:25.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): my global_id is 285037 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95500a3d20 con 0x7f955013b410 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478634235 0 0) 0x7f95300035d0 con 0x7f95500a6da0 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f955013e2e0 con 0x7f95500a6da0 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f9544001830 con 0x7f95500a6da0 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): dump: 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.739 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.724+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283898189 0 0) 0x7f95500a05f0 con 0x7f95500a55f0 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95300035d0 con 0x7f95500a55f0 2024-09-16T06:59:25.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95400015d0 con 0x7f95500a55f0 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient(hunting): dump: 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3542374323 0 0) 0x7f955013e2e0 con 0x7f95500a6da0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 >> v1:172.21.15.76:6790/0 conn(0x7f95500a55f0 legacy=0x7f955009c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 >> v1:172.21.15.189:6789/0 conn(0x7f955013b410 legacy=0x7f95501d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 monclient: found mon.a 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955006d430 con 0x7f95500a6da0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 5 monclient: authenticate success, global_id 334560 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 10 monclient: _renew_subs 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95500a0310 con 0x7f95500a6da0 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 10 monclient: _renew_subs 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f955009e290 con 0x7f95500a6da0 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95440036c0 con 0x7f95500a6da0 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f95440039d0 con 0x7f95500a6da0 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954cd99700 4 set_mon_vals no callback set 2024-09-16T06:59:25.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:25.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: dump: 2024-09-16T06:59:25.744 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.744 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.744 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.744 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9534000f10 con 0x7f95500a6da0 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f9544003b30 con 0x7f95500a6da0 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f9544043db0 con 0x7f95500a6da0 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.189:6804/407912501 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9530043bb0 con 0x7f953003fee0 2024-09-16T06:59:25.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954e59c700 10 monclient: get_auth_request con 0x7f9530036230 auth_method 0 2024-09-16T06:59:25.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=255) v2 ==== 24+0+0 (unknown 2784100248 0 0) 0x7f95440440b0 con 0x7f95500a6da0 2024-09-16T06:59:25.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f953effd700 10 monclient: handle_get_version_reply finishing 1 version 255 2024-09-16T06:59:25.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.728+0000 7f954ed9d700 10 monclient: get_auth_request con 0x7f953003fee0 auth_method 0 2024-09-16T06:59:25.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.732+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== osd.4 v1:172.21.15.189:6804/407912501 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9530043bb0 con 0x7f953003fee0 2024-09-16T06:59:25.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 --> v1:172.21.15.189:6804/407912501 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9534002c00 con 0x7f953003fee0 2024-09-16T06:59:25.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f953effd700 1 -- 172.21.15.76:0/502914879 <== osd.4 v1:172.21.15.189:6804/407912501 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f9534002c00 con 0x7f953003fee0 2024-09-16T06:59:25.749 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:25.750 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:25.750 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:25.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 >> v1:172.21.15.189:6804/407912501 conn(0x7f953003fee0 legacy=0x7f9530042340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 >> v1:172.21.15.189:6812/3098810674 conn(0x7f9530036230 legacy=0x7f95300386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 10 monclient: shutdown 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 >> v1:172.21.15.76:6789/0 conn(0x7f95500a6da0 legacy=0x7f955009cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954ed9d700 1 -- 172.21.15.76:0/502914879 reap_dead start 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 shutdown_connections 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 >> 172.21.15.76:0/502914879 conn(0x7f95500a42f0 msgr2=0x7f955013db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 shutdown_connections 2024-09-16T06:59:25.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.748+0000 7f954f59e700 1 -- 172.21.15.76:0/502914879 wait complete. 2024-09-16T06:59:25.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T06:59:25.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: get_monmap_and_config 2024-09-16T06:59:25.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: monmap: 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:25.890354+0000 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:25.890354+0000 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.887 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:25.888 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:25.888 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 1 Processor -- start 2024-09-16T06:59:25.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 1 -- start start 2024-09-16T06:59:25.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: init 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-c con 0x7f74cc09f230 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-a con 0x7f74cc09fd20 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-b con 0x7f74cc139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc09cd80 con 0x7f74cc09fd20 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc099fd0 con 0x7f74cc09f230 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc09a410 con 0x7f74cc139b70 2024-09-16T06:59:25.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d46f1700 10 monclient(hunting): authenticate will time out at 2113.023589s 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d1c8c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f74cc09fd20 0x7f74cc138450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48100/0 (socket says 172.21.15.76:48100) 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.884+0000 7f74d1c8c700 1 -- 172.21.15.76:0/4136137311 learned_addr learned my addr 172.21.15.76:0/4136137311 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250172599 0 0) 0x7f74cc09cd80 con 0x7f74cc09fd20 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): my global_id is 334563 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b80035d0 con 0x7f74cc09fd20 2024-09-16T06:59:25.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595203318 0 0) 0x7f74cc09a410 con 0x7f74cc139b70 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): my global_id is 374213 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc09cd80 con 0x7f74cc139b70 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545088103 0 0) 0x7f74cc099fd0 con 0x7f74cc09f230 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient(hunting): my global_id is 285040 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc09a410 con 0x7f74cc09f230 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1876424688 0 0) 0x7f74cc09cd80 con 0x7f74cc139b70 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 >> v1:172.21.15.189:6789/0 conn(0x7f74cc09f230 legacy=0x7f74cc09f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 >> v1:172.21.15.76:6789/0 conn(0x7f74cc09fd20 legacy=0x7f74cc138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 monclient: found mon.noname-b 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc09de90 con 0x7f74cc139b70 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 5 monclient: authenticate success, global_id 374213 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74c4001970 con 0x7f74cc139b70 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: dump: 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.895 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74c4001c20 con 0x7f74cc139b70 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 4 set_mon_vals no callback set 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 1 -- 172.21.15.76:0/4136137311 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74c4001ff0 con 0x7f74cc139b70 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d148b700 10 monclient: dump: 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.896 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 10 monclient: shutdown 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 1 -- 172.21.15.76:0/4136137311 >> v1:172.21.15.76:6790/0 conn(0x7f74cc139b70 legacy=0x7f74cc13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 1 -- 172.21.15.76:0/4136137311 shutdown_connections 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 1 -- 172.21.15.76:0/4136137311 >> 172.21.15.76:0/4136137311 conn(0x7f74cc0a3e50 msgr2=0x7f74cc0a0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 1 -- 172.21.15.76:0/4136137311 shutdown_connections 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 1 -- 172.21.15.76:0/4136137311 wait complete. 2024-09-16T06:59:25.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 10 monclient: build_initial_monmap 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.888+0000 7f74d46f1700 10 monclient: monmap: 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:25.898 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 Processor -- start 2024-09-16T06:59:25.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- start start 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: init 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-a con 0x7f74cc09f230 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-b con 0x7f74cc09fd20 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient(hunting): picked mon.noname-c con 0x7f74cc139b70 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc136de0 con 0x7f74cc09f230 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc1377e0 con 0x7f74cc09fd20 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d248d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f74cc09f230 0x7f74cc1c78e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48116/0 (socket says 172.21.15.76:48116) 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d248d700 1 -- 172.21.15.76:0/401522580 learned_addr learned my addr 172.21.15.76:0/401522580 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc139460 con 0x7f74cc139b70 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient(hunting): authenticate will time out at 2113.030380s 2024-09-16T06:59:25.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692576659 0 0) 0x7f74cc136de0 con 0x7f74cc09f230 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): my global_id is 334566 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00035d0 con 0x7f74cc09f230 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609331135 0 0) 0x7f74cc139460 con 0x7f74cc139b70 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): my global_id is 374216 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc136de0 con 0x7f74cc139b70 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961589391 0 0) 0x7f74b00035d0 con 0x7f74cc09f230 2024-09-16T06:59:25.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74cc139460 con 0x7f74cc09f230 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74bc001930 con 0x7f74cc09f230 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): dump: 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.905 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389743267 0 0) 0x7f74cc1377e0 con 0x7f74cc09fd20 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): my global_id is 285043 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00035d0 con 0x7f74cc09fd20 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973912961 0 0) 0x7f74cc136de0 con 0x7f74cc139b70 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74cc1377e0 con 0x7f74cc139b70 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74c4001930 con 0x7f74cc139b70 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:25.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient(hunting): dump: 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 741647929 0 0) 0x7f74cc139460 con 0x7f74cc09f230 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 >> v1:172.21.15.76:6790/0 conn(0x7f74cc139b70 legacy=0x7f74cc1cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 >> v1:172.21.15.189:6789/0 conn(0x7f74cc09fd20 legacy=0x7f74cc1335e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:25.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 monclient: found mon.a 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc1388a0 con 0x7f74cc09f230 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: _finish_auth 0 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 5 monclient: authenticate success, global_id 334566 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _renew_subs 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc139140 con 0x7f74cc09f230 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _renew_subs 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc138e60 con 0x7f74cc09f230 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74bc001cc0 con 0x7f74cc09f230 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f74bc003b10 con 0x7f74cc09f230 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74d0c8a700 4 set_mon_vals no callback set 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.892+0000 7f74c2ffd700 10 monclient: dump: 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:25.909 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f74bc003cf0 con 0x7f74cc09f230 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f74bc043f20 con 0x7f74cc09f230 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74d46f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74d46f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74d46f1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:25.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b4005100 con 0x7f74cc09f230 2024-09-16T06:59:25.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.896+0000 7f74d1c8c700 10 monclient: get_auth_request con 0x7f74b0036490 auth_method 0 2024-09-16T06:59:25.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.900+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f74bc014630 con 0x7f74cc09f230 2024-09-16T06:59:25.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.900+0000 7f74c2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:25.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:25.900+0000 7f74c2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:26.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.064+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f74b4000b90 con 0x7f74b0036490 2024-09-16T06:59:26.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.064+0000 7f74c2ffd700 1 -- 172.21.15.76:0/401522580 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7f74b4000b90 con 0x7f74b0036490 2024-09-16T06:59:26.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 >> v1:172.21.15.189:6812/3098810674 conn(0x7f74b0036490 legacy=0x7f74b0038950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 10 monclient: shutdown 2024-09-16T06:59:26.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 >> v1:172.21.15.76:6789/0 conn(0x7f74cc09f230 legacy=0x7f74cc1c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 shutdown_connections 2024-09-16T06:59:26.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 >> 172.21.15.76:0/401522580 conn(0x7f74cc0a3e50 msgr2=0x7f74cc0a1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 shutdown_connections 2024-09-16T06:59:26.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.068+0000 7f74d46f1700 1 -- 172.21.15.76:0/401522580 wait complete. 2024-09-16T06:59:26.070 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T06:59:26.080 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T06:59:26.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: get_monmap_and_config 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: monmap: 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:26.208892+0000 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:26.208892+0000 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.204 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.205 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.205 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:26.205 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:26.205 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 1 Processor -- start 2024-09-16T06:59:26.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 1 -- start start 2024-09-16T06:59:26.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: init 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-b con 0x7f65c40a5800 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-c con 0x7f65c40a70a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-a con 0x7f65c413b410 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c406c640 con 0x7f65c413b410 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c406ca80 con 0x7f65c40a70a0 2024-09-16T06:59:26.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c4099a60 con 0x7f65c40a5800 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c359e700 10 monclient(hunting): authenticate will time out at 2113.342857s 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c2d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f65c413b410 0x7f65c413d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48118/0 (socket says 172.21.15.76:48118) 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c2d9d700 1 -- 172.21.15.76:0/2266861071 learned_addr learned my addr 172.21.15.76:0/2266861071 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140489608 0 0) 0x7f65c406c640 con 0x7f65c413b410 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): my global_id is 334569 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b00035d0 con 0x7f65c413b410 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099472316 0 0) 0x7f65c4099a60 con 0x7f65c40a5800 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): my global_id is 374219 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c406c640 con 0x7f65c40a5800 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089262482 0 0) 0x7f65c406ca80 con 0x7f65c40a70a0 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient(hunting): my global_id is 285046 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c4099a60 con 0x7f65c40a70a0 2024-09-16T06:59:26.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 247155657 0 0) 0x7f65c406c640 con 0x7f65c40a5800 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 >> v1:172.21.15.189:6789/0 conn(0x7f65c40a70a0 legacy=0x7f65c40a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 >> v1:172.21.15.76:6789/0 conn(0x7f65c413b410 legacy=0x7f65c413d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 monclient: found mon.noname-b 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.204+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4099d40 con 0x7f65c40a5800 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65ac0015f0 con 0x7f65c40a5800 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 5 monclient: authenticate success, global_id 374219 2024-09-16T06:59:26.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: dump: 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:26.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65ac001890 con 0x7f65c40a5800 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 4 set_mon_vals no callback set 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 1 -- 172.21.15.76:0/2266861071 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65ac001c60 con 0x7f65c40a5800 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c159a700 10 monclient: dump: 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.214 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: shutdown 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- 172.21.15.76:0/2266861071 >> v1:172.21.15.76:6790/0 conn(0x7f65c40a5800 legacy=0x7f65c40a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- 172.21.15.76:0/2266861071 shutdown_connections 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- 172.21.15.76:0/2266861071 >> 172.21.15.76:0/2266861071 conn(0x7f65c40a7e20 msgr2=0x7f65c40a8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- 172.21.15.76:0/2266861071 shutdown_connections 2024-09-16T06:59:26.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- 172.21.15.76:0/2266861071 wait complete. 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: monmap: 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:26.219 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 Processor -- start 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- start start 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: init 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-a con 0x7f65c40a5800 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-b con 0x7f65c40a70a0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 10 monclient(hunting): picked mon.noname-c con 0x7f65c413b410 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.208+0000 7f65c359e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c406d5f0 con 0x7f65c40a5800 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f65c40a5800 0x7f65c4138bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48134/0 (socket says 172.21.15.76:48134) 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c40a5040 con 0x7f65c40a70a0 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c259c700 1 -- 172.21.15.76:0/3034657360 learned_addr learned my addr 172.21.15.76:0/3034657360 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c409a950 con 0x7f65c413b410 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient(hunting): authenticate will time out at 2113.348072s 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120596486 0 0) 0x7f65c409a950 con 0x7f65c413b410 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): my global_id is 374222 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a40035d0 con 0x7f65c413b410 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994317752 0 0) 0x7f65c406d5f0 con 0x7f65c40a5800 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): my global_id is 334572 2024-09-16T06:59:26.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c409a950 con 0x7f65c40a5800 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412665031 0 0) 0x7f65c40a5040 con 0x7f65c40a70a0 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): my global_id is 285049 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c406d5f0 con 0x7f65c40a70a0 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910538052 0 0) 0x7f65c409a950 con 0x7f65c40a5800 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65c40a5040 con 0x7f65c40a5800 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191267732 0 0) 0x7f65a40035d0 con 0x7f65c413b410 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65c409a950 con 0x7f65c413b410 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65ac001ba0 con 0x7f65c40a5800 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): dump: 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65bc001490 con 0x7f65c413b410 2024-09-16T06:59:26.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient(hunting): dump: 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3666169994 0 0) 0x7f65c40a5040 con 0x7f65c40a5800 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 >> v1:172.21.15.76:6790/0 conn(0x7f65c413b410 legacy=0x7f65c4139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 >> v1:172.21.15.189:6789/0 conn(0x7f65c40a70a0 legacy=0x7f65c4139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 monclient: found mon.a 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c409af40 con 0x7f65c40a5800 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 5 monclient: authenticate success, global_id 334572 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient: _renew_subs 2024-09-16T06:59:26.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c409beb0 con 0x7f65c40a5800 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient: _renew_subs 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c41d3110 con 0x7f65c40a5800 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65ac0011b0 con 0x7f65c40a5800 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f65ac003c30 con 0x7f65c40a5800 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:26.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: dump: 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c0d99700 4 set_mon_vals no callback set 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.76:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f65a8000f10 con 0x7f65c40a5800 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f65ac003e10 con 0x7f65c40a5800 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f65ac043fb0 con 0x7f65c40a5800 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.189:6800/2128366135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f65a4043dd0 con 0x7f65a40401a0 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_get_version_reply(handle=1 version=255) v2 ==== 24+0+0 (unknown 2784100248 0 0) 0x7f65ac0443e0 con 0x7f65c40a5800 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65baffd700 10 monclient: handle_get_version_reply finishing 1 version 255 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c1d9b700 10 monclient: get_auth_request con 0x7f65a40364f0 auth_method 0 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.212+0000 7f65c2d9d700 10 monclient: get_auth_request con 0x7f65a40401a0 auth_method 0 2024-09-16T06:59:26.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.216+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== osd.5 v1:172.21.15.189:6800/2128366135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f65a4043dd0 con 0x7f65a40401a0 2024-09-16T06:59:26.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 --> v1:172.21.15.189:6800/2128366135 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f65a8002c00 con 0x7f65a40401a0 2024-09-16T06:59:26.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65baffd700 1 -- 172.21.15.76:0/3034657360 <== osd.5 v1:172.21.15.189:6800/2128366135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f65a8002c00 con 0x7f65a40401a0 2024-09-16T06:59:26.231 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-09-16T06:59:26.232 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T06:59:26.232 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 >> v1:172.21.15.189:6800/2128366135 conn(0x7f65a40401a0 legacy=0x7f65a4042600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 >> v1:172.21.15.189:6812/3098810674 conn(0x7f65a40364f0 legacy=0x7f65a40389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 10 monclient: shutdown 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 >> v1:172.21.15.76:6789/0 conn(0x7f65c40a5800 legacy=0x7f65c4138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c2d9d700 1 -- 172.21.15.76:0/3034657360 reap_dead start 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 shutdown_connections 2024-09-16T06:59:26.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 >> 172.21.15.76:0/3034657360 conn(0x7f65c40a7e20 msgr2=0x7f65c40a3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 shutdown_connections 2024-09-16T06:59:26.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.232+0000 7f65c359e700 1 -- 172.21.15.76:0/3034657360 wait complete. 2024-09-16T06:59:26.244 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.360+0000 7fdd45e1c700 10 monclient: get_monmap_and_config 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.360+0000 7fdd45e1c700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.360+0000 7fdd45e1c700 10 monclient: monmap: 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:26.367770+0000 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:26.367770+0000 2024-09-16T06:59:26.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.364 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.364 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.364 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:26.364 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:26.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 Processor -- start 2024-09-16T06:59:26.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- start start 2024-09-16T06:59:26.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient: init 2024-09-16T06:59:26.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-b con 0x7fdd4009ed60 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-c con 0x7fdd400709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-a con 0x7fdd40070de0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4009f140 con 0x7fdd40070de0 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd40072530 con 0x7fdd400709d0 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd40097f00 con 0x7fdd4009ed60 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient(hunting): authenticate will time out at 2113.501406s 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd44e1a700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdd4009ed60 0x7fdd400702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40452/0 (socket says 172.21.15.76:40452) 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd44e1a700 1 -- 172.21.15.76:0/3760643317 learned_addr learned my addr 172.21.15.76:0/3760643317 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580662093 0 0) 0x7fdd40097f00 con 0x7fdd4009ed60 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): my global_id is 374225 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd280035d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385561791 0 0) 0x7fdd4009f140 con 0x7fdd40070de0 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): my global_id is 334575 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40097f00 con 0x7fdd40070de0 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250847475 0 0) 0x7fdd40072530 con 0x7fdd400709d0 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient(hunting): my global_id is 285052 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4009f140 con 0x7fdd400709d0 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219204483 0 0) 0x7fdd280035d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 >> v1:172.21.15.189:6789/0 conn(0x7fdd400709d0 legacy=0x7fdd4013a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 >> v1:172.21.15.76:6789/0 conn(0x7fdd40070de0 legacy=0x7fdd4013de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 monclient: found mon.noname-b 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4009a320 con 0x7fdd4009ed60 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd38001990 con 0x7fdd4009ed60 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 5 monclient: authenticate success, global_id 374225 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: dump: 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.370 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd38001c70 con 0x7fdd4009ed60 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 4 set_mon_vals no callback set 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 1 -- 172.21.15.76:0/3760643317 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd38002000 con 0x7fdd4009ed60 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd3f7fe700 10 monclient: dump: 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 10 monclient: shutdown 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- 172.21.15.76:0/3760643317 >> v1:172.21.15.76:6790/0 conn(0x7fdd4009ed60 legacy=0x7fdd400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- 172.21.15.76:0/3760643317 shutdown_connections 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- 172.21.15.76:0/3760643317 >> 172.21.15.76:0/3760643317 conn(0x7fdd400a0030 msgr2=0x7fdd400a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- 172.21.15.76:0/3760643317 shutdown_connections 2024-09-16T06:59:26.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.364+0000 7fdd45e1c700 1 -- 172.21.15.76:0/3760643317 wait complete. 2024-09-16T06:59:26.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient: monmap: 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 1 Processor -- start 2024-09-16T06:59:26.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 1 -- start start 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient: init 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-b con 0x7fdd400709d0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-c con 0x7fdd40070de0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient(hunting): picked mon.noname-a con 0x7fdd4009ed60 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd400a2730 con 0x7fdd4009ed60 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4009a540 con 0x7fdd400709d0 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd4561b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdd4009ed60 0x7fdd401d1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:48158/0 (socket says 172.21.15.76:48158) 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3ffff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7fdd40070de0 0x7fdd40096620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40464/0 (socket says 172.21.15.76:40464) 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd4561b700 1 -- 172.21.15.76:0/2533949513 learned_addr learned my addr 172.21.15.76:0/2533949513 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd400910b0 con 0x7fdd40070de0 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd45e1c700 10 monclient(hunting): authenticate will time out at 2113.506820s 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469399263 0 0) 0x7fdd400a2730 con 0x7fdd4009ed60 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): my global_id is 334578 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd2c0035d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255082162 0 0) 0x7fdd400910b0 con 0x7fdd40070de0 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): my global_id is 374228 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd400a2730 con 0x7fdd40070de0 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756785668 0 0) 0x7fdd4009a540 con 0x7fdd400709d0 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): my global_id is 285055 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd400910b0 con 0x7fdd400709d0 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226753637 0 0) 0x7fdd2c0035d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd4009a540 con 0x7fdd4009ed60 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd300018b0 con 0x7fdd4009ed60 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.380 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328783857 0 0) 0x7fdd400a2730 con 0x7fdd40070de0 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd2c0035d0 con 0x7fdd40070de0 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd34001830 con 0x7fdd40070de0 2024-09-16T06:59:26.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient(hunting): dump: 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 143425590 0 0) 0x7fdd4009a540 con 0x7fdd4009ed60 2024-09-16T06:59:26.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 >> v1:172.21.15.76:6790/0 conn(0x7fdd40070de0 legacy=0x7fdd40096620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 >> v1:172.21.15.189:6789/0 conn(0x7fdd400709d0 legacy=0x7fdd401c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 monclient: found mon.a 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40090810 con 0x7fdd4009ed60 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.368+0000 7fdd3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 5 monclient: authenticate success, global_id 334578 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _renew_subs 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd40090af0 con 0x7fdd4009ed60 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd300036c0 con 0x7fdd4009ed60 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _renew_subs 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7fdd300039d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd40096d30 con 0x7fdd4009ed60 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3effd700 4 set_mon_vals no callback set 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 10 monclient: dump: 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.384 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd400008d0 con 0x7fdd4009ed60 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7fdd30003b30 con 0x7fdd4009ed60 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7fdd30043ce0 con 0x7fdd4009ed60 2024-09-16T06:59:26.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.372+0000 7fdd44e1a700 10 monclient: get_auth_request con 0x7fdd2c0362e0 auth_method 0 2024-09-16T06:59:26.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.376+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7fdd300143f0 con 0x7fdd4009ed60 2024-09-16T06:59:26.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.376+0000 7fdd3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.376+0000 7fdd3d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:26.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.544+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd400a2260 con 0x7fdd2c0362e0 2024-09-16T06:59:26.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.544+0000 7fdd3d7fa700 1 -- 172.21.15.76:0/2533949513 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7fdd400a2260 con 0x7fdd2c0362e0 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 >> v1:172.21.15.189:6812/3098810674 conn(0x7fdd2c0362e0 legacy=0x7fdd2c0387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 10 monclient: shutdown 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 >> v1:172.21.15.76:6789/0 conn(0x7fdd4009ed60 legacy=0x7fdd401d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 shutdown_connections 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 >> 172.21.15.76:0/2533949513 conn(0x7fdd400a0030 msgr2=0x7fdd401bd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 shutdown_connections 2024-09-16T06:59:26.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.548+0000 7fdd45e1c700 1 -- 172.21.15.76:0/2533949513 wait complete. 2024-09-16T06:59:26.550 INFO:teuthology.orchestra.run.smithi076.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T06:59:26.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:59:26.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.668+0000 7f13b0d58700 10 monclient: get_monmap_and_config 2024-09-16T06:59:26.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.668+0000 7f13b0d58700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.668+0000 7f13b0d58700 10 monclient: monmap: 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:26.675472+0000 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:26.675472+0000 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:26.671 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 Processor -- start 2024-09-16T06:59:26.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- start start 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient: init 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-c con 0x7f13ac1342c0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-b con 0x7f13ac06cfb0 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-a con 0x7f13ac06d410 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac0a31f0 con 0x7f13ac06d410 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac06c4f0 con 0x7f13ac1342c0 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac06bbe0 con 0x7f13ac06cfb0 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient(hunting): authenticate will time out at 2113.809345s 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aaffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13ac06cfb0 0x7f13ac13a280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40470/0 (socket says 172.21.15.76:40470) 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aaffd700 1 -- 172.21.15.76:0/1935104403 learned_addr learned my addr 172.21.15.76:0/1935104403 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512600472 0 0) 0x7f13ac06bbe0 con 0x7f13ac06cfb0 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): my global_id is 374231 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13940035d0 con 0x7f13ac06cfb0 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618641434 0 0) 0x7f13ac0a31f0 con 0x7f13ac06d410 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): my global_id is 334581 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac06bbe0 con 0x7f13ac06d410 2024-09-16T06:59:26.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496841571 0 0) 0x7f13ac06c4f0 con 0x7f13ac1342c0 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient(hunting): my global_id is 285058 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac0a31f0 con 0x7f13ac1342c0 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501630478 0 0) 0x7f13ac06bbe0 con 0x7f13ac06d410 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 >> v1:172.21.15.76:6790/0 conn(0x7f13ac06cfb0 legacy=0x7f13ac13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 >> v1:172.21.15.189:6789/0 conn(0x7f13ac1342c0 legacy=0x7f13ac1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 monclient: found mon.noname-a 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac09ace0 con 0x7f13ac06d410 2024-09-16T06:59:26.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 5 monclient: authenticate success, global_id 334581 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a4001970 con 0x7f13ac06d410 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: dump: 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.678 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.0 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a4001b90 con 0x7f13ac06d410 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 1 -- 172.21.15.76:0/1935104403 <== mon.0 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a4001f60 con 0x7f13ac06d410 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 4 set_mon_vals no callback set 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13aa7fc700 10 monclient: dump: 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.679 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 10 monclient: shutdown 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- 172.21.15.76:0/1935104403 >> v1:172.21.15.76:6789/0 conn(0x7f13ac06d410 legacy=0x7f13ac13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- 172.21.15.76:0/1935104403 shutdown_connections 2024-09-16T06:59:26.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- 172.21.15.76:0/1935104403 >> 172.21.15.76:0/1935104403 conn(0x7f13ac0a41c0 msgr2=0x7f13ac0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- 172.21.15.76:0/1935104403 shutdown_connections 2024-09-16T06:59:26.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.672+0000 7f13b0d58700 1 -- 172.21.15.76:0/1935104403 wait complete. 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: build_initial_monmap 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: monmap: 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:26.685 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 Processor -- start 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- start start 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: init 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-b con 0x7f13ac06cfb0 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-c con 0x7f13ac06d410 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient(hunting): picked mon.noname-a con 0x7f13ac1342c0 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac1ca7e0 con 0x7f13ac1342c0 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac09e490 con 0x7f13ac06cfb0 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ac12e7b0 con 0x7f13ac06d410 2024-09-16T06:59:26.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient(hunting): authenticate will time out at 2113.813763s 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13aaffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f13ac06d410 0x7f13ac1cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:40476/0 (socket says 172.21.15.76:40476) 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13aaffd700 1 -- 172.21.15.76:0/3432073747 learned_addr learned my addr 172.21.15.76:0/3432073747 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112405900 0 0) 0x7f13ac12e7b0 con 0x7f13ac06d410 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): my global_id is 374234 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13980035d0 con 0x7f13ac06d410 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591889400 0 0) 0x7f13ac1ca7e0 con 0x7f13ac1342c0 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): my global_id is 334584 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac12e7b0 con 0x7f13ac1342c0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299568975 0 0) 0x7f13ac09e490 con 0x7f13ac06cfb0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): my global_id is 285061 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac1ca7e0 con 0x7f13ac06cfb0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262763354 0 0) 0x7f13ac12e7b0 con 0x7f13ac1342c0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13ac09e490 con 0x7f13ac1342c0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a4001ea0 con 0x7f13ac1342c0 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T06:59:26.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): dump: 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580345712 0 0) 0x7f13980035d0 con 0x7f13ac06d410 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13ac12e7b0 con 0x7f13ac06d410 2024-09-16T06:59:26.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a00018d0 con 0x7f13ac06d410 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient(hunting): dump: 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 11230658 0 0) 0x7f13ac09e490 con 0x7f13ac1342c0 2024-09-16T06:59:26.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 >> v1:172.21.15.76:6790/0 conn(0x7f13ac06d410 legacy=0x7f13ac1cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 >> v1:172.21.15.189:6789/0 conn(0x7f13ac06cfb0 legacy=0x7f13ac132850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: _finish_hunting 0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 monclient: found mon.a 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac09e8b0 con 0x7f13ac1342c0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: _finish_auth 0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: _check_auth_tickets 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 5 monclient: authenticate success, global_id 334584 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _renew_subs 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac09df80 con 0x7f13ac1342c0 2024-09-16T06:59:26.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _renew_subs 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac132f60 con 0x7f13ac1342c0 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a4003cd0 con 0x7f13ac1342c0 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f13a4004060 con 0x7f13ac1342c0 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f13a9ffb700 4 set_mon_vals no callback set 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.676+0000 7f1393fff700 10 monclient: dump: 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.692 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f13b0d58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f13b0d58700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f13b0d58700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c005100 con 0x7f13ac1342c0 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f13a4004280 con 0x7f13ac1342c0 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f13a4043130 con 0x7f13ac1342c0 2024-09-16T06:59:26.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.680+0000 7f13ab7fe700 10 monclient: get_auth_request con 0x7f1398036390 auth_method 0 2024-09-16T06:59:26.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.684+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f13a4013b30 con 0x7f13ac1342c0 2024-09-16T06:59:26.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.684+0000 7f1393fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:26.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.684+0000 7f1393fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:26.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.856+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f138c000b90 con 0x7f1398036390 2024-09-16T06:59:26.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.860+0000 7f1393fff700 1 -- 172.21.15.76:0/3432073747 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98558 (unknown 2965378022 0 1207652097) 0x7f138c000b90 con 0x7f1398036390 2024-09-16T06:59:26.864 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 >> v1:172.21.15.189:6812/3098810674 conn(0x7f1398036390 legacy=0x7f1398038850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 10 monclient: shutdown 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 >> v1:172.21.15.76:6789/0 conn(0x7f13ac1342c0 legacy=0x7f13ac1d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 shutdown_connections 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 >> 172.21.15.76:0/3432073747 conn(0x7f13ac0a41c0 msgr2=0x7f13ac134e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 shutdown_connections 2024-09-16T06:59:26.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:26.864+0000 7f13b0d58700 1 -- 172.21.15.76:0/3432073747 wait complete. 2024-09-16T06:59:26.867 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:59:26.884 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-16T06:59:26.041323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191536,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039504,"statfs":{"total":579820584960,"available":579624452096,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003815"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:21.071754+0000","last_change":"2024-09-16T06:44:52.856113+0000","last_active":"2024-09-16T06:55:21.071754+0000","last_peered":"2024-09-16T06:55:21.071754+0000","last_clean":"2024-09-16T06:55:21.071754+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:55:21.071754+0000","last_undegraded":"2024-09-16T06:55:21.071754+0000","last_fullsized":"2024-09-16T06:55:21.071754+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:26.598082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191190+0000","last_change":"2024-09-16T06:44:52.853100+0000","last_active":"2024-09-16T06:55:45.191190+0000","last_peered":"2024-09-16T06:55:45.191190+0000","last_clean":"2024-09-16T06:55:45.191190+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:55:45.191190+0000","last_undegraded":"2024-09-16T06:55:45.191190+0000","last_fullsized":"2024-09-16T06:55:45.191190+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:16.588106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":96,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.072562+0000","last_change":"2024-09-16T06:44:52.855874+0000","last_active":"2024-09-16T06:55:26.072562+0000","last_peered":"2024-09-16T06:55:26.072562+0000","last_clean":"2024-09-16T06:55:26.072562+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:55:26.072562+0000","last_undegraded":"2024-09-16T06:55:26.072562+0000","last_fullsized":"2024-09-16T06:55:26.072562+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:55.805033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:45.191145+0000","last_change":"2024-09-16T06:44:52.854782+0000","last_active":"2024-09-16T06:55:45.191145+0000","last_peered":"2024-09-16T06:55:45.191145+0000","last_clean":"2024-09-16T06:55:45.191145+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:55:45.191145+0000","last_undegraded":"2024-09-16T06:55:45.191145+0000","last_fullsized":"2024-09-16T06:55:45.191145+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:49.724068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":94,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:55.212610+0000","last_change":"2024-09-16T06:44:52.859034+0000","last_active":"2024-09-16T06:57:55.212610+0000","last_peered":"2024-09-16T06:57:55.212610+0000","last_clean":"2024-09-16T06:57:55.212610+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:57:55.212610+0000","last_undegraded":"2024-09-16T06:57:55.212610+0000","last_fullsized":"2024-09-16T06:57:55.212610+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056468+0000","last_change":"2024-09-16T06:44:52.859319+0000","last_active":"2024-09-16T06:55:25.056468+0000","last_peered":"2024-09-16T06:55:25.056468+0000","last_clean":"2024-09-16T06:55:25.056468+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:55:25.056468+0000","last_undegraded":"2024-09-16T06:55:25.056468+0000","last_fullsized":"2024-09-16T06:55:25.056468+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:36.527527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":163,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:01.211371+0000","last_change":"2024-09-16T06:44:52.857054+0000","last_active":"2024-09-16T06:56:01.211371+0000","last_peered":"2024-09-16T06:56:01.211371+0000","last_clean":"2024-09-16T06:56:01.211371+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:56:01.211371+0000","last_undegraded":"2024-09-16T06:56:01.211371+0000","last_fullsized":"2024-09-16T06:56:01.211371+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:13.406841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796081+0000","last_change":"2024-09-16T06:44:52.854292+0000","last_active":"2024-09-16T06:55:18.796081+0000","last_peered":"2024-09-16T06:55:18.796081+0000","last_clean":"2024-09-16T06:55:18.796081+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:55:18.796081+0000","last_undegraded":"2024-09-16T06:55:18.796081+0000","last_fullsized":"2024-09-16T06:55:18.796081+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:55.962678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201207+0000","last_change":"2024-09-16T06:44:52.857665+0000","last_active":"2024-09-16T06:56:55.201207+0000","last_peered":"2024-09-16T06:56:55.201207+0000","last_clean":"2024-09-16T06:56:55.201207+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:56:55.201207+0000","last_undegraded":"2024-09-16T06:56:55.201207+0000","last_fullsized":"2024-09-16T06:56:55.201207+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:58:55.693264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":165,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207121+0000","last_change":"2024-09-16T06:44:52.857164+0000","last_active":"2024-09-16T06:55:31.207121+0000","last_peered":"2024-09-16T06:55:31.207121+0000","last_clean":"2024-09-16T06:55:31.207121+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:55:31.207121+0000","last_undegraded":"2024-09-16T06:55:31.207121+0000","last_fullsized":"2024-09-16T06:55:31.207121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:24.291104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":170,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:31.207067+0000","last_change":"2024-09-16T06:44:52.854453+0000","last_active":"2024-09-16T06:55:31.207067+0000","last_peered":"2024-09-16T06:55:31.207067+0000","last_clean":"2024-09-16T06:55:31.207067+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:55:31.207067+0000","last_undegraded":"2024-09-16T06:55:31.207067+0000","last_fullsized":"2024-09-16T06:55:31.207067+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:49.308777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796044+0000","last_change":"2024-09-16T06:44:52.856374+0000","last_active":"2024-09-16T06:55:18.796044+0000","last_peered":"2024-09-16T06:55:18.796044+0000","last_clean":"2024-09-16T06:55:18.796044+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:55:18.796044+0000","last_undegraded":"2024-09-16T06:55:18.796044+0000","last_fullsized":"2024-09-16T06:55:18.796044+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:52.978402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:20.055715+0000","last_change":"2024-09-16T06:44:53.857627+0000","last_active":"2024-09-16T06:55:20.055715+0000","last_peered":"2024-09-16T06:55:20.055715+0000","last_clean":"2024-09-16T06:55:20.055715+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:55:20.055715+0000","last_undegraded":"2024-09-16T06:55:20.055715+0000","last_fullsized":"2024-09-16T06:55:20.055715+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:18:22.103962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.795979+0000","last_change":"2024-09-16T06:44:53.857508+0000","last_active":"2024-09-16T06:55:18.795979+0000","last_peered":"2024-09-16T06:55:18.795979+0000","last_clean":"2024-09-16T06:55:18.795979+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:55:18.795979+0000","last_undegraded":"2024-09-16T06:55:18.795979+0000","last_fullsized":"2024-09-16T06:55:18.795979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:09.945570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":116,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:16.079853+0000","last_change":"2024-09-16T06:44:53.857597+0000","last_active":"2024-09-16T06:56:16.079853+0000","last_peered":"2024-09-16T06:56:16.079853+0000","last_clean":"2024-09-16T06:56:16.079853+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:56:16.079853+0000","last_undegraded":"2024-09-16T06:56:16.079853+0000","last_fullsized":"2024-09-16T06:56:16.079853+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:59.218223+0000","last_clean_scrub_stamp":"2024-09-16T06:42:59.218223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:53.321544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":94,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:00.070430+0000","last_change":"2024-09-16T06:44:52.857229+0000","last_active":"2024-09-16T06:57:00.070430+0000","last_peered":"2024-09-16T06:57:00.070430+0000","last_clean":"2024-09-16T06:57:00.070430+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:57:00.070430+0000","last_undegraded":"2024-09-16T06:57:00.070430+0000","last_fullsized":"2024-09-16T06:57:00.070430+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:03.913790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":431,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:58:55.221587+0000","last_change":"2024-09-16T06:44:55.870279+0000","last_active":"2024-09-16T06:58:55.221587+0000","last_peered":"2024-09-16T06:58:55.221587+0000","last_clean":"2024-09-16T06:58:55.221587+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:58:55.221587+0000","last_undegraded":"2024-09-16T06:58:55.221587+0000","last_fullsized":"2024-09-16T06:58:55.221587+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":246,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:30.208916+0000","last_change":"2024-09-16T06:44:52.855544+0000","last_active":"2024-09-16T06:57:30.208916+0000","last_peered":"2024-09-16T06:57:30.208916+0000","last_clean":"2024-09-16T06:57:30.208916+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:57:30.208916+0000","last_undegraded":"2024-09-16T06:57:30.208916+0000","last_fullsized":"2024-09-16T06:57:30.208916+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:56:55.201179+0000","last_change":"2024-09-16T06:44:52.858722+0000","last_active":"2024-09-16T06:56:55.201179+0000","last_peered":"2024-09-16T06:56:55.201179+0000","last_clean":"2024-09-16T06:56:55.201179+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:56:55.201179+0000","last_undegraded":"2024-09-16T06:56:55.201179+0000","last_fullsized":"2024-09-16T06:56:55.201179+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:45.458295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:18.796021+0000","last_change":"2024-09-16T06:44:52.856469+0000","last_active":"2024-09-16T06:55:18.796021+0000","last_peered":"2024-09-16T06:55:18.796021+0000","last_clean":"2024-09-16T06:55:18.796021+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:55:18.796021+0000","last_undegraded":"2024-09-16T06:55:18.796021+0000","last_fullsized":"2024-09-16T06:55:18.796021+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:31.216149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":280,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:26.206353+0000","last_change":"2024-09-16T06:44:52.866089+0000","last_active":"2024-09-16T06:55:26.206353+0000","last_peered":"2024-09-16T06:55:26.206353+0000","last_clean":"2024-09-16T06:55:26.206353+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:55:26.206353+0000","last_undegraded":"2024-09-16T06:55:26.206353+0000","last_fullsized":"2024-09-16T06:55:26.206353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:55.504470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:25.056440+0000","last_change":"2024-09-16T06:44:52.856612+0000","last_active":"2024-09-16T06:55:25.056440+0000","last_peered":"2024-09-16T06:55:25.056440+0000","last_clean":"2024-09-16T06:55:25.056440+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:55:25.056440+0000","last_undegraded":"2024-09-16T06:55:25.056440+0000","last_fullsized":"2024-09-16T06:55:25.056440+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:20:02.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:25.208203+0000","last_change":"2024-09-16T06:44:52.844681+0000","last_active":"2024-09-16T06:57:25.208203+0000","last_peered":"2024-09-16T06:57:25.208203+0000","last_clean":"2024-09-16T06:57:25.208203+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:57:25.208203+0000","last_undegraded":"2024-09-16T06:57:25.208203+0000","last_fullsized":"2024-09-16T06:57:25.208203+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:55:46.075513+0000","last_change":"2024-09-16T06:44:52.848857+0000","last_active":"2024-09-16T06:55:46.075513+0000","last_peered":"2024-09-16T06:55:46.075513+0000","last_clean":"2024-09-16T06:55:46.075513+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:55:46.075513+0000","last_undegraded":"2024-09-16T06:55:46.075513+0000","last_fullsized":"2024-09-16T06:55:46.075513+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:46.816057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":234,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:58:05.214203+0000","last_change":"2024-09-16T06:44:53.857350+0000","last_active":"2024-09-16T06:58:05.214203+0000","last_peered":"2024-09-16T06:58:05.214203+0000","last_clean":"2024-09-16T06:58:05.214203+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:58:05.214203+0000","last_undegraded":"2024-09-16T06:58:05.214203+0000","last_fullsized":"2024-09-16T06:58:05.214203+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575101,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.71699999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.79300000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":0.872},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575103,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":0.71299999999999997}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":0.747}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":1.036},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":1.099}]}]},{"osd":5,"up_from":12,"seq":51539607808,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":1.012}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.872},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.79900000000000004}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.96299999999999997}]}]},{"osd":2,"up_from":12,"seq":51539607807,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":1.044},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.997},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.60699999999999998},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.83799999999999997,"15min":1.0549999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.60999999999999999,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.86399999999999999,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.71399999999999997,"5min":0.61899999999999999,"15min":0.57499999999999996},"max":{"1min":0.96499999999999997,"5min":1.073,"15min":1.3500000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.84699999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97099999999999997,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.82299999999999995}]},{"osd":5,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79900000000000004,"15min":0.83699999999999997},"min":{"1min":0.65700000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.5780000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84099999999999997,"15min":0.85399999999999998},"min":{"1min":0.71699999999999997,"5min":0.60699999999999998,"15min":0.59599999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.77000000000000002}]}]},{"osd":1,"up_from":12,"seq":51539607809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.878},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.98999999999999999}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.79200000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640510,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46800000000000003,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.60299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0109999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.58699999999999997,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":1.0149999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.872,"15min":0.88500000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":1.0529999999999999,"5min":1.153,"15min":1.4910000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.74099999999999999,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.629},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":1.008,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.018},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.121},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.88,"15min":0.86699999999999999},"min":{"1min":0.65900000000000003,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1140000000000001,"5min":1.125,"15min":1.1899999999999999},"last":0.93799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:59:26.889 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.889 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.889 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.890 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-16T06:42:59.218223+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.891 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:26.892 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T06:59:46.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: get_monmap_and_config 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: build_initial_monmap 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: monmap: 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:59:47.017542+0000 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:59:47.017542+0000 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.013 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:47.014 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T06:59:47.014 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T06:59:47.014 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 1 Processor -- start 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 1 -- start start 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: init 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-c con 0x7ff538132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-a con 0x7ff53806d480 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-b con 0x7ff53806cf50 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53806c280 con 0x7ff53806d480 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53806bfa0 con 0x7ff538132330 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53809ace0 con 0x7ff53806cf50 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff53cfaf700 10 monclient(hunting): authenticate will time out at 2134.151422s 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff537fff700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff53806cf50 0x7ff53813da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47300/0 (socket says 172.21.15.76:47300) 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff537fff700 1 -- 172.21.15.76:0/2335114026 learned_addr learned my addr 172.21.15.76:0/2335114026 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff536ffd700 1 --1- 172.21.15.76:0/2335114026 >> v1:172.21.15.76:6789/0 conn(0x7ff53806d480 0x7ff53813a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57388/0 (socket says 172.21.15.76:57388) 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.012+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880455654 0 0) 0x7ff53809ace0 con 0x7ff53806cf50 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): my global_id is 374237 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5200035d0 con 0x7ff53806cf50 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859181597 0 0) 0x7ff53806c280 con 0x7ff53806d480 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): my global_id is 334587 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53809ace0 con 0x7ff53806d480 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328762549 0 0) 0x7ff53806bfa0 con 0x7ff538132330 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient(hunting): my global_id is 285064 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53806c280 con 0x7ff538132330 2024-09-16T06:59:47.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634440240 0 0) 0x7ff5200035d0 con 0x7ff53806cf50 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 >> v1:172.21.15.189:6789/0 conn(0x7ff538132330 legacy=0x7ff538132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 >> v1:172.21.15.76:6789/0 conn(0x7ff53806d480 legacy=0x7ff53813a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: _finish_hunting 0 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 monclient: found mon.noname-b 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53809ae70 con 0x7ff53806cf50 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: _finish_auth 0 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: _check_auth_tickets 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:47.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff530001990 con 0x7ff53806cf50 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 5 monclient: authenticate success, global_id 374237 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: dump: 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.021 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff530001bb0 con 0x7ff53806cf50 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 10 monclient: get_monmap_and_config success 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 1 -- 172.21.15.76:0/2335114026 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff530001de0 con 0x7ff53806cf50 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 4 set_mon_vals no callback set 2024-09-16T06:59:47.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff5367fc700 10 monclient: dump: 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 10 monclient: shutdown 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:47.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 -- 172.21.15.76:0/2335114026 >> v1:172.21.15.76:6790/0 conn(0x7ff53806cf50 legacy=0x7ff53813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 -- 172.21.15.76:0/2335114026 shutdown_connections 2024-09-16T06:59:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 -- 172.21.15.76:0/2335114026 >> 172.21.15.76:0/2335114026 conn(0x7ff5380a41c0 msgr2=0x7ff5380a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 -- 172.21.15.76:0/2335114026 shutdown_connections 2024-09-16T06:59:47.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 -- 172.21.15.76:0/2335114026 wait complete. 2024-09-16T06:59:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 10 monclient: build_initial_monmap 2024-09-16T06:59:47.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 10 monclient: monmap: 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.016+0000 7ff53cfaf700 1 Processor -- start 2024-09-16T06:59:47.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- start start 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: init 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _reopen_session rank -1 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-a con 0x7ff53806cf50 addr v1:172.21.15.76:6789/0 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-c con 0x7ff53806d480 addr v1:172.21.15.76:6790/0 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient(hunting): picked mon.noname-b con 0x7ff538132330 addr v1:172.21.15.189:6789/0 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff538130d40 con 0x7ff53806cf50 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff538131100 con 0x7ff538132330 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5381ca6e0 con 0x7ff53806d480 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient(hunting): _renew_subs 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient(hunting): authenticate will time out at 2134.157146s 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff5377fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff53806cf50 0x7ff5381c98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:57396/0 (socket says 172.21.15.76:57396) 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff536ffd700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ff53806d480 0x7ff5381c9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:47310/0 (socket says 172.21.15.76:47310) 2024-09-16T06:59:47.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff5377fe700 1 -- 172.21.15.76:0/4102998008 learned_addr learned my addr 172.21.15.76:0/4102998008 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306130601 0 0) 0x7ff5381ca6e0 con 0x7ff53806d480 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): my global_id is 374240 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5240035d0 con 0x7ff53806d480 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150573444 0 0) 0x7ff538130d40 con 0x7ff53806cf50 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): my global_id is 334590 2024-09-16T06:59:47.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5381ca6e0 con 0x7ff53806cf50 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730312416 0 0) 0x7ff538131100 con 0x7ff538132330 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _negotiate 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): my global_id is 285067 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538130d40 con 0x7ff538132330 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 849915425 0 0) 0x7ff5240035d0 con 0x7ff53806d480 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff538131100 con 0x7ff53806d480 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff528001850 con 0x7ff53806d480 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): dump: 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T06:59:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3412372771 0 0) 0x7ff5381ca6e0 con 0x7ff53806cf50 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5240035d0 con 0x7ff53806cf50 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff52c0019c0 con 0x7ff53806cf50 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): dump: 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.034 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164006663 0 0) 0x7ff538130d40 con 0x7ff538132330 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5381ca6e0 con 0x7ff538132330 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff530001d20 con 0x7ff538132330 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient(hunting): dump: 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 671350155 0 0) 0x7ff538131100 con 0x7ff53806d480 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 >> v1:172.21.15.189:6789/0 conn(0x7ff538132330 legacy=0x7ff5381d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 >> v1:172.21.15.76:6789/0 conn(0x7ff53806cf50 legacy=0x7ff5381c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: _finish_hunting 0 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 monclient: found mon.c 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T06:59:47.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538132f70 con 0x7ff53806d480 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: _finish_auth 0 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: _check_auth_tickets 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 5 monclient: authenticate success, global_id 374240 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _renew_subs 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53812f410 con 0x7ff53806d480 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _renew_subs 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53812f6f0 con 0x7ff53806d480 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff528004890 con 0x7ff53806d480 2024-09-16T06:59:47.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ff528004ba0 con 0x7ff53806d480 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff51ffff700 10 monclient: dump: 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T06:59:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.020+0000 7ff535ffb700 4 set_mon_vals no callback set 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff53cfaf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff53cfaf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff53cfaf700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53804ef50 con 0x7ff53806d480 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ff52800f610 con 0x7ff53806d480 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ff528043d80 con 0x7ff53806d480 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.024+0000 7ff5377fe700 10 monclient: get_auth_request con 0x7ff524036540 auth_method 0 2024-09-16T06:59:47.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.032+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ff528014490 con 0x7ff53806d480 2024-09-16T06:59:47.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.032+0000 7ff51ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T06:59:47.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.032+0000 7ff51ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T06:59:47.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.200+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5380a3450 con 0x7ff524036540 2024-09-16T06:59:47.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff51ffff700 1 -- 172.21.15.76:0/4102998008 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+99061 (unknown 2965378022 0 1181211963) 0x7ff52c0088a0 con 0x7ff524036540 2024-09-16T06:59:47.210 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 >> v1:172.21.15.189:6812/3098810674 conn(0x7ff524036540 legacy=0x7ff524038a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 10 monclient: shutdown 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 >> v1:172.21.15.76:6790/0 conn(0x7ff53806d480 legacy=0x7ff5381c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 shutdown_connections 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 >> 172.21.15.76:0/4102998008 conn(0x7ff5380a41c0 msgr2=0x7ff53809d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 shutdown_connections 2024-09-16T06:59:47.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T06:59:47.212+0000 7ff53cfaf700 1 -- 172.21.15.76:0/4102998008 wait complete. 2024-09-16T06:59:47.212 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T06:59:47.226 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-09-16T06:59:46.048096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191416,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039624,"statfs":{"total":579820584960,"available":579624574976,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003972"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":109,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:29.344502+0000","last_change":"2024-09-16T06:59:29.344502+0000","last_active":"2024-09-16T06:59:29.344502+0000","last_peered":"2024-09-16T06:59:29.344502+0000","last_clean":"2024-09-16T06:59:29.344502+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:59:29.344502+0000","last_undegraded":"2024-09-16T06:59:29.344502+0000","last_fullsized":"2024-09-16T06:59:29.344502+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:29.344252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:29.344252+0000","last_clean_scrub_stamp":"2024-09-16T06:59:29.344252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:08.955368+0000","scrub_duration":0.0032722929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":92,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:57:55.210008+0000","last_change":"2024-09-16T06:44:52.853100+0000","last_active":"2024-09-16T06:57:55.210008+0000","last_peered":"2024-09-16T06:57:55.210008+0000","last_clean":"2024-09-16T06:57:55.210008+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:57:55.210008+0000","last_undegraded":"2024-09-16T06:57:55.210008+0000","last_fullsized":"2024-09-16T06:57:55.210008+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:42:58.210457+0000","last_clean_scrub_stamp":"2024-09-16T06:42:58.210457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":109,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.363511+0000","last_change":"2024-09-16T06:59:30.363511+0000","last_active":"2024-09-16T06:59:30.363511+0000","last_peered":"2024-09-16T06:59:30.363511+0000","last_clean":"2024-09-16T06:59:30.363511+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:59:30.363511+0000","last_undegraded":"2024-09-16T06:59:30.363511+0000","last_fullsized":"2024-09-16T06:59:30.363511+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.363248+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.363248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:49:23.024575+0000","scrub_duration":0.003112999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":103,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.049181+0000","last_change":"2024-09-16T06:59:26.049404+0000","last_active":"2024-09-16T06:59:26.049181+0000","last_peered":"2024-09-16T06:59:26.049181+0000","last_clean":"2024-09-16T06:59:26.049181+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:59:26.049181+0000","last_undegraded":"2024-09-16T06:59:26.049181+0000","last_fullsized":"2024-09-16T06:59:26.049181+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.049126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.049126+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.049126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:26.538974+0000","scrub_duration":0.0033088900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:27.433535+0000","last_change":"2024-09-16T06:59:27.433754+0000","last_active":"2024-09-16T06:59:27.433535+0000","last_peered":"2024-09-16T06:59:27.433535+0000","last_clean":"2024-09-16T06:59:27.433535+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:59:27.433535+0000","last_undegraded":"2024-09-16T06:59:27.433535+0000","last_fullsized":"2024-09-16T06:59:27.433535+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:27.433488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:27.433488+0000","last_clean_scrub_stamp":"2024-09-16T06:59:27.433488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:17:09.259330+0000","scrub_duration":0.0039497769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:37.080486+0000","last_change":"2024-09-16T06:59:37.080729+0000","last_active":"2024-09-16T06:59:37.080486+0000","last_peered":"2024-09-16T06:59:37.080486+0000","last_clean":"2024-09-16T06:59:37.080486+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:59:37.080486+0000","last_undegraded":"2024-09-16T06:59:37.080486+0000","last_fullsized":"2024-09-16T06:59:37.080486+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:37.080440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:37.080440+0000","last_clean_scrub_stamp":"2024-09-16T06:59:37.080440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:53.563942+0000","scrub_duration":0.0032182030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":175,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:29.866259+0000","last_change":"2024-09-16T06:59:29.866493+0000","last_active":"2024-09-16T06:59:29.866259+0000","last_peered":"2024-09-16T06:59:29.866259+0000","last_clean":"2024-09-16T06:59:29.866259+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:59:29.866259+0000","last_undegraded":"2024-09-16T06:59:29.866259+0000","last_fullsized":"2024-09-16T06:59:29.866259+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:29.866212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:29.866212+0000","last_clean_scrub_stamp":"2024-09-16T06:59:29.866212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:50.033803+0000","scrub_duration":0.0032964639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.751435+0000","last_change":"2024-09-16T06:59:26.751435+0000","last_active":"2024-09-16T06:59:26.751435+0000","last_peered":"2024-09-16T06:59:26.751435+0000","last_clean":"2024-09-16T06:59:26.751435+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:59:26.751435+0000","last_undegraded":"2024-09-16T06:59:26.751435+0000","last_fullsized":"2024-09-16T06:59:26.751435+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.751188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.751188+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.751188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:09.452446+0000","scrub_duration":0.0028652500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.106766+0000","last_change":"2024-09-16T06:59:30.107003+0000","last_active":"2024-09-16T06:59:30.106766+0000","last_peered":"2024-09-16T06:59:30.106766+0000","last_clean":"2024-09-16T06:59:30.106766+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:59:30.106766+0000","last_undegraded":"2024-09-16T06:59:30.106766+0000","last_fullsized":"2024-09-16T06:59:30.106766+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.106718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.106718+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.106718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:18.126317+0000","scrub_duration":0.0032356490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":178,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:25.843593+0000","last_change":"2024-09-16T06:59:25.843593+0000","last_active":"2024-09-16T06:59:25.843593+0000","last_peered":"2024-09-16T06:59:25.843593+0000","last_clean":"2024-09-16T06:59:25.843593+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:59:25.843593+0000","last_undegraded":"2024-09-16T06:59:25.843593+0000","last_fullsized":"2024-09-16T06:59:25.843593+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:25.843316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:25.843316+0000","last_clean_scrub_stamp":"2024-09-16T06:59:25.843316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:32.721226+0000","scrub_duration":0.0049066730000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":183,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.867912+0000","last_change":"2024-09-16T06:59:26.867912+0000","last_active":"2024-09-16T06:59:26.867912+0000","last_peered":"2024-09-16T06:59:26.867912+0000","last_clean":"2024-09-16T06:59:26.867912+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:59:26.867912+0000","last_undegraded":"2024-09-16T06:59:26.867912+0000","last_fullsized":"2024-09-16T06:59:26.867912+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.867771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.867771+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.867771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:22.057765+0000","scrub_duration":0.0029068090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:31.808654+0000","last_change":"2024-09-16T06:59:31.808856+0000","last_active":"2024-09-16T06:59:31.808654+0000","last_peered":"2024-09-16T06:59:31.808654+0000","last_clean":"2024-09-16T06:59:31.808654+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:59:31.808654+0000","last_undegraded":"2024-09-16T06:59:31.808654+0000","last_fullsized":"2024-09-16T06:59:31.808654+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:31.808602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:31.808602+0000","last_clean_scrub_stamp":"2024-09-16T06:59:31.808602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:03.722067+0000","scrub_duration":0.0027213519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.064286+0000","last_change":"2024-09-16T06:59:38.064487+0000","last_active":"2024-09-16T06:59:38.064286+0000","last_peered":"2024-09-16T06:59:38.064286+0000","last_clean":"2024-09-16T06:59:38.064286+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:59:38.064286+0000","last_undegraded":"2024-09-16T06:59:38.064286+0000","last_fullsized":"2024-09-16T06:59:38.064286+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:38.064240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:38.064240+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.064240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:18:58.509591+0000","scrub_duration":0.003270996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:40.724240+0000","last_change":"2024-09-16T06:59:40.724460+0000","last_active":"2024-09-16T06:59:40.724240+0000","last_peered":"2024-09-16T06:59:40.724240+0000","last_clean":"2024-09-16T06:59:40.724240+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:59:40.724240+0000","last_undegraded":"2024-09-16T06:59:40.724240+0000","last_fullsized":"2024-09-16T06:59:40.724240+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:40.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:40.724192+0000","last_clean_scrub_stamp":"2024-09-16T06:59:40.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:16.426041+0000","scrub_duration":0.0032690359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":128,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.251241+0000","last_change":"2024-09-16T06:59:38.251470+0000","last_active":"2024-09-16T06:59:38.251241+0000","last_peered":"2024-09-16T06:59:38.251241+0000","last_clean":"2024-09-16T06:59:38.251241+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:59:38.251241+0000","last_undegraded":"2024-09-16T06:59:38.251241+0000","last_fullsized":"2024-09-16T06:59:38.251241+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:38.251193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:38.251193+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.251193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:37:07.850631+0000","scrub_duration":0.0031383280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:31.059756+0000","last_change":"2024-09-16T06:59:31.059969+0000","last_active":"2024-09-16T06:59:31.059756+0000","last_peered":"2024-09-16T06:59:31.059756+0000","last_clean":"2024-09-16T06:59:31.059756+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:59:31.059756+0000","last_undegraded":"2024-09-16T06:59:31.059756+0000","last_fullsized":"2024-09-16T06:59:31.059756+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:31.059702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:31.059702+0000","last_clean_scrub_stamp":"2024-09-16T06:59:31.059702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:04:54.250001+0000","scrub_duration":0.0031244519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'119","reported_seq":444,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:33.363747+0000","last_change":"2024-09-16T06:59:33.363974+0000","last_active":"2024-09-16T06:59:33.363747+0000","last_peered":"2024-09-16T06:59:33.363747+0000","last_clean":"2024-09-16T06:59:33.363747+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:59:33.363747+0000","last_undegraded":"2024-09-16T06:59:33.363747+0000","last_fullsized":"2024-09-16T06:59:33.363747+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'119","last_scrub_stamp":"2024-09-16T06:59:33.363697+0000","last_deep_scrub":"255'119","last_deep_scrub_stamp":"2024-09-16T06:59:33.363697+0000","last_clean_scrub_stamp":"2024-09-16T06:59:33.363697+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:03.096741+0000","scrub_duration":0.0088077209999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"255'2","reported_seq":259,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.398231+0000","last_change":"2024-09-16T06:59:38.398420+0000","last_active":"2024-09-16T06:59:38.398231+0000","last_peered":"2024-09-16T06:59:38.398231+0000","last_clean":"2024-09-16T06:59:38.398231+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:59:38.398231+0000","last_undegraded":"2024-09-16T06:59:38.398231+0000","last_fullsized":"2024-09-16T06:59:38.398231+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-09-16T06:59:38.398175+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-09-16T06:59:38.398175+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.398175+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:48.616703+0000","scrub_duration":0.0056884550000000002,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":93,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:00.088924+0000","last_change":"2024-09-16T06:44:52.858722+0000","last_active":"2024-09-16T06:59:00.088924+0000","last_peered":"2024-09-16T06:59:00.088924+0000","last_clean":"2024-09-16T06:59:00.088924+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:59:00.088924+0000","last_undegraded":"2024-09-16T06:59:00.088924+0000","last_fullsized":"2024-09-16T06:59:00.088924+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:35.740906+0000","last_change":"2024-09-16T06:59:35.741108+0000","last_active":"2024-09-16T06:59:35.740906+0000","last_peered":"2024-09-16T06:59:35.740906+0000","last_clean":"2024-09-16T06:59:35.740906+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:59:35.740906+0000","last_undegraded":"2024-09-16T06:59:35.740906+0000","last_fullsized":"2024-09-16T06:59:35.740906+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:35.740860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:35.740860+0000","last_clean_scrub_stamp":"2024-09-16T06:59:35.740860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:30.350785+0000","scrub_duration":0.003186501,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"255'3","reported_seq":294,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:24.825568+0000","last_change":"2024-09-16T06:59:24.825568+0000","last_active":"2024-09-16T06:59:24.825568+0000","last_peered":"2024-09-16T06:59:24.825568+0000","last_clean":"2024-09-16T06:59:24.825568+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:59:24.825568+0000","last_undegraded":"2024-09-16T06:59:24.825568+0000","last_fullsized":"2024-09-16T06:59:24.825568+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'3","last_scrub_stamp":"2024-09-16T06:59:24.825396+0000","last_deep_scrub":"255'3","last_deep_scrub_stamp":"2024-09-16T06:59:24.825396+0000","last_clean_scrub_stamp":"2024-09-16T06:59:24.825396+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.274993+0000","scrub_duration":0.005644769,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":263,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.043534+0000","last_change":"2024-09-16T06:59:30.043534+0000","last_active":"2024-09-16T06:59:30.043534+0000","last_peered":"2024-09-16T06:59:30.043534+0000","last_clean":"2024-09-16T06:59:30.043534+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:59:30.043534+0000","last_undegraded":"2024-09-16T06:59:30.043534+0000","last_fullsized":"2024-09-16T06:59:30.043534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.043102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.043102+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.043102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:21:42.062586+0000","scrub_duration":0.0030844100000000001,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.372856+0000","last_change":"2024-09-16T06:59:30.373118+0000","last_active":"2024-09-16T06:59:30.372856+0000","last_peered":"2024-09-16T06:59:30.372856+0000","last_clean":"2024-09-16T06:59:30.372856+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:59:30.372856+0000","last_undegraded":"2024-09-16T06:59:30.372856+0000","last_fullsized":"2024-09-16T06:59:30.372856+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.372811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.372811+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.372811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:06:03.112869+0000","scrub_duration":0.0028233899999999998,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:37.255560+0000","last_change":"2024-09-16T06:59:37.255768+0000","last_active":"2024-09-16T06:59:37.255560+0000","last_peered":"2024-09-16T06:59:37.255560+0000","last_clean":"2024-09-16T06:59:37.255560+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:59:37.255560+0000","last_undegraded":"2024-09-16T06:59:37.255560+0000","last_fullsized":"2024-09-16T06:59:37.255560+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:37.255509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:37.255509+0000","last_clean_scrub_stamp":"2024-09-16T06:59:37.255509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:57.725303+0000","scrub_duration":0.0033020100000000002,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":234,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:58:05.214203+0000","last_change":"2024-09-16T06:44:53.857350+0000","last_active":"2024-09-16T06:58:05.214203+0000","last_peered":"2024-09-16T06:58:05.214203+0000","last_clean":"2024-09-16T06:58:05.214203+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:58:05.214203+0000","last_undegraded":"2024-09-16T06:58:05.214203+0000","last_fullsized":"2024-09-16T06:58:05.214203+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:38:26.064747+0000","last_clean_scrub_stamp":"2024-09-16T06:38:26.064747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575105,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.872},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.78000000000000003}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":1.02},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":1.157}]}]},{"osd":0,"up_from":13,"seq":55834575108,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.73499999999999999}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.81799999999999995}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.878},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":0.84399999999999997}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":1.284},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":1.2410000000000001}]}]},{"osd":5,"up_from":12,"seq":51539607812,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.75700000000000001}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.877},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":0.70199999999999996}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.72599999999999998}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":12,"seq":51539607811,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.42699999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":1.044},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.49299999999999999,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.89100000000000001,"5min":0.94899999999999995,"15min":0.997},"last":0.70299999999999996}]},{"osd":1,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.60699999999999998},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.33300000000000002},"max":{"1min":0.746,"5min":0.83799999999999997,"15min":1.0549999999999999},"last":0.501},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.60999999999999999,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.86399999999999999,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.71399999999999997,"5min":0.61899999999999999,"15min":0.57499999999999996},"max":{"1min":0.96499999999999997,"5min":1.073,"15min":1.3500000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.78600000000000003,"15min":0.79800000000000004},"min":{"1min":0.63300000000000001,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.84099999999999997}]},{"osd":4,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.70199999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":0.97099999999999997,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.90600000000000003}]},{"osd":5,"last update":"Mon Sep 16 06:58:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.79900000000000004,"15min":0.83699999999999997},"min":{"1min":0.65700000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.5780000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84099999999999997,"15min":0.85399999999999998},"min":{"1min":0.71699999999999997,"5min":0.60699999999999998,"15min":0.59599999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0960000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.68600000000000005}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.755}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.93899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640514,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77300000000000002,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46800000000000003,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.60299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0109999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.91100000000000003}]},{"osd":1,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78700000000000003},"min":{"1min":0.58699999999999997,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":1.0149999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.745},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.872,"15min":0.88500000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":1.0529999999999999,"5min":1.153,"15min":1.4910000000000001},"last":0.97499999999999998}]},{"osd":2,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.74099999999999999,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76200000000000001},"min":{"1min":0.621,"5min":0.57999999999999996,"15min":0.52100000000000002},"max":{"1min":1.008,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.86399999999999999}]},{"osd":4,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.92400000000000004,"5min":0.99299999999999999,"15min":1.018},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Sep 16 06:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.121},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.88,"15min":0.86699999999999999},"min":{"1min":0.65900000000000003,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1140000000000001,"5min":1.125,"15min":1.1899999999999999},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T06:59:47.229 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-16T06:42:58.210457+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=42, tm_sec=58, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:47.229 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:47.230 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T06:38:26.064747+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=38, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=6, tm_min=59, tm_sec=22, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T06:59:47.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T07:00:07.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:00:07.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: get_monmap_and_config 2024-09-16T07:00:07.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: build_initial_monmap 2024-09-16T07:00:07.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: monmap: 2024-09-16T07:00:07.353 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T07:00:07.358005+0000 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T07:00:07.358005+0000 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T07:00:07.354 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 1 Processor -- start 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 1 -- start start 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: init 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: _reopen_session rank -1 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-c con 0x7ffbbc097b60 addr v1:172.21.15.189:6789/0 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-a con 0x7ffbbc099090 addr v1:172.21.15.76:6789/0 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-b con 0x7ffbbc139500 addr v1:172.21.15.76:6790/0 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc098990 con 0x7ffbbc099090 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc097f40 con 0x7ffbbc097b60 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc09d610 con 0x7ffbbc139500 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbbfff700 10 monclient(hunting): authenticate will time out at 2154.491274s 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbb27fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffbbc099090 0x7ffbbc09f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33768/0 (socket says 172.21.15.76:33768) 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbbb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ffbbc139500 0x7ffbbc13b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41562/0 (socket says 172.21.15.76:41562) 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.355+0000 7ffbb27fc700 1 -- 172.21.15.76:0/4133991922 learned_addr learned my addr 172.21.15.76:0/4133991922 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263434987 0 0) 0x7ffbbc09d610 con 0x7ffbbc139500 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): my global_id is 374243 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba80035d0 con 0x7ffbbc139500 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051636916 0 0) 0x7ffbbc098990 con 0x7ffbbc099090 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient(hunting): my global_id is 334593 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc09d610 con 0x7ffbbc099090 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792875235 0 0) 0x7ffba80035d0 con 0x7ffbbc139500 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 >> v1:172.21.15.189:6789/0 conn(0x7ffbbc097b60 legacy=0x7ffbbc09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 >> v1:172.21.15.76:6789/0 conn(0x7ffbbc099090 legacy=0x7ffbbc09f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: _finish_hunting 0 2024-09-16T07:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 monclient: found mon.noname-b 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc0986b0 con 0x7ffbbc139500 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: _finish_auth 0 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffbb4001990 con 0x7ffbbc139500 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: dump: 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 5 monclient: authenticate success, global_id 374243 2024-09-16T07:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbb4001c30 con 0x7ffbbc139500 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 10 monclient: get_monmap_and_config success 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 4 set_mon_vals no callback set 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 1 -- 172.21.15.76:0/4133991922 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffbb4001fc0 con 0x7ffbbc139500 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbba7fc700 10 monclient: dump: 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.362 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 10 monclient: shutdown 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4133991922 >> v1:172.21.15.76:6790/0 conn(0x7ffbbc139500 legacy=0x7ffbbc13b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4133991922 shutdown_connections 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4133991922 >> 172.21.15.76:0/4133991922 conn(0x7ffbbc0a0030 msgr2=0x7ffbbc0a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4133991922 shutdown_connections 2024-09-16T07:00:07.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4133991922 wait complete. 2024-09-16T07:00:07.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 10 monclient: build_initial_monmap 2024-09-16T07:00:07.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 10 monclient: monmap: 2024-09-16T07:00:07.368 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 Processor -- start 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 1 -- start start 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.359+0000 7ffbbbfff700 10 monclient: init 2024-09-16T07:00:07.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _reopen_session rank -1 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-b con 0x7ffbbc097b60 addr v1:172.21.15.189:6789/0 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-a con 0x7ffbbc099090 addr v1:172.21.15.76:6789/0 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient(hunting): picked mon.noname-c con 0x7ffbbc139500 addr v1:172.21.15.76:6790/0 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc06cc90 con 0x7ffbbc099090 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc06d250 con 0x7ffbbc097b60 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc06d530 con 0x7ffbbc139500 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient(hunting): authenticate will time out at 2154.496500s 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb27fc700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffbbc099090 0x7ffbbc137540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33780/0 (socket says 172.21.15.76:33780) 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbb7fe700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7ffbbc139500 0x7ffbbc137c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41576/0 (socket says 172.21.15.76:41576) 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb27fc700 1 -- 172.21.15.76:0/4000367183 learned_addr learned my addr 172.21.15.76:0/4000367183 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T07:00:07.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271173382 0 0) 0x7ffbbc06cc90 con 0x7ffbbc099090 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): my global_id is 334596 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c0035d0 con 0x7ffbbc099090 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130679815 0 0) 0x7ffbbc06d530 con 0x7ffbbc139500 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): my global_id is 374246 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc06cc90 con 0x7ffbbc139500 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253617549 0 0) 0x7ffbbc06d250 con 0x7ffbbc097b60 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): my global_id is 285073 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc06d530 con 0x7ffbbc097b60 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598993722 0 0) 0x7ffbbc06cc90 con 0x7ffbbc139500 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbbc06d250 con 0x7ffbbc139500 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffbb4001f40 con 0x7ffbbc139500 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): dump: 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1611051898 0 0) 0x7ffb9c0035d0 con 0x7ffbbc099090 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbbc06cc90 con 0x7ffbbc099090 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffba4001840 con 0x7ffbbc099090 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient(hunting): dump: 2024-09-16T07:00:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2451951258 0 0) 0x7ffbbc06cc90 con 0x7ffbbc099090 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 >> v1:172.21.15.76:6790/0 conn(0x7ffbbc139500 legacy=0x7ffbbc137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 >> v1:172.21.15.189:6789/0 conn(0x7ffbbc097b60 legacy=0x7ffbbc136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: _finish_hunting 0 2024-09-16T07:00:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 monclient: found mon.a 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc06cf70 con 0x7ffbbc099090 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: _finish_auth 0 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: _check_auth_tickets 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 5 monclient: authenticate success, global_id 334596 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _renew_subs 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc138890 con 0x7ffbbc099090 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba4004890 con 0x7ffbbc099090 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7ffba4004ba0 con 0x7ffbbc099090 2024-09-16T07:00:07.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _renew_subs 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb9ffb700 4 set_mon_vals no callback set 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc06c070 con 0x7ffbbc099090 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 10 monclient: dump: 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.363+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7ffba400f610 con 0x7ffbbc099090 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbbbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbbbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbbbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.76:6789/0 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0008d0 con 0x7ffbbc099090 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7ffba4043e20 con 0x7ffbbc099090 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.367+0000 7ffbbaffd700 10 monclient: get_auth_request con 0x7ffb9c036450 auth_method 0 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.371+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mon.0 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7ffba40147c0 con 0x7ffbbc099090 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.371+0000 7ffbb3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.371+0000 7ffbb3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:00:07.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.539+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 --> v1:172.21.15.189:6812/3098810674 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbbc09fb80 con 0x7ffb9c036450 2024-09-16T07:00:07.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbb3fff700 1 -- 172.21.15.76:0/4000367183 <== mgr.4109 v1:172.21.15.189:6812/3098810674 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+99195 (unknown 2965378022 0 1197679995) 0x7ffbac007890 con 0x7ffb9c036450 2024-09-16T07:00:07.544 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 >> v1:172.21.15.189:6812/3098810674 conn(0x7ffb9c036450 legacy=0x7ffb9c038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 10 monclient: shutdown 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 >> v1:172.21.15.76:6789/0 conn(0x7ffbbc099090 legacy=0x7ffbbc137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 shutdown_connections 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 >> 172.21.15.76:0/4000367183 conn(0x7ffbbc0a0030 msgr2=0x7ffbbc0a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 shutdown_connections 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.547+0000 7ffbbbfff700 1 -- 172.21.15.76:0/4000367183 wait complete. 2024-09-16T07:00:07.549 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-09-16T07:00:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":887,"stamp":"2024-09-16T07:00:06.054559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1586841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":191416,"kb_used_data":4744,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566039624,"statfs":{"total":579820584960,"available":579624574976,"internally_reserved":0,"allocated":4857856,"data_stored":5814892,"data_compressed":666916,"data_compressed_allocated":1269760,"data_compressed_original":3850240,"omap_allocated":9562,"internal_metadata":191093414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003850"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":109,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:29.344502+0000","last_change":"2024-09-16T06:59:29.344502+0000","last_active":"2024-09-16T06:59:29.344502+0000","last_peered":"2024-09-16T06:59:29.344502+0000","last_clean":"2024-09-16T06:59:29.344502+0000","last_became_active":"2024-09-16T06:44:52.855630+0000","last_became_peered":"2024-09-16T06:44:52.855630+0000","last_unstale":"2024-09-16T06:59:29.344502+0000","last_undegraded":"2024-09-16T06:59:29.344502+0000","last_fullsized":"2024-09-16T06:59:29.344502+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:29.344252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:29.344252+0000","last_clean_scrub_stamp":"2024-09-16T06:59:29.344252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:08.955368+0000","scrub_duration":0.0032722929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:48.022560+0000","last_change":"2024-09-16T06:59:48.022560+0000","last_active":"2024-09-16T06:59:48.022560+0000","last_peered":"2024-09-16T06:59:48.022560+0000","last_clean":"2024-09-16T06:59:48.022560+0000","last_became_active":"2024-09-16T06:44:52.852665+0000","last_became_peered":"2024-09-16T06:44:52.852665+0000","last_unstale":"2024-09-16T06:59:48.022560+0000","last_undegraded":"2024-09-16T06:59:48.022560+0000","last_fullsized":"2024-09-16T06:59:48.022560+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:48.022296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:48.022296+0000","last_clean_scrub_stamp":"2024-09-16T06:59:48.022296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:55.812732+0000","scrub_duration":0.0027128389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":109,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.363511+0000","last_change":"2024-09-16T06:59:30.363511+0000","last_active":"2024-09-16T06:59:30.363511+0000","last_peered":"2024-09-16T06:59:30.363511+0000","last_clean":"2024-09-16T06:59:30.363511+0000","last_became_active":"2024-09-16T06:44:52.855392+0000","last_became_peered":"2024-09-16T06:44:52.855392+0000","last_unstale":"2024-09-16T06:59:30.363511+0000","last_undegraded":"2024-09-16T06:59:30.363511+0000","last_fullsized":"2024-09-16T06:59:30.363511+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.363248+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.363248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:49:23.024575+0000","scrub_duration":0.003112999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.049404+0000","last_change":"2024-09-16T06:59:26.049404+0000","last_active":"2024-09-16T06:59:26.049404+0000","last_peered":"2024-09-16T06:59:26.049404+0000","last_clean":"2024-09-16T06:59:26.049404+0000","last_became_active":"2024-09-16T06:44:52.854509+0000","last_became_peered":"2024-09-16T06:44:52.854509+0000","last_unstale":"2024-09-16T06:59:26.049404+0000","last_undegraded":"2024-09-16T06:59:26.049404+0000","last_fullsized":"2024-09-16T06:59:26.049404+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.049126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.049126+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.049126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:26.538974+0000","scrub_duration":0.0033088900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:27.433754+0000","last_change":"2024-09-16T06:59:27.433754+0000","last_active":"2024-09-16T06:59:27.433754+0000","last_peered":"2024-09-16T06:59:27.433754+0000","last_clean":"2024-09-16T06:59:27.433754+0000","last_became_active":"2024-09-16T06:44:52.858689+0000","last_became_peered":"2024-09-16T06:44:52.858689+0000","last_unstale":"2024-09-16T06:59:27.433754+0000","last_undegraded":"2024-09-16T06:59:27.433754+0000","last_fullsized":"2024-09-16T06:59:27.433754+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:27.433488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:27.433488+0000","last_clean_scrub_stamp":"2024-09-16T06:59:27.433488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:17:09.259330+0000","scrub_duration":0.0039497769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:37.080486+0000","last_change":"2024-09-16T06:59:37.080729+0000","last_active":"2024-09-16T06:59:37.080486+0000","last_peered":"2024-09-16T06:59:37.080486+0000","last_clean":"2024-09-16T06:59:37.080486+0000","last_became_active":"2024-09-16T06:44:52.858973+0000","last_became_peered":"2024-09-16T06:44:52.858973+0000","last_unstale":"2024-09-16T06:59:37.080486+0000","last_undegraded":"2024-09-16T06:59:37.080486+0000","last_fullsized":"2024-09-16T06:59:37.080486+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:37.080440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:37.080440+0000","last_clean_scrub_stamp":"2024-09-16T06:59:37.080440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:53.563942+0000","scrub_duration":0.0032182030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":175,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:29.866259+0000","last_change":"2024-09-16T06:59:29.866493+0000","last_active":"2024-09-16T06:59:29.866259+0000","last_peered":"2024-09-16T06:59:29.866259+0000","last_clean":"2024-09-16T06:59:29.866259+0000","last_became_active":"2024-09-16T06:44:52.856706+0000","last_became_peered":"2024-09-16T06:44:52.856706+0000","last_unstale":"2024-09-16T06:59:29.866259+0000","last_undegraded":"2024-09-16T06:59:29.866259+0000","last_fullsized":"2024-09-16T06:59:29.866259+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:29.866212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:29.866212+0000","last_clean_scrub_stamp":"2024-09-16T06:59:29.866212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:50.033803+0000","scrub_duration":0.0032964639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.751435+0000","last_change":"2024-09-16T06:59:26.751435+0000","last_active":"2024-09-16T06:59:26.751435+0000","last_peered":"2024-09-16T06:59:26.751435+0000","last_clean":"2024-09-16T06:59:26.751435+0000","last_became_active":"2024-09-16T06:44:52.854002+0000","last_became_peered":"2024-09-16T06:44:52.854002+0000","last_unstale":"2024-09-16T06:59:26.751435+0000","last_undegraded":"2024-09-16T06:59:26.751435+0000","last_fullsized":"2024-09-16T06:59:26.751435+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.751188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.751188+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.751188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:09.452446+0000","scrub_duration":0.0028652500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.106766+0000","last_change":"2024-09-16T06:59:30.107003+0000","last_active":"2024-09-16T06:59:30.106766+0000","last_peered":"2024-09-16T06:59:30.106766+0000","last_clean":"2024-09-16T06:59:30.106766+0000","last_became_active":"2024-09-16T06:44:52.857289+0000","last_became_peered":"2024-09-16T06:44:52.857289+0000","last_unstale":"2024-09-16T06:59:30.106766+0000","last_undegraded":"2024-09-16T06:59:30.106766+0000","last_fullsized":"2024-09-16T06:59:30.106766+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.106718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.106718+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.106718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:18.126317+0000","scrub_duration":0.0032356490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":178,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:25.843593+0000","last_change":"2024-09-16T06:59:25.843593+0000","last_active":"2024-09-16T06:59:25.843593+0000","last_peered":"2024-09-16T06:59:25.843593+0000","last_clean":"2024-09-16T06:59:25.843593+0000","last_became_active":"2024-09-16T06:44:52.856861+0000","last_became_peered":"2024-09-16T06:44:52.856861+0000","last_unstale":"2024-09-16T06:59:25.843593+0000","last_undegraded":"2024-09-16T06:59:25.843593+0000","last_fullsized":"2024-09-16T06:59:25.843593+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:25.843316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:25.843316+0000","last_clean_scrub_stamp":"2024-09-16T06:59:25.843316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:32.721226+0000","scrub_duration":0.0049066730000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":183,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:26.867912+0000","last_change":"2024-09-16T06:59:26.867912+0000","last_active":"2024-09-16T06:59:26.867912+0000","last_peered":"2024-09-16T06:59:26.867912+0000","last_clean":"2024-09-16T06:59:26.867912+0000","last_became_active":"2024-09-16T06:44:52.854090+0000","last_became_peered":"2024-09-16T06:44:52.854090+0000","last_unstale":"2024-09-16T06:59:26.867912+0000","last_undegraded":"2024-09-16T06:59:26.867912+0000","last_fullsized":"2024-09-16T06:59:26.867912+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:26.867771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:26.867771+0000","last_clean_scrub_stamp":"2024-09-16T06:59:26.867771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:22.057765+0000","scrub_duration":0.0029068090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:31.808654+0000","last_change":"2024-09-16T06:59:31.808856+0000","last_active":"2024-09-16T06:59:31.808654+0000","last_peered":"2024-09-16T06:59:31.808654+0000","last_clean":"2024-09-16T06:59:31.808654+0000","last_became_active":"2024-09-16T06:44:52.855989+0000","last_became_peered":"2024-09-16T06:44:52.855989+0000","last_unstale":"2024-09-16T06:59:31.808654+0000","last_undegraded":"2024-09-16T06:59:31.808654+0000","last_fullsized":"2024-09-16T06:59:31.808654+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:31.808602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:31.808602+0000","last_clean_scrub_stamp":"2024-09-16T06:59:31.808602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:03.722067+0000","scrub_duration":0.0027213519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.064286+0000","last_change":"2024-09-16T06:59:38.064487+0000","last_active":"2024-09-16T06:59:38.064286+0000","last_peered":"2024-09-16T06:59:38.064286+0000","last_clean":"2024-09-16T06:59:38.064286+0000","last_became_active":"2024-09-16T06:44:53.857178+0000","last_became_peered":"2024-09-16T06:44:53.857178+0000","last_unstale":"2024-09-16T06:59:38.064286+0000","last_undegraded":"2024-09-16T06:59:38.064286+0000","last_fullsized":"2024-09-16T06:59:38.064286+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:38.064240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:38.064240+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.064240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:18:58.509591+0000","scrub_duration":0.003270996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:40.724240+0000","last_change":"2024-09-16T06:59:40.724460+0000","last_active":"2024-09-16T06:59:40.724240+0000","last_peered":"2024-09-16T06:59:40.724240+0000","last_clean":"2024-09-16T06:59:40.724240+0000","last_became_active":"2024-09-16T06:44:53.857084+0000","last_became_peered":"2024-09-16T06:44:53.857084+0000","last_unstale":"2024-09-16T06:59:40.724240+0000","last_undegraded":"2024-09-16T06:59:40.724240+0000","last_fullsized":"2024-09-16T06:59:40.724240+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:40.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:40.724192+0000","last_clean_scrub_stamp":"2024-09-16T06:59:40.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:16.426041+0000","scrub_duration":0.0032690359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":128,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.251241+0000","last_change":"2024-09-16T06:59:38.251470+0000","last_active":"2024-09-16T06:59:38.251241+0000","last_peered":"2024-09-16T06:59:38.251241+0000","last_clean":"2024-09-16T06:59:38.251241+0000","last_became_active":"2024-09-16T06:44:53.857171+0000","last_became_peered":"2024-09-16T06:44:53.857171+0000","last_unstale":"2024-09-16T06:59:38.251241+0000","last_undegraded":"2024-09-16T06:59:38.251241+0000","last_fullsized":"2024-09-16T06:59:38.251241+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:38.251193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:38.251193+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.251193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:37:07.850631+0000","scrub_duration":0.0031383280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:31.059756+0000","last_change":"2024-09-16T06:59:31.059969+0000","last_active":"2024-09-16T06:59:31.059756+0000","last_peered":"2024-09-16T06:59:31.059756+0000","last_clean":"2024-09-16T06:59:31.059756+0000","last_became_active":"2024-09-16T06:44:52.856944+0000","last_became_peered":"2024-09-16T06:44:52.856944+0000","last_unstale":"2024-09-16T06:59:31.059756+0000","last_undegraded":"2024-09-16T06:59:31.059756+0000","last_fullsized":"2024-09-16T06:59:31.059756+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:31.059702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:31.059702+0000","last_clean_scrub_stamp":"2024-09-16T06:59:31.059702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:04:54.250001+0000","scrub_duration":0.0031244519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"255'119","reported_seq":444,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:33.363747+0000","last_change":"2024-09-16T06:59:33.363974+0000","last_active":"2024-09-16T06:59:33.363747+0000","last_peered":"2024-09-16T06:59:33.363747+0000","last_clean":"2024-09-16T06:59:33.363747+0000","last_became_active":"2024-09-16T06:44:55.869622+0000","last_became_peered":"2024-09-16T06:44:55.869622+0000","last_unstale":"2024-09-16T06:59:33.363747+0000","last_undegraded":"2024-09-16T06:59:33.363747+0000","last_fullsized":"2024-09-16T06:59:33.363747+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'119","last_scrub_stamp":"2024-09-16T06:59:33.363697+0000","last_deep_scrub":"255'119","last_deep_scrub_stamp":"2024-09-16T06:59:33.363697+0000","last_clean_scrub_stamp":"2024-09-16T06:59:33.363697+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:03.096741+0000","scrub_duration":0.0088077209999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"255'2","reported_seq":259,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:38.398231+0000","last_change":"2024-09-16T06:59:38.398420+0000","last_active":"2024-09-16T06:59:38.398231+0000","last_peered":"2024-09-16T06:59:38.398231+0000","last_clean":"2024-09-16T06:59:38.398231+0000","last_became_active":"2024-09-16T06:44:52.850302+0000","last_became_peered":"2024-09-16T06:44:52.850302+0000","last_unstale":"2024-09-16T06:59:38.398231+0000","last_undegraded":"2024-09-16T06:59:38.398231+0000","last_fullsized":"2024-09-16T06:59:38.398231+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-09-16T06:59:38.398175+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-09-16T06:59:38.398175+0000","last_clean_scrub_stamp":"2024-09-16T06:59:38.398175+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:57:48.616703+0000","scrub_duration":0.0056884550000000002,"objects_trimmed":0,"snaptrim_duration":0.0016814460000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:51.074777+0000","last_change":"2024-09-16T06:59:51.074993+0000","last_active":"2024-09-16T06:59:51.074777+0000","last_peered":"2024-09-16T06:59:51.074777+0000","last_clean":"2024-09-16T06:59:51.074777+0000","last_became_active":"2024-09-16T06:44:52.858372+0000","last_became_peered":"2024-09-16T06:44:52.858372+0000","last_unstale":"2024-09-16T06:59:51.074777+0000","last_undegraded":"2024-09-16T06:59:51.074777+0000","last_fullsized":"2024-09-16T06:59:51.074777+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:51.074726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:51.074726+0000","last_clean_scrub_stamp":"2024-09-16T06:59:51.074726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:13:36.805775+0000","scrub_duration":0.0033210430000000001,"objects_trimmed":0,"snaptrim_duration":0.0017473499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:35.740906+0000","last_change":"2024-09-16T06:59:35.741108+0000","last_active":"2024-09-16T06:59:35.740906+0000","last_peered":"2024-09-16T06:59:35.740906+0000","last_clean":"2024-09-16T06:59:35.740906+0000","last_became_active":"2024-09-16T06:44:52.856138+0000","last_became_peered":"2024-09-16T06:44:52.856138+0000","last_unstale":"2024-09-16T06:59:35.740906+0000","last_undegraded":"2024-09-16T06:59:35.740906+0000","last_fullsized":"2024-09-16T06:59:35.740906+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:35.740860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:35.740860+0000","last_clean_scrub_stamp":"2024-09-16T06:59:35.740860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:30.350785+0000","scrub_duration":0.003186501,"objects_trimmed":0,"snaptrim_duration":0.00083416600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"255'3","reported_seq":294,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:24.825568+0000","last_change":"2024-09-16T06:59:24.825568+0000","last_active":"2024-09-16T06:59:24.825568+0000","last_peered":"2024-09-16T06:59:24.825568+0000","last_clean":"2024-09-16T06:59:24.825568+0000","last_became_active":"2024-09-16T06:44:52.856990+0000","last_became_peered":"2024-09-16T06:44:52.856990+0000","last_unstale":"2024-09-16T06:59:24.825568+0000","last_undegraded":"2024-09-16T06:59:24.825568+0000","last_fullsized":"2024-09-16T06:59:24.825568+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'3","last_scrub_stamp":"2024-09-16T06:59:24.825396+0000","last_deep_scrub":"255'3","last_deep_scrub_stamp":"2024-09-16T06:59:24.825396+0000","last_clean_scrub_stamp":"2024-09-16T06:59:24.825396+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.274993+0000","scrub_duration":0.005644769,"objects_trimmed":0,"snaptrim_duration":0.00082059600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":263,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.043534+0000","last_change":"2024-09-16T06:59:30.043534+0000","last_active":"2024-09-16T06:59:30.043534+0000","last_peered":"2024-09-16T06:59:30.043534+0000","last_clean":"2024-09-16T06:59:30.043534+0000","last_became_active":"2024-09-16T06:44:52.856162+0000","last_became_peered":"2024-09-16T06:44:52.856162+0000","last_unstale":"2024-09-16T06:59:30.043534+0000","last_undegraded":"2024-09-16T06:59:30.043534+0000","last_fullsized":"2024-09-16T06:59:30.043534+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.043102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.043102+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.043102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:21:42.062586+0000","scrub_duration":0.0030844100000000001,"objects_trimmed":0,"snaptrim_duration":0.00081478300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:30.372856+0000","last_change":"2024-09-16T06:59:30.373118+0000","last_active":"2024-09-16T06:59:30.372856+0000","last_peered":"2024-09-16T06:59:30.372856+0000","last_clean":"2024-09-16T06:59:30.372856+0000","last_became_active":"2024-09-16T06:44:52.843856+0000","last_became_peered":"2024-09-16T06:44:52.843856+0000","last_unstale":"2024-09-16T06:59:30.372856+0000","last_undegraded":"2024-09-16T06:59:30.372856+0000","last_fullsized":"2024-09-16T06:59:30.372856+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:30.372811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:30.372811+0000","last_clean_scrub_stamp":"2024-09-16T06:59:30.372811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:06:03.112869+0000","scrub_duration":0.0028233899999999998,"objects_trimmed":0,"snaptrim_duration":0.00097950899999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:37.255560+0000","last_change":"2024-09-16T06:59:37.255768+0000","last_active":"2024-09-16T06:59:37.255560+0000","last_peered":"2024-09-16T06:59:37.255560+0000","last_clean":"2024-09-16T06:59:37.255560+0000","last_became_active":"2024-09-16T06:44:52.848471+0000","last_became_peered":"2024-09-16T06:44:52.848471+0000","last_unstale":"2024-09-16T06:59:37.255560+0000","last_undegraded":"2024-09-16T06:59:37.255560+0000","last_fullsized":"2024-09-16T06:59:37.255560+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:37.255509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:37.255509+0000","last_clean_scrub_stamp":"2024-09-16T06:59:37.255509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:57.725303+0000","scrub_duration":0.0033020100000000002,"objects_trimmed":0,"snaptrim_duration":0.00072471500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-16T06:59:48.412562+0000","last_change":"2024-09-16T06:59:48.412765+0000","last_active":"2024-09-16T06:59:48.412562+0000","last_peered":"2024-09-16T06:59:48.412562+0000","last_clean":"2024-09-16T06:59:48.412562+0000","last_became_active":"2024-09-16T06:44:53.856706+0000","last_became_peered":"2024-09-16T06:44:53.856706+0000","last_unstale":"2024-09-16T06:59:48.412562+0000","last_undegraded":"2024-09-16T06:59:48.412562+0000","last_fullsized":"2024-09-16T06:59:48.412562+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T06:59:48.412517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T06:59:48.412517+0000","last_clean_scrub_stamp":"2024-09-16T06:59:48.412517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:13.393664+0000","scrub_duration":0.0027329889999999999,"objects_trimmed":0,"snaptrim_duration":0.00094006700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":2115712,"data_compressed":24392,"data_compressed_allocated":131072,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575109,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.72599999999999998,"15min":0.748},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.41899999999999998},"max":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.542},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77900000000000003,"15min":0.76800000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":1.048,"15min":1.1859999999999999},"last":0.84399999999999997}]},{"osd":1,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81399999999999995},"min":{"1min":0.624,"5min":0.55200000000000005,"15min":0.499},"max":{"1min":0.98799999999999999,"5min":1.671,"15min":1.671},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79800000000000004,"15min":0.78200000000000003},"min":{"1min":0.54100000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.93600000000000005,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.93000000000000005}]},{"osd":2,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.84499999999999997},"min":{"1min":0.67900000000000005,"5min":0.65700000000000003,"15min":0.55000000000000004},"max":{"1min":0.99199999999999999,"5min":1.5,"15min":1.7170000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.49299999999999999},"max":{"1min":0.85499999999999998,"5min":0.94999999999999996,"15min":1.4099999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42599999999999999},"max":{"1min":0.83799999999999997,"5min":0.97099999999999997,"15min":1.196},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72299999999999998,"15min":0.74399999999999999},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.47099999999999997},"max":{"1min":0.90600000000000003,"5min":1.0209999999999999,"15min":1.3120000000000001},"last":0.872}]},{"osd":5,"last update":"Mon Sep 16 06:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.46000000000000002},"max":{"1min":1.0069999999999999,"5min":1.0229999999999999,"15min":1.5900000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.628,"5min":0.58799999999999997,"15min":0.53200000000000003},"max":{"1min":1.0509999999999999,"5min":1.1399999999999999,"15min":1.48},"last":0.69399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575112,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":798720,"data_stored":792852,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.55500000000000005},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.63800000000000001,"5min":0.80600000000000005,"15min":1.079},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":0.81899999999999995,"5min":0.84999999999999998,"15min":2.319},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":1.0429999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64200000000000002,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.84799999999999998,"15min":2.395},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56100000000000005},"max":{"1min":0.90700000000000003,"5min":1.1719999999999999,"15min":1.1870000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.81599999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.54300000000000004},"max":{"1min":0.90900000000000003,"5min":0.996,"15min":1.764},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81499999999999995,"15min":0.81200000000000006},"min":{"1min":0.59899999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.98499999999999999,"5min":1.234,"15min":1.236},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81699999999999995,"15min":0.83799999999999997},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.53000000000000003},"max":{"1min":1.01,"5min":1.0329999999999999,"15min":1.8080000000000001},"last":0.80200000000000005}]},{"osd":5,"last update":"Mon Sep 16 06:59:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.60499999999999998},"max":{"1min":1.008,"5min":1.1990000000000001,"15min":8.0709999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.89300000000000002,"15min":0.90300000000000002},"min":{"1min":0.749,"5min":0.65100000000000002,"15min":0.64100000000000001},"max":{"1min":1.0349999999999999,"5min":1.272,"15min":2.4660000000000002},"last":0.90300000000000002}]}]},{"osd":5,"up_from":12,"seq":51539607816,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":1.0089999999999999,"15min":1.476},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.505},"max":{"1min":0.92200000000000004,"5min":1.0649999999999999,"15min":1.5529999999999999},"last":0.82399999999999995}]},{"osd":1,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77200000000000002,"15min":0.80100000000000005},"min":{"1min":0.58199999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.117,"15min":1.4610000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.86099999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.45800000000000002},"max":{"1min":0.98599999999999999,"5min":1.335,"15min":1.49},"last":0.877}]},{"osd":2,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.78400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0249999999999999,"5min":1.0900000000000001,"15min":1.236},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.78900000000000003,"15min":0.78200000000000003},"min":{"1min":0.64500000000000002,"5min":0.503,"15min":0.46000000000000002},"max":{"1min":0.877,"5min":1.3979999999999999,"15min":1.4590000000000001},"last":0.69799999999999995}]},{"osd":3,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.47899999999999998,"15min":0.46400000000000002},"max":{"1min":0.85599999999999998,"5min":0.92900000000000005,"15min":1.516},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.71599999999999997},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.48699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0129999999999999,"15min":1.079},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Sep 16 06:59:07 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.375},"max":{"1min":0.93400000000000005,"5min":0.97999999999999998,"15min":1.389},"last":0.622},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.82299999999999995},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.45400000000000001},"max":{"1min":0.98999999999999999,"5min":1.254,"15min":1.3480000000000001},"last":0.77800000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607815,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.41699999999999998,"5min":0.34999999999999998,"15min":0.33500000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":1.044},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.499,"5min":0.45300000000000001,"15min":0.308},"max":{"1min":0.82799999999999996,"5min":0.91700000000000004,"15min":0.997},"last":0.626}]},{"osd":1,"last update":"Mon Sep 16 07:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.48099999999999998,"5min":0.41099999999999998,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.83599999999999997,"15min":1.0549999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.71799999999999997,"15min":0.69999999999999996},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.36899999999999999},"max":{"1min":0.91500000000000004,"5min":0.98399999999999999,"15min":1.2629999999999999},"last":0.72599999999999998}]},{"osd":3,"last update":"Mon Sep 16 07:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.79800000000000004},"min":{"1min":0.621,"5min":0.621,"15min":0.57499999999999996},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.3500000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.79800000000000004,"15min":0.79900000000000004},"min":{"1min":0.66200000000000003,"5min":0.59399999999999997,"15min":0.47099999999999997},"max":{"1min":1.2689999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.873}]},{"osd":4,"last update":"Mon Sep 16 07:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.83099999999999996,"15min":0.83599999999999997},"min":{"1min":0.66700000000000004,"5min":0.60799999999999998,"15min":0.51000000000000001},"max":{"1min":1.0529999999999999,"5min":1.4610000000000001,"15min":1.5049999999999999},"last":0.999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.88600000000000001,"15min":0.877},"min":{"1min":0.79400000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.024,"5min":1.0469999999999999,"15min":1.2749999999999999},"last":1.024}]},{"osd":5,"last update":"Mon Sep 16 07:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.81000000000000005,"15min":0.83299999999999996},"min":{"1min":0.73499999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0649999999999999,"5min":1.254,"15min":1.5780000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.86499999999999999,"15min":0.85599999999999998},"min":{"1min":0.77000000000000002,"5min":0.63,"15min":0.59599999999999997},"max":{"1min":1.2170000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.96699999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":835584,"data_stored":1321761,"data_compressed":115218,"data_compressed_allocated":233472,"data_compressed_original":991232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.54800000000000004},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.29099999999999998},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.99399999999999999},"last":0.505}]},{"osd":2,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.505,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.78000000000000003,"15min":0.878},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.53000000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88800000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54400000000000004}]},{"osd":3,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.67000000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.68500000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.80500000000000005,"15min":0.80200000000000005},"min":{"1min":0.66500000000000004,"5min":0.53800000000000003,"15min":0.48899999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.88100000000000001,"15min":0.88},"min":{"1min":0.76900000000000002,"5min":0.68100000000000005,"15min":0.53700000000000003},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.619},"last":0.93300000000000005}]},{"osd":5,"last update":"Mon Sep 16 06:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.85499999999999998,"15min":0.86199999999999999},"min":{"1min":0.73099999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.585},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.92800000000000005,"15min":0.90900000000000003},"min":{"1min":0.72899999999999998,"5min":0.67900000000000005,"15min":0.64500000000000002},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.89900000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":794624,"data_stored":792833,"data_compressed":109120,"data_compressed_allocated":200704,"data_compressed_original":466944,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 06:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.57199999999999995,"5min":0.48699999999999999,"15min":0.39300000000000002},"max":{"1min":0.94699999999999995,"5min":1.1279999999999999,"15min":1.147},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83299999999999996,"15min":0.84899999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.99299999999999999,"5min":1.1899999999999999,"15min":1.4219999999999999},"last":0.999}]},{"osd":1,"last update":"Mon Sep 16 06:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.77400000000000002,"15min":0.78400000000000003},"min":{"1min":0.61799999999999999,"5min":0.53400000000000003,"15min":0.501},"max":{"1min":0.94599999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.875,"15min":0.88300000000000001},"min":{"1min":0.64100000000000001,"5min":0.54900000000000004,"15min":0.49299999999999999},"max":{"1min":0.97599999999999998,"5min":1.153,"15min":1.4910000000000001},"last":0.93400000000000005}]},{"osd":2,"last update":"Mon Sep 16 06:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.73899999999999999,"15min":0.75},"min":{"1min":0.57199999999999995,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88200000000000001,"5min":0.98699999999999999,"15min":1.0309999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77900000000000003,"15min":0.76400000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.52100000000000002},"max":{"1min":1.0029999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon Sep 16 06:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.54600000000000004,"5min":0.501,"15min":0.38900000000000001},"max":{"1min":0.871,"5min":0.99299999999999999,"15min":1.018},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.745},"min":{"1min":0.56899999999999995,"5min":0.51300000000000001,"15min":0.46999999999999997},"max":{"1min":0.876,"5min":0.95499999999999996,"15min":1.1479999999999999},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Sep 16 06:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.80000000000000004,"15min":0.76200000000000001},"min":{"1min":0.60499999999999998,"5min":0.51800000000000002,"15min":0.45500000000000002},"max":{"1min":1.04,"5min":1.054,"15min":1.121},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88400000000000001,"15min":0.86799999999999999},"min":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.47099999999999997},"max":{"1min":1.016,"5min":1.125,"15min":1.1899999999999999},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6098,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:00:07.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: get_monmap_and_config 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: build_initial_monmap 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: monmap: 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.680 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T07:00:07.684698+0000 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T07:00:07.684698+0000 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-b 2024-09-16T07:00:07.681 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 1 Processor -- start 2024-09-16T07:00:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 1 -- start start 2024-09-16T07:00:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: init 2024-09-16T07:00:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: _reopen_session rank -1 2024-09-16T07:00:07.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-a con 0x7f0b18135500 addr v1:172.21.15.76:6789/0 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-b con 0x7f0b1809a9d0 addr v1:172.21.15.76:6790/0 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-c con 0x7f0b1813b980 addr v1:172.21.15.189:6789/0 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b180a67b0 con 0x7f0b18135500 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b180a7320 con 0x7f0b1813b980 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1809ade0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient(hunting): _renew_subs 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient(hunting): authenticate will time out at 2154.818694s 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1659c700 1 --1- >> v1:172.21.15.76:6790/0 conn(0x7f0b1809a9d0 0x7f0b1813a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6790/0 says I am v1:172.21.15.76:41590/0 (socket says 172.21.15.76:41590) 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b16d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0b18135500 0x7f0b181358e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33790/0 (socket says 172.21.15.76:33790) 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1659c700 1 -- 172.21.15.76:0/635846006 learned_addr learned my addr 172.21.15.76:0/635846006 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267263616 0 0) 0x7f0b1809ade0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): my global_id is 374249 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b000035d0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108974544 0 0) 0x7f0b180a67b0 con 0x7f0b18135500 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient(hunting): my global_id is 334599 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b1809ade0 con 0x7f0b18135500 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3357651510 0 0) 0x7f0b000035d0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 >> v1:172.21.15.189:6789/0 conn(0x7f0b1813b980 legacy=0x7f0b1813dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 >> v1:172.21.15.76:6789/0 conn(0x7f0b18135500 legacy=0x7f0b181358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 monclient: found mon.noname-b 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18099d10 con 0x7f0b1809a9d0 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: _finish_auth 0 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: _check_auth_tickets 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 5 monclient: authenticate success, global_id 374249 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0b0c001630 con 0x7f0b1809a9d0 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: dump: 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.2 v1:172.21.15.76:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b0c001850 con 0x7f0b1809a9d0 2024-09-16T07:00:07.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 10 monclient: get_monmap_and_config success 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b1d1fb700 4 set_mon_vals no callback set 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 1 -- 172.21.15.76:0/635846006 <== mon.2 v1:172.21.15.76:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0b0c001be0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.683+0000 7f0b15d9b700 10 monclient: dump: 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.690 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: shutdown 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/635846006 >> v1:172.21.15.76:6790/0 conn(0x7f0b1809a9d0 legacy=0x7f0b1813a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/635846006 shutdown_connections 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/635846006 >> 172.21.15.76:0/635846006 conn(0x7f0b180a42f0 msgr2=0x7f0b180a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/635846006 shutdown_connections 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/635846006 wait complete. 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: build_initial_monmap 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: monmap: 2024-09-16T07:00:07.691 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.noname-a 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.noname-c 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 Processor -- start 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- start start 2024-09-16T07:00:07.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: init 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: _reopen_session rank -1 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-a con 0x7f0b1809a9d0 addr v1:172.21.15.76:6789/0 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-c con 0x7f0b18135500 addr v1:172.21.15.76:6790/0 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient(hunting): picked mon.noname-b con 0x7f0b1813b980 addr v1:172.21.15.189:6789/0 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b18136ac0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b18137080 con 0x7f0b1813b980 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 1 -- --> v1:172.21.15.76:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b18137360 con 0x7f0b18135500 2024-09-16T07:00:07.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient(hunting): _renew_subs 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b1d1fb700 10 monclient(hunting): authenticate will time out at 2154.822905s 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b16d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0b1809a9d0 0x7f0b181ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:33794/0 (socket says 172.21.15.76:33794) 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0b16d9d700 1 -- 172.21.15.76:0/2524886642 learned_addr learned my addr 172.21.15.76:0/2524886642 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441457395 0 0) 0x7f0b18137360 con 0x7f0b18135500 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): my global_id is 374252 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b040035d0 con 0x7f0b18135500 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.0 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640629712 0 0) 0x7f0b18136ac0 con 0x7f0b1809a9d0 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): my global_id is 334602 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18137360 con 0x7f0b1809a9d0 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182595 0 0) 0x7f0b040035d0 con 0x7f0b18135500 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b18136ac0 con 0x7f0b18135500 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0b0c001b60 con 0x7f0b18135500 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.687+0000 7f0aff7fe700 10 monclient(hunting): dump: 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.700 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157028350 0 0) 0x7f0b18137080 con 0x7f0b1813b980 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): _negotiate 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): my global_id is 285079 2024-09-16T07:00:07.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b040035d0 con 0x7f0b1813b980 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.0 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892631435 0 0) 0x7f0b18137360 con 0x7f0b1809a9d0 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b18137080 con 0x7f0b1809a9d0 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.0 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0b08001d70 con 0x7f0b1809a9d0 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient(hunting): dump: 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.702 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1494125075 0 0) 0x7f0b18136ac0 con 0x7f0b18135500 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 >> v1:172.21.15.189:6789/0 conn(0x7f0b1813b980 legacy=0x7f0b181d2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 >> v1:172.21.15.76:6789/0 conn(0x7f0b1809a9d0 legacy=0x7f0b181ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 monclient: found mon.c 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18136da0 con 0x7f0b18135500 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: _finish_auth 0 2024-09-16T07:00:07.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 5 monclient: authenticate success, global_id 374252 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _renew_subs 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b18132ef0 con 0x7f0b18135500 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _renew_subs 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b181329c0 con 0x7f0b18135500 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b0c004b40 con 0x7f0b18135500 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1559a700 4 set_mon_vals no callback set 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 719386342 0 0) 0x7f0b0c001230 con 0x7f0b18135500 2024-09-16T07:00:07.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 10 monclient: dump: 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 53fe3d11-e8d6-4d19-8f38-046f335b6765 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-09-16T06:37:45.879044+0000 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 18 (reef) 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.76:6789/0 mon.a 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.76:6790/0 mon.c 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-09-16T07:00:07.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af8005100 con 0x7f0b18135500 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 7 ==== mgrmap(e 5) v1 ==== 43272+0+0 (unknown 3855792468 0 0) 0x7f0b0c00f7e0 con 0x7f0b18135500 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 8 ==== osd_map(255..255 src has 230..255) v4 ==== 6745+0+0 (unknown 1122178620 0 0) 0x7f0b0c043fb0 con 0x7f0b18135500 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.691+0000 7f0b16d9d700 10 monclient: get_auth_request con 0x7f0b04036450 auth_method 0 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.699+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (unknown 1092875540 0 3652978863) 0x7f0b0c014a10 con 0x7f0b18135500 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.699+0000 7f0aff7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:00:07.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.699+0000 7f0aff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:00:07.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.867+0000 7f0b1d1fb700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:00:07.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.867+0000 7f0b1d1fb700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:00:07.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.867+0000 7f0b1d1fb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.76:6790/0 2024-09-16T07:00:07.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.867+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 --> v1:172.21.15.76:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0af8005930 con 0x7f0b18135500 2024-09-16T07:00:07.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7f0b0c014a10 con 0x7f0b18135500 2024-09-16T07:00:07.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0aff7fe700 10 monclient: handle_config config(1 keys) v1 2024-09-16T07:00:07.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0b1559a700 4 set_mon_vals no callback set 2024-09-16T07:00:07.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0b1559a700 10 set_mon_vals mon_health_to_clog = false 2024-09-16T07:00:07.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0aff7fe700 1 -- 172.21.15.76:0/2524886642 <== mon.2 v1:172.21.15.76:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) v1 ==== 123+0+0 (unknown 3414686427 0 0) 0x7f0b0c012e80 con 0x7f0b18135500 2024-09-16T07:00:07.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0aff7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:00:07.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.875+0000 7f0aff7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:00:07.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 >> v1:172.21.15.189:6812/3098810674 conn(0x7f0b04036450 legacy=0x7f0b04038910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 10 monclient: shutdown 2024-09-16T07:00:07.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:00:07.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 >> v1:172.21.15.76:6790/0 conn(0x7f0b18135500 legacy=0x7f0b181cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:00:07.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 shutdown_connections 2024-09-16T07:00:07.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 >> 172.21.15.76:0/2524886642 conn(0x7f0b180a42f0 msgr2=0x7f0b1812c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:00:07.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 shutdown_connections 2024-09-16T07:00:07.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-09-16T07:00:07.879+0000 7f0b1d1fb700 1 -- 172.21.15.76:0/2524886642 wait complete. 2024-09-16T07:00:07.893 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T07:00:07.893 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T07:00:07.894 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T07:00:07.894 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:07.989 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T07:00:07.989 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T07:00:07.989 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:14.091 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T07:00:14.091 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T07:00:14.091 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:14.186 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T07:00:14.186 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T07:00:14.187 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:14.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-16T07:00:20.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-16T07:00:20.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-16T07:00:20.288 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T07:00:20.289 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T07:00:20.289 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:20.372 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T07:00:20.372 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T07:00:20.373 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:25.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-16T07:00:25.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-16T07:00:26.474 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T07:00:26.474 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T07:00:26.475 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T07:00:26.475 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:26.507 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T07:00:26.508 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T07:00:26.508 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T07:00:26.508 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:26.548 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T07:00:26.548 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T07:00:26.548 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:26.589 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T07:00:26.589 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T07:00:26.589 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:00:26.637 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T07:00:26.637 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T07:00:26.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-16T07:00:26.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T07:00:26.658 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T07:00:26.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T07:00:26.884 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T07:00:27.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T07:00:27.152 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T07:00:27.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T07:00:27.309 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T07:00:27.527 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T07:00:27.527 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T07:00:27.778 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T07:00:27.779 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T07:00:27.970 INFO:tasks.ceph:Archiving mon data... 2024-09-16T07:00:27.971 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/data/mon.a.tgz 2024-09-16T07:00:27.972 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T07:00:27.978 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.DQIHbvsrUm 2024-09-16T07:00:27.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DQIHbvsrUm 2024-09-16T07:00:28.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.DQIHbvsrUm 2024-09-16T07:00:28.170 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.DQIHbvsrUm is 973KB 2024-09-16T07:00:28.237 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.DQIHbvsrUm 2024-09-16T07:00:28.244 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/data/mon.c.tgz 2024-09-16T07:00:28.244 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-09-16T07:00:28.251 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.Guq6HFzBxe 2024-09-16T07:00:28.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Guq6HFzBxe 2024-09-16T07:00:28.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.Guq6HFzBxe 2024-09-16T07:00:28.487 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.Guq6HFzBxe is 973KB 2024-09-16T07:00:28.558 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.Guq6HFzBxe 2024-09-16T07:00:28.564 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/data/mon.b.tgz 2024-09-16T07:00:28.565 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-16T07:00:28.571 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.VWeiDvsWlU 2024-09-16T07:00:28.571 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VWeiDvsWlU 2024-09-16T07:00:28.733 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.VWeiDvsWlU 2024-09-16T07:00:28.803 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.VWeiDvsWlU is 975KB 2024-09-16T07:00:28.876 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.VWeiDvsWlU 2024-09-16T07:00:28.883 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T07:00:28.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T07:00:28.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T07:00:28.901 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T07:00:28.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T07:00:28.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T07:00:28.936 INFO:teuthology.orchestra.run.smithi076.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T07:00:28.972 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T07:00:28.978 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T07:00:28.980 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi076/crash 2024-09-16T07:00:28.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T07:00:29.003 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi189/crash 2024-09-16T07:00:29.004 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T07:00:29.021 INFO:tasks.ceph:Compressing logs... 2024-09-16T07:00:29.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:00:29.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:00:30.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-16T07:00:35.127 INFO:tasks.ceph:Archiving logs... 2024-09-16T07:00:35.128 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi076/log 2024-09-16T07:00:35.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T07:00:36.415 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi189/log 2024-09-16T07:00:36.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T07:00:37.515 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T07:00:37.525 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T07:00:37.525 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T07:00:37.528 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T07:00:37.595 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T07:00:37.606 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T07:00:37.607 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:00:37.609 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================== 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout:+hv01.front.sepi 67.205.162.81 3 u 47 64 377 0.081 -5.906 0.467 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout:+hv02.front.sepi 96.245.170.99 3 u 48 64 377 0.078 -1.584 0.831 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout:*hv03.front.sepi 74.6.168.72 3 u 41 64 377 0.080 -5.219 0.483 2024-09-16T07:00:37.630 INFO:teuthology.orchestra.run.smithi076.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout:+hv01.front.sepi 67.205.162.81 3 u 52 64 377 0.090 -0.303 0.196 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout:+hv02.front.sepi 96.245.170.99 3 u 50 64 377 0.103 5.020 0.476 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout:*hv03.front.sepi 74.6.168.72 3 u 46 64 377 0.085 0.855 0.201 2024-09-16T07:00:37.632 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-16T07:00:37.633 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T07:00:37.642 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T07:00:37.642 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T07:00:37.677 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T07:00:37.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T07:00:37.746 INFO:teuthology.task.internal:Duration was 1831.512590 seconds 2024-09-16T07:00:37.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T07:00:37.761 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T07:00:37.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T07:00:37.764 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T07:00:37.816 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T07:00:37.816 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-09-16T07:00:37.816 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T07:00:37.829 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-09-16T07:00:37.829 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T07:00:37.842 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T07:00:37.843 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:00:37.872 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:00:37.924 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T07:00:37.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T07:00:37.927 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T07:00:38.002 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T07:00:38.013 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T07:00:38.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T07:00:38.028 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T07:00:38.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T07:00:38.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:00:38.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:00:38.085 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-09-16T07:00:38.102 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-09-16T07:00:38.118 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:00:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:00:38.124 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:00:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:00:38.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T07:00:38.140 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T07:00:38.141 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi076 2024-09-16T07:00:38.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T07:00:38.168 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906553/remote/smithi189 2024-09-16T07:00:38.168 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T07:00:38.191 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T07:00:38.191 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T07:00:38.208 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T07:00:38.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T07:00:38.249 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T07:00:38.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T07:00:38.534 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T07:00:38.534 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T07:00:38.537 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T07:00:38.542 INFO:teuthology.orchestra.run.smithi076.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:00 /home/ubuntu/cephtest 2024-09-16T07:00:38.545 INFO:teuthology.orchestra.run.smithi189.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:00 /home/ubuntu/cephtest 2024-09-16T07:00:38.547 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T07:00:38.586 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T07:00:38.705 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/many workloads/rados_mon_workunits} duration: 1831.5125901699066 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T07:00:38.705 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T07:00:38.786 INFO:teuthology.run:pass